Igor Kromin |   Consultant. Coder. Blogger. Tinkerer. Gamer.

So you've been developing an app that run's on Google's App Engine and now you have a need to run background or scheduled tasks. No problem, that is covered in Google's documentation already here (Cron) and here (task queues). What isn't covered however are the knock-on effects of doing it the naive way...in the worst case going over your daily quota limits (if you're using the always free GCP tier).

I've recently ran into these issues myself when I implemented a Twitter auto-unfollow functionality for my new website (AtariGamer.com). I ran into this issue due to the nature of Twitter's API and its own usage limits - I had to split work up into a Cron job that kicked everything off, followed by a number of tasks on the push queue. What I didn't realise was due to the default (F1 autoscaling) I had new instances of my app started to serve these 'background' requests as a front-end instance and that quickly made me go over the 28hr/day quota and the entire site went down shortly after.
over_quota.png


With over 5 hours until my quota refreshed I started to look into this deeper. I knew that there were two separate quotas, 28 hours for the front-end and 9 hours for the back-end instances. I realised then that what I needed to do was to make sure that all of my Cron and Push Queue tasks were being executed on the back-end instance. Simple!

That led me to this page - Configuration Files. Specifically, I was interested in defining a separate service for my backend jobs. So, I created a file called backend-jobs.yaml and added the following content to it...
 backend-jobs.yaml
runtime: php55
api_version: 1
service: backend-jobs
instance_class: B1
basic_scaling:
idle_timeout: 10m
max_instances: 1
handlers:
- url: /xxxxxx/.*
script: atga/routes.php
login: admin
- url: /_ah/start
script: atga/routes.php


What the above file does is it defines a B1 instance that responds to some handlers ie. URLs that get requested by the Cron and Push Queue tasks. The main part here is the handler for "url: /xxxxxx/.*" - this routes all my requests to routes.php, which in turn interprets the URL and calls the appropriate controller (I'm using Silex for this). The "url: /_ah/start" handler is also required and must return a HTTP status of 200 or the backend instance of the app will not start.



My PHP code for the backend jobs stayed the same, it was just the service definition that changed. I then deployed this backend service using the gcloud tool...
 Command
gcloud app deploy backend-jobs.yaml

over_quota2.png


With the backend service defined I could now go ahead with the necessary changes to the queue.yaml file. The only change I had to do was add a target definition that pointed to the backend service I've just created...
 queue.yaml
queue:
- name: xxxxx
...
target: backend-jobs


Then each of the Cron jobs in the cron.yaml file was also updated in a similar fashion...
 cron.yaml
cron:
- description: XXXXX
...
target: backend-jobs



This was followed by redeploying my Cron and Task Queue definitions...
 Command
gcloud app deploy queue.yaml
gcloud app deploy cron.yaml


That was it! From this point on all of my backend processing tasks were being run on a separate B1 instance and all of the usual front-end requests were being served by the F1 default instance I've always had. I did notice that while I was still over quota on the F1 instance, the B1 instance would not start, but once the quota was refreshed, everything worked as expected.

Best of all, no PHP code needed changing! I could also see, in the App Engine Quotas page that both the front-end and back-end instances were being used as expected.
over_quota3.png


Now I don't have to worry about exhausting my front-end instance hours with my background processing tasks. All it took was a few configuration file changes and voila!

-i

Skip down to comments...
Hope you found this post useful...

...so please read on! I love writing articles that provide beneficial information, tips and examples to my readers. All information on my blog is provided free of charge and I encourage you to share it as you wish. There is a small favour I ask in return however - engage in comments below, provide feedback, and if you see mistakes let me know.

If you want to show additional support and help me pay for web hosting and domain name registration, donations, no matter how small, are always welcome!

Use of any information contained in this blog post/article is subject to this disclaimer.
 
comments powered by Disqus
Other posts you may like...