How To Restart Google App Engine Standard Service

Context: I have an app that serves interactive graphs and data analysis. In order to calculate plots and data summaries, it uses a dataset that is loaded upon App initialization by

Solution 1:

It's likely good practice to cache your dataset as you're doing; if you know the data hasn't changed. then there's no need to requery BigQuery for it.

However, your dataset does change, just once per day.

So, I think your approach should be to revise your app so that it refreshes the cached copy of your BigQuery dataset every day and stop|block your users from querying the dataset as it changes.

You actually need only change the dataset if a user requests it (there's no need to refresh the dataset on days when no users need it), so, depending on the time the refresh takes and your users' expectations on latency, you could trigger the refresh by a user request: has the dataset changed? If so, block this request, refresh the data and then respond to the user.

I assume you've already solved the problem that your users' data plots and calculations will differ for different datasets.

Solution 2:

One possible approach would be to trigger the running instances to exit (by themselves, i.e. commit suicide) once the BQ dataset is updated and leave GAE start new/replacement instances, which will load the updated dataset.

The trigger can be based on memcache, datastore or cloud storage/GCS (all faster than BQ - less penalty for checking them in every request). You want to be certain that the trigger doesn't also affect the freshly started instances:

• make the trigger be, for example, the timestamp of the most recent BQ dataset update
• add a global variable with the timestamp of the dataset loading in memory
• the trigger would fire when memcache/datastore timestamp is ~24h (or just "a lot") newer than the one in memory

For the action causing the exit I'd try:

• a regular sys.exit(0) call (not quite sure if/how this works on GAE)

• raising an exception (not so nice, it'll leave nasty traces in the logs). If you use it try to make it as clear as possible to minimize the chances of being accidentally interpreted as a real failure. Maybe something like:

  assertFalse, "Intentional crash to force an instance restart"

Another possible approach would be to force an instance restart from outside - by re-deploying the application using the same version string. The outage associated with the instances' restarts caused by re-deploying the same version is actually why I dislike using the service version based environment implementations, see Continuous integration/deployment/delivery on Google App Engine, too risky?

But for this to work you need some other environment(s) to trigger and execute the deployment. It could be some other GAE service or even a Cloud Function (in which case using a Storage event trigger would eliminate the need for explicitly polling for the dataset updated condition).

Solution 3:

I finally found a way to restart all instances programatically, by using the Python API discovery Client and a service account. It first gets the list of active instances and delets all of them. Then, performs a simple request to initiate one of them.

import requests
from apiclient.discovery import build
from google.oauth2 import service_account

credentials = service_account.Credentials.from_service_account_file('credentials.json')
scoped_credentials = credentials.with_scopes(['https://www.googleapis.com/auth/appengine.admin',"https://www.googleapis.com/auth/cloud-platform"])
appengine = build(serviceName="appengine",version="v1",credentials=scoped_credentials)

VERSION_ID = "version_id"
PROJECT_ID = "project_id"
SERVICE_ID = "appengine_service_name"
APP_URL = "http://some_url.com"

active_instances_dict = appengine.apps().services().versions().instances().list(servicesId=SERVICE_ID,appsId=PROJECT_ID,versionsId=VERSION_ID).execute()
list_of_instances = active_instances_dict["instances"]

for instance in list_of_instances:
    appengine.apps().services().versions().instances().delete(servicesId=SERVICE_ID,appsId=PROJECT_ID,
                  versionsId=VERSION_ID,instancesId=instance["id"]).execute()

requests.get(url=APP_URL)