On the most part what I saw were errors with file_get_contents() whenever I was trying to serve what essentially equates to a '404 page'. The errors looked like this...
ERROR: file_get_contents(../retroarch/ag404.lnx): failed to open stream: No such file or directory
The code was trying to open a file from the application file system. The code causing the above error looked like this...
$content = file_get_contents('../retroarch/ag404.lnx');
Of course the file that the code was trying to open was present on the file system. This was tested on the dev server and worked as expected.
To be sure that the production behaviour was different I decided to have a look at the deployed source code for my app, and was surprised that the 'retroarch' directory that the code was expecting, was not shown in the source code browser. I guess I overlooked testing this on the production server once it eventually got deployed, so it's been running like that for months. Lesson learned - never trust the dev server.
This got me thinking that the problem was most likely with app.yaml. The entry that I had for the 'retroarch' directory looked like this...
- url: /retroarch
Since I was using the static_dir directive, there was was a small detail that I overlooked. It was plain as day in the documentation however...
All files in this directory are uploaded with your app as static files. App Engine stores and serves static files separately from your app's files. Static files are not available in the app's file system by default. This can be changed by setting the application_readable option to true.
So in order to read the file programatically all I needed to do was set application_readable to true! Quite an obvious fix, but it does annoy me that the dev server does not have this behaviour. Anyway, I changed the handler to look like this...
- url: /retroarch
After deploying the app again, the source code listing included the directory and the file that the code was trying to open...
It should be noted that the application_readable directive is deprecated in later versions of the AppEngine SDK. I've not looked at the migration guide to find out more details as I'm most likely going to be moving away from AppEngine before I'm forced to upgrade.