Igor Kromin |   Consultant. Coder. Blogger. Tinkerer. Gamer.

I've been working on simple file manager for an Google App Engine project of mine when I came across some odd behaviour in Google Cloud Store. It appeared that the rename() call on a directory was always failing. At first I thought this was due to the dev server not supporting all Data Store operations but the same behaviour was exhibited when I deployed my app in App Engine.

The Advanced File Management documentation page for the standard PHP environment did say that rename() was supported however...
rename — Renames a file or directory. Supported.


There was no mention about partial support or that directories are treated any differently to files. However whenever I tried to rename a directory, I received an error like this...
 Error
Error: Unable to rename: gs://app_default_bucket/new_dir. Cloud Storage Error: NOT FOUND (/Applications/devtools/google-cloud-sdk/platform/google_appengine/php/sdk/google/appengine/ext/cloud_storage_streams/CloudStorageRenameClient.php:75) Code: 500 Path: /ag-api/admin/files


I did find a documented way to rename an object in Cloud Storage which could be done via a client library. There were even code samples included - in PHP! I didn't want to go down this path since it added yet another dependency and couldn't be tested on a local dev environment.

There had to be another way.

Something I noticed was that Google's code example for renaming an object via a client library did was not do any renaming but instead it copied the object and gave it a new name, then deleted the old object. This gave me an idea. Recursive directory copy and delete of the source files/directories!

In fact checking the rename() method documentation in CloudStorageRenameClient.php indicates the same behaviour...
 CloudStorageRenameClient.php
/**
* Perform the actual rename of a GCS storage object.
* Renaming an object has the following steps.
* 1. stat the 'from' object to get the ETag and content type.
* 2. Use x-goog-copy-source-if-match to copy the object.
* 3. Delete the original object.
*/


The end result was this function (simplified for the purpose of this article, additional error handling and exception throwing removed)...
 PHP
function rename_gae($path, $newPath) {
$isDir = is_dir($path);
/* regular file copy */
if (!$isDir) {
rename($path, $newPath);
}
/* directory copy */
else {
mkdir($newPath);
$dirsToDelete = [$path];
$dirIter = new \RecursiveDirectoryIterator($path,
\RecursiveDirectoryIterator::SKIP_DOTS);
$iterator = new \RecursiveIteratorIterator($dirIter,
\RecursiveIteratorIterator::SELF_FIRST);
foreach ($iterator as $item) {
$p = str_replace('//', '/', $iterator->getSubPathname());
if ($item->isDir()) {
mkdir($newPath . DIRECTORY_SEPARATOR . $p);
array_unshift($dirsToDelete, $item->getPathname());
}
else {
copy($item, $newPath . DIRECTORY_SEPARATOR . $p);
unlink($item);
}
}
array_map('rmdir', $dirsToDelete);
}
}




I could then call it like this...
 PHP
$path = 'gs://#default#/old_dir';
$newPath = 'gs://#default#/new_dir';
rename_gae($path, $newPath);


So lets look at the function closer. In a nutshell it can rename a file or directory present in the Google Cloud Store. The same function should work just fine running in any PHP environment however (though you really don't need to use it if dealing with a standard file system). If the function detects that its dealing with something that is not a directory i.e. a file, it simply calls rename().

If a directory is detected, the function creates a new directory with a new name. It adds the old directory to an array of directories to delete later. Then it recursively traverses the old directory and copies everything it finds into the new directory. If it comes across a sub-directory, it creates a new sub-directory, a file is simply copied.

Whenever a file is copied, it is immediately deleted from the source. Directories are handled a little differently since they can't be deleted until all of the files inside them are deleted, so directories are added to an array in reverse order or array traversal i.e. to the front of the array.

After everything is copied, array_map is called with rmdir to remove the old directory and all of its sub-directories.

The odd code on line 19 is a result of another strange behaviour I observed. Whenever a call to RecursiveDirectoryIterator ::getSubPathname() was made the path would become mangled with additional forward slashes. I couldn't find a nicer way of dealing with that so decided it was easier to just replace all the double slashes with a single slash.

So that was my solution. It's not a nice way of doing directory renames and it would have been more effective to disable renaming anything other than a file, but I wanted to have the file manager 'complete'. Doing renames this way will incur huge quota costs too so I wouldn't recommend doing it unless it was really necessary.

-i

Hope you found this post useful...

...so please read on! I love writing articles that provide beneficial information, tips and examples to my readers. All information on my blog is provided free of charge and I encourage you to share it as you wish. There is a small favour I ask in return however - engage in comments below, provide feedback, and if you see mistakes let me know.

If you want to show additional support and help me pay for web hosting and domain name registration, donations, no matter how small, are always welcome!

Use of any information contained in this blog post/article is subject to this disclaimer.
comments powered by Disqus
Other posts you may like...