Modern PHP applications up to 110x slower on network storage

The downside is that a single request have to touch a lot of PHP files, sometimes up to several hundred. In the "bad old days", each file must be read from the disk, parsed into byte code and executed by the PHP interpreter if byte code caches like APC wasn't installed. That took a lot of time and PHP applications could require up to several seconds to finish a request.

Since PHP 5.5, performance improved very much, mostly by the integrated OPcode cache, since PHP7 also by various optimizations inside the PHP interpreter itself. Now, request times of even big PHP applications can be only several 100ms or below. PHP applications can be almost as fast as Java one. Great stuff!

There's only one problem PHP developers and administrators don't really think about: The applications are only fast as long as the file system the files are stored is fast too! We're not talking about the sequential read or write performance, which can be fast on network file systems too. It's the time required for file access that matters if your application consists of several thousand files!

Why the file access you might wondering? The OPcache keeps the files in memory so there's no file access at all? Wrong! Even if you configured the available memory of the OPcache to be big enough for all files, it checks for file modifications regularly. By default, this check is executed every two seconds. A better value would be 60 seconds or more:

Things will get even worse. For every part of the file path, the meta data is retrieved by the file system. The path "/var/www/html/index.php" will result in requests for "/var", "/var/wwww", "/var/www/html" and "/var/www/html/index.php". You see, this can result in a lot of request for hundreds of files stored deep down in a directory tree.

PHP has a feature called "real path cache" that should mitigate the problem. The thought about the this cache is to reduce the amount of file system requests by storing the absolute path of the requested files. So far so good but it's value is 16KB by default and this is ridiculously low for modern PHP applications. This results in a constant trashing of the cached entries and renders the cache useless with this size. To avoid this, you should start and test with values of 128KB and more:

So, where's the 110x performance loss? Well, retrieving file meta data is fast when it's already in the cache of the operating system and don't have to be revalidated. Then, it doesn't matter much if the OPcache and real path cache settings are misconfigured.

For our tests, we've used the Aimeos Core plus the required libraries and extensions (ca. 6500 files and directories) as reference and executed the following command in the Aimeos directory:

Let's test the meta data performance of an Ext4 file system on a local hard disk.

First test:

Subsequent tests:

The first time retrieving the meta data of all files and directories from the hard disk lasted 2.722 seconds. If the data is cached by the operating system, only 0.043 seconds are needed.

Now we did the same for NFS, the standard Unix network file system:

First test:

Subsequent tests:

The first time, it takes almost 1.5x as long than from a local hard disk and afterwards it needs even 10x longer. Thus, performance of PHP applications over NFS will severely suffer!

Have a look at GlusterFS, another network file system:

First test:

Subsequent tests:

Not only that the first time it took more then three times longer than from a local hard disk, the subsequent test have shown a more than 110x worse performance compared to the local hard disk. You can imagine that this will lead to response times of several seconds!

The test results made clear that storing the files of a PHP application on a network storage is really a bad idea. It will make more than a noticeable difference and results in response times which are extremely bad when compared to applications stored on local hard disks.

If you have no other chance than using a network storage, you should at least optimize and test the OPcache and real path cache settings to mitigate the problem. If you use optimal settings, you can decrease the request time penalty to only 2-4 times of a local disk. But it will make a big difference if your customers wait 0.5 or up to 4 seconds!