Modern PHP applications up to 110x slower on network storage

Modern PHP applications consists of thousands of PHP files. Even if your own set of files is small, most of the functionality is nowadays provided by 3rd party libraries. Especially composer made it simpler than never before to integrate external libraries and full stack frameworks like Laravel or Symfony contains a lot of them. The Aimeos e-commerce components are not different in this case since this kind of code reuse is the best thing since sliced bread 🙂

Bad old days and brave new world

The downside is that a single request have to touch a lot of PHP files, sometimes up to several hundred. In the “bad old days”, each file must be read from the disk, parsed into byte code and executed by the PHP interpreter if byte code caches like APC wasn’t installed. That took a lot of time and PHP applications could require up to several seconds to finish a request.

Since PHP 5.5, performance improved very much, mostly by the integrated OPcode cache, since PHP7 also by various optimizations inside the PHP interpreter itself. Now, request times of even big PHP applications can be only several 100ms or below. PHP applications can be almost as fast as Java one. Great stuff!

When things turn wrong

There’s only one problem PHP developers and administrators don’t really think about: The applications are only fast as long as the file system the files are stored is fast too! We’re not talking about the sequential read or write performance, which can be fast on network file systems too. It’s the time required for file access that matters if your application consists of several thousand files!

OPcache

Why the file access you might wondering? The OPcache keeps the files in memory so there’s no file access at all? Wrong! Even if you configured the available memory of the OPcache to be big enough for all files, it checks for file modifications regularly. By default, this check is executed every two seconds. A better value would be 60 seconds or more:

opcache.revalidate_freq = 60

Things will get even worse. For every part of the file path, the meta data is retrieved by the file system. The path “/var/www/html/index.php” will result in requests for “/var”, “/var/wwww”, “/var/www/html” and “/var/www/html/index.php”. You see, this can result in a lot of request for hundreds of files stored deep down in a directory tree.

Real path cache

PHP has a feature called “real path cache” that should mitigate the problem. The thought about the this cache is to reduce the amount of file system requests by storing the absolute path of the requested files. So far so good but it’s value is 16KB by default and this is ridiculously low for modern PHP applications. This results in a constant trashing of the cached entries and renders the cache useless with this size. To avoid this, you should start and test with values of 128KB and more:

realpath_cache_size = 128KB

The network stoage disaster

So, where’s the 110x performance loss? Well, retrieving file meta data is fast when it’s already in the cache of the operating system and don’t have to be revalidated. Then, it doesn’t matter much if the OPcache and real path cache settings are misconfigured.

For our tests, we’ve used the Aimeos Core plus the required libraries and extensions (ca. 6500 files and directories) as reference and executed the following command in the Aimeos directory:

# time (find . > /dev/null)

Ext4

Let’s test the meta data performance of an Ext4 file system on a local hard disk.

First test:

real    0m2.722s
user    0m0.017s
sys     0m0.130s

Subsequent tests:

real    0m0.043s
user    0m0.003s
sys     0m0.040s

The first time retrieving the meta data of all files and directories from the hard disk lasted 2.722 seconds. If the data is cached by the operating system, only 0.043 seconds are needed.

NFS

Now we did the same for NFS, the standard Unix network file system:

First test:

real    0m4.197s
user    0m0.032s
sys     0m0.188s

Subsequent tests:

real    0m0.490s
user    0m0.019s
sys     0m0.080s

The first time, it takes almost 1.5x as long than from a local hard disk and afterwards it needs even 10x longer. Thus, performance of PHP applications over NFS will severely suffer!

GlusterFS

Have a look at GlusterFS, another network file system:

First test:

real    0m9.031s
user    0m0.045s
sys     0m0.176s

Subsequent tests:

real    0m4.769s
user    0m0.052s
sys     0m0.156s

Not only that the first time it took more then three times longer than from a local hard disk, the subsequent test have shown a more than 110x worse performance compared to the local hard disk. You can imagine that this will lead to response times of several seconds!

Conclusions

The test results made clear that storing the files of a PHP application on a network storage is really a bad idea. It will make more than a noticeable difference and results in response times which are extremely bad when compared to applications stored on local hard disks.

If you have no other chance than using a network storage, you should at least optimize and test the OPcache and real path cache settings to mitigate the problem. If you use optimal settings, you can decrease the request time penalty to only 2-4 times of a local disk. But it will make a big difference if your customers wait 0.5 or up to 4 seconds!