Since 2022.01 beta, the Aimeos core is using Upscheme for updating the database schema and migrating data between new releases. Upscheme is composer package for schema management based on Doctrine DBAL which offers an easy to use API. You can also integrate Upscheme it in your own application easily and this article explains the differences and how you can write migrations with only a few lines of code

Why Upscheme

Whenever you need to upgrade database schemas in more than one installation and your application supports more than one database (even MySQL and MariaDB differ in some details), you need a tool for schema management. And even if you only support one database vendor, it simplifies live because you don’t have to create messy custom scripts with raw SQL statements.

Aimeos used the Doctrine DBAL API directly up to 2021.10 LTS but DBAL has a very verbose API and requires a lot of code to do simple things. For example, creating/updating a table usually looks similar to these lines:

$dbalManager = $conn->createSchemaManager();
$from = $manager->createSchema();
$to = $manager->createSchema();
 
if( $to->hasTable( 'test' ) ) {
	$table = $to->getTable( 'test' );
} else {
	$table = $to->createTable( 'test' );
}
 
$table->addOption( 'engine', 'InnoDB' );
 
$table->addColumn( 'id', 'integer', ['autoincrement' => true] );
$table->addColumn( 'domain', 'string', ['length' => 32] );
 
if( $conn->getDatabasePlatform()->getName() === 'mysql' ) {
	$table->addColumn( 'code', 'string', ['length' => 64, 'customSchemaOptions' => ['charset' => 'binary']] );
} else {
	$table->addColumn( 'code', 'string', ['length' => 64]] );
}
 
$table->addColumn( 'label', 'string', ['length' => 255] );
$table->addColumn( 'pos', 'integer', ['default' => 0] );
$table->addColumn( 'status', 'smallint', [] );
$table->addColumn( 'mtime', 'datetime', [] );
$table->addColumn( 'ctime', 'datetime', [] );
$table->addColumn( 'editor', 'string', ['length' => 255] );
 
$table->setPrimaryKey( ['id'] );
$table->addUniqueIndex( ['domain', 'code'] );
$table->addIndex( ['status', 'pos'] );
 
foreach( $from->getMigrateToSql( $to, $conn->getDatabasePlatform() ) as $sql ) {
	$conn->executeStatement( $sql );
}

Contrary, Upscheme offers a very terse API which does the same in only a few lines of code:

$this->db()->table( 'test', function( $t ) {
	$t->engine = 'InnoDB';
 
	$t->id();
	$t->string( 'domain', 32 );
	$t->string( 'code', 64 )->opt( 'charset', 'binary', 'mysql' );
	$t->string( 'label', 255 );
	$t->int( 'pos' )->default( 0 );
	$t->smallint( 'status' );
	$t->meta();
 
	$t->unique( ['domain', 'code'] );
	$t->index( ['status', 'pos'] );
} );

Upscheme also offers much better support for migrating data. Instead of using a “migration” table to keep track executed migrations which can get out of sync, Upscheme checks the current state of the schema and applies only necessary changes to update the tables, columns and other schema objects.

If your application allows 3rd party plugins you will love Upscheme’s dependency management based on tasks that must run before or after the current one. Thus, extensions can modify existing tables and migrate data without interfering with core tasks.

Create/update database schema

Creating and updating a table for example requires:

• an Upscheme task
• the table definition

The code below creates an Upscheme task named “Messages” by extending from the Upscheme base task and implements the up() method which is executed to perform the schema update:

namespace Aimeos\Upscheme\Task;
use Aimeos\Upscheme\Schema\Table;
 
class Messages extends Base
{
    public function up()
    {
        $this->db( 'db-message' )->table( 'message', function( Table $t ) {
                $t->id();
                $t->type();
                $t->string( 'label' );
                $t->bool( 'status' );
                $t->meta();
        } );
    }
}

Within the up() method, the database schema for the “db-message” database is retrieved (with a fall-back to the standard database if “db-message” isn’t explicitely configured) and the “message” table defined. It consists of an “id” autoincrement integer column (with primary key), a label and status column as well as “ctime”, “mtime” and “editor” columns automatically added by the meta() function offered by the Aimeos core.

The Table section in the Upscheme documentation contains the list of pre-defined methods for creating the most often used column types. Nevertheless, you can always use the Upscheme col() method to create custom columns too and there’s also a list of Upscheme column modifiers to customize any column according to your needs.

Whenever you add, change or remove a column in your task, Upscheme will update your table schema automatically without any need to write an additional line of code!

Migrate data

Upscheme tasks can also migrate data between new releases and offers simple methods to insert, update and delete records. Also, there are methods available to check for tables, columns and other schema object, to rename them and to drop them if they are not required any more:

$this->db()->delete(
    'message',
    ['type' => 'custom']
)->dropColumn(
    'message',
    'type'
)->update(
    'message',
    ['status' => false],
    ['status' => true, 'mtime' => date('Y-m-d H:i:s')]
)->renameColumn(
    'message',
    'label',
    'content'
)->insert(
    'message',
    ['label' => 'test message', 'status' => true, 'ctime' => date('Y-m-d H:i:s'), 'mtime' => date('Y-m-d H:i:s'), 'editor => 'setup']
);

This statements will delete the messages of type “custom”, drop the messages column afterwards, set the status to TRUE of all remaining records which updating the “mtime” column, rename the “label” column to “content” and insert a new test message into the table. All method calls can be concatenated because the Upscheme API offers a fluent interface and there’s also a select() method to query data:

$this->db()->select(
    'message',
    ['status' => false, 'type' => 'old']
);

For more complex statement, Upscheme provides the stmt() method which gives you direct access to the Doctrine DBAL API:

$result = $this->db()->stmt()->select( 'id', 'content' )
    ->from( 'message' )
    ->where( 'status = 1' )
    ->execute();
 
while( $row = $result->fetchAssociative() ) {
    $map[$row['id']] = $row['content'];
}

Summary

Compared to Doctrine DBAL, Upscheme is very easy to use and also contains some workarounds for pitfalls in DBAL. Creating and updating the database schema is extremely simple and Upscheme saved up to 80% of the code in some setup tasks compared to the previous DBAL code in the Aimeos core.

The new 2.0 major release of the PHP map package is available which includes new methods and support for working with multi-dimensional arrays in an easy way. Additionally, it comes with the best documentation ever so you find everything you need quickly!

Instead of applying independent PHP functions on arrays, use the PHP Map object to stream array data through several methods to transform, order and shorten its elements until the result exactly conforms to your requirements, for example:

$data = [
    ['name' => 'Tom', 'dep' => 'Dev', 'qty' => 30],
    ['name' => 'Bob', 'dep' => 'Sales', 'qty' => 50],
    ['name' => 'Joe', 'dep' => 'Dev', 'qty' => 10]
];
 
// get the person with the highest quantity in "Dev" department
$name = map( $data )
    ->groupBy( 'dep' )->only( 'Dev' )
    ->flat( 1 )->col( 'name', 'qty' )
    ->ksort()->last();

To get the result in one statement is easier and much more elegant than the traditional way in PHP. The PHP Map object is perfectly suited to work on result sets returned by relational databases. You can transform them until you get a collection that exactly fits for your template generating the HTML for the frontend.

The PHP Map package combines the best from Laravel and CakePHP collections, jQuery, Backbone.js, and Lodash in an easy to use PHP package for every PHP application.

What’s new in PHP Map 2.0

One of the biggest new feature is the ability to pass a path of keys as parameter to several PHP Map methods to work on nested arrays directly. Instead of writing own code for diving into multi-dimensional arrays, you can now use the available methods to e.g. get a value by using:

map( ['key' => ['to' => ['value' => 'yes!']]] )
    ->get( 'key/to/value' ); // returns "yes!"

This also works for more advanced methods like col() to reindex an array by using a nested key:

$data = [
    ['name' => 'Tom', 'addr' => ['zip' => '12AB', 'city' => 'NY']],
    ['name' => 'Bob', 'addr' => ['zip' => '23WD', 'city' => 'LA']]
];
 
$map = map( $data )->col( 'name', 'addr/zip' );
// ['12AB' => 'Tom', '23WD' => 'Bob']

By default, a dot (“/”) is used as seperator between the array key names but you can change that for the current map using the sep() method:

$map = map( $data )->sep( '.' )->col( 'name', 'addr.zip' );

If you want to use another seperator in all new PHP Map objects (existing ones will keep their separator), you can change the delimiter globally with the delimiter() method:

Map::delimiter( ',' );

It also returns the delimiter used before if you want to revert the used delimiter later in your code.

Breaking changes in PHP Map 2.0

The new major version contains a few breaking changes to avoid potential problems that came up in the past. You won’t be affected by most of them but you should have a look to prevent unexpected behaviour.

New slash separator

The most important change affects parameter values including slashes because they are now treated as path separators for nested arrays.

$data = [
    'item/id' => 1,
    'item/code' => 'test',
    'item/name' => 'Test'
];
 
map( $data )->col( 'item/code', 'item/id' );
 
// 1.x
[1 => 'test']
 
// 2.x
[]
// use instead
map( $data )->sep( '.' )->col( 'item/code', 'item/id' );

jQuery style method calls

You can call methods of objects in a map like this:

// MyClass implements setStatus() (returning $this) and
// getCode() (initialized by constructor)
 
$map = Map::from( [
    'a' => new MyClass( 'x' ),
    'b' => new MyClass( 'y' )
] );
$map->setStatus( 1 )->getCode()->toArray();

Before, the objects had been checked if they really implement the setStatus() and getCode() methods.

To avoid returning an empty map if the method name is wrong or the called method is catched by the __call() magic method, this isn’t the case any more. If the method isn’t implemented by all objects, PHP generates a fatal error now for better error tracing.

Second equals() parameter

The $assoc parameter of the equals() method to compare keys too has been removed and now, you must use the is() method instead:

// 1.x
map( ['one' => 1] )->equals( ['one' => 1], true );
 
// 2.x
map( ['one' => 1] )->is( ['one' => 1] );

New find() argument

The find() method now accepts a default value or exception object as second argument and the $reverse argument is at the third position now:

// 1.x
Map::from( ['a', 'c', 'e'] )->find( function( $value, $key ) {
    return $value >= 'b';
}, true );
 
// 2.x
Map::from( ['a', 'c', 'e'] )->find( function( $value, $key ) {
    return $value >= 'b';
}, null, true );

Semantic change in groupBy()

Before, items have been grouped by the key string itself if the key passed to groupBy() didn’t exist. To offer easier checking and sorting of the keys, an empty string is used as key now:

Map::from( [
    10 => ['aid' => 123, 'code' => 'x-abc'],
] )->groupBy( 'xid' );
 
// 1.x
[
    'xid' => [
        10 => ['aid' => 123, 'code' => 'x-abc']
    ]
]
 
// 2.x
[
    '' => [
        10 => ['aid' => 123, 'code' => 'x-abc']
    ]
]

Semantic change in offsetExists()

The offsetExists() method uses isset() instead of array_key_exists() now to be in line with typical PHP behavior. Thus, keys with NULL values are now treated differently:

$m = Map::from( ['foo' => null] );
 
// 1.x
isset( $m['foo'] ); // true
 
// 2.x
isset( $m['foo'] ); // false

Renamed split() to explode()

To avoid conflicts with the Laravel split() method and to be in line with the PHP explode() method, the static Map::split() method has been renamed to Map::explode(). Also, the argument order has changed:

// 1.x
Map::split( 'a,b,c', ',' );
 
// 2.x
Map::explode( ',', 'a,b,c' );

The PHP Map package contains many methods to ease your day to day work. They are extremely helpful in a lot of situations where you’ve written custom code up to now. This tutorial explains some of the most useful ones you don’t want to miss any more!

• Documentation at Github
• Composer package at Packagist

Installation

To use the PHP Map package, you have to add it to your composer based application:

composer req aimeos/map

Afterwards, there will be a map() function available, that creates a Map object which offers the methods explained.

col(): Map key/value pairs

There are numerous times when you need to transform an existing array into a list of key/value pairs. Think about a list of database records for example which should be indexed by ID instead of a sequential index. The col() method creates that without the necessity of a loop:

$rows = [['id' => 1, 'code' => 'a'], ['id' => 3, 'code' => 'd']];
$result = map( $rows )->col( 'code', 'id' );
 
// Result equals:
map( [1 => 'a', 3 => 'd'] );

This does also work with objects if they implement the magic PHP __get() method.

collapse() and flat(): Transform multi-dimensional arrays

Do you remember the last time when you had a multi-dimensional array and you needed a flat array of all elements? If you know the maximum depth, you can nest several loops but if not, you need to write a recursive function, which is very cumbersome. The Map object offers the flat() method, that can to do that in one line:

$array = [[1, 2], [3, [4, 5]]];
$result = map( $array )->flat();
 
// Result equals:
map( [1, 2, 3, 4, 5] );

There’s a similar method named collapse(), that does almost the same but keeps the key of each value. Thus, it will overwrite elements with duplicate keys (also numeric keys):

$array = [['a' => 1, 'b' => 2], ['c' => 3, ['a' => 4, 'b' => 5]]];
$result = map( $array )->collapse();
 
// Result equals:
map( ['a' => 4, 'b' => 5, 'c' => 3] );

Both methods accept the number of levels they should collapse or flatten as parameter. That way it’s easy to keep e.g. the third level and above unflattened or uncollapsed.

concat(), merge() and union(): Differences in combining arrays

In PHP, you can combine arrays using array_merge() or the + operator but their results are different. The PHP map package also offers the concat() method:

$one = [0 => 'a', 'b' => 2];
$two = ['b' => 3, 0 => 4];
 
$result = map( $one )->concat( $two );
//equals map( [0 => 'a', 'b' => 2, 1 => 3, 2 => 4] );
 
$result = map( $one )->merge( $two );
//equals map( [0 => 'a', 'b' => 3, 1 => 4] );
 
$result = map( $one )->union( $two );
//equals map( [0 => 'a', 'b' => 2] );

concat() combines the elements of both arrays but doesn’t care about their keys and the resulting list will contain all elements of both arrays. merge() which uses the array_merge() function overwrites existing string keys but renumbers numeric keys. union() is the same as using the + operator and doesn’t add elements whose keys does already exist.

equals(): Test if arrays contains the same elements

To compare arrays, PHP only offers the == and === operators. They return only true if both array contains the same elements with the same keys (and the same order in case of ===). Contrary, the equals() method of the Map object can also test arrays without checking the keys:

$one = ['a', 'b'];
$two = ['b', 'a'];
 
map( $one )->equals( $two ); // true
map( $one )->equals( $two, true ); // false

When passing true as second parameter, equals() works like the == operator.

find(): Return the first matching element

Sometimes, you need the first element from an array that matches your needs. The Map object offers the find() method which allows you to pass a closure function as parameter. This anonymous function can check whose element matches your needs:

map( ['a', 'c', 'e'] )->find( function( $value, $key ) {
    return $value >= 'b';
} );
 
// Result equals 'c'

If no matching value is found, null is returned. If you pass true as second parameter, the search for the matching element will be reversed and you will get the last matching element.

first(), last(), get() and pull(): Advanced ways to retrieve an element

Fetching the first, last or any element of a list of elements is very handy and most useful in many cases. But the PHP Map object offers more than that. What if those methods don’t return null if no element is found but throw an exception or execute a closure? Instead of:

if( ( $item = map( [] ) )->first() === null ) {
    throw new \Exception( 'error' );
}
 
if( ( $item = map( [] ) )->last() === null ) {
    // fetch and return record from database
}
$id = $item->getId();

you can simply write:

$id = map( [] )->first( new \Exception( 'error' ) )->getId();
 
$id = map( [] )->last( function() {
    // fetch and return record from database
} )->getId();

All methods (first(), last(), get() and pull()) methods accept either an exception that’s thrown or a closure that’s executed if no element is available. Thus, it saves you the annoying checks that make you code full of if/else statements.

If you pass an exception, the exception object is created whether it’s used or not. This is no problem as long as you don’t do that in a loop. Then, create one exception object before the loop and pass only the variable to the methods.

in(): Check if several elements are in the list

To test if an value is in a list of elements with in() isn’t much different than using in_array(). But what if you need to do that several times with different elements? Then, in() takes away the burden of doing that within a loop:

map( ['a', 'b', 'c'] )->in( 'b' ); // true
map( ['a', 'b', 'c'] )->in( ['b', 'a'] ); // true
map( ['a', 'b', 'c'] )->in( ['d', 'a'] ); // false

The in() method of the Map object checks if all elements of the passed array are available in the list and only then it returns true.

toJson(): Encode everything into JSON

When working with APIs, you almost always need to encode the result into its JSON representation. The PHP map object offers a convenient way to transform all elements to JSON:

map( ['a', 'b', ['c', 'd'] )->toJson();

Like the PHP json_encode() method, it accepts a parameter for passing options that are used during encoding the elements. There are two that you may find extremely useful:

map( ['a', 'b', ['c', 'd'] )->toJson( JSON_FORCE_OBJECT|JSON_HEX_QUOT );

JSON_FORCE_OBJECT always creates {}, even for empty lists to avoid being encoded as []. If you add JSON output to HTML data attributes for storing something you need for dynamic Javascript features later on, JSON_HEX_QUOT saves you from generating invalid HTML.

walk(): Visit each value in a multi-dimensional array

You receive a multi-dimensional array from an HTML form and want to make sure that all values are of a certain type or doesn’t contain HTML tags? Then, the walk() method of the PHP Map object is your friend:

map( ['a', 'b', ['c', '<b>d</b>'] )->walk( function( &$val ) {
    $val = strip_tags( $val );
} );

You can also create a mapping based on the list you pass as second parameter:

$mapping = [1 => 'one', 2 => 'two', 3 => 'three', 4 => 'four'];
 
map( [1, 2, [3, 4] )->walk( function( &$val, $key, $data ) {
    $val = $data[$val] ?? $val;
}, $mapping );
 
// Result equals
map( ['one', 'two', ['three', 'four']] )

method(): Register your own methods

The best of all comes last: You can extend the PHP Map object by your own methods! This is extremely handy if you need the same code several times and different places. To register your own method, just do this:

\Aimeos\Map::method( 'strrev', function( $sep ) {
    return strrev( join( '-', $this->list ) );
} );

Your closure function has access to the internal array of elements using $this->list and you can read and/or modify that internal array of the Map object. You can use your new method the same way as any other method of the Map object afterwards:

map( ['a', 'b'] )->strrev( '-' );
// returns "b-a"

Conclusion

The PHP Map object offers some great methods that can simplify your code drastically and you will never want to miss them any more. Especially the possibility to register own methods to extend the Map object dynamically gives your great power in a few lines of code.

Arrays are the working horse of PHP, the programming language used by the vast majority of web applications. An array seems to be incredible simple to use but things get complicated and ugly when PHP functions for array manipulation come into play. Are you tired of hard to read code with excessive error handling too? Then maps can be the answer.

• Documentation at Github
• Composer package at Packagist

Why?

Here’s a common example of how array manipulation is done in most PHP code:

$list = [['id' => 'one', 'value' => 'value1'], ['id' => 'two', 'value' => 'value2'], null];
 
$list[] = ['id' => 'three', 'value' => 'value3'];    // add element
unset( $list[0] );                                   // remove element
$list = array_filter( $list );                       // remove empty values
sort( $list );                                       // sort elements
$pairs = array_column( $list, 'value', 'id' );       // create ['three' => 'value3']
$value = reset( $pairs ) ?: null;                    // return first value

It adds/removes elements, sorts and transforms the content and retrieves the first element. You can’t write that shorter or more elegant and moreover, you need to check almost all array functions for errors.

Using maps

The map composer package offers an easy API comparable to jQuery or Laravel collections:

composer req aimeos/map

When using the Map object, you can do the same as above more elegant with less code:

$list = [['id' => 'one', 'value' => 'value1'], ['id' => 'two', 'value' => 'value2'], null];
 
$value = map( $list )                                // create Map
    ->push( ['id' => 'three', 'value' => 'value3'] ) // add element
    ->remove( 0 )                                    // remove element
    ->filter()                                       // remove empty values
    ->sort()                                         // sort elements
    ->col( 'value', 'id' )                           // create ['three' => 'value3']
    ->first();                                       // return first value

And contrary to PHP arrays, you don’t have to care about error handling because either, there are no errors possible or the Map object will throw an exception if something is really wrong.

Of course, you can use the Map object still like a plain PHP array:

$map[] = ['id' => 'three', 'value' => 'value3'];
$value = $map[0];
count( $map );
foreach( $map as $key => value );

In addition, Map objects allow you to pass anonymous functions to a lot of methods. Thus, you can process and transform the elements according to your current needs, e.g. using the each() method:

$map->each( function( $val, $key ) {
    echo $key . ': ' . $val;
} );

You can even extend Maps dynamically by own methods. This is especially useful if you need a transformation of the Map elements more often and at several places of your code:

Map::method( 'strrev', function( $sep ) {
    return strrev( join( '-', $this->list ) );
} );
Map::from( ['a', 'b'] )->strrev( '-' );

The best part is using a Map like a jQuery object, where methods are applied to all objects of the Map. Then, you can create expressive code in one line:

class MyClass {
    private $code;
    private $status;
 
    public function __construct( $code ) { $this->code = $code; }
    public function setStatus( $s ) { $this->stat = $s; return $this; }
    public function getCode() { return $this->code; }
}
 
$objects = Map::from( ['a' => new MyClass( 'x' ), 'b' => new MyClass( 'y' )] );
$result = $objects->setStatus( 1 )->getCode()->toArray();

The example will call setStatus( 1 ) on all objects and creates a new Map with the returned values. Because setStatus() returns the same object ($this), we can apply another method on all objects afterwards (getCode()). The results of getCode() is wrapped in a Map object again and toArray() returns a plain PHP array which consists of:

['a' => 'x', 'b' => 'y']

Available methods

The Map object contains methods for almost all array functions available and more like collapse(), flat(), toJson() or methods found in the Javascript Array prototype. Here’s a list of its methods and for documentation, have a look at the Map readme at GitHub where you can also check the code.

• function is_map() : Tests if the variable is a map object
• function map() : Creates a new map from elements
• arsort() : Reverse sort elements with keys
• asort() : Sort elements with keys
• chunk() : Splits the map into chunks
• clear() : Removes all elements
• col() : Creates a key/value mapping
• collapse() : Collapses multi-dimensional elements
• concat() : Combines the elements
• copy() : Creates a new copy
• count() : Returns the number of elements
• diff() : Returns the missing elements
• diffAssoc() : Returns the missing elements and checks keys
• diffKeys() : Returns the missing elements by keys
• each() : Applies a callback to each element
• empty() : Tests if map is empty
• equals() : Tests if map contents are equal
• filter() : Applies a filter to the map elements
• find() : Returns the first matching element
• first() : Returns the first element
• firstKey() : Returns the first key
• flip() : Exchanges keys with their values
• flat() : Flattens multi-dimensional elements
• from() : Creates a new map from passed elements
• get() : Returns an element by key
• has() : Tests if a key exists
• in() : Tests if element is included
• includes() : Tests if element is included
• intersect() : Returns the shared elements
• intersectAssoc() : Returns the shared elements and checks keys
• intersectKeys() : Returns the shared elements by keys
• isEmpty() : Tests if map is empty
• join() : Returns concatenated elements as string
• keys() : Returns the keys
• krsort() : Reverse sort elements by keys
• ksort() : Sort elements by keys
• last() : Returns the last element
• lastKey() : Returns the last key
• map() : Applies a callback to each element and returns the results
• merge() : Combines elements overwriting existing ones
• method() : Registers a custom method
• pipe() : Applies a callback to the map
• pop() : Returns and removes the last element
• pull() : Returns and removes an element by key
• push() : Adds an element to the end
• random() : Returns random elements
• reduce() : Computes a value for the map content
• remove() : Removes an element by key
• replace() : Replaces elements recursively
• reverse() : Reverses the array order
• rsort() : Reverse sort elements
• search() : Find the key of an element
• set() : Overwrites an element
• shift() : Returns and removes the first element
• shuffle() : Randomizes the element order
• slice() : Returns a slice of the map
• sort() : Sorts elements
• split() : Splits a string into map elements
• splice() : Replaces a slice by new elements
• toArray() : Returns the plain array
• toJson() : Returns the elements in JSON format
• uasort() : Sorts elements with keys using callback
• uksort() : Sorts elements by keys using callback
• union() : Combines the element without overwriting
• unique() : Returns unique elements
• unshift() : Adds an element at the beginning
• usort() : Sorts elements using callback
• values() : Returns all elements with new keys

Again, the Map documentation offers descriptions, explains parameters/return values and has examples for each method. Some methods are even better documented than the corresponding PHP array function.

Conclusion

Map objects are an easy to use and elegant way to handle PHP arrays. After using it for a while you won’t want to remember the times before. If you’ve already worked with jQuery and like it’s simplicity, you will feel like home when using appropriate lists of objects in your application.

As the Map methods are using the native array functions or are highly optimized, it’s also save to use them in large scale applications. The documentation contains a section about the Map performance and hints to avoid performance bottlenecks.

Modern PHP applications consists of thousands of PHP files. Even if your own set of files is small, most of the functionality is nowadays provided by 3rd party libraries. Especially composer made it simpler than never before to integrate external libraries and full stack frameworks like Laravel or Symfony contains a lot of them. The Aimeos e-commerce components are not different in this case since this kind of code reuse is the best thing since sliced bread

Bad old days and brave new world

The downside is that a single request have to touch a lot of PHP files, sometimes up to several hundred. In the “bad old days”, each file must be read from the disk, parsed into byte code and executed by the PHP interpreter if byte code caches like APC wasn’t installed. That took a lot of time and PHP applications could require up to several seconds to finish a request.

Since PHP 5.5, performance improved very much, mostly by the integrated OPcode cache, since PHP7 also by various optimizations inside the PHP interpreter itself. Now, request times of even big PHP applications can be only several 100ms or below. PHP applications can be almost as fast as Java one. Great stuff!

When things turn wrong

There’s only one problem PHP developers and administrators don’t really think about: The applications are only fast as long as the file system the files are stored is fast too! We’re not talking about the sequential read or write performance, which can be fast on network file systems too. It’s the time required for file access that matters if your application consists of several thousand files!

OPcache

Why the file access you might wondering? The OPcache keeps the files in memory so there’s no file access at all? Wrong! Even if you configured the available memory of the OPcache to be big enough for all files, it checks for file modifications regularly. By default, this check is executed every two seconds. A better value would be 60 seconds or more:

opcache.revalidate_freq = 60

Things will get even worse. For every part of the file path, the meta data is retrieved by the file system. The path “/var/www/html/index.php” will result in requests for “/var”, “/var/wwww”, “/var/www/html” and “/var/www/html/index.php”. You see, this can result in a lot of request for hundreds of files stored deep down in a directory tree.

Real path cache

PHP has a feature called “real path cache” that should mitigate the problem. The thought about the this cache is to reduce the amount of file system requests by storing the absolute path of the requested files. So far so good but it’s value is 16KB by default and this is ridiculously low for modern PHP applications. This results in a constant trashing of the cached entries and renders the cache useless with this size. To avoid this, you should start and test with values of 128KB and more:

realpath_cache_size = 128KB

The network stoage disaster

So, where’s the 110x performance loss? Well, retrieving file meta data is fast when it’s already in the cache of the operating system and don’t have to be revalidated. Then, it doesn’t matter much if the OPcache and real path cache settings are misconfigured.

For our tests, we’ve used the Aimeos Core plus the required libraries and extensions (ca. 6500 files and directories) as reference and executed the following command in the Aimeos directory:

# time (find . > /dev/null)

Ext4

Let’s test the meta data performance of an Ext4 file system on a local hard disk.

First test:

real    0m2.722s
user    0m0.017s
sys     0m0.130s

Subsequent tests:

real    0m0.043s
user    0m0.003s
sys     0m0.040s

The first time retrieving the meta data of all files and directories from the hard disk lasted 2.722 seconds. If the data is cached by the operating system, only 0.043 seconds are needed.

NFS

Now we did the same for NFS, the standard Unix network file system:

First test:

real    0m4.197s
user    0m0.032s
sys     0m0.188s

Subsequent tests:

real    0m0.490s
user    0m0.019s
sys     0m0.080s

The first time, it takes almost 1.5x as long than from a local hard disk and afterwards it needs even 10x longer. Thus, performance of PHP applications over NFS will severely suffer!

GlusterFS

Have a look at GlusterFS, another network file system:

First test:

real    0m9.031s
user    0m0.045s
sys     0m0.176s

Subsequent tests:

real    0m4.769s
user    0m0.052s
sys     0m0.156s

Not only that the first time it took more then three times longer than from a local hard disk, the subsequent test have shown a more than 110x worse performance compared to the local hard disk. You can imagine that this will lead to response times of several seconds!

Conclusions

The test results made clear that storing the files of a PHP application on a network storage is really a bad idea. It will make more than a noticeable difference and results in response times which are extremely bad when compared to applications stored on local hard disks.

If you have no other chance than using a network storage, you should at least optimize and test the OPcache and real path cache settings to mitigate the problem. If you use optimal settings, you can decrease the request time penalty to only 2-4 times of a local disk. But it will make a big difference if your customers wait 0.5 or up to 4 seconds!