#gigacommerce – Aimeos shop performance with 1 billion items


Aimeos has recently proven to be capable of handling one billion articles and more in an online shop using the #gigacommerce extension. This is far more than the number of articles at the Amazon market place which is currently estimated by around 562 million items. But the sheer numbers doesn’t say anything about the performance so how fast can Aimeos delivery content when users browsing the shop pages?

The setup

For a realistic test Aimeos has been set up in a standard cloud hosting environent sponsored by Profihost, an e-commerce hoster from Hannover, Germany. They provided two of their “Flexserver Expert” packages, which are virtual machines running on a cluster with Intel Xeon Skylake 2,6 GHz CPUs.


  • LAMP VM: 8 Core, 16GB RAM, 17GB on SSD

  • ElasticSearch VM: 8 Core, 32GB RAM, 465GB on SSD


  • Debian Linux, Apache 2.4, MySQL 5.7, PHP 7.2, ElasticSearch 5.4

  • Laravel 5.7, Symfony 4.1, TYPO3 8.7

  • Aimeos 2018.10, ai-elastic extension 2018.10

Test procedure

All tests have been done with Loader.io, a SaaS provider for load tests. The latency between the clients and the server is about 32ms, which is included in the response times. A typical page request of a detail page therefore looks as follows:

Stalled 5.26 ms
Request sent 0.12 ms
Waiting (TTFB) 50.74 ms
Content Download 18.33 ms
Sum 75.28 ms

The response time therefore consists of the network latency (32ms), the processing on the server (50.7ms – 32ms = 18.7ms) and the transmission time of the HTML page (18.3ms). The network latency is the largest influencing factor here and can be reduced only with difficulty in real operation. In general it is lower the closer the server is geographically to the customer and the better the connection of the hoster is.

The Aimeos list and detail page were tested, both with cache and without. The cache does NOT contain the complete page (“full page cache”) as in other shop systems, but only page fragments such as the product list or the product details. The page frame is provided by Laravel, Symfony and TYPO3, in which the fragments are inserted before delivering the page to the client.

Test results

The tests showed that Laravel 5.7 and Symfony 4.1 are about the same speed. In the evaluations of Loader.io, there were no significant differences between the two and therefore the results were summarized. The only measurable difference is about 5ms, which Symfony 4.1 can generate on an unloaded system faster than Laravel 5.7. Under load there is no more difference.

With TYPO3 as frontend, the server can process fewer requests per second, but TYPO3 is not directly comparable to Laravel/Symfony. As a complete content management system, TYPO3 allows the maintenance of pages by editors, which is not easily possible with Laravel/Symfony.


With over 600 PI/second and a processing time on the server less than 20ms, the Aimeos E-Commerce System is certainly one of the fastest shop systems. The differences between list and detail view are mainly due to the different page size (189KB to 66KB) and the resulting longer time for data transfer.

Competitors can only achieve similar results by storing complete pages in a Varnish cache and loading shopping carts and other dynamic content through AJAX queries. However, Aimeos is so fast that this would make response times even worse, because the separate query for e.g. the shopping cart is not necessary at all with Aimeos.

Even without cache the number of 130-150 PI/second and the response times of 162ms incl. network latency (130ms without latency) are excellent values compared to other shop systems, especially since the system can scale up to 1 billion and more items. This makes Aimeos particularly suitable for high-load scenarios such as TV advertising.

More about #gigacommerce: https://aimeos.com/gigacommerce

Aimeos 2019.01 major release


The new beta version contains a lot of improvements and focuses on:

  • Even more performance
  • Developer experience
  • Code cleanup

Performance improvements

Aimeos is already known as ultra fast e-commerce solution for Laravel, Symfony, SlimPHP, TYPO3 and Flow/NeosCMS even for complex requirements. This release pushes the limits teared down by Aimeos #gigacommerce further with render times of 20ms and below if content caching is enabled. Without caching, response times are exceptional too compared to other shop systems and in the 2019.01 release, we reduced that time further.

First, we denormalized tables that use types, so the type code is now stored directly instead of the type ID. This saves several database queries per request. Additionally, it’s possible to retrieve related data for e.g. products filtered by type:

$manager->searchItems( $filter, ['text', 'price', 'product' => ['suggestion'] );

This fetches related products of type “suggestion” only instead of all related products. Especially for shops with many relations, this can be a huge performance improvement if components can’t be cached.

PHP 7.3 promises another huge performance improvement and we are testing that version every time. Unfortunately, there’s a bug in PHP 7.3 with prevents production usage yet. It seems that the PHP developers over-optimized opcode caching so PHP 7.3 produces wrong results: https://bugs.php.net/bug.php?id=77310

Simplifications for developers

One of the major goals for this Aimeos release was to simplify interfaces so the first steps for new developers are easier. As a result, creating managers, controllers and clients requires only:

$manager = \Aimeos\MShop::create( $context, 'product' );
$cntl = \Aimeos\Controller\Frontend::create( $context, 'catalog' );
$client = \Aimeos\Client\Html::create( $context, 'catalog/lists' );

Items with relations are fetched recursively now and a decorator limits the recursion depth to two levels. The allowed depth is configurable like everything else in Aimeos. This allows you to get all releated data you need at one without fetching and merging data from several calls to searchItems():

$products = $manager->searchItems( $filter, ['product', 'price', 'text] );

For bundle and selection products, the items in the result list contains the bundled and variant articles including prices, texts and related products.

Also, all newly created items have the status “enabled” by default and must be explicitly disabled if required. This saves many calls to “setStatus()” especially when implementing a new importer. The frontend controllers allow multiple category IDs for filtering so it’s possible to search for
products in multiple categories with one method call.

Code cleanups

Finally, a lot of old code has been removed due to the switch to document oriented managment of items. Especially in the index managers and the filter criteria implementation was some code left after the changes required to use ElasticSearch natively.

The 2019.x releases are the last ones that will support PHP 5 as 5.6 isn’t maintained any more since this year. You should use PHP 7 nevertheless to get the best performance.

For a full list of backward incompatible changes, please have a look at the Wiki article. BC breaking changes in 2019.01 are listed up to 2018-12-31:



#gigacommerce – Scalable online shops with 1 billion items and more


Does an online shop with 1 billion items work on a standard cloud server? And that even with the demand for very low response times? We investigated this question during the #gigacommerce project.

The #gigacommerce setup contains 1 billion articles, which are combined into 10 million products. Each item contains attributes, images, prices and text like in any online store. The basis is a Symfony, Laravel and TYPO3 installation with the Aimeos E-Commerce extension, which access an ElasticSearch server.

All three frontends have been extensively tested to find out where the limits are. The Aimeos components showed that they scale almost infinitely, with response times of around 100 milliseconds. Interestingly, the only absolute limit in the tests was not the available hardware, but the network bandwidth, which was used up at over 600 PI/second. This makes it particularly interesting for peak load situations and TV advertising.

How it this possible?

The special feature of the Aimeos e-commerce system is its high flexibility. Data can be stored in different backends, which use the appropriate technology depending on the requirements – like here ElasticSearch for the product data. This makes the system ultra fast and scalable. Nevertheless, Aimeos is not a special solution, but comparable with other major players in the industry.

The open source software Aimeos can be adapted to the most diverse needs, in particular to complex B2B requirements and delivers everything that is important for an online shop: product catalogs, shopping baskets, payment integration, order management, e-mails and all optimized for mobile devices. Aimeos is available for TYPO3, Laravel, Symfony, SlimPHP and Flow/Neos but the software can be integrated into any PHP application.

More about Aimeos #gigacommerce

What is an e-commerce framework?

The term e-commerce framework is related to software frameworks for e-commerce applications. They offer an environment for building e-commerce applications quickly.

E-Commerce frameworks are flexible enough to adapt them to your specific requirements. As result, they are suitable for building virtually all kinds of online shops and e-commerce related (web) applications.

An e-commerce framework must

  • allow replacing all parts of the framework code
  • forbid changes in the framework code itself
  • contain bootstrap code to start the application
  • be extensible by user-written code

E-Commerce frameworks should

  • define the general program flow
  • consist of reusable components
  • be organized in functional domains

They provide an overall structure for e-commerce related applications.

Furthermore, they implement the general program flow e.g. how the checkout process works. Contrary to monolithic shop systems, existing program flow can not only be extended but completely changed according to your needs.

Evolution of e-commerce systems

Since the beginning of (internet) e-commerce around 1995, a lot has changed on the technology side. The first generation of e-commerce systems evolved from existing ERP and related systems. This was followed by the 2. generation of standalone shop systems between 2004 and 2008. E-commerce frameworks are the latest generation of e-commerce systems and started around 2012.

Hybris, the shop system owned by SAP is one of the representatives of the 1. generation. It’s strongly connected to the SAP ERP system and Hybris is mainly a shop front-end for SAP. Customer relationship (CRM) and content management (CMS) tools are available in the ERP system but very limited.

The Magento shop system represents the 2. generation of standalone e-commerce systems. They usually contain CRM and CMS and some other functionality but also only at a very basic level. They might be enough for the smallest shops but are unusable for shop owners who run for real profits.

Aimeos is one of the few real e-commerce frameworks that are currently available. These 3rd generation systems excel in their own domain: Present and sell products. For all other e-commerce related tasks, they connect to specialized systems and exchange data in both ways. Thus, shop owners can choose the best systems for their needs.

E-Commerce framework architecture

E-commerce frameworks must be based on a strong architectural model. Usually, they make heavy use of interfaces and design patterns like

  • Dependency Injection (make components independent of used object implementation)
  • Factories (create objects at a central place that instantiates the actual implementation)
  • Decorators (dynamically add functionality to existing objects)
  • Publish/Subscribe model (notify listening objects about changes instead of polling for updates)

A “design pattern” is re-usable solution that solves similar software design problems in an elegant way. They require programming language templates which enforce public class methods and their signatures called “interfaces”.

The basic requirement are independent, side-effect free components that form the building blocks. One or more components care about the functionality of a business “domain”. Such a domain can be the HTML front-end, a JSON REST API and the administration interface.

The vertical separation of code into business domains enables scaling out applications by deploying the application as a bunch of micro services.

Micro services are loosely coupled applications that offer their service over lightweight protocols like HTTP and JSON. They are easier to extend, maintain and especially deploy than monolithic applications.

More about Aimeos

Was ist ein E-Commerce Framework?

Der Term E-Commerce Framework bezieht sich auf Software-Frameworks für E-Commerce Applikationen. Sie ermöglichen es in relativ kurzer Zeit komplette E-Commerce Applikationen zu bauen.

E-Commerce Frameworks sind flexibel genug um sie an kundenspezifische Anforderungen anzupassen. Dadurch können sie als Basis für praktisch alle Arten von Onlineshops und E-Commerce bezogene (Web-)Applikationen dienen.

Ein E-Commerce Framework muss

  • es erlauben, jeden seiner Bestandteile zu ersetzen
  • Änderungen am Framework-Code selbst verbieten
  • Bootstrap-Code beinhalten um die Applikation zu starten
  • erweiterbar für kundenspezifischen Code sein

E-Commerce Frameworks sollen

  • den allgemeinen Programmablauf definieren
  • aus wiederverwendbaren Komponenten bestehen
  • in funktionalen Domänen unterteilt sein

Sie bieten eine komplette Infrastruktur für E-Commerce bezogene Applikationen.

Weiterhin implementieren sie den allgemeinen Programmfluss, z.B. wie der Checkout-Prozess funktioniert. Im Gegensatz zu monolithischen Shopsystemen kann der bestehende Programmfluss nicht nur erweitert werden, sondern auch komplett für die eigenen Bedürfnisse geändert werden.

Evolution von E-Commerce Systemen

Seit dem Beginn des E-Commerce im Internet um 1995 hat sich vieles auf Seiten der Technologie verändert. Die erste Generation von E-Commerce Systemen entwickelte sich aus existierenden Warenwirtschaftsprogrammen und ähnlichen Systemen. Daraufhin folgte die zweite Generation von eigenständigen Shopsystemen zwischen 2004 und 2008. E-Commerce Frameworks sind die neuste Generation von E-Commerce Systemen und begannen ab 2012 auf den Markt zu kommen.

Hybris, das Shopsystem von SAP ist einer der Vertreter der ersten Generation. Es ist eng mit dem SAP ERP System verzahnt und Hybris ist hauptsächlich eine Shopoberfläche für SAP. Customer Relationship (CRM) und Content Management (CMS) Werkzeuge sind im Umfeld von SAP verfügbar aber mit eingeschränkter Funktionalität im Gegensatz zu spezialisierten Systemen.

Das Magento Shopsystem repräsentiert die zweite Generation von eigenständigen E-Commerce Systemen. Sie beinhalten CRM und CMS Komponenten sowie ein paar weitere Funktionalitäten, aber ebenfalls nur mit einem beschränkten Funktionsumfang. Sie können für Kleinstunternehmen ausreichend sein, für Shopsbetreiber mit größeren Umsätzen aber meistens unbrauchbar.

Aimeos ist eines der wenigen echten E-Commerce Frameworks die im Moment verfügbar sind. Diese Systeme der dritten Generation glänzen in ihrer Kerndomäne: Der Präsentation und dem Verkauf der Produkte. Für alle weiteren E-Commerce bezogenen Aufgaben verbinden sie sich mit spezialisierten Systemen und tauschen Daten in beide Richtungen aus. Dadurch können Shopbetreiber das jeweils beste System für ihre Bedürfnisse wählen.

E-Commerce Framework Architektur

Gute E-Commerce Frameworks müssen einer stringenten Architektur folgen. Dafür verwenden sie ausgiebig Interfaces und Design Patterns wie

  • Dependency Injection (Komponenten sind unabhängig von der konkreten Objektimplementierung)
  • Factories (erzeugen zentral Objekte und instanzieren die tatsächliche Implementierung)
  • Decorators (fügen dynamisch zusätzliche Funktionalität zu existierenden Objekten hinzu)
  • Publish/Subscribe (benachrichtigen registrierte Objekte über Änderungen anstatt wiederholt auf Änderungen zu testen)

Ein “Design Pattern” ist eine wiederverwendbare Lösung, die ähnliche Softwaredesign-Probleme elegant löst. Sie benötigen Schablonen names “Interfaces” in den Programmiersprachen, die öffentliche Klassenmethoden und ihre Signaturen erzwingen können.

Die wichtigste Anforderung sind unabhängige Bausteine ohne Seiteneffekte, die zu Komponenten zusammen gefügt werden können. Ein oder mehrere Bausteine sind für die Funktionalität einer Fachdomäne zuständig. Eine Domäne kann eine HTML Oberfläche, eine JSON REST API oder eine Administrationsoberfläche sein.

Die vertikale Trennung von Code in Fachdomänen ermöglicht es Applikationen beliebig zu skalieren, indem sie als Microservices auf mehreren Serverinstanzen laufen können.

Microservices sind lose gekoppelte Applikationen, die ihre Dienste über leichtgewichtige Protokolle wie HTTP oder JSON anbieten. Sie sind leichter zu erweitern, zu warten und auszurollen als monolithische Applikationen.

Mehr über Aimeos

Aimeos on Content Commerce Summit

Content Commerce Summit

Content commerce, the fusion of editorial content and e-commerce, is a growing area gaining more and more importance for businesses.

We have been on the first European Content Commerce Summit in Leipzig (Germany) for two days spotting the latest trends and and held a session about Aimeos as solution for integrating content and e-commerce to interested participants.

The offered sessions had a wide range of topics, from the very interesting talk from Danny Nauth about the secrets of successful content based on the experience of neural marketing, over content along the customer journey by Kai Wermer and the importance of the right content for SEO by Markus Hoevener to tops and flops in content commerce projects by Hannes Richter from Projecter.

Roman Zenner had some bad luck with the beamer but used that problem as a smart excuse for having no presentation at all 😉

To sum up: A really cool event for everyone interested in all aspects of content and e-commerce!

Modern PHP applications up to 110x slower on network storage


Modern PHP applications consists of thousands of PHP files. Even if your own set of files is small, most of the functionality is nowadays provided by 3rd party libraries. Especially composer made it simpler than never before to integrate external libraries and full stack frameworks like Laravel or Symfony contains a lot of them. The Aimeos e-commerce components are not different in this case since this kind of code reuse is the best thing since sliced bread 🙂

Bad old days and brave new world

The downside is that a single request have to touch a lot of PHP files, sometimes up to several hundred. In the “bad old days”, each file must be read from the disk, parsed into byte code and executed by the PHP interpreter if byte code caches like APC wasn’t installed. That took a lot of time and PHP applications could require up to several seconds to finish a request.

Since PHP 5.5, performance improved very much, mostly by the integrated OPcode cache, since PHP7 also by various optimizations inside the PHP interpreter itself. Now, request times of even big PHP applications can be only several 100ms or below. PHP applications can be almost as fast as Java one. Great stuff!

When things turn wrong

There’s only one problem PHP developers and administrators don’t really think about: The applications are only fast as long as the file system the files are stored is fast too! We’re not talking about the sequential read or write performance, which can be fast on network file systems too. It’s the time required for file access that matters if your application consists of several thousand files!


Why the file access you might wondering? The OPcache keeps the files in memory so there’s no file access at all? Wrong! Even if you configured the available memory of the OPcache to be big enough for all files, it checks for file modifications regularly. By default, this check is executed every two seconds. A better value would be 60 seconds or more:

opcache.revalidate_freq = 60

Things will get even worse. For every part of the file path, the meta data is retrieved by the file system. The path “/var/www/html/index.php” will result in requests for “/var”, “/var/wwww”, “/var/www/html” and “/var/www/html/index.php”. You see, this can result in a lot of request for hundreds of files stored deep down in a directory tree.

Real path cache

PHP has a feature called “real path cache” that should mitigate the problem. The thought about the this cache is to reduce the amount of file system requests by storing the absolute path of the requested files. So far so good but it’s value is 16KB by default and this is ridiculously low for modern PHP applications. This results in a constant trashing of the cached entries and renders the cache useless with this size. To avoid this, you should start and test with values of 128KB and more:

realpath_cache_size = 128KB

The network stoage disaster

So, where’s the 110x performance loss? Well, retrieving file meta data is fast when it’s already in the cache of the operating system and don’t have to be revalidated. Then, it doesn’t matter much if the OPcache and real path cache settings are misconfigured.

For our tests, we’ve used the Aimeos Core plus the required libraries and extensions (ca. 6500 files and directories) as reference and executed the following command in the Aimeos directory:

# time (find . > /dev/null)


Let’s test the meta data performance of an Ext4 file system on a local hard disk.

First test:

real    0m2.722s
user    0m0.017s
sys     0m0.130s

Subsequent tests:

real    0m0.043s
user    0m0.003s
sys     0m0.040s

The first time retrieving the meta data of all files and directories from the hard disk lasted 2.722 seconds. If the data is cached by the operating system, only 0.043 seconds are needed.


Now we did the same for NFS, the standard Unix network file system:

First test:

real    0m4.197s
user    0m0.032s
sys     0m0.188s

Subsequent tests:

real    0m0.490s
user    0m0.019s
sys     0m0.080s

The first time, it takes almost 1.5x as long than from a local hard disk and afterwards it needs even 10x longer. Thus, performance of PHP applications over NFS will severely suffer!


Have a look at GlusterFS, another network file system:

First test:

real    0m9.031s
user    0m0.045s
sys     0m0.176s

Subsequent tests:

real    0m4.769s
user    0m0.052s
sys     0m0.156s

Not only that the first time it took more then three times longer than from a local hard disk, the subsequent test have shown a more than 110x worse performance compared to the local hard disk. You can imagine that this will lead to response times of several seconds!


The test results made clear that storing the files of a PHP application on a network storage is really a bad idea. It will make more than a noticeable difference and results in response times which are extremely bad when compared to applications stored on local hard disks.

If you have no other chance than using a network storage, you should at least optimize and test the OPcache and real path cache settings to mitigate the problem. If you use optimal settings, you can decrease the request time penalty to only 2-4 times of a local disk. But it will make a big difference if your customers wait 0.5 or up to 4 seconds!

Posted in PHP

How to use the TYPO3 caching framework with cache groups

The TYPO3 caching framework is a great way to speed up your site if your extension needs to perform tasks that can be cached. In the Aimeos TYPO3 extension, we use it to store content like list or detail views of the web shop to get lightning fast response times. Since version 6.2, a cache can belong to one or more cache groups in the caching framework and it enables editors to control more precisely which caches should be flushed. Unfortunately, this features isn’t well documented yet in the TYPO3 documentation but here’s the remedy!

What are cache groups

In the TYPO3 caching framework, there are currently three types of caches:

  • pages : Front-end related caches
  • system : Low-level caches that shouldn’t normally be flushed
  • all : All other caches that don’t fall into one of the two categories before

Based on the cache type (a.k.a cache group) the caches are flushed on different occasions. This makes a difference for the editors because depending on what they have changed, most often one of the cache groups will be flushed as well. Also, there are at least two options available when editors decide to flush the caches manually:

Aimeos-typo3-cache7The first option will only flush front-end related caches while the second one flushes all caches that are not in the “system” or “pages” cache group. Flushing the “system” caches isn’t that easy any more but those don’t need to be cleared at all by editors. Instead, they are automatically flushed if e.g. a new extension is installed.

Therefore, it makes sense to tell TYPO3 which cache group your cache should be in if you use the caching framework to store data temporarily. This allows at least editors to selectively flush caches instead of the famous “flush all caches to be sure”! More information about which cache in in which cache group can be found in the article about the TYPO3 caching architecture.

Set up your extension cache

To create your own cache in your TYPO3 extension, there are only a few lines in the ext_localconf.php of your extension necessary. You can copy and paste these lines and only have to replace the aimeos string by the name of your own TYPO3 extension:

if( !is_array( $TYPO3_CONF_VARS['SYS']['caching']['cacheConfigurations']['aimeos'] ) ) {
    $TYPO3_CONF_VARS['SYS']['caching']['cacheConfigurations']['aimeos'] = array();
if( !isset($TYPO3_CONF_VARS['SYS']['caching']['cacheConfigurations']['aimeos']['frontend'] ) ) {
    $TYPO3_CONF_VARS['SYS']['caching']['cacheConfigurations']['aimeos']['frontend'] = 'TYPO3\\CMS\\Core\\Cache\\Frontend\\StringFrontend';
if( !isset($TYPO3_CONF_VARS['SYS']['caching']['cacheConfigurations']['aimeos']['backend'] ) ) {
    $TYPO3_CONF_VARS['SYS']['caching']['cacheConfigurations']['aimeos']['backend'] = 'TYPO3\\CMS\\Core\\Cache\\Backend\\DatabaseBackend';
if( !isset($TYPO3_CONF_VARS['SYS']['caching']['cacheConfigurations']['aimeos']['options'] ) ) {
	$TYPO3_CONF_VARS['SYS']['caching']['cacheConfigurations']['aimeos']['options'] = array( 'defaultLifetime' => 0 );
if( !isset($TYPO3_CONF_VARS['SYS']['caching']['cacheConfigurations']['aimeos']['groups'] ) ) {
	$TYPO3_CONF_VARS['SYS']['caching']['cacheConfigurations']['aimeos']['groups'] = array( 'pages' );

The whole if/is_array/isset thing is necessary to allow administrators to overwrite your settings in the typo3conf/LocalConfiguration.php file as the configuration of your extension is loaded afterwards. Thus, it would overwrite any previous configuration settings if you don’t check first if they exist.

There are four settings you can use to configure your cache in the TYPO3 caching framework:

  • frontend : What type of data is accepted
  • backend : Where is the data stored
  • options : Additional configuration settings for the back-end
  • groups : The cache groups your extension cache belongs to

There’s the choice between two cache front-ends and several cache back-ends. A complete list of both including their options is available in the TYPO3 cache front-end/back-end article. The groups setting can be either pages, system, all or a combination thereof. Normally, you should use only one cache group and the group all is the default value if you don’t specify one.

If you use the default database back-end, you should be aware that the default lifetime of cached entries is only ONE HOUR if you don’t set it to a different value or to unlimited (“0”) like in the example above. Don’t forget to set up the “Caching framework garbage collection” scheduler task to clean up old entries. Otherwise, your cache tables will grow infinitely!

Use your new cache

Once set up, you can access the cache everywhere in you TYPO3 extension and you only have to write a few lines:

$name = 'TYPO3\\CMS\\Core\\Cache\\CacheManager';
$manager = \TYPO3\CMS\Core\Utility\GeneralUtility::makeInstance( $name );
$cache = $manager->getCache( 'aimeos' );

They create an instance of the cache manager and retrieve your cache by name. Replace aimeos with the name of your extension resp. the name you’ve used in your ext_localconf.php in the cache setup code. Now you are able to set and get cached entries or remove them from the cache again:

$cache->set( 'auniqueidentifier', $data, array( 'tag1', 'tag2' ) );
$data = $cache->get( 'auniqueidentifier' );
$cache->remove( 'auniqueidentifier' );

So, let’s go and make your TYPO3 extension fast! 🙂

Language mapping in TYPO3 with RealURL and bootstrap package

The cool thing about the TYPO3 bootstrap_package extension is that you can set up a new site within minutes because everything is pre-configured. This also includes the RealURL setup so you already see the page titles in your URLs and the language parameter value is mapped too.

The dark side

You will face the dark side of the bootstrap package when you try to use different languages than those configured as examples. There’s only a mapping between the value of sys_language_id and English, German and Danish available and if you try to add a mapping manually, RealURL immediately switches to the “manual mode” where you have to configure everything yourself once again.

Loosing the auto-configuration implies a lot of work, especially if you are using the Aimeos web shop extension or similar ones, which add a lot of RealURL rules out of the box. Thus, it’s very compelling to try to overwrite only the RealURL configuration which is responsible for the language ID/code mapping.

The bright side

Fortunately, the bootstrap package allows you to overwrite its RealURL auto-configuration rules including the preVars section which is responsible for the language mapping.

First of all, you need an extension for your project where you can add the RealURL rules to. You can either use the Kickstarter extension or the Aimeos extension creator. The automatic configuration requires a PHP class and method that will add new or overwrite existing RealURL rules. You should add it in the Classes/Realurl.php file of your extension. Copy this code to your new file:

  1. <?php
  3. namespace <vendor>\<extname>;
  5. class Realurl
  6. {
  7.         public function addAutoConfig( array $params, $pObj )
  8.         {
  9.                 $params['config']['preVars'][1] = array(
  10.                         'GETvar' => 'L',
  11.                         'valueMap' => array(
  12.                                 'en' => '0',
  13.                                 'de' => '1',
  14.                                 '...' => '2',
  15.                         ),
  16.                         'noMatch' => 'bypass',
  17.                 );
  19.                 return $params['config'];
  20.         }
  21. }

The addAutoConfig() method replaces the second entry of the “preVars” array, which are exactly the language mapping rules in the bootstrap package. You can change or add as many entries in the “valueMap” array as you like. The array keys are the strings that will be shown in the URL while the values are the sys_language_id values of the languages you’ve configured in your TYPO3 page tree and TypoScript configuration.

Remember to replace the <vendor> and <extname> placeholders in the namespace statement in the third line of the file. <vendor> can be an arbitrary string while <extname> must be your extension name which the first letter in upper case as well as all letters that follow an underscore, e.g. “bootstrap_package” becomes “BootstrapPackage”.

Afterwards, you have to add your class to the RealURL hook that will call your method when collecting the rules for auto-configuration. This is done in the ext_localconf.php of your extension by adding these lines:

  1. $GLOBALS['TYPO3_CONF_VARS']['SC_OPTIONS']['ext/realurl/class.tx_realurl_autoconfgen.php']['extensionConfiguration']['<extname>'] =
  2. 	'EXT:<extname>/Classes/Realurl.php:<vendor>\\<extname>\\Realurl->addAutoConfig';

As above, you have to replace the <vendor> and <extname> placeholders with the same values you’ve used for the namespace statement.

The order you will install the extensions matters! Make sure the RealURL extension is installed first, then the bootstrap package and afterwards the rest of the extensions which contain RealURL rules including your new one.

Finally, clear your caches (TYPO3 general and frontend caches) and delete the typo3conf/realurl_autoconf.php file. Then, your languages should appear correctly mapped in your URLs.

Setup a working Neos CMS environment

This article is for Neos 2.0 and thing migt have changed in the meantime. At least the software packages are required in newer versions.

Neos is well known for its well designed interface for editors who quickly fall in love with the system. On the other hand, it’s also infamous for being hard to set up by starters due to its high demands towards the hosting environment. Someone also compared it to walking on a slack line: It’s easy to stumble and to get frustrated.

Neos error page

This article focuses on creating a hosting environment suited for Neos so you will be able to finish the Neos setup and get a running instance afterwards. I’m using Ubuntu 14.04 as reference but it also applies to other Linux distributions like Fedora, only the path names might differ a bit (/etc/apache2/ vs. /etc/httpd/).

Required software

At first, you need to make sure that the required software is installed. Apache, MySQL and PHP might be obvious but that’s not enough. Especially the additional Apache and PHP modules are important to get it running at all. You will have a good start with these packages:

  • Apache 2.2+
  • MySQL 5.1+
  • PHP 5.3.2+
    • mbstring
    • tokenizer
    • pdo_mysql

Even if the minimum requirement for PHP is version 5.3.2, it’s better to use newer versions for performance reasons. Also, the requirements for Neos will be PHP 5.5+ for Neos 2.0.

You can install the required packages via the package manager of your distribution (apt-get for Ubuntu/Debian, rpm for Fedora/SuSE):

  1. sudo apt-get install apache2 mysql-server libapache2-mod-php5 php5 php5-mysql

To check if all required PHP extensions are available, the easiest way is to place a short PHP file named info.php in your Apache document root directory (usually /var/www/) containing this content:

  1. <?php phpinfo();

If you open http://localhost/info.php in your browser, the output will contain the environment, the configuration settings and the installed modules available in your installation. Search for the strings “mbstring”, “tokenizer” and “pdo_mysql”. There should be sections with this names in the rendered page.

Apache configuration

Most often, the first problems occur due to an insufficient Apache configuration. Neos uses .htaccess files and needs some modules that are not activated by default all the times. The “mod_rewrite” and “mod_env” modules are the best examples for this. To enable them in your Apache configuration, you need to execute these commands on the command line:

  1. sudo a2enmod rewrite
  2. sudo a2enmod env

If you got the message that one or both modules had been already enabled then this was set up correctly before.

The VHost configuration is another common source of problems and you have to adapt it for sure. The default configuration is located in /etc/apache2/sites-enabled/000-default.conf (in Fedora or other Linux distributions the files might be located somewhere else). In this file, change the DocumentRoot to the Neos ./Web/ directory, e.g.

  1. DocumentRoot /var/www/Neos/Web

Furthermore, you need to add a “Directory” directive below the “DocumentRoot” line to allow the Neos .htaccess file to add rewrite rules and some settings:

  1. <Directory /var/www/Neos/Web/ >
  2. 	AllowOverride All
  3. </Directory>

If there’s already a “Directory” section available, replace it with the lines above.

Keep care that the named directory in the “Directory” directive is exactly the same as in the “DocumentRoot” AND contains a trailing slash (/).

The last task is to restart Apache:

  1. sudo service apache2 restart

Now you should be at able to open at least http://localhost/setup in your browser but beware: http://localhost/index.php/setup still won’t work!

PHP configuration

There are various PHP settings that can cause your Neos installation to fail. First of all you should have a look into the output of the info.php file again and check if one of the required PHP functions is disabled. This is often the case for security reasons in shared hosting environments and this leads to errors afterwards. Especially check the disable_functions line in the core section for:

  • system
  • shell_exec
  • escapeshellcmd
  • escapeshellarg

If non of these function is listed there, then you are on the safe side.

The next problems are lurking in your PHP configuration but that can be fixed either by adding a few lines to your /etc/php5/apache2/php.ini or to the .htaccess file in the Neos ./Web/ directory. When using the php.ini file you should add these lines at the end of the file:

  1. memory_limit = 256M
  2. magic_quotes_gpc = off # only for PHP 5.3 to be sure
  3. date.timezone = Europe/Berlin # or any other valid timezone
  4. xdebug.max_nesting_level = 500 # only if XDebug extension is installed

In case you are allowed to overwrite settings also via the .htaccess file, you can add these lines to the ./Web/.htaccess of your Neos installation:

  1. php_value memory_limit = 256M
  2. php_flag magic_quotes_gpc off # only for PHP 5.3 to be sure
  3. php_value date.timezone Europe/Berlin # or any other valid timezone
  4. php_value xdebug.max_nesting_level 500 # only if XDebug extension is installed

Neos can be very memory consuming so even the standard memory limit of 128MB can be to low. Errors due to an insufficient memory limit are written to the Apache error log file (usually in /var/log/apache2/error.log) so it’s a good idea to keep an eye on the content of that file.

The magic_quotes_gpc directive has been removed in PHP 5.4 and was already deprecated before. To disable it for PHP 5.3 is only a safety measure for installations that have been constantly updated during the last years.

For the timezone value you can use any valid timezone if it fits better for your location. A list of valid timezones is available at the PHP web site.

When you’ve installed the PHP XDebug extension you might run into error messages like Maximum function nesting level of '...' reached. Only in this case you need to increase the XDebug nesting level for functions.


Finally, you should be able to open the Neos setup page in your browser, that is available at http://localhost/setup. The rest of the installation is usually a non-brainer due to the nice setup routine provided by the Neos installer.

Afterwards you can enjoy Neos. Happy editing! 🙂