Gigacommerce

From Aimeos documentation

<languages/>


<translate>

Setup

To create your own #gigacommerce setup, you should use at least hardware fulfilling that specs:

  • 8 CPU cores
  • 16+ GB RAM
  • 550 GB SSD

Furthermore, you need these software:

For the #gigacommerce setup, please install:

Configuration

This section contains the setup of the #gigacommerce demo setup.

Setup resources

You need a MySQL 5.7+ and ElasticSearch 5+ server installed. Afterwards, you have to create database (e.g. "aimeos") and configure how they can be accessed:

'resource' => [
    'db' => [
        'adapter' => 'mysql',
        'host' => '127.0.0.1',
        'port' => '3306',
        'database' => 'aimeos',
        'username' => '<user>',
        'password' => '<pass>',
        'stmt' => [
            "SET SESSION sort_buffer_size=2097144; SET NAMES 'utf8mb4'; SET SESSION sql_mode='ANSI'"
        ],
        'limit' => 3,
        'defaultTableOptions' => [
            'charset' => 'utf8mb4',
            'collate' => 'utf8mb4_bin',
        ],
    ],
    'es' => [
        'hosts' => ['127.0.0.1:9200'],
        'index' => 'aimeos',
        'norefresh' => true, // only for indexing
        'selectorClass' => '\Elasticsearch\ConnectionPool\Selectors\StickyRoundRobinSelector',
        'settings' => [
            'max_result_window' => 10000,
            'number_of_shards' => 8,
            'number_of_replicas' => 0,
            'refresh_interval' => '-1',
        ]
    ],
    'es-stock' => [
        'hosts' => ['127.0.0.1:9200'],
        'index' => 'aimeos_stock',
        'norefresh' => true, // only for indexing
        'selectorClass' => '\Elasticsearch\ConnectionPool\Selectors\StickyRoundRobinSelector',
        'settings' => [
            'max_result_window' => 10000,
            'number_of_shards' => 8,
            'number_of_replicas' => 0,
            'refresh_interval' => -1,
        ],
    ],
],

The database setup is pretty straight forward, in the two ElasticSearch configuration blocks there are a few settings to explain:

number_of_shards 
An index is divided into shards and they can be distributed across an ElasticSearch cluster. Eight shards for product and stock data are a good value for a #gigacommerce setup
number_of_replicas 
The number of replica shards. Good for load distribution and failover in a cluster but bad if you configure them during creating the index because they are synced between the cluster
refresh_interval 
How often indexed documents are written to disk. The value "-1" disables refreshing and data will be only written if the index buffer of the ElasticSearch server is filled up with documents on a refresh is explicitely requested
max_result_window 
Maximum number of documents that will be fetched when searching for products or stocks. This also limits the products shown in the product list views
norefresh 
Avoid sending refresh requests. This is only useful to speed up indexing but must not be used afterwards
selectorClass 
How ElasticSearch nodes in a cluster are selected. StickyRoundRobinSelector is best for normal operation, RoundRobinSelector for indexing

Use ElasticSearch

By default, Aimeos will use the database configured as db. To use ElasticSearch for product and stock data, you need to add these settings:

'mshop' => [
    'index' => [
        'manager' => [
            'name' => 'Elastic',
        ]
    ],
    'product' => [
        'manager' => [
            'name' => 'Elastic',
        ]
    ],
    'stock' => [
        'manager' => [
            'name' => 'Elastic',
        ]
    ]
],

Frontend optimization

In a standard setup, Aimeos caches list and detail views in the database. Due to the amount of data and varations possible in the #gigacommerce setup, this isn't desired. Use these settings to disable caching list and detail views:

'client' => [
    'html' => [
        'catalog' => [
            'lists' => [
                'cache' => false,
                'standard' => [
                    'subparts' => ['items'],
                ],
            ],
            'detail' => [
                'cache' => false,
            ],
            'stage' => [
                'standard' => [
                    'subparts' => [],
                ],
            ],
        ],
    ],
],

It will also disable two features (promotional product list in list view and last/next product link in detail view) that require at least one more ElasticSearch query and are turned of for raw speed.

Demo data

To generate the same set of demo data as in the #gigacommerce demo, use these settings:

'setup' => [
    'unitperf' => [
        'max-batch' => 10000,
        'num-catlevels' => 3,
        'num-categories' => 1000,
        'num-catproducts' => 10000,
        'num-prodvariants' => 100,
    ],
],

The meaning of the configuration values is:

max-batch 
Maximum number of ElasticSearch documents (for products and articles) sent at once. This is mainly useful to reduce the memory consumption
num-catlevels 
Number of levels the category tree will consist of not including the root node. This correlates to the total number of categories
num-categories 
Total number of leaf categories that will be created. They will be split into a tree of categories depending the number of level, e.g. in the demo setup a category tree of pow( 1000, 1/3) = 10 so 10 * 10 * 10 = 1000 categories will be created. The category tree will have 10 second level categories, 100 third level categories and 1000 forth level categories
num-catproducts 
Number of unique products per leaf category. Each 10th product will also be added to the parent category
num-prodvariants 
Number of unique articles per product using width/length/size as variant attributes

You can use different settings for the categories and products but you should use multiples of 10 for each setting. You can also use another distribution of categories and products like 100 categories, 10000 products/category, 1000 articles/product or 10000 categories, 10000 products/category, 10 articles/product. Only keep in mind that if the product or article value is above 10000, you have to adapt the max_result_window settings of your ElasticSearch configuration and MySQL could start choking due to the number of records in the mshop_catalog_list table (category -> product references).

Performance

'apc_enabled' => true,
'pcntl_max' => 24,

For maximum performance APCu must be enabled and this requires the PHP apcu extension to be installed. This will cache the used configuration settings and translations in the shared memory of the web server. How to enable APCu in Aimeos depends on the integration. Please check the sample configuration files for Laravel and Symfony resp. the Aimeos settings in the Extension Manager for TYPO3.

The optimal number of parallel processes for task like generating the demo data depends on your server hardware. A good starting point for tests is twice the number of processors used by ElasticSearch, so the processors are always fully occupied.

Nevertheless, indexing the products is an IO intensive task and most likely, the storage will be your bottleneck, not the CPUs.

Generating data

Now you can start generating the #gigacommerce demo data. Depending on your host application, you have to use:

Laravel 
php -d memory_limit=-1 ./artisan aimeos:setup unitperf default
Symfony 
php -d memory_limit=-1 ./bin/console aimeos:setup unitperf default
TYPO3 
It's currently not possible to generate the demo data set by the scheduler but you can use the setup script from the Aimeos Core

Now you have to wait. Depending on your hardware it will take a long time until the data is indexed in Elasticsearch. Using a single ElasticSearch node with 16 CPU cores, 48 GB RAM and SSDs it takes approx. 35h until the process finishes. If you have a cluster of several nodes, you only need a fraction of that time because the IO is distributed across all nodes.

More performance

To get the best performance in the frontend, you should merge the segments in the index to only one segment by using:

curl -X POST http://127.0.0.1:9200/aimeos/_forcemerge?max_num_segments=1

curl -X POST http://127.0.0.1:9200/aimeos_stock/_forcemerge?max_num_segments=1

After merging the segments in all shards, you will have ca. 465 GB occupied on your SSDs.

</translate>