Product import takes too long

[kdim95]

Laravel framework version: 11.3.1
Aimeos Laravel version: 2023.10.8
PHP Version: 8.2.17
Environment: Linux
aimeoscom/ai-elastic: 2023.04.*

Hello,

I am running an import of 122,000 products and the entire import process takes around 2 hours.
Is it normal for the import process to take this long?
What can I do to make the import time shorter?

I'm using only the elastic index, this is my configuration:

Code: Select all

return [
    'resource' => [
        'es' => [
            'hosts' => [
                '127.0.0.1:9200',
            ],
            'index' => 'aimeos',
            // 'SSLVerification' => false, // for self-signed certificates
            // 'basicAuthentication' => ['elastic', '<password>'], // ElasticSearch 8+
            'selectorClass' => '\Elasticsearch\ConnectionPool\Selectors\StickyRoundRobinSelector',
            'settings' => [
                'number_of_shards' => 4, // Distribute data across multiple nodes ( large indexes are split into smaller 'shards' )
                'number_of_replicas' => 3, // Number of copies of primary shards ( redundancy and search speed )
                'max_result_window' => 200000, // maximum number of results retrieved
                // 'refresh_interval' => -1, // for initial indexing only
            ],
            // 'norefresh' => false, // for initial indexing only
        ],
    ],

    'mshop' => [
        'index' => [
            'manager' => [
                'name' => 'Elastic',
                'attribute' => [
                    'name' => 'Elastic',
                ],
                'catalog' => [
                    'name' => 'Elastic',
                ],
                'price' => [
                    'name' => 'Elastic',
                ],
                'supplier' => [
                    'name' => 'Elastic',
                ],
                'text' => [
                    'name' => 'Elastic',
                ],
            ],
        ],
        'product' => [
			'manager' => [
				'name' => 'Elastic',
				'lists' => [
					'name' => 'Elastic',
					'type' => [
						'name' => 'Elastic',
					],
				],
				'property' => [
					'name' => 'Elastic',
					'type' => [
						'name' => 'Elastic',
					],
				],
				'type' => [
					'name' => 'Elastic',
				],
			]
		],
    ]
];

Best regards

[aimeos]

Which importer do you use? CSV, XML or an own implementation?

[kdim95]

Hello,

I am using the default XML importer included with Aimeos.
With the command php artisan aimeos:jobs product/import/xml.

[kdim95]

Hello,

Do you have any update on this?

Best regards

[nos3]

The XML importer is the fastest standard option but it still fetches the products, updates them and stores everything back. When using ElasticSearch and your extension, it's much faster to assign an ID locally and just store/overwrite the data in ES without fetching the products first. Then, it's possible to import 100k products in minutes instead of two hours.

[kdim95]

By "storing locally" you mean having the products in both the database and elastic?
E.g. changing the config to have the products in both the database and in the elastic index?

[nos3]

No, I've said "assign an ID locally and just overwrite the data in ES (without fetching the products first)". Products need to be in ES only but not with an autogenerated ID from ES. Then, you can overwrite the products in ES without the need to fetch them first.

[kdim95]

Hello,

I'm afraid I don't understand you, am I supposed to change the default import logic of your XML importer?
What and where exactly should I change?

Best regards

[aimeos]

We've added a new config option for the product XML importer in the master branch which allows replacing products by their "ref" value when using document-oriented storages like ElasticSearch. You can check the commit so see what that means:
https://github.com/aimeos/ai-controller ... 3f96094a41