Importing product images takes too long.

[Doku]

I need to import product images to my Aimeos shop.
I have about 50,000 products inside my Shop, each product should have one of two pictures.

I'm using Aimeos 22.10 inside TYPO3 11.5. I'm importing through the default csv product import with custom mappings (see below).

Linux: Ubuntu 22.04.4 LTS
PHP : 8.2.23

While importing 40 pictures takes just 30 sec, importing 100 doesn't stop running even after two hours.

Which configurations should I check to speed up the import? Is there a faster way to import pictures?
If not, is there a way to set pictures for all product of a category?

Is there a way to log progress or timings?
Could this be a memory Problem? (Looking at the numbers, it shouldn't be. Anyway, I tried lowering the max-size value)

noteable Settings:

Code: Select all

skip-lines = 2

// when importing less than 200 products this should have an effekt?
max-size = 200 // I've also tried the default 1000 but it seems to make no difference. 

Mapping (TypoScript):

Code: Select all

            
mapping{
    item {
        0 = product.code
    }
    media {
        6 = media.type
        13 = media.label
        39= media.url
    }
}

// Example
0:   1463755,
6:   "default",
13:  "picture",
39:  "https://example.org/fileadmin/picture.jpg"

Code: Select all

Memory:
               total        used        free      shared  buff/cache   available
Mem:         7923528     2085600      290136      474616     5547792     5157248
Swap:              0           0           0

[aimeos]

If you import images from external domains, the network latency is usually the biggest source of delays and long import times. You can add timing information for the media import in the aimeos/ai-controller-jobs package here:
https://github.com/aimeos/ai-controller ... #L217-L245

Adding output to the scale() method of the media controller can give you detailed information:
https://github.com/aimeos/aimeos-core/b ... #L182-L208

Note that Aimeos 22.10 is supported only by the ELTS program of the Aimeos company:
https://aimeos.com/support#c427

[Doku]

Thanks for the reply!
After looking at some measurements. I noticed my PHP maximum execution time was set to 30s
I roughly get two products per second now.

Do I have to increase the "maximum execution time" to the time the import takes, even if that would mean raising it to multiple hours or days?

If you import images from external domains, the network latency is usually the biggest source of delays and long import times. You can add timing information for the media import in the aimeos/ai-controller-jobs package here:
https://github.com/aimeos/ai-controller ... #L217-L245

Is there a way to import local images? I tried the absolute /path/to/file.jpeg, but couldn't get it working.
Is there a way to find all images without pictures?

Note that Aimeos 22.10 is supported only by the ELTS program of the Aimeos company:
https://aimeos.com/support#c427

We are currently in the process of upgrading.

[aimeos]

You should never execute an importer via a HTTP request. Instead, start importers only via the command line or cronjobs where the max execution time should be unlimited.

The image paths for importing from local file system must be always relative to the virtual file system defined here:
https://github.com/aimeos/aimeos-typo3/ ... hp#L46-L51

[Doku]

I started the Import via the TYPO3 Scheduler. But executed it manually, could this cause my script to depend on the execution time? Because after setting it to a higher value, my execution worked.

What is the recommended TYPO3 way of starting an import? Should I run the Symfony command (JobsCommand) via console?

Thank You for your help!

[aimeos]

I started the Import via the TYPO3 Scheduler. But executed it manually, could this cause my script to depend on the execution time? Because after setting it to a higher value, my execution worked.

Yes

What is the recommended TYPO3 way of starting an import? Should I run the Symfony command (JobsCommand) via console?

Yes