Import CSV with HTTP:// images

VerteXVaaR · Post by **VerteXVaaR** » 01 Jun 2021, 12:56

Hello,

i am currently evaluating aimeos for a complex project.
My Setup:
* TYPO3 v10.4.16
* aimeos/aimeos-typo3 21.4.2
* Dockerized Dev Env
* PHP 7.4.19
* MySQL 5.7.34

I am trying to import products from a CSV file, which works just fine.
The images, however, are only provided via HTTP Url, like "https://cdn.example.com/images/products ... 31dd97ce51" (this is a random URL, just to show an example). The URL returns an image and the response header contains the correct mime type.

I know external URLs are supported, because the demo shop's images are HTTP Urls and i can set them in the database.

My Import config:

Code: Select all

controller {
    jobs {
        product {
            import {
                csv {
                    skip-lines = 1
                    location = fileadmin/shop-import/
                    mapping {
                        item {
                            0 = product.label
                            3 = product.code
                        }

                        text {
                            4 = text.content
                        }

                        media {
                            17 = media.url
                        }

                        price {
                            18 = price.currencyid
                            10 = price.quantity
                            6 = price.value
                            7 = price.taxrate
                        }

                        supplier {
                            2 = supplier.code
                        }
                    }
                }
            }
        }
    }
}

First of all: The import fails when i add "17 = media.url". I debugged the code and found an exception thrown in vendor/aimeos/aimeos-core/lib/mwlib/src/MW/Filesystem/Standard.php@148

Code: Select all

Couldn't get file mtime for "/project/app/private/uploads/tx_aimeos/https://cdn.example.com/images/products?id=fe9cb6ee-b8bd-4440-82a5-df31dd97ce51"

The message is pretty clear. aimeos prefixes the external url with the local base "/project/app/private/uploads/tx_aimeos/" because it thinks it is a relative file path.

First of all, i think the scheduler task should output some errors when an exception was thrown. This would help to discover and investigate import errors.

My actual question:
How can i configure the importer to accept the HTTP URL as is and to treat it as such? (Just save the URL, do not download the file to the local file system)
Additionally: how can i configure the importer to fetch the mime type from the response headers?

Post by **aimeos** » 01 Jun 2021, 13:14

VerteXVaaR wrote: ↑01 Jun 2021, 12:56 How can i configure the importer to accept the HTTP URL as is and to treat it as such? (Just save the URL, do not download the file to the local file system)

Can you investigate, why the has() method returns TRUE for the URL. Here are the relevant lines:

- https://github.com/aimeos/ai-controller ... .php#L228
- https://github.com/aimeos/aimeos-core/b ... #L182-L185
- https://github.com/aimeos/aimeos-core/b ... #L370-L379

VerteXVaaR wrote: ↑01 Jun 2021, 12:56 Additionally: how can i configure the importer to fetch the mime type from the response headers?

That's not necessary because the mimetype of the generated preview images is used to set that field automatically:
https://github.com/aimeos/aimeos-core/b ... #L162-L206

VerteXVaaR · Post by **VerteXVaaR** » 02 Jun 2021, 09:27

Can you investigate, why the has() method returns TRUE for the URL. Here are the relevant lines:

Code: Select all

$fs->has( $url )

is never executed because the first part of the condition already evaluates to true, which is

Code: Select all

$refItem->getPreviews() === []

When i change the code so that the whole condition evaluates to false and no refItem is created, the media will not be imported.

I changed the code to make it work, but since my first contact with aimeos was yesterday, i'm not quite sure about the code style and contribution workflow.

Code: Select all

			} elseif( $fs->has( $url )
				&& (
					$refItem->getPreviews() === []
					|| $refItem->getUrl() !== $url
					|| !( $fs instanceof \Aimeos\MW\Filesystem\MetaIface )
					|| date( 'Y-m-d H:i:s', $fs->time( $url ) ) > $refItem->getTimeModified()
				)
			) {
				$refItem = \Aimeos\Controller\Common\Media\Factory::create( $context )->scale( $refItem->setUrl( $url ) );
			} elseif( 0 === strpos( $url, 'http://' ) || 0 === strpos( $url, 'https://' ) ) {
				$refItem->setUrl( $url );
			}

What's missing here is the determination of the mimetype. The "Rescale product images" job will not resize those images.

Post by **aimeos** » 02 Jun 2021, 09:31

Just create a pull request at the htttps://github.com/aimeos/ai-controller-jobs repository. It's easier to care about coding style there.

VerteXVaaR · Post by **VerteXVaaR** » 02 Jun 2021, 10:28

Here you go: https://github.com/aimeos/ai-controller-jobs/pull/30

This topic can be closed, as my question is the result of a bug and a PR is submitted.

Post by **aimeos** » 29 Jun 2021, 05:58

In Aimeos, there are objects available for dealing with CSV files:
https://aimeos.org/docs/latest/infrastr ... ite-files/

For saving/fetching data, use the appropriate manager:
https://aimeos.org/docs/latest/models/managing-items/

Help for Aimeos