AWS S3 filesystem CDN

Yafar_iSi · 13. August 2020 um 10:24

I followed the S3 documentation (https://docs.shopware.com/en/shopware-platform-dev-en/how-to/use-s3-datastorage) but is not very complete
I manage to store the resources in the AWS bucket but it has some performance issues
It does not compress the files

This is without the CDN

And this is with the CDN active

This is my code in config/packages/shopware.yml:

shopware:
    filesystem:
      public:
        url: "https://s3.eu-central-1.amazonaws.com/%env(AWS_BUCKET)%"
        type: "amazon-s3"
        config:
            bucket: "%env(AWS_BUCKET)%"
            region: "eu-central-1"
            endpoint: "https://s3.eu-central-1.amazonaws.com/%env(AWS_BUCKET)%"
            root: "/"
            options:
              visibility: "public" # On private adapters need this to be private
      theme:
        url: "https://s3.eu-central-1.amazonaws.com/%env(AWS_BUCKET)%"
        type: "amazon-s3"
        config:
            bucket: "%env(AWS_BUCKET)%"
            region: "eu-central-1"
            endpoint: "https://s3.eu-central-1.amazonaws.com/%env(AWS_BUCKET)%"
            root: "/"
            options:
              visibility: "public" # On private adapters need this to be private
      asset:
        url: "https://s3.eu-central-1.amazonaws.com/%env(AWS_BUCKET)%"
        type: "amazon-s3"
        config:
            bucket: "%env(AWS_BUCKET)%"
            region: "eu-central-1"
            endpoint: "https://s3.eu-central-1.amazonaws.com/%env(AWS_BUCKET)%"
            root: "/"
            options:
              visibility: "public" # On private adapters need this to be private
      sitemap:
        url: "https://s3.eu-central-1.amazonaws.com/%env(AWS_BUCKET)%"
        type: "amazon-s3"
        config:
            bucket: "%env(AWS_BUCKET)%"
            region: "eu-central-1"
            endpoint: "https://s3.eu-central-1.amazonaws.com/%env(AWS_BUCKET)%"
            root: "/"
            options:
              visibility: "public" # On private adapters need this to be private

Do I miss something?

It pushes my site performance (lighthouse) like 20points down

musikdiscount24 · 19. Oktober 2020 um 14:43

Hello,

have you found a solution for your problem?

Would be glad to know.

Greetings

Yafar_iSi · 23. Oktober 2020 um 08:06

Yes, I found the reason and the solution

Feel free to find me in Slack „shopwarecommunity.slack.com“ @ TaiKamilla

The problem was that my SW config file config/packages/shopware.yml for the CDN and the CDN was incomplete.

The problem is that I was using the ‚url: „https://s3.eu-central-1.amazonaws.com/%env(AWS_BUCKET)%“‘

and that address is direct to the bucket and does not accept headers, you need to do it to the cloudfront service instead

SOLUTION

For this, you need an AWS bucket and an AWS cloudfront to the bucket, make sure you can access the files from the bucket from the cloudfront provided URL

Then

Go to the .env file on your SW installation and add this

AWS_DOMAIN=„CLOUDFRONT_PROVIDED_URL.com“
AWS_ACCESS_KEY_ID=HERE_YOUR_AWS_ACCESS_KEY_ID
AWS_SECRET_ACCESS_KEY=HERE_YOUR_AWS_SECRET_ACCESS_KEY
AWS_BUCKET=THE_NAME_OF_YOUR_BUCKET

2 Then go to config/packages/shopware.yml and replace the content with this (feel free to modify „region“, „endpoint“ and „endpoint“ or anything else)

shopware:
    filesystem:
      public:
        url: "https://%env(AWS_DOMAIN)%"
        type: "amazon-s3"
        config:
            bucket: "%env(AWS_BUCKET)%"
            region: "eu-central-1"
            endpoint: "https://s3.eu-central-1.amazonaws.com/%env(AWS_BUCKET)%"
            root: "/"
            options:
              visibility: "public" # On private adapters need this to be private
      private:
        type: "amazon-s3"
        config:
            bucket: "%env(AWS_BUCKET)%"
            region: "eu-central-1"
            endpoint: "https://s3.eu-central-1.amazonaws.com/%env(AWS_BUCKET)%"
            root: "/"
            options:
              visibility: "private" # On private adapters need this to be private
      theme:
        url: "https://%env(AWS_DOMAIN)%"
        type: "amazon-s3"
        config:
            bucket: "%env(AWS_BUCKET)%"
            region: "eu-central-1"
            endpoint: "https://s3.eu-central-1.amazonaws.com/%env(AWS_BUCKET)%"
            root: "/"
            options:
              visibility: "public" # On private adapters need this to be private
      asset:
        url: "https://%env(AWS_DOMAIN)%"
        type: "amazon-s3"
        config:
            bucket: "%env(AWS_BUCKET)%"
            region: "eu-central-1"
            endpoint: "https://s3.eu-central-1.amazonaws.com/%env(AWS_BUCKET)%"
            root: "/"
            options:
              visibility: "public" # On private adapters need this to be private
      sitemap:
        url: "https://%env(AWS_DOMAIN)%"
        type: "amazon-s3"
        config:
            bucket: "%env(AWS_BUCKET)%"
            region: "eu-central-1"
            endpoint: "https://s3.eu-central-1.amazonaws.com/%env(AWS_BUCKET)%"
            root: "/"
            options:
              visibility: "public" # On private adapters need this to be private

musikdiscount24 · 26. Oktober 2020 um 13:12

Hello,

ah, i see. Thank you!

But as far as i can see there are no variables for „AWS_SECRET_ACCESS_KEY“ and „AWS_ACCESS_KEY_ID“ in the shopware.yml file. Or is it not necessary to set these at options or wherever?

Is it not possible like this?!

shopware:
    filesystem:
      public:
        url: "https://%env(AWS_DOMAIN)%"
        type: "amazon-s3"
        config:
            bucket: "%env(AWS_BUCKET)%"
            region: "eu-central-1"
            endpoint: "https://s3.eu-central-1.amazonaws.com/%env(AWS_BUCKET)%"
            root: "/"
            options:
              visibility: "public" # On private adapters need this to be private
              credentials:
                key: "%env(AWS_ACCESS_KEY_ID)%"
                secret: "%env(AWS_SECRET_ACCESS_KEY)%"

Thank you for your answers!!!

Yafar_iSi · 26. Oktober 2020 um 13:49

Hey, did you try my solution anyway?

“AWS_SECRET_ACCESS_KEY” and “AWS_ACCESS_KEY_ID” it does not show in shopware.yml
but is an SW naming for the bucket credentials in /vendor/aws/aws-sdk-php/src/Credentials/CredentialProvider.php

musikdiscount24 · 26. Oktober 2020 um 14:54

Hello,

yes, i tried it and it works. I was just wondering about the fact, that i don’t need the Access-Key.

Thank you! You helped me a lot!

Feel free to find me in Slack „shopwarecommunity.slack.com“ @ TaiKamilla

Btw. i am not invited

Best regards!

shyim · 26. Oktober 2020 um 14:57

@musikdiscount24‍ You can join with https://slack.shopware.com

Yafar_iSi · 26. Oktober 2020 um 14:58

You don’t need to be invited
Just go here https://slack.shopware.com
and create an account

Also would be great if you can mark the answer as solved

Yafar_iSi · 28. Oktober 2020 um 09:06

Hi @musikdiscount24‍

I have a question since you use an AWS CDN

I am experiencing slow down on theme and plugin installation

without the CDN the theme compilation takes around 30-40 seconds

but with the CDN is like 180 seconds (3min)

are you experiencing a slow down in this?

klug · 11. August 2021 um 09:10

Hi Yafar_iSi,

i have tried your solution but it doesn’t work.

In the .env file:
Should the parameter be inside ‚xxx‘ or without ‚‘?

like:
AWS_DOMAIN=‚XXX‘
AWS_ACCESS_KEY_ID=‚XXX‘
AWS_SECRET_ACCESS_KEY=‚xxx‘
AWS_BUCKET=‚xxx‘

are these parameters correct?

public:
        url: "https://%env(AWS_DOMAIN)%"
        type: "amazon-s3"
        config:
            bucket: "%env(AWS_BUCKET)%"
            region: "eu-central-1"
            endpoint: "https://s3.eu-central-1.amazonaws.com/%env(AWS_BUCKET)%"
            root: "/"
            options:
              visibility: "public" # On private adapters need this to be private

Best regards!

toco3 · 5. November 2023 um 13:04

Thank you for all your hints.
I’d had also some Issues connecting a Shopware 6.5.6.1 to AWS S3 / CloudFront

I need the following adaptions to config/packages/z-shopware.yml in comparison to the one of @Yafar_iSi :

endpoint: "https://s3.eu-central-1.amazonaws.com"

if I add the bucket as proposed the bucket name is twice in the url and does not work

           root: ""

If I add „/“ for root I get a directory with the name „/“ in the root of my s3 bucket with all stuff inside…

Beside configuring the CNAMES for S3 and CloudFront I had some CORS issues.

For CloudFront I use this settings for the behaivor:

and added a custom Header to the S3 origin:

[
    {
        "AllowedHeaders": [
            "Authorization"
        ],
        "AllowedMethods": [
            "GET",
            "HEAD"
        ],
        "AllowedOrigins": [
            "*"
        ],
        "ExposeHeaders": [],
        "MaxAgeSeconds": 3000
    }
]

For the S3 Bucket I have this access policies:

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Principal": "*",
            "Action": [
                "s3:GetObjectVersionAcl",
                "s3:GetObjectVersionTorrent",
                "s3:GetObject",
                "s3:GetObjectAcl",
                "s3:GetObjectTorrent",
                "s3:GetObjectVersion"
            ],
            "Resource": "arn:aws:s3:::static.shirt-helden.shop/*"
        }
    ]
}

and this CORS settings:

[
    {
        "AllowedHeaders": [
            "Authorization"
        ],
        "AllowedMethods": [
            "GET",
            "HEAD"
        ],
        "AllowedOrigins": [
            "*"
        ],
        "ExposeHeaders": [],
        "MaxAgeSeconds": 3000
    }
]

With all my tests, sometimes the directories and their content like bundles, theme, thumbnail got created - some times not.

These commands helped to get everything up and running:
./bin/console cache:clear
./bin/console theme:compile
./bin/console media:generate-thumbnails --strict -v
./bin/console system:update:finish

Perhaps this can help someone having the same issues as me - took me about an day to figure this all out

toco3 · 22. Dezember 2023 um 16:51

one additional information:

for everyone who can open the backend only in Firefox:

make sure that you disabled the Admin Worker in config/packages/z-shopware.yaml:
shopware:
admin_worker:
enable_admin_worker: false

(Admin worker should be disabled on production server anyway)

AlexGalax · 6. März 2024 um 07:05

I have the same issue – some thumbnails just do not get generated.

When I run media:generate-thumbnails --strict -v, I get the following output for a lot of files:

  Cannot process file 2505600920-06 (id: f57c3d5ae20d4b208a111cc90d71d7fe) due error: The file for media object with id f57c3d5ae20d4b208a111cc90d71d7fe is not supported for creating thumbnails.

It sounds like there is something wrong with the original image, which I doubt, because its just too many and they are good enough for Shopware 5. Now I have to figure out, whats wrong with these images or why sw6 does not like them.

Did you found out already, @toco3 ?

Thema		Antworten	Aufrufe
Problem using AWS S3 for storage International (English Only) general	0	1122	19. Dezember 2019
S3 Amazon CDN und fehlerhafte URLs Installation/Einstieg	6	978	5. Februar 2018
Shopware6 mit Amazon S3 Storage verbinden Shopware 6 (German) installation--gettin	1	1103	5. April 2023
AWS-S3 Filesystem upload in falschen Ordner Programmierung	0	204	11. Dezember 2023
AWS CloudFront mit Shopware Administration	2	1718	7. Mai 2019

AWS S3 filesystem CDN

Verwandte Themen