Included files referenced as http instead of https

Running into this issue over and over.

Doing a default installation of the production version, if I load the site (with a valid SSL certificate) through HTTPS, all included files/links are pointing to HTTP. This leads to issues starting from the web installer to whatever I do.

I have CloudFlare as DNS and turning on forcing/redirecting requests through HTTPS helps with CSS and images, but some requests are just blocked with CORS/mixed-content errors.

Am I missing some config value to generate all the links as HTTPS?

As a context:
I installed Shopware 6.5 on a clean Ubuntu 22 server running nginx. Nginx is running the site as HTTP and Cloudfront is taking care of providing SSL and redirecting http request to https.
This works fine on other servers if I use https:// links for all my assets, but where can I set Shopware to always generate https links instead of http?