Sort product listing by manufacturer name

I would like to sort the products on any navigation page by manufacturer name. So, this would be the association product.manufacturer.name.

It seems this field is already mapped like this:

"manufacturer":
  {
    "type": "nested",
    "properties": {
      "_count": {
        "type": "long"
      },
      "id": {
        "type": "keyword",
        "normalizer": "sw_lowercase_normalizer"
      },
      "name": {
        "type": "keyword",
        "normalizer": "sw_lowercase_normalizer",
        "fields": {
          "ngram": {
            "type": "text",
            "analyzer": "sw_ngram_analyzer"
          },
          "search": {
            "type": "text"
          }
        }
      }
    }
  }

Currently I have added a sorting to the product_sorting table like so:

[{"field": "product.manufacturer.name", "order": "asc", "priority": 1, "naturalSorting": 0}]

This sorting appears in the dropdown, but when applied this does not change the product sort direction at all. Is this possible at all? If so how would I change the product sort to be able to sort on manufacturer name?

Many thanks!