Issue product variants and listing

I have products with several variants (based on colors). They also have several other properties that you can filter on, on the product listing page.

If no filter is selected all is good: you must go to the product detail page to be able to add it to the cart.

But, if you select a property to filter by (not the color property), you can now add the product directly to the cart, from the product listing page (with a random(?) selected color). That is, the „Detail“ button is changed to „Add to cart button“.

Is there a way to avoid this issue? If you filter on a property, you must still go to the product detail page, select the right color and then be able to add the product to the cart.

/ Magnus