Grand Country Houses, worth the misery?

There has been much discussion around the revelation that slavery financed the lavish buildings and lifestyles of the ancestors of many current Establishment English families.
Little has been said about the slave like conditions of native, British, workers employed in agriculture, mining etc.

I do enjoy the beauty of the estates, gardens and houses that this exploitation facilitated. Plus, of course, the majesty of churches and cathedrals built and furnished by the labouring class.
I can't "not like" something that is beautiful because of how it came about.

Are you wearing gold, even diamonds, should you feel guilty by how they were and still are produced today?