@starfrost For me it's a design decision to make use of biased data and unsupervised training in your manufacturing process. Of course these companies are aware that vast amounts of their data is prejudiced.

The Google photos incident where a black person was tagged as a gorilla happened in 2015 and 8 years later they haven't fixed this, just blocked the ability to search for gorilla. These companies have had time to become aware of the bias problem.

There are definitely many useful applications for AI and it's possible to work around these issues and mitigate impact to some extent before harm happens. Many companies however choose not to. To work around these moral dilemmas I believe you need to be open and honest about their presence and how you are addressing them.

If you aren't actively looking for them as a user, prejudices will also make their way into everyday use of generative models.