OpenAI uses hundreds of contractors to read real ChatGPT conversations and score the replies, according to reporting by 404 Media on 14 September.
The work is known internally as Project Lily. Reviewers summarise what the person was asking for and rate the response, on a scale of one to seven according to TechRepublic. 404 Media obtained the reviewer instructions, internal messages and the scoring system. Some of the conversations reviewers saw contained sensitive personal information, including chats where the person had asked ChatGPT to keep the exchange confidential.
OpenAI says reviewers do not see usernames and that it tries to remove personal information before conversations reach them. According to 404 Media, it also acknowledged that sensitive details can still get through.
This is not a story about one company
Before going further, one thing needs saying. Human review of AI conversations is not unique to OpenAI. Anthropic confirmed to 404 Media that it also uses human review to improve its models, for people who have the relevant setting switched on. TechRepublic reports that Google does something similar under certain settings.
Having people look at real conversations is one of the ordinary ways these products get better. The question is not whether it happens. It is whose conversations it can include, and who decided.
The setting that decides it
This is where it becomes a management question.
In ChatGPT, the setting that lets your conversations be used to improve the model is called “Improve the model for everyone”. According to TechRepublic, it is switched on by default for the Free, Plus and Pro plans. It is switched off by default for Enterprise, Business and Edu.
So the same tool, used for the same work, behaves differently depending on who pays for it. If a member of your team uses their personal Plus account to tidy up a client email, the default is that the conversation can be used for training, and so may be seen by a reviewer. If they do the same thing on a company Business account, the default is that it will not.
Most managers have not looked at this, and most staff have not either. The setting sits in the data controls section and is easy to change, which also means nothing ever prompts anyone to think about it.
The cheapest argument you will get
Every manager with a team has a version of the shadow AI problem. People are using AI tools for work, often on personal accounts, often without anyone having said what is and is not acceptable.
It is usually discussed in abstract terms: data leakage, compliance exposure, governance. Those words rarely move anyone to act.
This makes it concrete. On a personal account with the default settings, something a colleague pasted in could be read by a contractor. On a managed business plan, the default runs the other way.
That is a clearer case for moving a team onto a managed plan than most. It does not depend on believing any supplier is careless. It depends only on knowing how the defaults are set.
A managed plan brings a second thing that matters as much as the default: someone in your organisation controls the settings. On a personal account, each person decides for themselves, usually by never deciding at all. On a business plan, an administrator decides once for everyone and can see what has been chosen. Suppliers are starting to build around that difference. Anthropic said this month that Enterprise administrators will get at least 30 days’ notice before new features arrive, and will choose when to switch them on.
None of that is free. A managed plan costs more per head than a personal one, and somebody has to administer it. But the cost is known and small, and it buys you the one thing a collection of personal accounts cannot give you, which is a single answer to the question of where your team’s work goes.
Three practices, not one
Retention, training and human review are three different things, and it is worth keeping them apart.
Retention is how long a supplier keeps what you typed. Training is whether it can be used to improve the model. Human review is whether a person may read it along the way. A supplier can keep conversations for a month for safety monitoring without training on them. It can train on them while trying to remove personal details first.
Last week I wrote about two mathematicians who could not rule out that a year of their drafts had helped improve a model. That was the training question. This is the human review question. When you ask a supplier about your data, ask about all three.
What to do this week
Three things, none of which needs anyone technical.
Find out which plan your team is on. If the honest answer is “a mixture of personal accounts”, that is the finding.
If personal accounts are in use for work, ask people to open their data controls and check the training setting. It takes a minute. Do not make it a disciplinary matter. Most people have simply never looked, and a manager who asks calmly will learn more than one who announces a crackdown.
Then write a short list of what should not go into any AI tool on a personal account. Client names. Personal details about colleagues. Anything under a confidentiality agreement. Not a policy, just a list someone can remember.
If that list turns out to cover most of your team’s work, you have your case for a managed plan, and you did not need a supplier to make it for you.
On the source
404 Media’s report is partly behind a paywall, and the plan defaults come from TechRepublic’s follow-up rather than from OpenAI directly. Settings and defaults change. Check them in your own account before relying on them, and check again at renewal.
The detail that makes this worth your time is not the contractors. It is that a single default, set differently on personal and business plans, decides whether your team’s work can end up in front of them.