OpenAI's Project Lily: The Contractor Pipeline Behind ChatGP

OpenAI is paying hundreds of contractors to read genuine user prompts and grade the chatbot's answers, a programme run internally under the name Project Lily. The reviewers sit at the end of a pipeline that shapes how ChatGPT responds to a user base that has passed 900 million weekly users. OpenAI states that personal information is stripped out before prompts reach those workers, while acknowledging that sensitive details can still get through.

The operation surfaced this week through leaked internal documents, reviewer instructions and Slack messages that describe how the work is organised. Contractors are recruited through Crossing Hurdles and paid via Mercor. Every task follows the same shape: a reviewer reads a user's prompt, summarises what that person was trying to accomplish, and scores four candidate replies on a scale from 1 to 7.

One objective stated in the materials is to cut "AI-speak" and sycophancy, the habit models have of flattering whoever is typing. That makes the tone of a mass-market product the output of an hourly contract workforce reading conversations users treated as private. People type about breakups, tax trouble, medical symptoms and career panic into the same box.

What Project Lily Reviewers Actually See

Reviewers cannot see usernames. They can see a memory summary displayed above each prompt, a running record that may show what a person has previously used the chatbot for and roughly where in the world they live. The summary travels with the prompt, so a single review can expose months of context instead of one isolated question.

What makes the arrangement uncomfortable is the gap between how the product is used and how it is staffed. A chat window feels like a diary; the memory summary makes it closer to a customer profile. Someone who spent an evening describing a debt problem and then asked for a recipe has both topics inside the same reviewable record.

OpenAI says it tries to remove personal information before prompts are routed to reviewers. The company has acknowledged that sensitive details can still reach them, and the leaked material contains prompts carrying exactly that kind of content. The distance between the filtering policy and the filtered output is where the exposure lives.

Users can opt out of having their data used for this kind of review, but the control is on by default and does not apply retroactively. A prompt reviewed last month stays in the evaluation record even if the toggle is flipped today.

The scoring itself is subjective by design. Reviewers judge whether a reply answered the question and whether it leaned on phrasing that reads as machine-generated. Turning that judgement into a usable training signal takes thousands of repeated comparisons, which is why the workforce has to be large and why the written instructions carry more weight than any individual reviewer's taste.

The Gap Between Policy and Practice

Three points of friction stand out when the company's public position is set against what the internal documents describe.

Question	OpenAI's stated position	What the internal material describes
Personal data	Stripped before review	Sensitive details can still reach reviewers
Identity	Reviewers do not see usernames	A memory summary can reveal prior usage and rough location
User control	Opt-out is available	Enabled by default, not retroactive

The compliance question is whether OpenAI can identify a lawful basis under the GDPR for contractors, working through two intermediary companies, to read prompts that may contain health information, financial details or other special-category data. Data minimisation and purpose limitation both apply, and a memory summary that discloses prior usage and approximate location sits awkwardly beside them.

Mercor and Crossing Hurdles function as sub-processors in that chain. Enterprise buyers who have signed data processing agreements with OpenAI need to know whether those two names appear on a sub-processor list and whether the contract gives them a right to object. It has a paper trail, which makes it answerable.

Standard enterprise agreements cover retention, regional processing and training opt-outs, but they rarely name the individuals or intermediaries touching the data. A DPA that describes processing in the abstract leaves the practical question open: which company employs the person reading the prompt, under what jurisdiction, and subject to what confidentiality terms.

For regulated industries the stakes are concrete. A bank that lets staff draft client correspondence inside a consumer assistant, or a clinic that uses one to summarise notes, has no control over which intermediary employee reads the result. The vendor's privacy policy governs its relationship with the user; it does not govern the worker at the other end of the chain.

Why Project Lily Is a Category-Wide Practice

Human labelling has been part of model development for years. The input is what changed. Earlier annotation pipelines worked on curated datasets assembled for a specific training run; Project Lily reads a continuous stream of live traffic, so the review queue tracks whatever users happen to be asking this week. That shift is what turns a routine data-labelling job into a standing privacy commitment.

Anthropic has confirmed that it also uses human review to improve its models. That fact reshapes the decision for buyers, because switching vendors does not remove the exposure when the exposure comes from the method rather than the supplier. Any assistant that learns from real conversations needs human judgement somewhere in the loop, and contract labour has been the cheapest way to buy it.

OpenAI's scale sharpens the arithmetic. With more than 900 million weekly users, even a thin sample of prompts generates enough review work to keep hundreds of people busy. Those people are hired through intermediaries, which keeps them off the vendor's own payroll and outside the visibility of most enterprise risk assessments.

Neither OpenAI nor its intermediaries have published the size of the Project Lily workforce or what it costs to run. The labour budget behind the assistant's tone cannot be compared with the compute budget behind its answers, and for anyone building a vendor-risk model that is an unpriced input.

The commercial pressure runs in the same direction. Assistants are judged on whether they feel natural in conversation, and preference data from real users is the most direct way to measure that. Each round of tuning that makes a model sound warmer creates demand for more human comparison, which lengthens the queue of prompts waiting for a stranger to read.

There is also a switching-cost angle that buyers underweight. Once an organisation has tuned prompts, built retrieval around a specific assistant and trained staff on its quirks, changing vendor to escape a review programme is expensive. The cheaper path is contractual: narrower data flows, clearer sub-processor terms and a documented decision about which categories of information are allowed near a consumer-tier model.

The practical steps are narrow and can be checked this quarter.

Check sub-processor disclosures for Crossing Hurdles and Mercor, and use any right to object the contract provides.
Set the data-use control to off at the tenant level, then test the default state instead of assuming it.
Keep customer records, contract terms and source code out of consumer-tier assistants.
Negotiate deletion terms rather than relying on toggles, since an opt-out that is not retroactive leaves historic prompts in scope.
Ask for the retention window on human-reviewed prompts and whether reviewer annotations feed back into training.

Why this matters

Project Lily places a low-visibility labour pipeline inside the trust boundary of a product that 900 million people treat as a private notebook. Enterprise buyers already price model accuracy, latency and data residency into vendor risk; the human layer that tunes model behaviour belongs on that list too. The setting that governs it is on by default, so the safe configuration is one an organisation has to choose.

Photo by Brecht Corbeel on Unsplash

✔Human Verified

Researched and cross-referenced against primary sources by the Bytevyte editorial team. This article was generated with the assistance of artificial intelligence and reviewed by the Bytevyte editorial team.