- OpenAI tested AI persuasion using r/ChangeMyView, evaluating how well its models could change user opinions compared to human responses.
- The company claims this evaluation is separate from its Reddit data licensing deal, though details on how it accessed the subreddit’s data remain unclear.
- OpenAI’s AI models rank in the top 80-90% of human participants in persuasion, but the company aims to prevent models from becoming overly influential.
OpenAI has been using the subreddit r/ChangeMyView as a benchmark to test the persuasive capabilities of its AI reasoning models. This method was revealed in a system card released alongside the company’s new AI model, o3-mini. The system card, which outlines how the model operates, confirmed that OpenAI designed an evaluation to measure AI-generated persuasive arguments against human responses from the subreddit.
The r/ChangeMyView subreddit, with millions of users, serves as a discussion forum where individuals post opinions and invite others to challenge their viewpoints with counterarguments. This makes it a valuable resource for AI companies seeking high-quality, human-generated discussions. OpenAI has reportedly used posts from the subreddit in a controlled environment to generate AI responses aimed at changing users’ perspectives, with human testers then assessing the effectiveness of these arguments.
While OpenAI has a licensing agreement with Reddit allowing it to access user-generated content for AI training, the company states that its ChangeMyView evaluation is separate from this deal. Details on how OpenAI accessed the subreddit’s data remain unclear, and the company has no plans to make the evaluation publicly available. Other AI firms, including Google, have struck similar agreements with Reddit, with Google reportedly paying $60 million annually for data access.
The use of ChangeMyView as a benchmark highlights both the importance of human data in AI development and the ongoing controversy surrounding data collection practices in the tech industry. Reddit has previously criticized AI companies for scraping content without compensation, and OpenAI itself faces legal challenges related to data usage, including lawsuits alleging improper web scraping for training AI models.
Despite concerns over data sourcing, OpenAI’s latest reasoning models, including o3-mini and GPT-4o, demonstrate persuasive abilities ranking within the top 80th to 90th percentile of human participants in ChangeMyView discussions. However, OpenAI insists that its goal is not to create hyper-persuasive AI but to ensure such models do not become overly influential. The challenge of finding high-quality datasets for AI evaluation persists, underscoring the difficulty AI developers face in ethically sourcing and testing their models.
