Google DeepMind’s ‘Superhuman’ AI System is making waves in the field of fact-checking, cost efficiency, and accuracy. In a recent study, researchers from DeepMind found that their artificial intelligence system, known as SAFE (Search-Augmented Factuality Evaluator), outperformed human fact-checkers when evaluating the accuracy of information generated by large language models.
The study, titled “Long-form factuality in large language models,” introduces SAFE as a method that uses a large language model to break down generated text into individual facts. It then uses Google Search results to determine the accuracy of each claim. The researchers compared SAFE’s assessments with those of human annotators on a dataset of 16,000 facts and found that SAFE’s judgments matched the human ratings 72% of the time. Even more impressively, when there were disagreements between SAFE and human raters, SAFE’s judgment was correct in 76% of cases.
While the researchers claim that LLM agents can achieve “superhuman” rating performance, some experts are questioning the definition of “superhuman” in this context. AI researcher Gary Marcus suggests that “superhuman” may simply mean better than an underpaid crowd worker, rather than a true expert fact checker. Marcus argues that benchmarking SAFE against expert human fact-checkers is crucial to truly demonstrate its superhuman performance.
One clear advantage of SAFE is its cost-efficiency. The researchers found that using the AI system was about 20 times cheaper than employing human fact-checkers. As the volume of information generated by language models continues to increase, having an economical and scalable way to verify claims becomes increasingly vital.
The DeepMind team also used SAFE to evaluate the factual accuracy of 13 top language models across four families. They found that larger models generally produced fewer factual errors. However, even the best-performing models still generated a significant number of false claims. This highlights the risks of relying too heavily on language models that can fluently express inaccurate information. Automatic fact-checking tools like SAFE could play a key role in mitigating these risks.
Transparency is another important aspect of AI development. While the SAFE code and LongFact dataset have been open-sourced on GitHub, more transparency is still needed regarding the human baselines used in the study. Understanding the qualifications, compensation, and fact-checking process of the human raters is crucial for properly contextualizing the results.
As tech giants race to develop more powerful language models, the ability to automatically fact-check the outputs of these systems becomes increasingly important. Tools like SAFE represent a step towards building trust and accountability in AI systems. However, it is essential that the development of such technologies happens in an open and collaborative manner, with input from a broad range of stakeholders. Rigorous and transparent benchmarking against human experts will be crucial to measure true progress and assess the real-world impact of automated fact-checking in the fight against misinformation.