Listen Now
In this episode, Chris deep dives with Anup Iyer, a member of Moore & Van Allen’s AI task force on intellectual property, to discuss Apple’s recent research study on LLM performance on math problems, noting their sensitivity to irrelevant information and lack of true logical understanding. The conversation shifts to a legal case in Texas where a company was held accountable by the attorney state general for misleading consumers on hallucination rates.
Pieces, an AI company, was criticized for misrepresenting the hallucination rate of their model, which summarizes patient data. The Texas Attorney General issued an assurance of voluntary compliance, requiring Pieces to be more transparent about their methodology and clarify that their metrics are specific to their framework. While Pieces’ attempt to establish a benchmark is commendable, the lack of a uniform performance metric framework across industries and models makes it difficult for end-users to make informed comparisons.
Pieces used a human-in-the-loop model with an adversarial detection module to improve error rates in their summarization system. However, the use of a self-created metric for hallucination rate and the potential for blurring the line between specific and general error rates raises concerns. The Texas Attorney General’s involvement further complicates the situation, highlighting the challenges of marketing and communicating accuracy in a nascent industry.
Hallucinations in LLMs pose significant risks, especially in critical fields like healthcare and law, where accurate information is paramount. While LLMs can be valuable tools for mundane tasks and summarizing vast amounts of data, they lack the nuanced understanding and fact-checking capabilities required for complex, high-stakes applications. Software companies incorporating LLMs must be transparent about their performance metrics and avoid making exaggerated claims that could lead to legal consequences.
A near-zero error rate is not a reasonable selling proposition in healthcare, especially when claiming zero severe hallucinations. The conversation highlights the need for a more realistic and transparent approach to error rates in high stakes industries like healthcare where accuracy is paramount.
About Moore & Van Allen (MVA): One of the largest law firms in the Southeast, and the largest law firm in Charlotte, with over 400 lawyers and professionals in over 90 areas of focus represent clients across the country and around the globe. Blue-chip Fortune 500 organizations, financial services leaders, domestic and global manufacturers, retailers, individuals, and healthcare and technology companies benefit from their strategic, innovative approach to significant business transactions, complicated legal issues and difficult disputes.
MVA also specializes in software companies and their more complicated intellectual property issues.
About our guest, Anup Iyer: Anup specializes in patent preparation and prosecution for AI, fintech, quantum computing, cybersecurity, and high-performance computing. He also counsels clients on legal issues related to AI-based technology products and services and keep internal teams updated on legal developments in emerging technology. He is part of Moore & Van Allen’s AI Task Force where he helps draft internal AI policies and beta tests new AI functionalities. He routinely engages in discussions on the legal and ethical implications of AI, serving as a panelist at industry events.
Relevant links:
- Attorney General Ken Paxton Reaches Settlement in First-of-its-Kind Healthcare Generative AI Investigation (Sept 18, 2024)
- Pieces Technologies, Inc.
- The paper at the center of the dispute: System to Classify, Detect and Prevent Hallucinatory Error in Clinical Summarization: An Analysis of 5.4 Million AI-Generated Inpatient Clinical Summaries
- NY Lawyer faces discipline after AI chatbot invented case citation.
- Update 2024-12-28: Texas lawyer fined over fake citations generated by AI
