A California attorney learned an expensive lesson about artificial intelligence when he was slapped with a $10,000 fine for submitting a court appeal packed with fabricated case citations created by ChatGPT.
The hefty penalty, believed to be the largest AI-related fine issued by a California state court, serves as a stark warning to the legal profession about the dangers of blindly trusting AI-generated content. Los Angeles-area attorney Amir Mostafavi found himself at the center of this cautionary tale after filing an appeal in July 2023 that contained 21 fake quotes out of 23 case citations.
California Lawyer Fined for Citing Fake Cases from ChatGPT
The California 2nd District Court of Appeal didn’t mince words in its scathing opinion, published as a warning to other legal professionals. “Simply stated, no brief, pleading, motion, or any other paper filed in any court should contain any citations, whether provided by generative AI or any other source, that the attorney responsible for submitting the pleading has not personally read and verified,” the court declared.
Mostafavi admits he made a critical error in judgment. He told the court he used ChatGPT to improve his appeal, but never bothered to read the AI-generated text before submission.
This happened just months after OpenAI had marketed ChatGPT as capable of passing the bar exam, creating a false sense of security among legal professionals.
“I didn’t know it would add case citations or make things up,” Mostafavi explained to CalMatters. The three-judge panel wasn’t sympathetic, fining him for filing a frivolous appeal, violating court rules, citing fake cases, and wasting both the court’s time and taxpayers’ money.
The problem extends far beyond California. Damien Charlotin, who teaches AI and law in Paris and tracks cases of lawyers citing fake authority, says he’s witnessing an alarming trend. When he started monitoring these incidents earlier this year, he encountered just a few cases monthly. Now he sees multiple cases daily across the United States, Canada, Australia, and the United Kingdom.
“The harder your legal argument is to make, the more the model will tend to hallucinate, because they will try to please you,” Charlotin explains. “That’s where the confirmation bias kicks in.”
The Challenge of AI Hallucinations in the Legal Profession
The issue stems from how large language models work. These AI systems confidently present false information as fact, especially when supporting evidence is scarce. A Stanford University analysis found that while three out of four lawyers plan to use generative AI, some forms of the technology produce hallucinations in one out of every three queries.
The scale of the problem is staggering. Another tracking project has identified 52 cases in California alone where lawyers cited nonexistent legal authority due to AI use, with more than 600 cases documented nationwide. The situation is expected to worsen as AI innovation outpaces attorney education about these tools.
Jenny Wondracek, who leads one tracking project, regularly encounters lawyers who don’t understand that AI fabricates information or believe that legal technology can eliminate all false content. She’s even documented three instances where judges cited fake legal authority in their decisions.
“I think we’d see a reduction if lawyers just understood the basics of the technology,” Wondracek notes.
California’s Legal Profession Grapples with Generative AI
California’s legal establishment is scrambling to respond. The state’s Judicial Council recently required judges and court staff to either ban generative AI or adopt usage policies by December 15. Meanwhile, the California Bar Association is considering strengthening its conduct code to address various AI applications.
Despite the risks, Mostafavi believes it’s unrealistic to expect lawyers to abandon AI entirely. He views it as an essential tool, similar to how online databases replaced traditional law libraries. However, he advocates for extreme caution until AI systems stop generating false information.
“We’re going to have some victims, some damages, some wreckages,” Mostafavi said. “I hope this example will help others not fall into the hole. I’m paying the price.”
His expensive mistake serves as a wake-up call for the legal profession about the importance of verifying AI-generated content before submitting it to courts.
