In 2022, a lawyer working an airline liability case used an AI tool to help research and draft a brief. The citations looked right, specific case names, docket numbers, real judges' initials attached to real courts. Opposing counsel noticed something was off. When the judge requested verification, the lawyer admitted he had used AI and revealed he had asked the AI directly whether the cases were real. The AI confirmed they were. They weren't. Six of the cited cases were entirely fabricated. The judge sanctioned both the filing attorney and his colleague, ordered a $5,000 fine, and mandated further legal education on AI use.

That decision came in 2023. Since then, the number of similar incidents has not declined. It has exploded.

The Problem Isn't That Lawyers Are Using AI. It's That No One Thought Through the Implications.

There's a version of the AI-in-law story that's about bad actors: lawyers trying to cut corners, submitting work they know is sloppy. That version is the easier one to dismiss. The harder story, and the more common one, is about competent attorneys reaching for the most useful tool available and not fully understanding what it does with their inputs, or what it might hand back to them.

In a 2025 case, a California attorney filed an appeal citing 21 cases generated by ChatGPT. He told the court he had written the appeal himself, then used the AI to refine the language. He hadn't realized it would add citations, let alone fabricated ones. He hadn't read the final output carefully before filing. A three-judge panel fined him $10,000 for a frivolous appeal, court rule violations, and wasting both the court's and taxpayers' time. He told reporters he still thinks it's unrealistic to expect lawyers to stop using AI.

He's right about that. But "we can't stop using it" is not the same as "we've thought carefully about how we're using it." For most firms right now, the gap between those two things is wide open.

The Scale of the Problem Is Getting Hard to Ignore

Since mid-2023, more than 300 cases of AI-driven legal hallucinations have been documented in court filings, with the pace accelerating sharply: a researcher who tracks these incidents noted the trend had gone from roughly two cases per week to two or three per day by 2025. Courts in the US, UK, Australia, Canada, and Israel have all encountered the issue.

And it's not just solo practitioners or small firms scrambling to keep up with technology. In July 2025, a large and well-regarded firm had a motion sanctioned in federal court in Alabama for submitting AI-generated hallucinations. The judiciary's patience is running out. Courts are now explicitly distinguishing between intentional deception and inadvertent AI reliance, but they're making clear that neither excuses the result.

The most striking recent case may be from May 2025, when a plaintiff's law firm was sanctioned $31,100 after submitting fake AI citations in a California federal court. What made the incident notable wasn't the fine. It was the judge's admission. He wrote that he had initially found the citations convincing and had almost included them in a ruling. He wrote, "Plaintiff's use of AI affirmatively misled me."

The Exposure Goes Beyond Hallucinations

The fabricated-citations problem gets the headlines. But it's arguably not the most serious risk facing law firms that are deploying AI without a governance framework.

Law firms hold some of the most sensitive data in any industry: litigation strategy, M&A details, criminal defense materials, privileged client communications. When a lawyer feeds any of that into a centralized AI platform, they're sending it to a remote server they don't control. Depending on the platform's data policies, that information may be retained, used to train future models, or exposed to other users. Most attorneys using consumer AI tools have never read those policies carefully, and most firms haven't asked them to.

The potential consequences span several directions at once: malpractice liability if a client's confidential information is exposed, ethics violations under professional conduct rules governing confidentiality and competence, and contractual breaches if client engagement letters include data handling provisions. None of these risks require malicious intent. They just require a lawyer doing their job with whatever tool is in front of them.

The numbers reflect how poorly the profession has caught up with this reality: 79% of lawyers now use AI in their practice, but only 10% of firms have policies governing how it's used. That's a large governance gap.

What Leading Firms Are Starting to Do

The firms getting ahead of this share a common instinct. They're treating AI infrastructure the way they already treat client file systems and case management platforms, as something that requires access controls, usage policies, and auditability.

Concretely, that means a few things. It means being explicit about which workflows can touch external AI platforms and which cannot. Legal research on public information is a very different risk profile than drafting a motion that contains privileged client detail. It means looking seriously at private deployment options for sensitive work: AI models that run on the firm's own infrastructure, where data never leaves the environment.

Infrastructure like Aphanarc is built specifically for this: atomized deployment that keeps sensitive client data safe, with no third-party exposure and no risk of inputs feeding an external model's training data

It also means building the same verification habits around AI output that good lawyers already apply to work from junior associates. That means not assuming the output is right just because it looks right.

The bar itself is moving. Ethics opinions are beginning to address what competent AI use actually requires. Courts are establishing expectations through sanctions. The question for most firms isn't whether to engage with this. The questions is whether to engage with it proactively or reactively.

The Bottom Line

The lawyer in the Avianca case wasn't negligent in the ordinary sense. He was trying to do good work efficiently, using a tool that presented itself as helpful and authoritative. The problem was that neither he nor his firm had thought carefully about where the tool's confidence ended and its reliability began, or what it was doing with the information they handed it.

That's the risk profile for most law firms right now. Not bad actors. Not reckless disregard. Just the ordinary adoption of powerful tools, ahead of the frameworks needed to use them safely.

Firms built on client trust and confidentiality have more to lose than most when those frameworks are missing. The good news is that the path forward isn't complicated. The path just requires the same due diligence that good lawyers apply to everything else. For sensitive workflows, that due diligence increasingly means asking where your data goes, and choosing infrastructure like Aphanarc, which makes sure the data goes nowhere that it shouldn’t.

About Aphanarc

Aphanarc is agentic AI for enterprises and privacy-focused users who can't risk proprietary data, internal expertise, or company knowledge being absorbed by centralized AI systems. Rather than routing workloads through a single centralized server, Aphanarc atomizes compute, model execution, and data handling by design. Sensitive information is never fully processed, stored, or reconstructed in one place, leading to private, safe AI. Since 2024, Aphanarc gives organizations access to powerful, high-performance AI without paying centralized-cloud premiums or trading away privacy and control.