A company deploys an LLM to make consequential decisions. A court asks: show us how the system reached that conclusion. The company has nothing - no reasoning logs, no decision trace, no audit trail. The price tag: $10 million.
That scenario, outlined in a recent analysis from the "Air-Gapped Chronicles" series by AI infrastructure architect Piyoosh Rai, captures a legal risk that most companies using AI haven't thought through. The core problem is simple: LLMs are probabilistic systems that generate outputs by predicting the most likely next word in a sequence. They don't inherently record why they said what they said. And courts are starting to notice.
The Cases Piling Up
This isn't hypothetical. In March 2026, Nippon Life Insurance Company of America filed suit against OpenAI in the Northern District of Illinois, seeking $10 million in punitive damages after alleging that ChatGPT provided personalized legal assistance to a pro se litigant (someone representing themselves without a lawyer), effectively practicing law without a license and interfering with a settlement.
In the Estate of Lokken v. UnitedHealth Group case, a federal court allowed claims to proceed after plaintiffs alleged UnitedHealth's nH Predict algorithm - an AI system that forecast patient recovery timelines - was making final coverage decisions instead of clinical staff. Plaintiffs claimed the system had roughly a 90% error rate based on Medicare appeal reversals. UnitedHealth resisted producing the system's training data, source code, and decision logic. The court wasn't sympathetic.
Then there's the smaller but telling Air Canada case from 2024, where a tribunal ordered the airline to pay $812 after its chatbot gave a passenger wrong information about bereavement fares. Air Canada's defense - that the chatbot was "a separate legal entity responsible for its own actions" - was rejected outright.
The Explainability Gap
The pattern across these cases is consistent: companies deploy AI systems to make or influence decisions that affect real people, then scramble when asked to explain those decisions in a legal setting.
Traditional software is auditable. You can trace a decision through if/then logic, database queries, and business rules. LLMs don't work that way. A large language model processes input through billions of parameters (numerical weights adjusted during training) and produces output through statistical inference. There's no neat decision tree to show a judge.
Regulators are catching up. Colorado's AI discrimination statute now covers insurance decisions. California requires detailed records of AI-powered hiring decisions for at least four years. The EU AI Act mandates transparency for high-risk AI systems. And insurance companies are actively trying to exclude AI-related claims from their policies - AIG, WR Berkley, and Great American all filed with regulators in late 2025 requesting exactly that.
What This Means for Anyone Deploying AI
The practical takeaway isn't "don't use AI." It's that any company using LLMs for decisions that could end up in court - hiring, insurance claims, customer disputes, compliance - needs to build logging infrastructure now, not after they get a discovery request.
That means capturing inputs, outputs, model versions, confidence scores, and ideally some form of reasoning chain or retrieval-augmented generation (RAG) trace that shows what documents the AI consulted. The companies that will fare best in this new legal environment are the ones treating AI audit trails as a first-class engineering requirement, not an afterthought.
The $10 million question isn't whether courts will demand AI explainability. They already are. The question is whether your system can answer.