Where LLM data privacy becomes business-critical.

Where the PII actually leaks

Trace one ambient documentation session end to end. The clinician greets the patient by name, and the transcript captures it. The patient recites their date of birth for verification, mentions their address changed, describes their diagnosis history. All of it lands in the transcript, the transcript becomes the prompt, and the prompt lands in a model provider's infrastructure, under their logs, their retention, their jurisdiction. Multiply by every visit, every day. The same trace runs through chart abstraction, where the record itself is the input; through coding, where the encounter note carries the MRN; and through prior auth, where patient notes, documentation, and orders travel together. None of these leaks is a bug. Each one is the workflow operating as designed, which is exactly why nobody has mapped it.

That unmapped flow is the regulatory exposure. Enforcement in this space keeps tracing back to one finding: a defective risk analysis, meaning the organization could not say where its protected data went. A BAA with your model provider does not close the gap, because a BAA is a liability contract, not a control: it decides who pays after the data has moved, and it has never been the mitigating factor in a settlement. Regulators also pursue software vendors directly, not just the health systems using them.

Where the PII actually leaks

An adverse event case arrives as a narrative: the patient's name, age, medications, dosing history, and outcome, written as prose, because that is what a case narrative is. AI triage means that narrative becomes a prompt, verbatim. Trial operations leak the same way: pre-screening runs patient EHR data through eligibility logic, monitoring agents read patient records and lab results, and submission drafting is built on clinical study data. For US sponsors running EU trials, the European layer sharpens it: regulators have said plainly that a trained model cannot be assumed anonymous, and unlawful processing upstream can taint everything downstream. The only control that holds is removing identifiers before the data reaches the model provider.

Where the PII actually leaks

Open one claimant file: name, date of birth, SSN, address, complete medical history, employment records, financial details, lien data, sometimes a minor's records. AI-drafted chronologies and demand letters mean that entire file becomes prompt material, page after page. Document review leaks differently: it sweeps whole custodian corpora, entire mailboxes and HR folders, so the PII sent to the model is not curated, it is whatever happens to exist. Privilege review sends the most protected material a firm holds, by definition. Deposition summarization sends verbatim testimony, where witnesses volunteer medical history and family details no redaction policy ever anticipated. Client intake leaks earliest of all, often before an engagement letter exists. Meanwhile bar ethics opinions require confidentiality analysis before client data touches a generative tool, and court standing orders govern AI use directly, so "we did not know where the data went" is no longer an available answer.

Where the PII actually leaks

A support agent opens the copilot and pastes the case: customer name, full account number, recent transactions, the complaint. That paste is the leak, and it happens hundreds of times a day per team. Fraud narratives leak by design, because the narrative is the transaction history with a name attached. Compliance review leaks in bulk, because the documents under review are dense with customer identifiers. And one leak nobody budgets for: engineers and analysts paste connection strings, API keys, and credentials into internal copilots daily. All of it moves under GLBA, SOX, PCI DSS, and state privacy law simultaneously.

Where the PII actually leaks

An adjuster drops a claims file into the summarization tool. That single PDF holds medical records, financial details, an identity document, and payment data, four regulated categories, one prompt. Triage and correspondence drafting repeat the same transfer across the whole book of claims, and underwriting and fraud workflows carry the identical shape. No other industry concentrates this much regulated data into one routine AI action.

Where the PII actually leaks

Support AI runs at conversation speed. The customer types their name, order number, address, and card's last digits into chat; the ticket history carries every previous interaction; summarization, drafting, and routing each re-send the thread to the model, streaming, in real time. The exposure compounds with volume, and once customer data is distributed across provider logs, GDPR and CCPA erasure requests become promises you cannot keep.

Where the PII actually leaks

A recruiter pastes a resume into the assistant: name, address, work history, sometimes visa status. Feedback summarization reads performance reviews with names attached. People analytics reads compensation by employee. Every one of these is employee and candidate data leaving for a third-party model, with employment law exposure in every jurisdiction you operate, and it is exactly the data internal AI pilots reach for first, usually before legal has seen the tool.

Where the PII actually leaks

An engineer pastes a failing config into the AI assistant, and the config contains the database connection string. A stack trace carries a bearer token. A debugging session includes a cloud key. None of it was meant to leave; all of it did. This exposure crosses every vertical, because it is not a workflow, it is a habit.