AI agents that handle patient data need a different engineering approach. Here is how to build healthcare AI that is both useful and compliant.
Healthcare is where AI agents can create the most value and cause the most damage. Scheduling, clinical documentation, prior authorization, and patient triage are all candidates for agentic automation. But every one of these workflows touches Protected Health Information (PHI), which means HIPAA compliance is not optional. This guide covers the technical architecture for building healthcare AI agents that satisfy HIPAA requirements without crippling the system's usefulness.
Compliance Note:
This article provides technical guidance for engineering teams building healthcare AI. It does not constitute legal advice. Consult with a healthcare compliance attorney for your specific situation.
HIPAA has two main rules that directly impact how you build AI systems:
Controls who can access PHI and for what purposes. For AI systems, this means: the LLM provider should not have access to identifiable patient data unless they are a Business Associate with a signed BAA. Data sent to the model must be de-identified or processed under a BAA.
Requires administrative, physical, and technical safeguards. For AI systems: encryption at rest and in transit, access controls, audit logs of every data access, and incident response procedures.
The core architectural principle is simple: PHI never reaches the LLM in identifiable form. Build a PHI firewall between your clinical data and the AI layer.
The Pattern:
Clinical Data -> De-identification Layer -> LLM Processing -> Re-identification Layer -> Clinical Output
Before sending any text to an LLM (whether OpenAI, Anthropic, or self-hosted), run it through a de-identification pipeline. Replace patient names, dates of birth, MRNs, addresses, and other PHI identifiers with placeholder tokens. NER-based tools (AWS Comprehend Medical, Microsoft Presidio) automate this.
Use LLM providers that offer HIPAA-eligible services with a signed Business Associate Agreement (BAA). As of 2026: Azure OpenAI Service (BAA available), AWS Bedrock (BAA available), and Google Cloud Vertex AI (BAA available) all support this. Consumer APIs (api.openai.com direct) are not HIPAA-compliant.
Alternatively, self-host open-source models on your own HIPAA-compliant infrastructure. This gives you complete data control but requires significant operational effort. For deployment patterns, see deploying LLMs on AWS and Terraform with AWS Bedrock.
Every interaction with the AI system must be logged: who accessed it, what data was processed, what output was generated, and what actions were taken. These logs must be immutable, retained for 6 years (HIPAA requirement), and accessible for compliance audits.
Agent listens to doctor-patient conversation, generates structured clinical notes (SOAP format), and pre-fills the EHR. The de-identification layer strips identifiers before LLM processing, and the re-identification layer maps them back for the final note.
Agent receives a prior auth request, retrieves the patient's clinical data from the EHR, matches it against payer-specific criteria, and generates the submission. Uses function calling to interact with EHR APIs and payer portals.
Patient describes symptoms through a chat interface. Agent classifies urgency, recommends next steps, and uses RAG to ground recommendations in clinical guidelines. For data isolation in multi-clinic deployments, use multi-tenant RAG with Pinecone with per-clinic namespaces.
If your system serves multiple healthcare providers (hospitals, clinics, practices), strict data isolation between tenants is non-negotiable. A patient's data from Hospital A must never be retrievable by Hospital B's queries. Use namespace-level isolation in your vector store and separate encryption keys per tenant.
For the overall agent architecture that wraps these healthcare-specific patterns, see our production agent blueprint. For managing conversation history securely, see AI agent memory management.
Not the consumer API directly. Use Azure OpenAI Service, which offers the same models with a signed BAA, data residency guarantees, and no data retention for training. AWS Bedrock and Google Vertex AI are also HIPAA-eligible alternatives.
De-identification under the HIPAA Safe Harbor method (removing 18 types of identifiers) allows the data to be used without a BAA. However, the de-identification process itself must be validated to ensure no PHI leaks through. Expert determination is the more robust standard.
Clinical AI outputs must be treated as decision support, not clinical decisions. The provider reviews and approves all outputs. For engineering safeguards, see our full guide on handling AI hallucinations -- the techniques apply across regulated industries.
We build HIPAA-compliant AI systems for healthcare organizations. From architecture design to BAA management.
Discuss Your Healthcare Project