Bridging the Gap Between Doctors and Data: How EHRAgent Turns Medical Questions into Code

Imagine a busy clinician in an Intensive Care Unit. They need to know something specific, and they need to know it now: “How many patients were prescribed aspirin within two months of receiving a venous catheter procedure?”

In the world of modern medicine, the answer exists. It is sitting in the Electronic Health Record (EHR) system. However, getting that answer out is surprisingly difficult. The doctor cannot simply ask the database. Instead, they usually have to ask a data engineer, who then translates the request into a complex SQL query, executes it, checks the data, and sends it back. This loop is inefficient, slow, and creates a bottleneck that separates medical professionals from their own data.

Why hasn’t AI solved this yet? While Large Language Models (LLMs) like GPT-4 are impressive, they historically struggle with the specific, rigid, and high-stakes environment of medical databases. They “hallucinate” table names, write invalid SQL, or get confused by the sheer scale of medical data.

Enter EHRAgent, a new framework proposed by researchers from Georgia Tech, Emory University, and the University of Washington. EHRAgent transforms the LLM from a simple text generator into an autonomous agent capable of writing code, using tools, and—crucially—debugging its own mistakes to answer complex medical questions.

The Problem: Why EHRs are Hard for AI

To understand why we need a specialized agent, we first need to understand the data. Electronic Health Records are not simple spreadsheets. They are massive relational databases containing dozens of tables (e.g., admissions, prescriptions, lab_events) linked by obscure identifiers (HADM_ID, SUBJECT_ID, ITEMID).

In standard text-to-SQL tasks (like asking an AI to query a Wikipedia table), the data is usually small and clean. EHRs are the opposite. They are messy, vast, and require deep domain knowledge to navigate.

As shown in Figure 2 above, general domain datasets like WikiSQL or SPIDER usually involve small tables with few rows. In contrast, datasets derived from real-world EHRs (like MIMIC-III or eICU) contain tables with hundreds of thousands of rows. A single clinical question often requires “multi-hop reasoning”—jumping from a patient’s demographic data to their admission records, then to their lab results, and finally to a prescription table.

Standard LLMs often fail here because they lack the specific “schema” knowledge (knowing how the tables connect) and the ability to reason sequentially through these complex hops.

The Solution: EHRAgent

The researchers propose EHRAgent, an autonomous system that treats medical question answering not as a translation task, but as a tool-use planning process. Instead of trying to guess the answer or write a single SQL query in one go, EHRAgent acts like a programmer. It writes Python code, executes it against the database, checks if it worked, and fixes it if it didn’t.

The architecture consists of four distinct pillars that allow it to reason effectively:

1. Medical Information Integration: Teaching the model the “map” of the database.
2. Long-Term Memory: Retrieving the best past examples to solve new problems.
3. Code Interface: Using Python instead of just SQL for complex logic.
4. Interactive Coding (Rubber Duck Debugging): Learning from errors.

Let’s break down the architecture visualized below.

1. Medical Information Integration (The Map)

One of the biggest hurdles for an LLM is the “context window”—the limit on how much text it can process at once. You cannot simply feed an entire database schema with thousands of column descriptions into a prompt.

EHRAgent solves this by incorporating a Medical Information Integration module. When a query comes in (e.g., about “Aspirin”), the system first consults the metadata of the EHR. It identifies which tables and columns are relevant to the specific medical terms in the question.

If the question involves a drug, the agent knows to look at the prescriptions table. If it involves a lab test, it looks at labevents. This acts as a filter, feeding the LLM only the relevant “background knowledge” it needs to understand the specific database structure for that specific question.

2. Long-Term Memory (The Experience)

In AI, “few-shot learning” is a technique where you give the model a few examples (demonstrations) of a task to show it what to do. Usually, these examples are fixed.

EHRAgent, however, maintains a Long-Term Memory of past successful queries. It stores a library of questions and the code that successfully answered them. When a new question arrives, the system searches its memory for the most similar past questions.

This allows the agent to learn from experience. If it successfully answered a question about “calculating the duration of a stay” in the past, and the new question asks for “time between admission and discharge,” it retrieves that specific logic. This dynamic selection of examples is far more powerful than a static list.

3. The Code Interface: Why Python?

Most previous attempts at this problem tried to make LLMs write SQL (Structured Query Language). While SQL is great for querying databases, it is terrible for logical reasoning. It struggles with complex dates, multi-step calculations, and procedural logic.

EHRAgent instead uses a Code Interface based on Python. It treats the LLM as a planner that generates Python scripts. The agent is equipped with a specific set of tools (functions) it can call within its code:

• LoadDB(): To access a table.
• FilterDB(): To narrow down rows.
• GetValue(): To extract specific data points.
• SQLInterpreter(): To run raw SQL when necessary.

By generating Python, the agent can break a complex problem into steps: “First, get the patient ID. Second, loop through their visits. Third, calculate the time difference.”

4. Rubber Duck Debugging

This is arguably the most innovative part of EHRAgent. In software engineering, “Rubber Duck Debugging” is a method where a programmer explains their code line-by-line to an inanimate object (like a rubber duck) to find a bug. The act of explaining often reveals the error.

EHRAgent automates this. When it writes code, it executes it immediately. If the code fails (which it often does on the first try), the system captures the error message (traceback).

Instead of giving up, the agent enters a multi-turn dialogue. It looks at the error message, “thinks” about the cause, and generates a new plan to fix it. This creates a feedback loop:

1. Plan: Generate code.
2. Execute: Run code.
3. Debug: Analyze error.
4. Refine: Rewrite code.

The image above illustrates this process perfectly.

1. First Attempt: The agent tries to filter the database using a condition that combines a Subject ID and a max(INTIME) function. The system throws an “Invalid input query” error.
2. Second Attempt: The agent tries to fix it by splitting the logic but makes a type error (trying to subtract strings instead of date objects).
3. Third Attempt (Success): The agent realizes it needs to import the datetime module, convert the strings to time objects, and then perform the subtraction. It writes the correct Python code and solves the problem.

This mimics how a human data scientist works: iterative refinement until the code runs.

The Complete Workflow

To summarize how these pieces fit together, we can look at the complete workflow diagram below.

The process begins with the user’s question. The system integrates the relevant medical metadata (Step 1) and retrieves similar past examples from memory (Step 2). It generates the initial Python code (Step 3). If execution fails—perhaps due to a case-sensitivity issue like “csru” vs “CSRU”—the agent analyzes the error (Step 4), refines the code, and finally produces the correct answer.

Experimental Results

Does it actually work? The researchers tested EHRAgent against several strong baselines, including:

• Standard prompting (CoT): Just asking GPT-4 to think step-by-step.
• ReAct: A popular agent framework that interleaves reasoning and acting.
• Text-to-SQL methods: Models trained specifically to write SQL.

They used three datasets: MIMIC-III and eICU (large, real-world critical care databases) and TREQS.

Performance by Complexity

The results showed that EHRAgent significantly outperformed all baselines. One of the most telling metrics is how the model handles complexity. The researchers categorized questions based on how many “elements” (constraints or variables) were in the question and how many columns were needed to answer it.

In Figure 4, look at the purple line (EHRAgent).

• Graph (a) - Success Rate: As the number of elements in a question increases (getting harder), all models perform worse. However, EHRAgent maintains a significantly higher success rate than ReAct (blue) or Chameleon (orange).
• Graph (b) - Completion Rate: This measures if the model could even generate executable code (regardless of the answer’s correctness). EHRAgent’s completion rate remains incredibly high, often near 90%, even for complex queries. This proves that the Python code interface is much more robust than trying to write perfect SQL in one shot.

Sample Efficiency

Another major advantage of LLMs is that they don’t need thousands of training examples. The researchers tested how EHRAgent performs with different numbers of “shots” (examples provided in the prompt).

As shown in Figure 5, EHRAgent (purple line) achieves high performance with very few examples. Even with just 4 examples, it outperforms the AutoGen baseline (green) by a wide margin. This is critical for medical applications where annotating training data (having doctors explain queries) is extremely expensive.

Where does it fail?

No system is perfect. The researchers conducted an error analysis on the cases where EHRAgent failed to get the right answer.

Figure 6 shows that the biggest category of failure (26.7%) is “Fail to Debug.” This means the agent got stuck in a loop—it wrote code, got an error, tried to fix it, got another error, and eventually hit the maximum number of allowed steps (T=10).

Other common errors included Incorrect Logic (20.39%), where the code ran but calculated the wrong thing (e.g., averaging the wrong column), and Context Length issues (14.56%), where the conversation history became too long for the LLM to handle.

Conclusion and Implications

EHRAgent represents a significant step forward in the application of Large Language Models to healthcare. By moving away from simple text generation and towards agentic workflows—where the AI acts as a programmer with tools, memory, and debugging capabilities—we can bridge the gap between complex medical data and the clinicians who need it.

The implications are broad:

1. Efficiency: Clinicians can get answers in minutes rather than days.
2. Accessibility: You don’t need to know SQL to query a database of millions of patients.
3. Transparency: Because the agent writes code, a human engineer can review the exact logic used to derive an answer, ensuring safety and accuracy.

While challenges remain—specifically regarding privacy, cost, and the “Fail to Debug” loops—EHRAgent demonstrates that code generation is the key to unlocking the reasoning capabilities of LLMs in specialized, data-heavy domains.