I fed the same ICU patients to two AI models, one that read numbers and one that read text, and found they fail on different people.
Try it: where each model goes blind
Every number in this demo is a published aggregate from the paper. No patient-level data is shown, and nothing here is drawn from invented points.
Step 1: the study as it was
22,591 ICU patients from MIMIC-III, each patient's first 24 hours of ICU data, represented two ways. Numbers: 458 human-engineered numeric features into XGBoost. Meaning: the same variables rewritten as text and embedded by a large language model, into the same XGBoost. Six prediction tasks.
In-hospital mortality, headline scores (AUROC)
On the headline score, the plain numbers win. That is not the finding.
Step 2: pick a task, see the result
Step 3: the disagreement explorer
The paper studied two tasks in depth, mortality and readmission. Pick one, then pick an error type, and see how much the two models' errors overlap. The numbers on screen are the paper's published aggregates.
The two models do not fail on the same patients. A leaderboard score cannot show you that. This can.
Step 4: what the embeddings kept
A probe asks the representation a question it was not trained on, and measures how much of the answer it kept (R²). Pick a lab and compare how well the embedding kept how often it was measured against the last value itself.
The embedding remembered the watching, not the number: it kept how closely a patient was being measured far better than the value that was measured.
Every clinical AI paper I had read asked the same question: which model scores higher? Working at the Institute for Health Computing, I got to ask a ruder one. If two models read the exact same patients, one as a spreadsheet of numbers and one as text fed through a large language model, do they at least fail on the same people? Across 22,591 ICU patients and six prediction tasks, the answer was mostly no, and that answer changed how I think any clinical model should be judged.
Evidence
Paper: “Numeric versus Embedding Pipelines for Optimized Clinical Risk Prediction,” published in JAMIA Open. No DOI is listed for this paper in my source files, so none is printed here. Code and finished figure and table exports: github.com/Tensor-Lab-for-Computational-Medicine/EHR-Embeddings (manuscript_outputs/). Raw patient data, embedding arrays, and fitted model artifacts are not in that repository; they must be regenerated locally with PhysioNet credentialed data.
The setup
Same patients, same data, two ways of writing it down: 22,591 ICU patients from MIMIC-III, using each patient's first 24 hours of ICU data. The Numerical Model used 458 human-engineered numeric features (labs, vitals, counts, trends) into XGBoost. The Semantic Model rewrote the same variables as text, embedded them with a large language model, and fed them into the same XGBoost classifier. One classifier, held constant, so the only difference is how the patient is written down.
The team searched 6 embedding models, 3 text formats, and 6 prompt styles (108 configurations per task, 19,440 semantic model fits) and kept the best configuration per task. Winning embeddings included Google's text-embedding-004 and text-embedding-005. The six tasks were in-hospital mortality, 30-day readmission, ICU stay over 3 days, ICU stay over 7 days, new mechanical ventilation, and new vasopressor start.
Presenting problem
Leaderboards hide blind spots. A model can win on average while missing a distinct group of patients, and nobody checks whose errors overlap.
What I built
The two pipelines, plus three analysis tools the paper added: error-overlap analysis, linear probes that test what the embedding kept, and subgroup discovery for failure phenotypes.
Workup
Probe tests showed the embeddings remembered how often something was measured far better than the value itself. Potassium measurement count was recoverable from the embedding at R² 0.496; the last potassium value only 0.141. Failure shapes differed too: embedding-model mortality errors clustered in patients with clotting problems, low albumin, liver involvement, and very dense records; the numeric model's readmission errors clustered in creatinine-driven, sparse, heavily imputed records.
Discharge summary
Numbers won five of six tasks. Mortality AUROC 0.901 vs 0.835 for embeddings. Embeddings won the one task about what happens after discharge: 30-day readmission 0.662 vs 0.590. The paper itself flags the readmission number as the best of a large search, a ceiling, not a typical result. The models failed on different patients: 80.7% of mortality false alarms were unique to one model. The conclusion the paper argues for is hybrid systems and error audits over winner-take-all.
What I would do differently
Prespecify the hybrid model and test it on an external cohort instead of leaving it as a recommendation; probe more than two tasks; and report what a typical, not best, embedding configuration scores, so readers see the deployment reality next to the ceiling.