Medical RAG at home: secure local analysis of personal health data

What goes into the corpus

The household medical corpus, in our case, is everything we have on paper or in PDF, photographed where necessary, transcribed where automated extraction is feasible. The categories are unsurprising:

• GP letters and specialist correspondence going back as far as we have access to them.
• Test results — blood panels, imaging, the supporting commentary that came with each.
• Prescription histories and medication notes.
• Wearable telemetry summaries — sleep, heart-rate variability, activity patterns. Not the raw stream, but daily and weekly aggregates.
• Family medical history, recorded as structured notes per relative.
• Subjective health logs — symptoms noted at the time, with timestamps.

Volume across all of this for a household of two over several decades comes to a few thousand documents and a few hundred megabytes of structured data. Trivial at the scale of a workstation, sufficient for genuinely useful retrieval.

The ingestion pipeline

The ingestion pipeline is the unglamorous part that determines whether the system is actually useful. Documents arrive in mixed formats — paper letters scanned to PDF, screen-shot test results, free-text notes, structured exports from wearables. The pipeline normalises all of this.

For PDFs and images, a local OCR pass extracts the text. For structured exports, the relevant fields are pulled and tagged. For free-text notes, the original is preserved with date metadata. Every ingested document gets a record in the household document store with the raw file, the extracted text, the date, the source, the category, and the family member it relates to.

The chunking is by section heading where the documents have headings (most clinical correspondence does), by paragraph where they do not. Each chunk is embedded with the household's local embedding model and stored in the household vector store with its metadata.

The pipeline runs on a schedule, picking up new documents from a designated folder, processing them, and adding them to the index. The household member adds new material by dropping it into the folder. The rest is automatic.

Question shapes that are genuinely useful

Once the corpus is indexed, the questions worth asking are different from the ones a generic medical assistant would answer.

• “What were my last three iron panel results, and what was the trend?”
• “When did the GP first mention the medication I am currently on, and what was the context?”
• “Across the last two years of sleep data, what is the correlation between the nights I slept under six hours and the mornings I logged a headache?”
• “What did the consultant say in their letter after the scan in April, and how does it relate to the GP follow-up?”
• “Across all the family medical history I have logged, who else has the same condition my partner was just diagnosed with?”

None of these are diagnostic questions. They are memory and synthesis questions that the system can answer accurately because it has access to the source material. The model does not need to know medicine; it needs to retrieve the right documents and report what they say. That is a much more tractable task than diagnosis, and it is genuinely useful for managing a household's medical history over decades.

What the system is explicitly not for

The system does not provide medical advice. It does not recommend treatments. It does not interpret test results in the sense of telling the user what they mean. It is a memory and retrieval tool for the documents we have, with the model acting as a librarian rather than a clinician.

The discipline matters because language models will try to provide medical interpretation if not constrained. Our prompts explicitly tell the model that it is operating as a retrieval assistant over personal health records, that it should quote source documents when answering, that it should never offer diagnostic interpretation, and that it should escalate any synthesis question to the user with a recommendation to consult a clinician.

This boundary is not optional. A medical RAG system that drifts into giving advice is worse than no system at all. The local infrastructure makes the boundary easier to maintain because we control the prompt, the model, and the response surface.

The privacy boundary

The architectural commitment is that no medical data leaves the house. The OCR runs locally. The embedding model runs locally. The vector store runs locally. The language model that answers questions runs locally. No telemetry from the system reaches any third party.

The vector store and document archive are on encrypted storage. Backups go to encrypted offsite storage with keys the household controls. The system is reachable from outside the home only via the private mesh, with explicit identity-based access for household members.

The result is a personal medical assistant that is materially more private than the GP's own electronic system, and that operates against a more complete corpus than any single clinical system has access to. The household keeps the data the household generates, queries against it freely, and shares it only deliberately when interacting with the wider healthcare system.

The takeaway

Medical RAG at the household scale is not a research project. The components are mature, the data volume is small, and the architectural shape is the same as any other operator-grade RAG pipeline. The discipline is in the boundary — keeping the system as a memory and retrieval tool rather than a diagnostic one, and keeping the data inside the house under all circumstances.

For a household that has accumulated decades of medical correspondence and is tired of trying to remember which letter said what, the system pays back its setup cost the first time a real question is answered without a frantic search through paper folders. After that the value compounds with every new document added to the corpus.