Case study

Evidence-First RAG

At a glance

Who it is for: Engineers who answer questions about message, signal and mapping definitions by opening structured definition files and cross-checking them by hand.
The problem: The same message name can appear in two versions of the definition data with different values: in the synthetic data, SAMPLE_MSG_ENGINE_STATUS has a 10 ms cycle time in one version and 20 ms in another. A search that matches the name can return a correct value from the wrong version, and nothing in the answer shows it. A version here is one snapshot of the definition data. Each snapshot belongs to one project, revision and network. Those four, project, revision, network and snapshot, are the scope that, with the name, identifies an entry.
The design: A language model only helps find which entity the person means. The person makes the choice. Facts come from registered SQL queries over PostgreSQL, and the page shows them apart from the model's words.
What was built and checked: A read-only query runtime, entity search, HTTP and MCP interfaces, a chat relay, and a deployment to Azure. Evaluations were judged against pass marks registered before each run, and every deploy compares the deployed database with a local build.
Decisions I made: Most of the code and documents were written by AI coding sessions (Claude Code) and checked by a separate AI review loop. Every decision the workflow reserves for a person was mine: accepting each contract and architecture decision, setting the pass marks, accepting milestones, and merging. Each is recorded on the repository's issues and pull requests.
Scope and limits: A personal project on synthetic data. It does not show a comparison with vector search, accuracy on questions it was not tested on, or production operation.

Screenshot of the demo. The person's message asks the cycle time of SAMPLE_MSG_ENGINE_STATUS. Below it, the model's reply sits under the heading The model's words with its Not evidence note. A Lookups panel shows the relay contract served, relay-v0.1 0.7.0, the lookup result Lookup: scope not specified, and two candidate scopes, SAMPLE_SNAP_BASE and SAMPLE_SNAP_REVISED, each with a Use this scope button and neither selected. — In the demo, a person describes what they are looking for, the model searches for candidates, and the person fetches the facts of the one they choose. Here, asked for a cycle time, the model searched and found the name in two versions of the data (the page calls them snapshots); the page lists both and chooses neither. The person then chose one with "Use this scope". The page looked the name up again in that version and found one match, and still fetched nothing (this step is not pictured). Only when the person pressed "Fetch the message's facts" did the answer appear under "From the database": a 10 ms period, shown with the scope it belongs to, the registered query template that returned it and the read-only connection it used. The model's words stay in their own region, marked "Not evidence".Screenshots recorded from the deployed demo on 2026-10-08, at evidence-first-rag f8b82e6, under the contract versions shown on screen (relay-v0.1 0.7.0, api-v0.1 0.2.0, mvp-v0.1 0.6.1). This is a recording, not a live view.

Screenshot of the answer. Answered, with its returned values listed first: frame_identifier 256, message_key SAMPLE_MSG_ENGINE_STATUS, payload_byte_length 8, four superseded_by fields with no value, transmit_mode SAMPLE_MODE_CYCLIC and transmit_period_ms 10. Below them, the system's sentence gives the route message_facts, status success and resolved scope SAMPLE_SNAP_BASE. The open evidence lists the result contract api-v0.1 0.2.0, the bound parameters, contract mvp-v0.1 0.6.1, the read-only safeguards with role mvp_runtime, one row returned, and the message-facts template at version 2. — In the demo, a person describes what they are looking for, the model searches for candidates, and the person fetches the facts of the one they choose. Here, asked for a cycle time, the model searched and found the name in two versions of the data (the page calls them snapshots); the page lists both and chooses neither. The person then chose one with "Use this scope". The page looked the name up again in that version and found one match, and still fetched nothing (this step is not pictured). Only when the person pressed "Fetch the message's facts" did the answer appear under "From the database": a 10 ms period, shown with the scope it belongs to, the registered query template that returned it and the read-only connection it used. The model's words stay in their own region, marked "Not evidence".Screenshots recorded from the deployed demo on 2026-10-08, at evidence-first-rag f8b82e6, under the contract versions shown on screen (relay-v0.1 0.7.0, api-v0.1 0.2.0, mvp-v0.1 0.6.1). This is a recording, not a live view.

1. The problem

An engineer checking a signal or message definition often opens several structured files and compares them by hand. The work is slow to repeat, and another engineer cannot easily check the conclusion later.

This project is a bounded tool for that task. It answers scoped questions about messages, signals and mappings, shows where each fact came from, and says plainly when it cannot answer.

The problem comes from my work in automotive software. Everything in the project, including the data, was written new for it.

2. The failures this design avoids

Three failures shaped the design. For each, this section gives the design decision, how it is checked, and the trade-offs.

Mixing up same-named entries from different versions.

In the synthetic data, SAMPLE_MSG_ENGINE_STATUS has a 10 ms cycle time in one version and 20 ms in another.

Design decision: The scope (project, revision, network and snapshot) is part of an entry's identity. The system never fills in a "latest" or default value for any of them. If the scope is unclear, it asks which one is meant instead of answering.
How it is checked: The synthetic data deliberately contains same-named entries. Registered test and evaluation cases cover a missing scope and name collisions.
Trade-offs: The person has to choose a version more often, and requests carry more parameters.

Treating the model's words as facts.

Design decision: The model holds one tool, which searches for candidates. It has no tool to fetch facts or to make a choice. A fact appears only after the person clicks, comes from a database route, and is shown in its own region.
How it is checked: The relay contract fixes the toolset. A check on every deploy confirms the model's search call.
Trade-offs: One extra click. The model's own sentences are not checked and can still be wrong about anything that is not a fact. Other MCP clients see all three tools, and the risks recorded for them remain.

Behaving differently on Azure than locally.

Design decision: One container image and one provisioning module, used by CI, the local stack and the deploy. Every deploy rebuilds the database from the commit being deployed.
How it is checked: Each deploy compares table row counts, the registry digest and the query-template digests with a local build, and compares the test verdicts. Failed checks roll back both the database and the image.
Trade-offs: The database keeps no state, which works only because the data is synthetic. The demo can be unavailable for a few minutes during a deploy.

Records: Synthetic data with same-named entries · ADR-0002 (scoped identity) · ADR-0005 (the chat relay) · Relay contract · Deployment contract

3. The product boundary

It answers scoped questions about messages, signals and mappings, and only through a fixed set of registered, read-only queries over PostgreSQL. It is not a chatbot and has no free-form SQL.

When it cannot answer, it says why, with one of these results:

not_found: : the data covers the question but has no match
coverage_gap: : the data does not cover the question
ambiguous: : more than one candidate or scope
unsupported: : outside what it is built for
invalid_request: : the request's scope or arguments are malformed or contradictory

Records: Project charter

4. How the design changed

The first page was a form. A model could turn a sentence into a draft request, but before any search the form still asked for the kind of entry, message or signal, and its scope. A person uses this tool to find out what the definition data holds, and the form asked them to know it first. When I tried it myself, I searched for "temperature" as a message and got not_found. The term is a signal, and nothing on the page could say so.

So I moved the work of building the search from the person to a model. The existing search and fact routes became MCP tools: a person describes what they are looking for in their own words, a model turns that into a search, and the candidates come from the database. That also moved where the guarantees end. A tool's result now goes to a model, and what that model writes afterwards is outside what the tools can control.

MCP tools need the reader's own AI client, and there the model holds every tool, so it can choose for the person or fetch facts itself. For the demo, I therefore run the chat myself, through a relay that gives the model the search tool only. The person chooses, and a fact appears only after the person's click.

What stayed fixed throughout is where facts come from: registered, read-only SQL queries over PostgreSQL.

Records: ADR-0004 (MCP surface) · ADR-0005 (the chat relay) · Relay contract · Query runtime (acceptance record)

5. Architecture

Two paths are kept apart. The search path turns a person's words into candidates; the fact path turns a person's click into a fact. Only the fact path writes to the region labelled "From the database".

① How a question becomes a fact. The language model only suggests candidates, and the person chooses. The fact then comes from fixed, read-only SQL, with its source. When nothing matches, the page says so instead of guessing.

Open the diagram at full size

Flowchart, top to bottom, titled MCP tool: Find candidates, the only tool the demo's LLM can call. Your term, then Which version?, which if missing searches each and leads to Pick a version, only where found. Given a version: Exact name or approved alias, leading to Identified if exactly one. If none: Word match, all words found, leading to Pick a candidate, never automatic. If none: No match, leading to Not found, no guess. Footer: each match shows how it was found; facts come only after your click. — ② How candidates are found, inside the one MCP tool the demo's model can call. If the version is missing, each version is searched and only those where the term was found are offered. A match is settled only when exactly one entry matches its exact name or an approved alias. Otherwise the person picks, and when nothing matches, the search says so.

Records: ADR-0005 (the chat relay) · Relay contract · MCP contract · HTTP API contract

6. Behavior and evaluation

Every result, negative ones included, carries its evidence: the scope used, the query template and version, the bound parameters, the row count, the source trace and the limitations. Registered test cases compare each result with expected results built separately from the data.

Two evaluations — the model adapter against a deterministic baseline, and entity search on labelled cases — were judged against pass marks registered before the runs.

Their records state the limits plainly: the adapter was tuned on the same cases it was judged on, and the search cases were written from the same rules the search implements, so neither says anything about new questions. The adapter from the first evaluation is not on the demo's path.

Records: Model adapter evaluation (acceptance record) · Entity search evaluation (acceptance record)

7. Deployment

This site is a static Astro site on Cloudflare Pages. It holds no secret and calls the relay and the database routes from the browser.
Two apps on one Azure App Service (Linux) plan, both running the same image: surface serves the HTTP and MCP interfaces; relay handles chat.
Azure Database for PostgreSQL Flexible Server 17. surface connects with a read-only role; relay has no database credential.
Secrets in Key Vault, read through each app's managed identity. Only relay can read the model provider's key; only surface can read the database password.
One Docker image, stored in Azure Container Registry under a tag equal to its commit.
Deploys run from GitHub Actions with OIDC (no stored cloud secret), are started by hand, and come from main only. A failed check rolls back the database and the image.

③ Architecture. The AI path (purple) only finds candidates. The fact path (blue) runs from the visitor's click to fixed SQL on PostgreSQL. Both apps run one image from Container Registry on Azure App Service, and each reads only its own secret from Key Vault.

Open the diagram at full size

Records: Deploy records

8. Trade-offs and limitations

Data and evaluation

All data is synthetic and small: four versions (snapshots), seven messages, thirteen signals, three mappings. Behavior at this size says nothing about a real network database.
No comparison with vector search was run, and no evaluation on unseen questions exists.

The model and search

The model's sentences are not checked. Whether it calls the search tool is not in this project's control; the deployed check has seen the call on every run since the relevant change, which is evidence, not a guarantee.
Matching is by words, not meaning: a synonym is found only if it is registered as an approved alias, and otherwise the search reports no match.

Operation

Not yet observed on the deployed demo: the daily limits and the model-unavailable state.
No authentication and no availability commitment: one instance per app, one region, the platform's built-in metrics only.
Dependencies are version ranges, not locked versions.

Records: Limitations

9. Key decisions and how it was built

Two decisions in this project

The same name in a different scope is a different entry (ADR-0002)

What I chose: The same name in a different scope is a different entry. Project, revision, network and snapshot are part of its identity, and the system never fills in a missing scope; it asks which one is meant.
What I rejected: Answering by name alone, or filling in the latest or only loaded version.
Why: An answer about the wrong signal is easy to catch. The dangerous one is a true fact from the wrong scope: a value from one revision when the person meant another. Both are real requirements, and software built on the older one may still build and pass its own tests, so nobody notices. The concern comes from my work with engineering data.
How it is checked: The synthetic data deliberately holds same-named entries with different values. Registered evaluation cases with a missing scope or a name collision check that the system asks back instead of guessing, and a data check confirms that same-named entries stay separate in the loaded database.

The model searches; the person chooses (ADR-0005)

What I chose

In the demo, the model can only search for candidates. It cannot choose one or fetch facts. The person chooses, and the fact is the database's response, shown apart from the model's words.

What I rejected

Letting the model pick a candidate or a scope from the context of the question, and giving it the fact tool as well.

Why

When a question does not say which revision is meant, guessing it from context is the risk: the answer is a real fact, but maybe not the one the person meant. The system assumes that a person confirms anything ambiguous, and it asks again while it stays ambiguous.

Engineers use a tool like this for a specific investigation, not for a general answer. If they cannot trust each fact, they go back to the source documents. I did not want adding a model to cost that trust.

An MCP server can be connected to any model, and how those models behave is untested. On the path I operate, I did not rely on the model behaving well: the toolset and the relay's check make the system ask back when something is ambiguous.

How it is checked

A registered test checks that the call the relay sends to the model enables only the search tool, and that a scope the model takes from an earlier ambiguous result is marked as not the person's. On every deploy, a check sends a registered message to the production relay and confirms that the model called the search tool. Neither check reads the model's sentences.

Records: ADR-0002 (scoped identity) · ADR-0005 (the chat relay) · Relay contract

Not shown yet

This project does not show model training, operating a service with real users, a comparison between retrieval methods, or results on unseen questions.

How it was built

Most of the code and documents were written by AI coding sessions (Claude Code) and checked by a separate AI review loop. Every decision the workflow reserves for a person was mine: accepting each contract and architecture decision, setting the pass marks, accepting milestones, and merging. Each is recorded on the repository's issues and pull requests.