The validation gate: stopping bad data before it compounds

If you only implement one thing from this course, implement this.

Why this is the agentic part

It would be easy to read this lesson as database hygiene. It isn’t. Validation only becomes urgent once the agent is writing to the graph on its own, which is the loop Lesson 0 defined: retrieve, act, write back, retrieve again.

A human-curated graph gets bad rows. An agentic graph gets bad beliefs, because the thing that wrote the edge is the same thing that will later retrieve it and treat it as established fact. Nobody is between the write and the next read. That is the whole reason the gate has to live in code rather than in a prompt asking the model to be careful.

Why a bad edge is worse than a bad chunk

In a vector store, a bad chunk is inert. It surfaces occasionally, the model reads it, maybe it produces a wrong answer once. The damage is bounded by that single retrieval.

In a knowledge graph, a bad edge is generative:

Bad extraction
      ↓
Bad graph edge
      ↓
Bad retrieval          ← now it looks like verified evidence
      ↓
Bad reasoning          ← the model trusts it, because you told it to
      ↓
Bad memory written back
      ↓
More bad retrieval     ← and now it has friends

The loop closes. You built a system whose explicit purpose is to treat stored edges as ground truth, then you let unverified model output write to it. Every downstream component is faithfully doing its job while amplifying a mistake.

This is why ingestion is a data pipeline problem, not an LLM call. The LLM is one stage. The stages around it are what make the output trustworthy.

The gate

The rule is simple: no new entity or edge reaches the graph through the ingestion path without passing a validator that can explain its rejections.

Two things deliberately sit outside that rule, and it is worth knowing exactly what they are. The raw episode text is stored first, because it is the provenance record the rejections are reported against. And a temporal close runs before the gate, because closing an edge is not the assertion of a new fact, it is bounding one that is already in the graph and was validated when it arrived. Lesson 3 covers why that ordering is deliberate. GraphStore.add_edge() sits below the line too, as the raw insert the gate itself calls, and the lessons use it directly when demonstrating storage mechanics. In a real system you would keep that method private, or route it through the gate too.

LLM Extraction
      ↓
JSON schema validation      ← is it even the right shape?
      ↓
Entity normalization        ← "Apple Inc." and "Apple" are one node
      ↓
Duplicate detection
      ↓
Relationship validation     ← is this relation in my vocabulary?
      ↓
Endpoint consistency        ← are both endpoints declared?
      ↓
Temporal validation         ← is that actually a date?
      ↓
Graph write

Read those stage names as descriptions of intent, not as libraries. Every check is a hand-written Python predicate in validate.py; there is no JSON Schema dependency, and “JSON schema validation” here means “is this payload the shape we expect.” Duplicate detection is also narrower than the diagram implies: it deduplicates entities within a single payload. It does not compare against episodes already in the graph. Cross-episode edge deduplication is handled one layer down, by the store’s unique index, which is the COALESCE constraint from Lesson 3. Knowing which layer catches what matters the moment a duplicate slips through and you have to decide where to look.

Open labs/graphlab/validate.py. It implements exactly that, in under 200 lines of plain Python. Here are the checks that matter most, and why.

1. A closed relation vocabulary

ALLOWED_RELATIONS = {
    "works_at", "worked_on", "involved", "depends_on", "owns",
    "located_in", "reports_to", "part_of", "uses", "authored",
    "caused", "replaced",
}

An open vocabulary is how a graph turns to mush. Let the model choose freely and you will end up with works_at, worked at, employed_by, employment, and job as five distinct relations describing one thing. None of them match at query time. Your graph looks full and answers nothing.

Twelve relations is not a limitation, it is a schema. If you genuinely need a thirteenth, add it deliberately and re-extract.

2. Normalization before comparison

def normalize_name(name: str) -> str:
    name = re.sub(r"\s+", " ", str(name)).strip().strip(".,;:")
    name = re.sub(r"\b(Inc|Inc\.|LLC|Ltd|Corp|Corporation|Co)\b\.?$", "", name).strip()
    return name

normalize_name("Apple Inc.") returns "Apple". Without this, entity duplication kills you quietly: the graph has both nodes, each holds half the edges, and every query returns half an answer while looking perfectly healthy.

3. The endpoint consistency check

This is the most important check in the file and the one people skip:

# Endpoint consistency: an edge may only reference entities the same
# payload actually declared. This is what stops the model from
# quietly inventing a participant out of thin air.
if src not in known or tgt not in known:
    missing = src if src not in known else tgt
    result.rejected.append((f"edge references undeclared entity '{missing}'", raw))
    continue

Extraction hallucinations rarely look like nonsense. They look like a plausible extra participant appearing in the edges array who was never named in the entities array, because the model pattern-matched to what such a document usually contains. Requiring both endpoints to be independently declared in the same payload catches a large share of that for free.

4. Make the model cite its source

Consistency asks whether a payload agrees with itself. Grounding asks whether it agrees with the text. Those are different questions, and only the second one catches the hallucination above.

So every edge has to say where it came from, as character offsets into the episode:

{"source": "Alice", "relation": "works_at", "target": "Northwind",
 "valid_from": "2023",
 "evidence": {"start": 0, "end": 22, "text": "Alice joined Northwind"}}

The gate then re-reads the stored episode and checks the quote itself:

actual = episode_text[start:end]
if actual != quoted:
    return None, f"evidence text does not match the episode at [{start}, {end})"

No model is involved in that comparison. Either the characters match or they do not, which is what makes it a gate rather than a suggestion. Project Aurora never appeared in the episode, so no honest span can be produced for it, and a dishonest one fails on the string compare.

Three refinements matter more than they look:

Names must appear as whole tokens. A span mentioning Annie is not evidence about Ann. Substring matching quietly rebuilds the fabrication hole one level down.

The span must be one sentence. Quote a whole paragraph and any two entities that co-occur in it appear “cited” for any relation between them. That is document-level provenance again, which is the thing spans were supposed to improve on.

Closes need departure language. Alice joined Northwind is a real quote that mentions both endpoints, so it passes every check above as evidence that Alice left Northwind. Ending a fact is the most destructive write in the system, so it gets a polarity check the ordinary path does not need.

5. Reject, but record why

@dataclass
class ValidationResult:
    entities: list[dict]
    edges: list[dict]
    rejected: list[tuple[str, Any]]

Every rejection carries a reason. This is not politeness, it’s instrumentation. A rejection log grouped by reason tells you precisely what your extractor is bad at:

• Lots of relation not in allowed vocabulary? Your prompt’s relation list is drifting from your schema.
• Lots of undeclared entity? Your model is hallucinating participants, or your prompt isn’t clear that endpoints must be declared.
• Lots of not YYYY? Your date instruction is too vague.

Without reasons you have a number. With reasons you have a work queue. Lesson 9 turns this into a metric.

Hands-on

Run the test suite:

cd labs && .venv/bin/python -m pytest tests/ -q

............................                                             [100%]
50 passed in 0.10s

Now read the tests that describe the gate. This one is the hallucination trap:

def test_gate_rejects_ungrounded_edge():
    res = validate({
        "entities": [{"name": "A", "type": "person"}],
        "edges": [{"source": "A", "target": "Ghost Corp", "relation": "works_at"}],
    })
    assert res.edges == []
    assert any("undeclared entity" in r for r, _ in res.rejected)

Ghost Corp never appears in the entities array. It is exactly the shape of a confident hallucination, and the gate drops it while keeping the rest of the payload.

Note what the gate does not do: it doesn’t throw away the whole extraction because one edge was bad. It accepts what survives and reports what didn’t:

def test_commit_writes_only_valid_rows():
    g = GraphStore()
    res = commit(g, {
        "entities": [{"name": "A", "type": "person"}, {"name": "B", "type": "organization"}],
        "edges": [
            {"source": "A", "target": "B", "relation": "works_at"},   # good
            {"source": "A", "target": "Nope", "relation": "works_at"}, # ungrounded
        ],
    })
    assert g.stats()["edges"] == 1
    assert len(res.rejected) == 1

Partial acceptance is the right default. All-or-nothing rejection throws away good facts because of one bad neighbour, and in a large backfill that silently costs you most of your graph.

Exercises

1. Break it on purpose. Add an edge with relation: "vibes_with" and confirm the rejection reason names the relation. Then add it to ALLOWED_RELATIONS and watch it pass. Feel how deliberate a schema change should be.

2. Add a confidence floor. The Edge dataclass carries confidence, but nothing reads it: validate.py and extract.py do not mention the word, a confidence: 0.01 edge passes the gate, and commit stores the default 1.0 rather than the value you supplied. So this exercise is a full vertical slice, not a one-line threshold. You need four things, and skipping any one leaves you with a control that looks wired and is not:

   • Extract it. Have your extractor emit a real per-edge confidence instead of leaving it absent.
   • Gate on it. Reject below a configurable threshold in validate, with a rejection reason that names the score, so a dropped edge is explainable.
   • Persist it. Pass the value through commit to add_edge. This is the step people miss; a threshold that gates correctly and then stores 1.0 has thrown away the evidence for its own decision.
   • Test it. tests/test_graphlab.py asserts today’s behaviour (unenforced, not persisted) precisely so that wiring it up turns those tests red. Update them in the same commit, and update the Lesson 3 paragraph that tells readers the field enforces nothing.

   Then measure how many real edges you lose at 0.9 versus 0.7. There is a genuine precision/recall trade and you should see it in your own numbers rather than take a threshold on faith.

3. Add contradiction detection. If the graph already has an open works_at edge for a person and a new episode asserts a different employer with a later valid_from, that’s a job change, not a conflict: close the old edge instead of writing a parallel one. Today that close only fires when the text literally says “left”, and the logic lives in extract.py (which emits the close marker) and ingest.py (which applies it); pipeline.py just calls ingest_episode. Make it infer the close from a competing employer instead of waiting for the magic word.

4. Log rejections to a table. Persist every rejection with its reason and episode id. That table is the input to Lesson 9’s rejection-rate-by-reason metric.

Next: the routing policy that decides which model does which half of the work.