Human Rights Education AI

RAG answer quality is a weighted score of four measured components — semantic distance (0.35), topic coverage (0.25), best-match strength (0.25), cross-chunk consistency (0.15) — checked against five named thresholds: minimum relevance, minimum 3 chunks, minimum average quality, bounds on worst and best distances. All checks must pass.

If the quality gate fails, the reflect node forces the next tool to be web search — a hard-coded code path, deliberately not an LLM decision. The model plans; the system enforces. Measured on a 12-query labeled eval: all 3 out-of-corpus questions were correctly flagged for escalation — including one whose headline confidence of 0.754 would have passed a single-score threshold and was caught only by the best-match bound.

Tool selection parses the plan text by exact tool-name matching; the LLM is consulted only for the one genuinely ambiguous call — corpus versus web. The same instinct as Act I's story-lookup overrides, promoted to an architectural rule. It also carried the system through a real provider outage: when the API key's free quota died mid-eval, the agent completed all 12 queries with zero crashes on its deterministic fallback plan.

The Upstander bot shipped with its facts pasted into the system instruction — and two of them drifted. The story database existed in duplicate with manual sync, and an acronym got expanded incorrectly in shipped user-facing copy, unnoticed because prompt-embedded knowledge has no provenance to check against. A fact without a citation path is a liability — this single failure class is why the rebuild is retrieval-first with filenames in every answer.

The prototype began rule-based with an LLM fallback; the final build inverted to LLM-first with deterministic overrides at the accuracy-critical points — story retrieval, stage transitions. Neither extreme was right. The keeper insight is placement: put determinism exactly where being wrong is expensive, and let the model own tone and flexibility everywhere else.

The ingestion pipeline split documents on blank lines. Clean text chunks fine — but PDF-extracted legal text has few blank lines, so a 1,002,946-character guide became 9 chunks averaging 111,435 characters, and the entire corpus indexed into just 112 chunks — 75 of them past the embedding model's 256-token window, tails silently truncated. The demos worked because the well-chunked documents carried them. Then it was measured: a from-scratch rebuild reproduced the exact same 112 chunks, and the labeled eval put a number on the cost — 7/9 in-corpus queries pass the gate, both failures direct chunking casualties. The fix is the top roadmap item rather than a quiet patch, because owning it is the point.

A same-week audit of the repo found the README naming a search API the code never used, three different model names across README, code and debug payload, and an early test file still asserting the 4-tool era. None of it was lying — all of it was velocity residue. Documentation is code: it needs the same single-source-of-truth discipline, and unverified performance numbers don't belong in a README at all.

In high-stakes domains, citations, confidence scores and inspectable reasoning aren't AI-UX polish — they're what makes a chatbot usable by a museum at all.

The model gets everything else. This held from a 6-stage museum chatbot to a 5-tool agent.

The RAG pipeline is the easy 20%. Measure the index — chunk counts, size distributions, per-collection coverage — not the demo.

Every meaningful design decision in Act I traces to a stakeholder sentence. That's what made it a product exercise and not a tech demo.