Most of my RAG evals didn't need an LLM judge

ReviewLens AI is my live review-analytics product: upload, paste, or scrape customer reviews, then ask questions in a chat that answers through a RAG pipeline — embeddings in Pinecone, retrieval, GPT-4o. The product's entire credibility hangs on one behavior. Every factual claim in an answer carries a [Review N] citation that opens the underlying review, and the system prompt in the chat-rag Edge Function makes that a hard rule: "ALWAYS cite specific reviews using [Review N] notation when making claims. Every factual statement must have at least one citation."

When I sat down to write LLM evals for it, I assumed the work would be judge prompts — a second model grading the first against a rubric. What shipped instead is six promptfoo test cases in LLMtests/promptfoo/, passing 6/6 against the live pipeline, and not one of them uses a model-graded assertion. That became the finding: when the output contract is explicit, most LLM evaluation compiles down to deterministic assertions — and the judge is only needed where genuine judgment lives.

Grade the pipeline, not the prompt

The default promptfoo setup points at a model with a prompt template and some variables. That validates a prompt in isolation, which is not what users touch. RAG fails in the plumbing — ingestion, embedding, retrieval, context assembly, streaming — so the suite's providers are two small Python scripts instead of a model config. chat_rag_provider.py POSTs each test question to the deployed chat-rag Supabase Edge Function, collects the SSE token stream into one response string, and pulls out the citations_ready event payload. insight_provider.py calls the generate-insight function and returns its raw JSON. Every assertion in the suite runs against exactly the bytes a browser would receive from production infrastructure.

That one decision changes what a green run means. When TC_001 ("What are the main complaints in the reviews?") passes, it means a real question went through real retrieval over real ingested reviews and came back grounded and cited — not that a template scored well against a fixture.

Design the contract so a regex can grade it

The most valuable eval engineering happened in the system prompt, not the eval config. Two product decisions made the suite cheap to write. First, citations have fixed notation: because answers must cite as [Review N], "does this answer cite its sources?" stops being a semantic question and becomes a regex match, and the customer-praise case raises the bar to at least two citations. Second, refusals have an exact string: the scope guard does not say "politely decline off-topic questions" — it instructs the model to respond exactly with "I can only answer questions about {product}'s {platform} reviews. That information is not available in the ingested review data." So TC_005 asks the deployed product about the weather and asserts that the refusal phrasing appears — and that "degrees", "sunny", "celsius", and "forecast" do not. An answer that helpfully reports the temperature before apologizing still fails.

# ── TC_005 — Scope guard: off-topic question (HIGH)
- description: "TC_005: Scope guard — declines weather question"
  vars:
    message: "What is the weather today?"
    skill: "general"
  assert:
    - type: javascript
      value: output.toLowerCase().includes("can only answer") || output.toLowerCase().includes("not available in the ingested")
    - type: not-contains
      value: "degrees"
    - type: not-contains
      value: "sunny"
    - type: not-contains
      value: "celsius"
    - type: not-contains
      value: "forecast"

# ── TC_001 — Core complaints with citations (HIGH)
- description: "TC_001: Core complaints — General skill"
  vars:
    message: "What are the main complaints in the reviews?"
    skill: "general"
  assert:
    - type: contains
      value: "[Review"
    - type: javascript
      value: (output.match(/\[Review \d+\]/g) || []).length > 0
    … 3 more asserts: not-contains "weather" · not-contains "I can only answer" · icontains "issue"

Structured output is a schema, so assert the schema

The insight report endpoint returns JSON with themes, FAQs, and prioritized actions. TC_008 grades it structurally: valid JSON, one to six themes, one to eight FAQs, and every action's priority drawn from high, med, or low — any value outside that whitelist fails, no matter how sensible it sounds. TC_010 gives the executive summary a word budget with the [Review N] tags stripped first, so citing more never costs words, and requires at least two unique citations counted with a Set, so repeating one badge cannot fake coverage. One honest note: the test matrix targets "max 200 words, plain language, no jargon"; the automated bound is 300 words, because a script can count words but cannot judge plain language.

# ── TC_008 — Insight Report structure (HIGH)
- description: "TC_008: Insight Report — 3 sections present, valid priorities"
  provider: "python:insight_provider.py"
  assert:
    - type: is-json
    - type: javascript
      value: JSON.parse(output).themes.length > 0 && JSON.parse(output).themes.length <= 6
    - type: javascript
      value: JSON.parse(output).faqs.length > 0 && JSON.parse(output).faqs.length <= 8
    - type: javascript
      value: JSON.parse(output).actions.length > 0 && JSON.parse(output).actions.every(function(a) { return ["high","med","low"].includes(a.priority); })

# ── TC_010 — Executive summary (MED)
- description: "TC_010: Executive summary — concise, cited, plain language"
  assert:
    - type: contains
      value: "[Review"
    - type: javascript
      value: output.replace(/\[Review \d+\]/g, "").trim().split(/\s+/).length <= 300
    - type: javascript
      value: new Set(output.match(/\[Review \d+\]/g) || []).size >= 2

Where an LLM judge actually belongs

Deterministic asserts have a ceiling, and in this suite it shows up in one specific place. The test plan's case matrix expects TC_007 to catch sarcasm: reviews like "Broke in 2 days, great product!" must be classified as negative, with no false positives when someone asks what customers love. No substring check can see that. The runnable asserts for TC_007 only enforce grounding — at least two [Review N] citations — and the sarcasm judgment itself lives in a dated manual pass log in the test plan: 2026-03-17, "Sarcastic reviews correctly classified as negative sentiment."

That is exactly the check that deserves a model-graded llm-rubric assertion — a judge grading "does this answer treat sarcastic praise as negative?" — and this suite does not have one yet. Four planned cases are also still marked TODO in the test plan: image-based extraction (TC_002), competitor SWOT (TC_004), emoji sentiment mapping (TC_006), and UI-bug isolation (TC_009). The standard for the judge is already written down — the test plan weighs factual accuracy at 40%, citation correctness at 30%, scope compliance at 20%, and format adherence at 10%, with a 70% pass threshold. What is missing is the judge, not the rubric.

What changed

Three habits came out of this suite. I write the output contract before the feature — exact refusal strings, fixed citation notation, JSON bounds — because that is what turns "evaluate the LLM" from a research problem into assertions. I point evals at the deployed pipeline, not the prompt, so a green run certifies the product. And I publish the whole scoreboard: the 6/6 promptfoo result sits next to 168/173 Vitest unit tests, with the five failures documented in the README as pre-existing timeouts in NewProduct render tests rather than quietly excluded. A product that must cite its sources should hold its own test suite to the same rule — every pass/fail claim traceable to a file in the repo.