Internal engineering
Eval harness
Where it fits in Soundboard
The eval harness is internal tooling for testing Soundboard's agents before a change reaches clients. It runs an agent against cases with known correct answers, scores every result, and compares the run with a saved baseline, so a prompt or model change that quietly breaks something shows up as a failing case instead of a bookkeeper's correction weeks later. I built it for the transaction categorizer, and any agent can plug in through the same interface.
Problem and origin
The categorizer's production accuracy is measured from what bookkeepers confirm, which means a regression only shows up after it has already reached clients. A prompt or model change can fix one case and break another, and nothing flags it until someone corrects the output. Regular software has unit tests for this; LLM features usually don't. I wanted the same discipline for the agents: run the real agent against known answers before shipping, not after. I built the harness in two days in August 2026.
Constraints
Evaluation runs can't touch production records, so they can be run as often as needed without risk. The test cases have to be ones the agent has genuinely never seen, or the results mean nothing. A model's own output can never become a "correct answer" without a person checking it. And the framework's own tests have to run offline, with no API keys.
Architecture
Building a test set
once per client dataset
1. Export
A real client's QuickBooks history and chart of accounts
2. Split by time
Older months become history; the two most recent are held out
3. Seed
History is loaded into an isolated test client as retrieval context
4. Review
Drafted fixtures are marked unverified until a person checks them
Running an evaluation
before and after a change
1. Run the agent
Each held-out transaction goes through the real categorizer in preview mode
2. Score
Exact match against the confirmed category, or an LLM judge with partial credit
3. Compare
Each case is checked against a saved baseline to flag regressions
4. Report
Accuracy on suggestions, coverage, and every miss, case by case
The framework. Any agent that takes an input and returns an output can be evaluated. The harness doesn't interpret either; that's the scoring layer's job.
class Agent(Protocol):
def run(self, input: dict[str, Any]) -> dict[str, Any]:
"""
Take one fixture's input, return the agent's output.
The harness does not interpret their contents,
that's the scoring layer's job.
"""
...
Each case is scored either by exact match against the expected answer or by an LLM judge that can award partial credit for a reasonable alternative. Each run is compared case by case against a saved baseline, so the report shows exactly which cases got better or worse. A fake judge lets the whole suite run offline.
Running the real categorizer safely. The harness calls the production categorizer through its real API, with a preview flag I added: in preview mode, the categorizer runs exactly as it does for bookkeepers but saves nothing, and returns the exact prompts it sent to the model. That last part turned out to matter more than I expected. Test data lives in isolated test clients, seeded with history before a run and cleaned up afterward, including their vectors in Qdrant.
Bookkeeper run
- Transactions
- Same categorizer
- Suggestions saved for review
Evaluation run (preview)
- Held-out transactions
- Same categorizer
- Result + exact prompts returned, nothing saved
Building test sets from real data. Each test set comes from a real client's QuickBooks export, split by time: older transactions become the history the categorizer can retrieve from, and the two most recent months are held out as the test set, with the categories bookkeepers actually confirmed as the correct answers. Drafted test cases are marked unverified and kept apart from the real test set until a person reviews them, so a model's guess never becomes ground truth by accident.
One client's QuickBooks history
History
Seeded into an isolated test client. This is all the categorizer can retrieve from, just like a real client's past.
Test set (last two months)
Never seen by the categorizer. Each transaction is scored against the category the bookkeeper actually confirmed.
The first run scored 0%
The first time I ran the categorizer against a real test set, every transaction came back uncategorized. A broken categorizer would have been the easy conclusion, but production was clearly working. Because preview mode returns the exact prompts, I could read what the model had actually been given. The historical transactions in the test client had category names but not the real category IDs from the client's chart of accounts. The categorizer is only allowed to return categories from that chart, so with every example pointing at "Unknown," it had nothing valid to choose and correctly refused every time. The bug was in the test data, not the agent. Once the history carried real IDs, the results looked like production.
First run
SIMILAR HISTORICAL TRANSACTIONS:
1. Merchant: ACME SOFTWARE
Amount: $24.00 | Direction: debit
Category: Office expenses:Software & apps
Category ID: Unknown
Similarity: 100.0%Every example points at an invalid category, so the categorizer refuses every transaction: 0%.
After fixing the test data
SIMILAR HISTORICAL TRANSACTIONS:
1. Merchant: ACME SOFTWARE
Amount: $24.00 | Direction: debit
Category: Office expenses:Software & apps
Category ID: 42
Similarity: 100.0%Examples carry real category IDs: 99 of 101 suggestions correct.
It's the reason I now treat a surprising evaluation number as a question about the evaluation first.
Testing the tests
Before trusting the framework with real decisions, I went through it looking for ways it could report a pass that wasn't real, and found three:
A typo could pass forever. If a test case misspelled the field being scored, the comparison found nothing to compare and counted it as a perfect match, permanently. Test cases are now checked when they're loaded, so a scored field missing from the expected answer fails loudly.
The real judge didn't match the fake one. The offline judge used in tests awarded partial credit; the real LLM judge only returned pass or fail. Both now support partial credit, and the real judge runs at temperature 0 so results are repeatable.
One bad case could end the run. An error from the agent on a single case crashed the whole evaluation. Each case is now isolated, so it's recorded as one failure among many.
expected_output: category: "Office expenses:Software & apps" scoring: categroy: exact_match # typo
Before
Nothing named categroy to compare, so the case scores as a perfect match. Forever.
After
Rejected when the fixtures load: a scored field must exist in the expected output.
Each fix has a regression test, and the suite grew from 5 tests to 22, all passing in a tenth of a second with no network. A few of them, by name:
test_exact_match_catches_a_wrong_answer PASSED
test_judge_gives_partial_credit_for_reasonable_alternative PASSED
test_scoring_a_field_missing_from_expected_output_is_rejected PASSED
test_agent_exception_fails_that_fixture_without_crashing_the_run PASSED
test_unmatched_category_id_falls_back_to_uncategorized PASSED
======================= 22 passed in 0.10s =======================
Results
On two held-out client datasets, 115 transactions in total, the categorizer (running on Claude Haiku, as in production) suggested a category for 88% of them. When it did, it was right 99 times out of 101.
That's in line with the 99% bookkeepers confirm in production. Coverage is lower than production's 95% because a held-out test client has less history to match against than a client with years of data. The transactions it skipped went to the bookkeeper as uncategorized, which is the design: the categorizer abstains rather than guesses.
The three possible outcomes, with synthetic examples:
| Transaction | Suggested | Confirmed by bookkeeper | Result |
|---|---|---|---|
| ACME SOFTWARE, $24.00 | Office expenses: Software & apps | Office expenses: Software & apps | Correct |
| HARDWARE STORE #0001, $61.20 | Uncategorized | Supplies | Abstained (no history match) |
| WIRELESS CARRIER, $95.00 | Utilities | Home office: Internet + phone | Incorrect |
Key decisions and tradeoffs
Test the real agent, not a copy. The harness calls the same categorizer bookkeepers use, through its real API, rather than a reimplementation that could drift from production. The cost was adding a preview mode to the production endpoint, which also made every prompt inspectable, and that's what solved the 0% run.
Hold out by time, not at random. Holding out the most recent months mirrors how the categorizer is actually used: it only ever sees the past when categorizing new transactions. A random split would let it learn from transactions that came after the ones it's being tested on.
Abstentions are not misses. The report separates accuracy on the categories the agent suggested from how often it suggested one, the same way production is measured. Counting every "uncategorized" as wrong would punish the behavior the categorizer was designed to have.
What I'd do differently
I'd make the test data fully reproducible. The step that attached real category IDs to the test history, the fix for the 0% run, was a one-off script I never saved, and the datasets stay out of the repository because they come from client data. Rebuilding a test set today means redoing that step by hand. A committed, versioned pipeline from export to reviewed test set would let anyone rerun the exact evaluation.
I'd also wire it into deployment. Today I run evaluations by hand after every agent change and before each deployment, and CI only tests the framework itself. Running the categorizer's test sets automatically on every prompt or model change, and blocking the deploy on a regression, is what would turn the harness from a tool I use into a guarantee.