Bookkeeping

Transaction categorization

Where it fits in Soundboard

Each month, a bookkeeper works through every client's transactions in Soundboard. In that monthly task, they pull the client's transactions from QuickBooks and run the categorizer, which suggests an accounting category for each transaction from that client's own chart of accounts. The bookkeeper reviews and edits the suggestions, saves them to QuickBooks, and the confirmed transactions go back into the system as history for future runs.

The monthly task, with a demo client's transactions pulled in and ready to categorize.

The similar-transactions view behind a suggestion, showing the retrieved examples it was based on.

The review screen after a run, with one transaction still needing a category.

Problem and origin

The first version, built in 2023, ran on the OpenAI Assistants API. It worked, but it took about 20 seconds per transaction, accuracy plateaued around 90%, and it gave me no visibility into why it got things wrong or control over how to fix them. At about 6,300 transactions a month, bookkeepers were still spending about 3 minutes on each one. In 2024, I rebuilt it from scratch, working alone, in two weeks.

Constraints

The model can never invent a category: every suggestion must come from the client's existing chart of accounts, and that's checked in code after the call rather than trusted to the prompt. Nothing reaches QuickBooks until a bookkeeper confirms it. Each client's history must stay isolated from every other client's. And at thousands of transactions a month, cost per transaction matters.

Architecture

Retrieval runs first and fills up to five example slots from the client's own confirmed history. Exact matches on normalized description and transaction type come from Postgres. If there are fewer than five, Qdrant fills the remaining slots with similar transactions, filtered by client and transaction type and limited to a 0.95 similarity score.

The model (Claude Haiku 4.5 in production) receives those examples along with the transaction's bank description, amount, and date, and returns a category from the client's list, or "uncategorized" if nothing fits. Descriptions are normalized before they reach retrieval or the model, with account and reference numbers masked. Along with the category, the model returns a confidence level on a five-point scale, and it may only return a real category at high or very high confidence. Anything lower comes back as "uncategorized." When retrieval finds no history at all, which happens on about 40% of executions, the request goes to a separate agent configuration written for categorizing without examples.

After the call, the TypeScript service checks the result. A missing category, a category outside the client's list, or an error all become "Uncategorized," so nothing invalid can reach the review screen as a real suggestion. When the bookkeeper confirms a batch, those transactions are embedded and added to the client's history, so a merchant that needed the no-history path once gets real examples the next time.

One categorization execution record from the database, showing the model, execution time in milliseconds, tokens used, cost, and a completed status. The prompt and response columns are blurred. — One execution record from a demo client. Every run stores its model, time, tokens, cost, and status; the prompt and response columns are blurred.

Key decisions and tradeoffs

Retrieval and reasoning are separate. Retrieval is deterministic and narrow; the model only decides between candidates. Handing everything to the model in one pass would have been simpler but harder to debug and audit.

The 0.95 similarity threshold is deliberately strict. A loosely similar example can steer the model toward the wrong category, so I'd rather give it fewer examples than misleading ones.

The system is allowed to abstain, and does whenever its confidence is below high. Since a bookkeeper reviews every transaction, "uncategorized" costs a few seconds of their time, while a confident wrong category can slip through review. I chose precision over coverage.

Some transactions are ambiguous by nature, so the agent follows explicit rules for them instead of guessing:

Same merchant, different categories. A vendor can show up as a purchase, a fee, or a refund. The agent only compares against past transactions within 20% of the new amount, and abstains if those still disagree.
Peer-to-peer payments. A Venmo description says almost nothing about what the payment was for, so the agent needs at least three similar past transactions, all with the same category, before suggesting one.
Transactions a bookkeeper left uncategorized. If an exact match was confirmed as uncategorized, the agent returns uncategorized too rather than second-guessing that decision.
Merchant names over boilerplate. Generic bank text like "Withdrawal Debit Card" doesn't count as similarity; the merchant name does.

1
Is there an exact match (99%+ similarity) in the client's history?
- The exact match was confirmed as uncategorizedUncategorized
- All exact matches share one categorySuggest category(very high confidence)
- Exact matches disagreego to step 2
- No exact matchgo to step 3
2
Do past matches within 20% of the new amount agree on a category?
- YesSuggest category(very high confidence)
- They disagree, or none are within 20%Uncategorized
3
Is it a peer-to-peer payment (Venmo)?
- 3+ similar transactions, all the same categorySuggest category(high confidence)
- Fewer than 3, or they disagreeUncategorized
- Not a Venmo paymentgo to step 4
4
Do the similar transactions share the merchant, not just generic bank text?
- Only generic text matchesUncategorized
- Merchant names matchgo to step 5
5
How consistent are the similar transactions' categories?
- High or very high confidenceSuggest category
- Medium, low, or very low confidenceUncategorized

The rules the categorizer follows for each transaction. It only suggests a category at high or very high confidence; everything else goes to the bookkeeper as uncategorized.

I moved production from Claude Sonnet 4.5 to Claude Haiku 4.5 after Haiku matched its accuracy while costing less and running faster.

Evaluation and reliability

Accuracy is measured against what bookkeepers actually confirm: a suggestion counts as correct only if the final saved category matches it. Transactions the system leaves uncategorized are tracked separately, so the accuracy figure and the abstention rate can't hide each other.

Retrieval is evaluated on its own. Using confirmed categories as ground truth, I check whether the correct category was present among the retrieved examples. That separates a retrieval failure (the right example never reached the model) from a reasoning failure (it did, and the model chose wrong). When the correct category was among the examples, the model chose it about 99% of the time, so the remaining errors come mostly from retrieval, not reasoning.

Every execution records its inputs, the path it took, and its cost, so any suggestion can be traced back to the examples it was given.

Results

The categorizer suggests a category for 95% of transactions and leaves the rest for manual review. When it suggests one, bookkeepers keep it 99% of the time, up from about 90% with the original system, measured the same way.
Bookkeeper handling time dropped from about 3 minutes to about 30 seconds per transaction, saving roughly 260 hours a month across about 6,300 transactions.
Latency dropped from about 20 seconds to about 3 seconds per transaction.
LLM cost per transaction dropped about 15x.
About 60% of executions are grounded in the client's own history; the other 40% run without examples.

What I'd do differently

I'd build the evaluation set before the rebuild, not after. I rebuilt the system based on production accuracy and bookkeeper corrections, and only later measured it against a held-out golden dataset. Having that dataset first would have let me compare design choices during the two weeks of the rebuild instead of after it shipped.

I'd also move the deterministic parts of the decision rules into code. Counting matches, comparing amounts within 20%, and the Venmo minimum are exact rules, but today the model applies them. In code they'd be exact and testable, and the model would only handle the real judgment calls, like whether two merchant names are the same business.