Client operations
Ops Q&A agent
Where it fits in Soundboard
Tax preparers and admins communicate with clients through Soundboard's messaging, both to answer client questions and to ask clients for information. When writing a new message, they type "/" and describe what they want in a few words; inside an existing thread, they click "Suggest a draft." The agent writes a full message based on how the team has handled similar conversations before. The preparer edits it or sends it as is. Nothing reaches a client without a person reviewing it.
Problem and origin
Preparers answer the same kinds of questions over and over (where to upload a document, whether an expense is deductible, what's needed for a filing), and they ask clients for the same kinds of information. Writing each message from scratch takes time and makes quality depend on who happens to answer.
The preparers came to me with a solution in mind: set up a project in Claude with the team's instructions as context, and open it whenever they needed to draft a question or an answer. It would have worked, but it meant leaving Soundboard for every message, copying each client's conversation into another tool, and relying on a fixed set of instructions that someone would have to keep updated. The team's best answers already existed in their past conversations inside Soundboard. So I offered to build drafting into the messaging itself: it reads the thread it's in, finds how the team handled similar conversations before, and learns from every message they send. I built it alone in under a week in 2026.
Constraints
A draft can't carry another client's details into this client's conversation. Past messages have to be useful as examples without exposing who sent them. And since the agent learns from what gets sent, it must never learn from its own drafts.
Architecture
The interface sends the whole thread, with each message tagged as written by the client or by the team. The last message decides what kind of draft is needed. If the client wrote last, it's a reply, and the client's message becomes the search query. If the team wrote last, it's a follow-up, and the query comes from the preparer's notes, or failing that, the last message in the thread.
Every past message the team sent is stored as a precedent with two embeddings: one of the message itself, and one of a short instruction describing it, written by a model when the message is indexed (for example, "ask for the missing 1099"). Clients and preparers write very differently. A client's question is searched against the messages themselves, while a preparer's shorthand notes are searched against the instructions, so each query is compared with text written the same way.
Retrieval searches four scopes at once, from narrowest to broadest: same client, entity type, and tax year; same entity type and tax year; same tax year; and everything. Each scope returns up to 12 candidates above a minimum similarity, and the top three overall are kept, ranked by a fused score: raw similarity, plus a small bonus for each scope a precedent matches, plus a small bonus for precedents that use numbered or bulleted steps, which is how the team writes instructions. A strong match from a broad scope can outrank a weak match from a narrow one.
The model drafts from the thread, the preparer's notes, and the top precedents, and returns structured output. It's instructed that every specific fact, figure, and date must come from the current conversation or the notes, and that precedents only show tone and structure. In code, any precedent the model claims to have used is checked against the ones it was actually given.
When a preparer sends a message, it's added to the precedent corpus automatically through an event-driven sync, so the corpus grows with every conversation.
Privacy
Before a message becomes a precedent, the client's name and business name are replaced with placeholders, using the names Soundboard has on record for that client. Precedents are shared across clients, so this keeps one client's identity out of drafts for another. On a synthetic message:
Before: Hi, this is Maria Lopez from Lopez Design Studio LLC. Can you send me the 1099s for 2025? Thanks, Maria L.
After: Hi, this is [CLIENT] from [CLIENT_BUSINESS]. Can you send me the 1099s for 2025? Thanks, [CLIENT] L.
The similarity floor
Every scope only returns precedents above a minimum similarity score. I first set the floor at 0.75 for replies and 0.65 for questions, values that seemed reasonable, then tested them the same day against 820 real embedded precedents. Every true reply match in the test scored between 0.48 and 0.74, so the 0.75 floor would have rejected all of them, and the agent would have drafted without the precedents it was built around. For questions, the 0.65 floor rejected 8 of 15 true matches. I lowered the floors to 0.45 for replies, which sits in the gap between the true matches and the true negatives, and 0.50 for questions as a recall-favoring interim value. The samples were small, so these are tuned against an initial test, not fully calibrated.
Key decisions and tradeoffs
One shared corpus, with names removed. Each client could have had a private corpus, but most clients ask similar questions, and a shared corpus gives every client the benefit of everyone's history. Removing names isn't only about privacy: names and business details in the text would make messages look similar because of who wrote them, not what they ask, which hurts retrieval.
The agent never learns from itself. Any precedent that traces back to an accepted or edited AI draft is excluded from retrieval by a filter applied in every scope. Otherwise the agent's own phrasing would gradually become its examples, and quality could drift without anyone noticing.
# Contamination guard: a precedent whose message originated from an
# accepted suggestion is excluded entirely. The model must not cite its
# own prior output as expert ground truth. Applied in every tier.
must.append(IsNullCondition(is_null=PayloadField(key="source_suggestion_id")))
return Filter(must=must)
Always a draft. Even with no matching precedent, the agent writes a draft from judgment, since a starting point is more useful than nothing. If the drafting service fails or times out, the suggestion is marked unavailable instead.
Search by voice. Indexing an instruction for every precedent costs one extra model call per message at ingestion, in exchange for preparer shorthand matching precedents written the same way.
Evaluation and reliability
Every suggestion is logged with the precedents it retrieved, their scores, its token usage, and its cost. Outcomes are captured automatically: when a preparer sends a message, the system records whether the draft was sent as is or edited, and how much it changed, measured as an edit distance. Drafts nobody uses expire after 24 hours. A calibration report groups outcomes by retrieval scope and edit size, to show which kinds of matches preparers trust.
I also built a backtest that replays July and August conversations against an index built only from January through June, comparing the generated drafts with the messages preparers actually sent, alongside a version with retrieval turned off. Formal backtest results aren't published yet.
Requests carry an idempotency key, so a double click never generates two drafts. Calls to the drafting service fail fast, with no retries and a 20-second timeout, so a slow request can't quietly multiply cost.
Results
- Preparers report replying to clients about 40% faster.
What I'd do differently
I'd show the preparer how well a draft is grounded. The system knows when it found strong precedent and when it's writing from judgment alone, but every draft looks the same in the interface. Showing that signal would tell preparers which drafts to read closely.
I'd also check facts in code, not just in the prompt. The rule that every figure and date must come from the current conversation is an instruction to the model. A check after generation, confirming that every specific detail in the draft appears in the thread or the notes, would make it a guarantee. It's the same principle as the transaction categorizer, where anything outside the client's category list is rejected in code.