Notes from building LLM and payments systems that have to survive real users — retrieval, agents, and the constraints a regulated product puts on both.RSS.
Semantic search alone will not ground an LLM support assistant in a regulated product. Hybrid retrieval, exact-match recall, and a first-class "I don't know" are the parts that do the work.
You will not get a golden set before you ship. Four evaluation techniques that need no labels — inverted queries, negative controls, retrieval-only metrics, and pairwise diffs — in cost order.
Approval prompts bolted onto an agent produce rubber-stamping, not oversight. A four-question framework for deciding which actions pause, and where the line between reads and writes really sits.
Hand-rolled tool definitions are correct for one assistant and wrong for three. A direct comparison of MCP against per-agent integrations, and the point where the tradeoff flips.
JSON mode and typed schemas stop parse errors, not wrong answers. The validation that matters happens after the model, and it's boring deterministic code you have to write yourself.
Payments engineering solved unreliable retries a decade ago with idempotency keys and a ledger. LLM agents have the same failure mode and almost none of the discipline. Here's the transfer.
Open to senior AI / backend roles