What you will take away
- Retrieval over your own content beats a bigger model almost every time.
- An assistant that cannot do anything can only answer - give it tools.
- Measure resolution and handover quality, not conversation count.
Why the first generation of bots failed
Decision-tree bots failed for a structural reason: they required the user to phrase the problem in the categories the business already had. People do not arrive saying "billing enquiry, subtype three". They arrive saying they were charged twice and they are annoyed.
Language models remove that constraint. They handle phrasing, spelling, mixed languages and follow-up questions that refer back to earlier turns. That solves the understanding problem, which turns out to have been the smaller half of the work.
Retrieval is the part that makes it useful
A general model knows a great deal about the world and nothing about your refund policy. The pattern that works is retrieval-augmented generation: the user question is used to search your own documentation, policies, order records and past tickets, and the retrieved passages are given to the model as the material it must answer from.
Almost all of the engineering effort lands here rather than on the model. Content has to be chunked so a passage makes sense out of context, kept in sync as policies change, and scoped by permission so a customer cannot retrieve another customer's record.
Nine times out of ten, a disappointing assistant is a retrieval problem wearing a model costume.
Give it tools, or it can only talk
An assistant that can only produce text can explain your returns process. An assistant with access to a small set of well-defined actions can look the order up, check whether it is inside the window, and start the return.
- Read actions first - order status, delivery tracking, account details - since they are safe and cover most volume.
- Write actions behind explicit confirmation, with the change described back to the user before it is made.
- A hard boundary on anything financial or destructive, which routes to a human by design.
- Full logging of every tool call, so a disputed interaction can be reconstructed.
Handover, guardrails and honesty
The assistant must be able to say it does not know, and it must hand over cleanly when it does. A handover that dumps the customer into a queue with no context is worse than no bot at all, because the customer has now explained the problem twice.
Constrain answers to retrieved material, refuse confidently outside scope, and detect frustration as a routing signal rather than a sentiment score on a dashboard. Publish what the assistant can and cannot do, in the interface, before the first message.
Measuring the right thing
Conversation volume tells you nothing. Track resolution rate without handover, the proportion of handovers where the human agrees the bot was right to escalate, repeat contacts on the same issue within a week, and the questions the assistant answered badly.
That last list is the product roadmap. In every deployment we have run, the failure log has been more valuable than the success metric, because it names the content that does not exist yet.
OrbixEdge Engineering
The engineers who build and run the systems we write about - platform, cloud and product teams shipping for clients in retail, health, logistics and fintech.
