Good AI chatbot design is the work of deciding what the bot is for, what it must refuse, how it speaks, and when a person takes over, before anyone chooses a model. The interface matters, but most chatbots fail because of missing scope, weak knowledge sources and no plan for failure. This guide walks through the decisions in the order you should make them, with the evidence behind each one.
It is written for CTOs, VPs of Engineering and founders at US companies, including teams whose prototype from an AI builder now has to become a production product.
TL;DR
- Treat the chatbot as a system with separate layers for knowledge, policy and decisions, not as a single prompt.
- Tell users on the first screen what the bot can do, and give them suggested prompts as buttons.
- Design for transparency and a frictionless human handoff, not for sounding human.
- Score faithfulness, relevance and tool-call accuracy on live traffic, and keep a regression set.
- In regulated areas such as health, finance and legal, avoid unsupported claims. The FTC enforces them.
Table of Contents
- Start with scope, not with the chat window
- Conversation design that users can follow
- Voice, persona and honesty about being a bot
- Architecture: build layers, not a long prompt
- Trust, handoff and compliance in the US
- Evaluate before and after launch
- Our perspective
- How Wve Labs can help
- A pre-build checklist
- FAQ
- Sources
Start with scope, not with the chat window
The first design artifact is a one-page scope document. It answers four questions:
- Who is the user? A customer, a claims adjuster, a patient, an internal analyst.
- What job does the bot finish? Answering from documentation, booking, checking an order, drafting a message.
- What does it refuse? Medical advice, legal conclusions, pricing exceptions, anything outside your policy.
- When does a human take over? Define the triggers in advance: low confidence, repeated rephrasing, a high-risk action, an angry user.
The refusal list is not optional. The NIST Generative AI Profile calls for acceptable use policies for chatbots, including the kinds of queries the application should refuse, and for human-AI configurations that are defined up front. A bot with a narrow, well-defined job is easier to test, easier to trust and cheaper to run than a general assistant with a chat box on top.
Pick the interaction pattern that matches the job
| Pattern | Best for | Main risk |
|---|---|---|
| Scripted flow with buttons | Fixed tasks such as order status or appointment booking | Dead ends when the user goes off script |
| Retrieval-based assistant (RAG) | Answering from your policies, docs and help center | Confident answers from the wrong passage |
| Agentic assistant with tools | Taking actions: updating records, sending emails, filing requests | Wrong or irreversible actions |
| Hybrid | Most production products: scripted for critical steps, generative for open questions | More design and testing effort |
The industry has moved from basic question-and-answer flows toward agentic workflows built on retrieval-augmented generation, according to FutureAGI's 2026 development guide. That does not mean every bot should be agentic. Give the model tools only for jobs where a mistake is recoverable or a person approves the result.
Conversation design that users can follow
Conversation design is the part readers search for as chatbot conversation design, and it is mostly craft: opening message, prompts, turn length, repair and exit.
Say what the bot can do in the first message
Nielsen Norman Group's guidelines for site AI chatbots advise that on first open the chatbot should state clearly and briefly what it can do. Their research found that a generic "ask me anything" greeting leaves users guessing and sets expectations the bot cannot meet. Replace it with a specific line, such as what it can look up, plus two or three suggested prompts.
The same guidelines recommend offering suggested questions as buttons rather than plain text, keeping the chat window available across pages, and using progressive disclosure so long answers do not bury the point. These are small interface decisions with a large effect on whether people continue past the first turn.
Keep turns short and map the paths
For structured tasks, map the likely paths with a decision tree before writing any copy. Salesforce's chatbot best practices recommend logical conversational flows and decision trees so users reach solutions without wandering. For generative answers, set rules in the design brief:
- Lead with the answer, then the explanation.
- Ask one clarifying question at a time.
- Show the source of an answer when it came from a document.
- Offer a next step: a button, a link or a handoff.
Design the failure turn on purpose
Every bot will misunderstand someone. Write the response for "I don't know", for out-of-scope requests and for the third rephrase. A good failure turn does three things: admits the limit, says what the bot can do instead, and offers a person. Teams that skip this write it by accident, and users find the accident first.
Voice, persona and honesty about being a bot
A persona is a set of rules for tone, vocabulary and length. Define it in a configuration layer rather than scattering it through prompts. If you use generated copy for greetings and canned replies, edit it by hand, and a guide on humanizing AI-written text can help you give those messages a natural rhythm before they ship.
Natural tone is not the same as pretending to be a person. UX research published through the ACM found that over-emphasizing humanness can break trust when the bot fails. The better approach is transparency: state what the AI can do, acknowledge limits when it fails, and give users an easy route to a human. That produces appropriate trust, meaning users rely on the bot as much as it deserves and no more.
Architecture: build layers, not a long prompt
The most common mistake we see in prototypes is a single prompt carrying persona, policy, knowledge and logic. It works in a demo and fails the first time a document changes. Enterprise AI architects describe the alternative as a layered control plane with these layers: channel, identity, policy, knowledge, tool, model and decision. Each has its own owner, job and failure mode.
Two examples from that model are worth copying:
- The policy layer blocks restricted topics before the model tries to reason about them.
- The decision layer defines when to hand off to a person instead of automating a high-risk action.
The practical benefit is modularity. You can swap a model or update a source document without breaking the whole conversation, which is the maturity a product needs when it moves from prototype to production.
Retrieval that earns its answers
Production RAG is more than embedding documents and searching them. FutureAGI's guide describes a layered pipeline: ingestion that normalizes content and keeps metadata, hybrid retrieval that combines vector search with keyword search, and re-ranking that narrows roughly 30 candidates to the top 5 passed to the model. It reports that re-ranking adds about 30 to 80 ms of latency while improving precision. For most support and knowledge use cases, that trade is worth it. Keep document metadata such as product, date and audience, so the bot can prefer current and relevant sources and cite them.
Memory and sessions
Decide what the bot remembers. Common patterns are one session per authenticated user, one per support ticket, or a hybrid with long-lived preferences plus short-term conversation state, as described in the OpenAI Agents SDK guide from Techademy. In healthcare, insurance and fintech, memory choices are also privacy decisions, so involve security and compliance early.
Trust, handoff and compliance in the US
If your chatbot touches health, money or legal questions, design decisions become compliance decisions.
Do not market the bot as a professional. The FTC's action against DoNotPay alleged unsubstantiated claims that the service could operate like a human lawyer, as set out in the FTC's final consent documents. The agency's consumer guidance on Operation AI Comply also covers the case and the $193,000 in redress. The design lesson: do not label a bot an expert, an advisor or a replacement for a licensed person unless you can substantiate it, and say plainly where answers may be inaccurate.
Do not encourage a false sense of a relationship. An FTC staff statement, Early Thoughts on Generative AI, urges companies to think carefully before deploying products designed to make people feel they have a trusted relationship with them or are talking to a real person. Avatars, names and typing indicators are design choices that deserve that scrutiny.
Document risk controls. The NIST profile maps risks such as hallucinations, harmful bias and information security breaches across the lifecycle, and recommends independent evaluation such as red-teaming. For regulated products, ask your legal counsel which state and federal rules apply to your use case. If the bot handles protected health information, for example, your compliance team should review the data flow before launch.
Handoff checklist
- A visible "talk to a person" option at every stage, not only after failure.
- Automatic triggers: repeated misunderstanding, sensitive keywords, high-value or irreversible actions.
- The full conversation passes to the agent so the user never repeats themselves.
- A clear message about wait time or hours when no one is available.
Evaluate before and after launch
If you cannot answer "did this change make the bot better?" with a number, you are not ready to deploy. That is the standard practitioners recommend in the enterprise agent guide from Kommunicate, and it is a good one.
Build a regression set
Collect anonymized, production-derived cases and group them by category:
- Policy conflicts, where two sources disagree.
- Missing knowledge, to test that the fallback works.
- Prompt injection attempts.
- Multi-turn coherence, where the answer depends on earlier turns.
Run the set on every change to the prompt, model, retrieval settings or documents.
Score what matters
Production teams track faithfulness (is the answer grounded in the retrieved context), relevance, instruction following, tool-call accuracy and task success rate, not only response speed, as FutureAGI's guide notes. Add user-facing signals too: containment, handoff rate, and whether users who were handed off were satisfied.
Keep scoring after launch
Offline benchmarks give signals but are not a contract for production performance. One practitioner view, summarized in Techademy's SDK guide, is to score a small share of live traffic with an LLM as judge. This catches silent regressions, such as a prompt edit that quietly lowers faithfulness, which would otherwise hide in ordinary logs. Human review of a sample remains necessary to check that the judge itself is accurate.
Our perspective
We have built mobile products for more than a decade, and the pattern with chatbots is familiar: teams treat the conversation as a feature when it is a product surface. It needs the same discipline as any screen, meaning research, flows, states, edge cases and measurement.
Three things we push for on every engagement:
- Design and engineering in one room. The wording of a refusal, the retrieval filter that triggers it and the handoff screen are one decision. When design and engineering are split, they get decided three different ways.
- A narrow first release. One job done well, with a real regression set, earns more trust than ten jobs done unevenly.
- Support after launch. Documents change, models change and users surprise you. A chatbot needs ongoing care, with evaluation on live traffic and regular reviews, or it drifts.
If you built a working prototype in an AI builder, that is a real asset: it proves the idea and shows the flows users want. The production step is separating the layers, adding evaluation and designing the failure paths. Our UI/UX design work starts from the same belief: good design is how a product earns trust.
— Brian
How Wve Labs can help
Wve Labs designs, builds and scales mobile apps, web platforms and applied AI solutions. For chatbots, that includes product strategy, UI/UX design, chatbots and virtual assistants, LLM integration, AI API integrations, backend and cloud work, and ongoing product care. If you have a prototype that needs to become a production app, or a chatbot that must fit an existing mobile or web product, see our full list of services and tell us about your use case.
A pre-build checklist
Before a design sprint starts, confirm you can answer yes to each item:
- We have a written scope with a refusal list.
- The first message states capabilities and offers suggested prompts as buttons.
- Failure, out-of-scope and handoff turns are written and designed.
- Knowledge, policy and decision logic are separate and versioned.
- Marketing copy makes no claim we cannot substantiate.
- We have a regression set and agreed scores, with a plan to sample live traffic.
- Someone owns the bot after launch.
FAQ
What is the first step in designing an AI chatbot?
Write the scope: the user, the single job, the topics the bot refuses and the triggers for human handoff. Only then choose an interaction pattern and a model. This order keeps the design testable and avoids a generic bot that does everything poorly.
Should an AI chatbot sound human?
It should sound clear and natural, but it should not pretend to be a person. Research on trust shows that over-emphasizing humanness can backfire when the bot fails, and the FTC has urged caution about designs that make people think they are talking to a real person. Be open that it is an AI and make the path to a human easy.
How do you measure whether a chatbot design works?
Use a regression set of anonymized production cases and score faithfulness, relevance and tool-call accuracy. After launch, score a sample of live conversations and track task success and handoff rate. Review a sample by hand to confirm the automated scores match what a person would conclude.
Is a chatbot prompt enough, or do I need a full architecture?
A prompt is enough for a demo. For production, separate knowledge, policy and decision logic so you can update a document or change a model without breaking the flow. The layered approach also lets you test each part and find where failures start.
Sources
- 10 Guidelines for Designing Your Site's AI Chatbots, NN/G
- AI Chatbot Development 2026: LLMs, RAG, Agentic, FutureAGI
- How to Build an Enterprise AI Agent with OpenAI, Kommunicate
- OpenAI Agents SDK: Complete Guide for Developers, Techademy
- The Top Chatbot Best Practices for Service, Salesforce
- Artificial Intelligence Risk Management Framework: Generative AI Profile, NIST
- DoNotPay final consent, FTC
- Operation AI Comply, FTC Consumer Advice
- Early Thoughts on Generative AI, FTC
- Trust and naturalness in AI design, ACM Digital Library