I Built an AI Chatbot That Answered 40K Queries a Month — Here's What I Learned
I Built an AI Chatbot That Answered 40K Queries a Month — Here's What I Learned
At University Living, we built an AI-powered study abroad guidance chatbot that went from zero to 40,000 monthly queries. It cut inquiry response time from 24 hours to 30 seconds. But the real lessons weren't about the AI — they were about the product decisions that made the AI useful.
Why We Didn't Start with AI
The first version wasn't AI at all.
We started with a decision tree. Simple branching logic: What country? What budget? What course? Each answer narrowed the options. It was rigid, limited, and — critically — it worked well enough to validate demand.
The lesson: Don't build AI until you've proven the problem is worth solving. Decision trees are ugly, but they ship in a week and tell you everything you need to know about user intent.
Within two weeks, we had data:
3. Contextual Memory Was the Breakthrough
We didn't build a general-purpose chatbot. We built an intent classifier with fallback.
The bot handled 5 core intents extremely well:
The Human Handoff Problem
The hardest product decision wasn't about AI. It was about when to stop using AI.
Bad handoff: Bot says "I can't help with that. Please contact support." User feels abandoned.
Good handoff: Bot says "I've connected you with Priya, who specializes in UK student visas. I've shared your conversation so you won't need to repeat anything." User feels supported.
The difference is context transfer. Every agent who received a handoff got:
What 40K Monthly Queries Taught Me About AI Products
1. AI accuracy is table stakes. Speed is the differentiator. Users tolerate a slightly imperfect answer delivered in 30 seconds. They won't tolerate a perfect answer delivered in 24 hours.
2. Edge cases are the real product. The 70% of queries that fit neatly into categories are easy. The 30% that don't define your user experience. How you handle "I don't understand" is more important than how you handle "Tell me about UK visas."
3. Training data > model sophistication. We spent more time curating training data than selecting models. Clean, well-labeled data from real user conversations beat generic datasets every time.
4. Users don't care about AI — they care about answers. Nobody praised the technology. They praised the speed. The best AI product is one nobody notices is AI.
5. Build the feedback loop first. Every unanswered query was logged, categorized, and used to improve the next version. The chatbot didn't just answer questions — it learned which questions it couldn't answer yet.
The Metrics That Mattered
| Metric | Before | After |
|--------|--------|-------|
| Average response time | 24 hours | 30 seconds |
| Monthly queries handled | ~5,000 (manual) | 40,000+ (automated) |
| Resolution without escalation | 30% | 72% |
| User satisfaction (CSAT) | 3.2/5 | 4.4/5 |
| Support team headcount needed | 8 | 3 (for complex cases) |
The chatbot didn't replace people. It freed them to handle the conversations that actually required human judgment, empathy, and creativity.
For PMs Building AI Products
Start with the boring version. Validate demand with decision trees. Build AI only when you've proven the problem is real, the volume justifies automation, and you have enough data to train on.
The best AI products aren't built by choosing the best model. They're built by deeply understanding the problem, obsessing over edge cases, and measuring what actually matters to users.
Previous
Why Most CRM Integrations Fail — And How We Cut Our Sales Cycle by 42%
Next
The A/B Testing Playbook That Increased Engagement by 20%
Faizan didn't just study AI products — he built them.