LLM, RAG & AI Agent Automation

Support chatbots fail in two ways: they make up answers, or they can't tell when to stop. A useful AI agent must answer only from your own docs, show where each answer came from, use tools for live data, and hand off to a person when it should not answer.

Retrieval (RAG): BM25 search over chunked company docs, with light stemming and synonym expansion

Cited answers: every answer links to the exact source passage, e.g. [billing#1]

Tool calling: order numbers trigger a live order-status lookup

Guardrails: hand off when the customer asks for a person, shares a password or card number, requests a refund above $500, or when retrieval confidence is low

Evaluation harness: labelled question set plus a held-out set never used for tuning

LLM-ready: set an API key and an OpenAI-compatible model writes the final answer from the same retrieved sources

Correct route: 91% baseline → 97% after tuning; 87% on unseen questions

Right source ranked #1: 77% → 96%; 82% on unseen questions

0 wrong answers on the held-out set: every miss was a safe hand-off

Runs offline in about 0.1 ms per question, no paid API needed

Self-directed demo: the knowledge base describes a fictional company ("Northwind Cloud"). The same pattern works with your help center, Notion or Google Drive docs, and can be upgraded to embeddings + a vector database. Source code: https://github.com/cricket20052022/rag-support-agent