Bengali Book-Based RAG Chatbot Development
Rabiul Islam
đ āĻŦāĻŋāĻļā§āĻŦā§āĻ° āĻāĻĒāĻžāĻĻāĻžāĻ¨ â Knowledge Base Chatbot
A Bengali book-based Retrieval-Augmented Generation (RAG) chatbot built using LangChain, multilingual embeddings, FAISS, Groq, and Streamlit.
The chatbot answers questions using only the selected Bengali book âāĻŦāĻŋāĻļā§āĻŦā§āĻ° āĻāĻĒāĻžāĻĻāĻžāĻ¨â and provides the relevant chapter and source URL with each answer.
đ¯ Project Objective
The goal of this project is to build a Knowledge Base Chatbot with a Vector Database that can:
Crawl a complete Bengali book from Wikisource
Clean and preprocess the collected text
Split the book into meaningful chunks
Generate multilingual embeddings
Store embeddings in a FAISS vector database
Retrieve relevant book passages
Generate answers using an LLM
Provide chapter and source information
Avoid using outside knowledge for book-specific questions
Clearly respond when information is not available in the selected book
đ Knowledge Source
Selected Book
Book: āĻŦāĻŋāĻļā§āĻŦā§āĻ° āĻāĻĒāĻžāĻĻāĻžāĻ¨ Author: āĻļā§āĻ°ā§āĻāĻžāĻ°ā§āĻāĻ¨ā§āĻĻā§āĻ° āĻāĻā§āĻāĻžāĻāĻžāĻ°ā§āĻ¯ Publication Year: 1952 Publisher: Visva-Bharati Source: Bengali Wikisource
Source URL:
The selected book is a completed Bengali prose book containing six main chapters.
Chapters
āĻ
āĻŖā§, āĻĒāĻ°āĻŽāĻžāĻŖā§
āĻāĻ˛ā§āĻā§âāĻā§āĻ°āĻ¨ āĻ āĻĒā§āĻ°ā§āĻāĻ¨
āĻĒāĻāĻŋāĻā§āĻ°āĻ¨, āĻ¨āĻŋāĻāĻā§āĻ°āĻ¨, āĻ¨āĻŋāĻāĻā§āĻ°āĻŋāĻ¨ā§, āĻŽāĻŋāĻ¸ā§āĻā§āĻ°āĻ¨
āĻāĻĒāĻžāĻĻāĻžāĻ¨ā§āĻ° āĻĒā§āĻ°āĻā§āĻ¤āĻŋ
āĻļāĻā§āĻ¤āĻŋ āĻ āĻ¤āĻĄāĻŧāĻŋā§
āĻāĻĒāĻ¸āĻāĻšāĻžāĻ°
đ§ RAG Architecture
đ Retrieval Configuration
The final chunking configuration is:
The processed book produces:
The chatbot uses:
The retriever uses FAISS similarity search with:
This configuration was selected after testing different chunk sizes.
đ Retrieval Evaluation
The project includes 10 test questions:
9 answerable questions
1 no-answer question
For the 9 answerable questions, the expected chapter was found within the top-10 retrieved results.
Result
Note: This 100% result represents retrieval coverage at top-10, not 100% end-to-end answer accuracy.
The no-answer test is separately used to evaluate whether the chatbot can avoid answering questions outside the selected book.
đī¸ Project Structure
venv/and.envare intentionally excluded from Git because they are listed in.gitignore.
âī¸ Installation
1. Clone the repository
2. Create a virtual environment
Windows:
Activate the environment:
3. Install dependencies
đ Environment Variables
Create a
.env file in the project root:Never commit the real API key to GitHub.
The project
.gitignore contains:so the API key remains local.
đ Run the Chatbot
Activate the virtual environment:
Run the Streamlit application:
Open the application in your browser:
đ§Ē Test Questions
The project contains 10 test questions.
1. āĻĒāĻ°āĻŽāĻžāĻŖā§ āĻā§?
2. āĻāĻ˛ā§āĻā§āĻā§āĻ°āĻ¨ āĻā§?
3. āĻĒā§āĻ°ā§āĻāĻ¨ āĻā§?
4. āĻ¨āĻŋāĻāĻā§āĻ°āĻ¨ āĻā§?
5. āĻĒāĻāĻŋāĻā§āĻ°āĻ¨ āĻā§?
6. āĻŽāĻŋāĻ¸ā§āĻā§āĻ°āĻ¨ āĻŦāĻž āĻŽā§āĻ¸āĻ¨ āĻ¸āĻŽā§āĻĒāĻ°ā§āĻā§ āĻŦāĻāĻāĻŋāĻ¤ā§ āĻā§ āĻŦāĻ˛āĻž āĻšā§ā§āĻā§?
7. āĻŽā§āĻ˛āĻŋāĻ āĻĒāĻĻāĻžāĻ°ā§āĻĨā§āĻ° āĻĒāĻ°āĻŽāĻžāĻŖā§āĻā§āĻ˛ā§āĻ° āĻŽāĻ§ā§āĻ¯ā§ āĻā§ āĻĒāĻžāĻ°ā§āĻĨāĻā§āĻ¯ āĻāĻā§?
8. āĻļāĻā§āĻ¤āĻŋ āĻ āĻ¤ā§āĻŋā§ āĻ¸āĻŽā§āĻĒāĻ°ā§āĻā§ āĻŦāĻāĻāĻŋāĻ¤ā§ āĻā§ āĻāĻ˛ā§āĻāĻ¨āĻž āĻāĻ°āĻž āĻšā§ā§āĻā§?
9. āĻĒā§āĻ°āĻžāĻāĻā§āĻ° āĻŽāĻ¤ āĻā§ āĻāĻŋāĻ˛?
10. āĻŦāĻžāĻāĻ˛āĻžāĻĻā§āĻļā§āĻ° āĻŦāĻ°ā§āĻ¤āĻŽāĻžāĻ¨ āĻāĻ¨āĻ¸āĻāĻā§āĻ¯āĻž āĻāĻ¤?
Question 10 is intentionally outside the selected book and is used as a no-answer test.
Expected behavior:
đĄī¸ Hallucination Control
The chatbot is instructed to answer book-specific questions only from the retrieved book context.
The prompt enforces the following rules:
Use only the retrieved book context.
Do not use outside knowledge.
Use relevant information even if the wording differs from the question.
Answer in Bengali.
Include the relevant chapter and source.
If the information is not available in the selected book, clearly say:
This prevents the chatbot from behaving like a general-purpose knowledge engine.
đ Multilingual Embeddings
The source material is Bengali, so an English-only embedding model would not be appropriate for semantic retrieval.
The project uses:
This model supports multilingual semantic representations and works with the Bengali book content.
đī¸ Why FAISS?
FAISS is used as the vector database because it provides efficient similarity search over embedding vectors and is convenient for a local RAG application.
The vector store is generated from the processed book chunks.
đˇī¸ Metadata Preservation
Each document chunk preserves important metadata such as:
This allows the chatbot to identify where retrieved information came from and provide source references with answers.
𧊠Main Components
crawler.py
crawler.pyDownloads the selected book chapters from Bengali Wikisource.
processor.py
processor.pyCleans raw scraped text and preserves important metadata.
chunker.py
chunker.pySplits cleaned documents into overlapping text chunks.
Final configuration:
embedding.py
embedding.pyGenerates multilingual embeddings and creates the FAISS vector database.
retriever.py
retriever.pyLoads the FAISS database and retrieves the most relevant book chunks.
qa.py
qa.pyCombines retrieval with the Groq LLM and generates Bengali answers based only on retrieved context.
app.py
app.pyProvides the Streamlit chatbot interface.
𧰠Technologies Used
Technology Purpose Python Core programming language LangChain RAG pipeline LangChain Community FAISS and supporting integrations LangChain HuggingFace Embedding integration LangChain Text Splitters Document chunking BeautifulSoup Web scraping Requests HTTP requests Sentence Transformers Multilingual embeddings FAISS Vector similarity search Groq LLM inference Streamlit Web interface python-dotenv Environment variable management
đ Important Design Decisions
Multilingual embedding model
The source material is Bengali, so multilingual embeddings were selected instead of an English-only embedding model.
Chunk size
Different chunking configurations were tested.
The final configuration:
produced 103 chunks and achieved 9/9 expected-chapter retrieval coverage in the current test set.
Top-k retrieval
The retriever uses:
This was chosen because some questions, such as
āĻĒā§āĻ°ā§āĻāĻ¨ āĻā§?, did not consistently retrieve the expected chapter within a smaller top-k value but were found when using top-10 retrieval.đŽ Future Improvements
Possible improvements include:
Bengali-specific embedding models
Hybrid keyword + semantic retrieval
Reranking retrieved documents
Better no-answer detection
Streaming LLM responses
Improved source citation UI
Chunking strategy comparison
Embedding model comparison
Automated retrieval evaluation
End-to-end answer evaluation
Deployment using Streamlit Cloud or another hosting platform
đ¨âđģ Author
Rabiul Islam
AI/ML Developer | Computer Vision | Generative AI | AI Agents
GitHub:
Project Repository:
đ Disclaimer
This project is an educational RAG application built around the selected Bengali book âāĻŦāĻŋāĻļā§āĻŦā§āĻ° āĻāĻĒāĻžāĻĻāĻžāĻ¨â.
The chatbot is designed to answer questions using the selected knowledge source and should not be treated as a general-purpose factual search engine.
â Bonus â Chunking Strategy Comparison
Two different chunking strategies were evaluated to measure their effect on retrieval performance.
Approaches Tested
Strategy Chunk Size Chunk Overlap Total Chunks Strategy A 700 150 135 Strategy B 1000 200 103
Both strategies used the same multilingual embedding model:
sentence-transformers/paraphrase-multilingual-MiniLM-L12-v2The same 9 answerable test questions and the same top-10 retrieval setting were used for both strategies.
Evaluation Method
For each test question, the FAISS vector store retrieved the top 10 chunks.
A retrieval was counted as a hit when the expected chapter appeared in at least one of the top 10 retrieved chunks.
Hit Rate = Correct Retrievals / Total Answerable Questions à 100
Results
Strategy Correct Retrievals Hit Rate 700 / 150 7 / 9 77.8% 1000 / 200 9 / 9 100.0%
Result
The
1000 / 200 configuration achieved a 100.0% retrieval hit rate on the 9 answerable test questions, compared with 77.8% for the 700 / 150 configuration.Therefore, the
1000 / 200 configuration was selected for the final RAG pipeline based on this evaluation.Note: This hit-rate evaluation measures retrieval coverage only. It does not represent the accuracy of the final LLM-generated answers and should not be interpreted as a universal result for other datasets or books.
Like this project
Posted Sep 18, 2026
Developed a RAG chatbot for a Bengali book using LangChain and FAISS.
Likes
0
Views
1
