Embedded Files
Introduction
It is increasingly common for companies to use chatbots to handle customer queries and requests. Many developers use techniques like Retrieval Augmentation Generation (RAG) with LangChain to build such chatbots.
However, while building my chatbot application, several questions arose:
How can I verify if my chatbot is answering questions correctly before deployment?
Must I rely on manual testing by humans to create test questions and evaluate answers? This approach seemed inefficient.
Fortunately, LangChain offers tools to generate and evaluate question-answer pairs efficiently.
In this notebook, I demonstrate how to:
Use an LLM to generate new question-answer pairs based on context data.
Evaluate the chatbot's performance using these generated pairs.
Improve chatbot performance based on evaluation results.
When tested, my chatbot's performance improved from 79% to 93%, overcoming issues like failing to answer rephrased questions. This was achieved through one round of data augmentation, resulting in a 14% increase in correct answers.
Process Summary
See this notebook for the codes implementing the described process..
1. Augmenting QA Dataset with LLM
Key Import:
from langchain.evaluation.qa import QAGenerateChain
Steps:
1. Load your dataset.
2. Select an LLM model and define a generation prompt to instruct the model to generate questions based on provided context.
3. Use QAGenerateChain to generate new QA pairs.
4. Save the generated pairs into a DataFrame for review.
Example Output for generated QA pairs:
2. Evaluating Chatbot Performance
Key Import:
from langchain.evaluation.qa import ContextQAEvalChain
Steps:
1. Configure your chatbot’s model, prompt, vector database, and chain. Maintain consistency with the settings used during chatbot deployment. The chain here is referring to the chain you use for your chatbot. It can be something like a RetrievalQA chain or a ConversationalRetrieval chain.
2. Use the generated questions from the previous step to create predicted responses.
3. Save the questions, predicted responses, and expected responses (the ground truth) into a DataFrame.
4. Set up an evaluation prompt for the grading LLM assistant, using chains such as ContextQAEvalChain or QAEvalChain. The default prompt treats the LLM as a teacher grading student answers.
5. Specify which columns contain the predictions (prediction_key), ground truth (context_key), and questions (question_key) when calling the evaluation function. If using a single file, both examples and predictions can reference it. For multiple files, consult the documentation for proper configuration.
Note: Evaluation outputs sometimes misinterpret answers as incorrect. Human review is recommended to validate grading quality during testing. We can consider spending more time to edit the prompt better and using a smarter model.
Example Output for grading results:
3. Improving Chatbot Performance
After evaluation, identify questions the chatbot answered incorrectly or failed to address. Add these QA pairs, along with correct answers, to the dataset used in the chatbot’s retrieval chain.
To verify improvements:
1. Rerun data augmentation to generate new and unseen testing questions and rerun evaluation steps.
2. Compare performance metrics to confirm enhancements.
Evaluation Results
See this notebook for the codes implementing the described process..
Round 1 Results:
Machine Evaluation Score: 72%
Errors:
20% "I don’t know" responses
8% suspected incorrect answers
After human review:
Round 1 Final Evaluation Score: 79%
Round 2 Results:
Machine Evaluation Score: 74%
Errors:
2% "I don’t know" responses
24% suspected incorrect answers
After human review:
Round 1 Final Evaluation Score: 93%
Observations and Recommendations
I initially tested the process using llama3-8b-chinese-chat-ollama-q4 (8B parameters), the same model as my chatbot application, but deliberately switched to Llama 3.2 (3B parameters) in this notebook to induce more incorrect answers for demo purposes.
Key observations:
Question Generation: Llama 3.2 effectively generated rephrased questions, but its evaluation accuracy was lower. Higher-quality models may improve evaluation reliability.
LLM Grader Challenges: After data augmentation, the LLM sometimes struggled to distinguish context from generated answers, requiring refined prompts and smarter evaluators.
AI-Augmented Data Risks: Repeated augmentation using previously generated data can lead to nonsensical results. Carefully curate and validate augmented datasets to avoid data drift.
Final Thoughts:
This process demonstrates that LLM-generated data augmentation can significantly enhance chatbot performance when combined with systematic evaluation. However, human oversight remains critical for validation.
Future improvements could include:
Refining LLM prompts to improve grading consistency.
Using larger, more capable evaluation models for grading.
Exploring multi-language support for broader applicability.
By leveraging LLMs for data augmentation and evaluation, developers can streamline chatbot optimization while minimizing manual effort.
Google Sites
Report abuse