Legal Agent/Chatbot

This project is a continuation of work from last semester, where students built a proof-of-concept AI chatbot to answer housing-related legal questions using a Retrieval-Augmented Generation (RAG) pipeline. The existing system ingests curated legal sources and returns natural-language answers with citations. In Spring 2026, the student team will focus on improving, evaluating, and validating the existing chatbot rather than building a new system from scratch. The work will center on three tightly scoped phases: 1. Model & Data Improvements • Expand and refine the chatbot’s knowledge base using actual legal text (e.g., statutes, regulations, and selected primary sources), not just metadata or secondary summaries. • Improve document embeddings, retrieval quality, and citation grounding. • Incorporate additional vetted legal Q&A pairs developed by clinic students. 2. Evaluation & Testing Framework • Build a gold-standard test set consisting of legal questions, expected answers, and required citations. • Implement structured evaluation using frameworks such as RAGAS, Phoenix, or similar tools to measure retrieval accuracy, citation quality, and response consistency. • Design a system to log and track model outputs over time, enabling week-over-week comparison and iterative improvement. 3. Benchmarking & Comparison • Where permitted, compare the Spark-built chatbot’s responses against outputs from Harvey AI, a legal AI platform available to BU Law. • Analyze differences in accuracy, sourcing, explainability, and cost tradeoffs. • Use insights from benchmarking to inform recommendations for improving the open, lower-cost system. The goal is not to replace legal advice, but to create a transparent, well-evaluated tool that helps tenants understand their rights and prepares them to engage more effectively with legal systems.