Retrieval Augmented Generation

This is a continuation project building on substantial work completed in Fall 2025, when a Spark! team modernized Digital Commonwealth’s Retrieval-Augmented Generation (RAG) search pipeline. That work established a strong technical foundation, including a vector database, metadata-aware search, and a developer-friendly interface. In Spring 2026, the focus shifts from infrastructure to search quality, evaluation, and usability. Students will work on improving how well the system retrieves relevant materials, how performance is measured over time, and how results are presented to users. To keep the scope manageable, the project will focus on the BPL subset of the Digital Commonwealth collection. The ultimate goal is to support a natural-language search experience that returns results that are accurate, contextually relevant, and easy for non-technical users to understand. Project Goals 1. Incorporate Full-Text Document Content into Search • Extend the existing retrieval system to index and search the full text of documents, not just metadata fields • Experiment with combining document text embeddings and metadata signals (e.g., title, creator, date) to improve relevance and ranking 2. Track Inputs, Outputs, and Metrics in a Structured Way • Log queries, retrieved documents, model responses, and evaluation scores • Store results in structured, reusable formats (e.g., CSV, JSON, tables) to enable longitudinal analysis and future dashboarding 3. Build a Gold-Standard Evaluation Dataset • Create a curated set of representative natural-language queries with expected relevant documents • Use this dataset as a consistent benchmark for evaluating retrieval and generation quality 4. Implement an Evaluation and Testing Framework • Experiment with evaluation tools such as Ragas, ContextCheck, Phoenix, or similar frameworks • Establish baseline retrieval and generation metrics • Run evaluations iteratively throughout the semester to track improvements over time