Autograding Tool for Written Answers and Complex Assignments

In Fall 2025, the Spark! team focused on improving grading consistency for text-based short answer assessments using Azure AI Foundry. Through prompt iteration and testing with Mike’s teaching team, the AI-generated grades and feedback became acceptable by the end of the semester, and the current prompt/model configuration does not require major changes for text-only assessment grading at this time. However, the team also identified a key limitation in Azure AI Foundry’s multimodal capabilities. While Azure can extract text from PDFs and slides, including text embedded in images when preserved in the file, it cannot reliably interpret or summarize images that do not contain text. Since CS 581 includes course materials and assignment submissions that are often Excel-based and diagram/image-based, this limitation affects the feasibility of expanding the AI grading workflow beyond text-only assessments. This semester, the project will continue the grading pilot while focusing on selecting and implementing an AI platform that can reliably handle multimodal course materials and assignment submissions (PDFs with images, Excel files, and image/diagram submissions), with API access as a core requirement. Client Note: We want to parse Blackboard data (student submissions, lecture material, previous semester student submissions, etc.) in a way that a RAG system can read. Our main concern is that existing files are not readable using RAG, either in Azure or other tools. Whatever tool they choose (if Azure is not workable for our assignment use case), if needed please create a script to convert all lecture and student submissions to outputs interpretable by rag. This may need to be word documents or text. This project can be divided into 3 phases: Phase 1: Research and Tool Comparison (Multimodal Benchmarking) The team will evaluate multiple AI platforms/tools (Azure and non-Azure options) to determine which best supports CS 581 needs, especially multimodal interpretation. To keep testing consistent, ETI will provide three representative CS 581 files: • One Excel file (example student assignment submission) • One image submission (diagram, exported as PNG/JPEG) • One lecture PDF (“Principal Lectures”) containing a mix of text and images The team will run a structured comparison across tools, documenting: • whether the tool can read and interpret each file type correctly • whether it can use course materials as reference during grading • API access and integration feasibility (and rough cost considerations) Phase 2: Environment Setup and Configuration After selecting a platform, the team will: • set up a local testing environment using that platform’s API • implement a repeatable workflow for ingesting and evaluating the 3 sample files • document setup steps so the system is reproducible If Azure remains the chosen platform, ETI will provide documentation for the existing system so the team can mirror the current configuration as closely as possible (then iterate based on research findings). Phase 3: Pilot Support and Iteration (Live Course Data) Once the environment is running, the team will support active CS 581 pilots by: • continuing to improve grading for short-answer assessments • expanding analysis into discussion posts (if in scope for the pilot) • configuring and testing grading support for Excel assignments and diagram/image submissions • comparing AI outputs to facilitator/instructor grading and documenting improvements over time • recording any changes made to the environment, prompts, rubrics, or workflow Stretch / add-on within Phase 3: Develop clearer evaluation metrics for grading quality/consistency (potentially informed by IRR-style approaches and ETI research), and propose how to adopt them.