Integration of Public Academic Information and Personal Student Data in a Large Language Model-Based Chatbot Using a Retrieval-Augmented Generation Approach

Abstract

Objective: This study aims to develop an academic chatbot based on a Large Language Model (LLM) using a Retrieval-Augmented Generation (RAG) approach to integrate public academic information and students' personal academic data at Universitas Teknologi Yogyakarta (UTY). Methods: The system utilizes public academic documents processed through text extraction, chunking, and embedding, then stored in ChromaDB. Personal academic data are retrieved through authentication via the Academic Information System (SIA) and stored in PostgreSQL. Evaluation consists of Black Box Testing involving nine scenarios and RAG assessment using 50 questions across five evaluation metrics. Results: Functional testing achieved a success rate of 100% (9 out of 9 scenarios). RAG evaluation yielded 96.0% for Retrieval Hit Rate and Context Relevance, 93.2% for Answer Relevance, 94.0% for Faithfulness, and an equivalent Accuracy score of 92.8% of the maximum scale. Conclusion: Integrating RAG with personal academic data can provide academic information services through a single conversational interface with authenticated access mechanisms.