NYC Gazette

In an effort to improve digital education, researchers have introduced an AI-powered teaching system designed to integrate visual learning with interactive content. Led by Dr. Youshan Zhang, an assistant professor at the Katz School, the study was presented at the 2025 IEEE Integrated STEM Education Conference. The study, "Automatic Teaching Platform on Vision Language Retrieval Augmented Generation (VL-RAG)," addresses the challenges of replicating human teaching's adaptive nature.

Ruslan Gokhman, lead author of the study and a student at the Katz School, emphasized the difficulties in replicating personalized teaching feedback in AI platforms. Human instructors can provide adaptive, real-time responses, a gap found in current e-learning systems, especially in subjects such as artificial intelligence and machine learning. Traditional platforms rely heavily on text, often disadvantaging visual learners.

Co-author Jialu Li discussed the proposed VL-RAG system, which uses deep-learning retrieval mechanisms and visual question-answering technology to make learning more interactive. It aims to move beyond static responses, offering contextually relevant answers by analyzing tailored content, thus reducing human intervention.

The developed Automatic Teaching Platform supports courses in machine learning and deep learning. It allows students to interact with content using both text and visual queries. This dual-modality approach is particularly beneficial for STEM education, offering annotated images alongside written explanations to enrich understanding.

Testing using the SparrowVQE dataset compared multiple AI models, with Bart Large CNN proving most effective. Dr. Zhang noted that this technology has the potential to revolutionize not only higher education but also corporate and K-12 training by providing visually enriched explanations, illustrating its broad applicability.

“The implications of VL-RAG extend far beyond a single classroom,” stated Dr. Zhang. The team envisions diverse applications, such as aiding medical students in visualizing surgical procedures and helping high school students engage with complex scientific models.

The adaptable nature of VL-RAG could significantly impact digital learning, offering enhanced educational tools to institutions across various disciplines, transforming student engagement with complex materials.