A Factual and Child Centric LLM Based Chatbot for STEAM and Empathetic Conversations Architected in an Agentic FSM Inspired Framework

Abstract

This work aims to develop a chatbot tailored to teach kids STEAM concepts and engage in empathetic conversations. Historically, chatbots have relied on scripted traditional natural language processing (NLP) techniques. Lately, large language models (LLMs) have been shown to engage in more natural dialogue, but propitiatory models may pose privacy concerns, and the computational requirements of large open-source models are prohibitive. Moreover, small open-source LLMs are not trained to have empathetic and domain-specific conversations tailored for children and may produce false or harmful content. While retrieval augmented generation (RAG) has become popular for augmenting general-purpose LLMs with new knowledge, LLMs may still ignore the provided content and hallucinate misinformation. Furthermore, studies show that LLMs may non-consensually nudge the child from a factual topic towards an empathetic one. To address this gap, we propose an agentic finite state machine (FSM) inspired data generation workflow that uses existing factual datasets and empathetic conversations, search queries, and transition statements distilled from large LLM to produce conversations that separate factual RAG-based Q\&A from empathetic exchanges, while allowing for child-initiated transitions. Fine-tuning a LLAMA 3.2 1b model on the dataset, shows that the fine-tuned model can adhere to the defined flow, while the base model fails. We also apply quantization to reduce computational overhead, and the results show minimal performance degradation (<4\%) for 4 bit quantized models. Additionally, to assess the factuality of the model's content, this research proposes COSMIC, a novel metric, which captures semantic similarity and detects contradictions, overcoming limitations of popular lexical overlap metrics like BLEU, ROUGE that overlook semantics, and embedding-based similarity metrics that ignore contradictions. The results show that, unlike other metrics, COSMIC correlates well with both semantic similarity (82\%) and consistency (88\%). Finally, we compare document retrieval methods, and determine that contextualized embeddings is the most reliable method.