A Fine-Tuned Multimodal AI Chatbot for Dietary Health and Nutrition, Purrfessor: Development and Mixed Methods Evaluation

Background: The integration of Large Language and Vision Assistant models with food and nutrition data enables multimodal meal analysis and contextual dietary guidance. Despite this potential, the reliability and practical usefulness of such systems for supporting everyday dietary decision-making remain underexplored. Objective: This study introduces Purrfessor, an innovative artificial intelligence (AI) chatbot designed to provide personalized dietary guidance through interactive, multimodal engagement. The study aimed to evaluate its performance in ingredient recognition and recipe generation. Methods: The Purrfessor chatbot was trained using a combination of the FoodData Central database from the US Department of Agriculture (USDA), the Recipe2img dataset featuring food images and corresponding recipes, a curated human-annotated dataset derived from Recipe1M, and a customized question-and-answer dialogue dataset. The system operates under a session-based, multiturn interaction paradigm, with memory retained only within an active session and no cross-session memory persistence. We implemented a 2-phase evaluation framework combining AI-based performance assessment and human scoring. Results: Purrfessor achieved a high average cosine similarity of 0.90 in ingredient recognition with human-coded references. In GPT-4.1-based (OpenAI) evaluation of recipe generation quality, Purrfessor outperformed the raw Large Language and Vision Assistant model across all evaluated dimensions, with the largest improvements in completeness (7.44 vs 6.52), consistency (8.90 vs 7.81), and clarity (9.13 vs 8.39). Overall recipe quality improved from 7.66 to 8.35. Automatic metrics indicated strong ingredient coverage (0.78) and moderate step complexity (0.74), with lower coherence (0.62) and temperature and time specification (0.59), yielding an overall structured score of 0.68. Human evaluators rated Purrfessor's question-and-answer accuracy highly: correctness (mean 8.71, SD 1.15), relevance (mean 9.99, SD 0.10), and clarity (mean 9.33, SD 0.68). Error analysis indicated that 56% of responses contained minor hallucinations (ie, inclusion of inferred secondary details or invisible garnishes). At the same time, core food identification and overall recipe logic remained accurate. Conclusions: Findings highlight the role of anthropomorphic chatbot design and multimodal AI in supporting engaging dietary health conversations. This study offers an example of AI-driven, evidence-based dietary guidance and underscores the potential of health chatbots to nudge informed health decision-making. Insights contribute to the development of digital health interventions and personalized health communication strategies, with implications for the design of engaging, user-centered AI health assistants.