How should we design AI tools that handle personal information? Evaluating AI-generated personalized care advice based on deeply personal data

AI systems typically rely on commonly available knowledge from the internet—that is, public “common sense.” However, when applying AI in personalized service domains such as healthcare and elder care, it becomes essential to incorporate deeply personal information, such as individuals’ life histories. This paper introduces a case study of an AI tool that provides personalized care advice to care workers, aiming to derive insights for designing services that focus on the individuality of each service user. We developed a prototype tool using a profile sheet constructed from real narratives. Care workers’ evaluations, analyzed qualitatively, yielded insights regarding AI usefulness, individualized care, practical applicability, advice presentation, limitations and risks, and AI use contexts.