A Study on the Direction of Security-Enhanced On-Device AI Secretary Systems through Comparison of Machine Learning Context-Based and LLM Context-Based Models

This study explores the potential of Machine Learning (ML)-based models as a secure and network-independent alternative to current Large Language Model (LLM)-based AI secretary systems. While LLM-based assistants like Google Gemini and Apple Intelligence offer advanced context analysis, they face critical challenges regarding personal information leakage and dependency on external servers. To verify technical substitutability, we conducted a comparative performance analysis between representative ML models (RoBERTa-large and DeBERTa-v3-large) and LLM models (Gemini 2.5 Pro and GPT-4.1) using a message priority prediction task. The evaluation utilized metrics including NDCG@10, Kendall’s tau, and Exact Match Accuracy, based on reference priorities established by a consensus of 20 human evaluators. Key Findings: Contextual Understanding: All models, regardless of architecture, demonstrated high proficiency in identifying core message importance, with ML models showing marginal performance differences in the NDCG@10 metric (0.978+) compared to LLMs. Reasoning Limitations: LLM-based models significantly outperformed ML-based models in Exact Match Accuracy, highlighting the current limitations of ML architectures in precise reasoning and nuanced priority judgment. Security Potential: The results suggest that server-independent ML-based AI assistants can effectively resolve inherent security vulnerabilities while maintaining respectable performance in basic context understanding. This research contributes to the development of privacy-enhanced, on-device AI systems by confirming the foundational potential of ML architectures to handle secretary tasks locally.