Towards Capable and Secure Autonomous Computer-Use Agents (Student Abstract)

Autonomous computer-use agents (ACUAs) enable end-to-end computer operation with human-like capabilities, executing commands across applications and making independent decisions. However, their real-world effectiveness and security remain largely untested. A systematic evaluation of ACUAs from Anthropic, OpenAI, and open-source projects categorized them into full-computer-access and browser-based agents. Findings reveal substantial limitations, with success rates dropping as low as 28% in some cases. Additionally, a 100% rate of unauthorized software installation was observed in certain tasks. The agents also demonstrated clear susceptibility to prompt injection attacks. The impact of varied prompting strategies on performance was also examined. Building on these weaknesses, development of a specialized agent for office tasks is proposed. This work bridges agentic AI, human-computer interaction (HCI), and security to address the observed limitations of ACUAs, prioritizing both capability and safety.