CareEval: Evaluating Large Language Models for Decision-Making in Physical Robot Caregiving

We present CareEval, a benchmark for evaluating the physical caregiving decision-making abilities of Large Language Models. Developed with a licensed occupational therapist expert in caregiving and validated by eight clinical stakeholders, it contains 100 realistic scenarios spanning all six basic Activities of Daily Living. Instead of testing general reasoning, CareEval assesses whether model responses account for key physical caregiving factors, such as user function, agency, intent, communication, and safety, and align with expert practice. Across several state-of-the-art LLMs, the best model only scores 53.1%, revealing substantial gaps in current models’ ability to reason about physical caregiving. We release 80 of the CareEval scenarios and all prompts through our website: https://emprise.cs.cornell.edu/care-eval/.