JFTA-Bench: Evaluate LLM’s Ability of Tracking and Analyzing Malfunctions Using Fault Trees

Read full text Summarise with VIRAT
Authors: Yuhui Wang, Zhixiong Yang, Ming Zhang, Shihan Dou, Zhiheng Xi, Di Liang, Enyu Zhou, Senjie Jin, Yujiong Shen, Dingwei Zhu, Yi Dong, Yujia Chen
+4 more Haibo Shi, Tao Gui, Qi Zhang, Xuanjing Huang
Cited by: 0
References: 32

Summary

No summary yet. VIRAT can read this paper and write a plain-language summary of its aim, methods, key findings and limitations.

Summarise with VIRAT Free to try with your Science Catalogs account.

Abstract

In the maintenance of complex systems, fault trees are used to locate problems and provide targeted solutions.To enable fault trees stored as images to be directly processed by large language models, which can assist in tracking and analyzing malfunctions, we propose a novel textual representation of fault trees.Building on it, we construct a benchmark for multi-turn dialogue systems that emphasizes robust interaction in complex environments, evaluating a model's ability to assist in malfunction localization, which contains 3130 entries and 40.75 turns per entry on average.We train an endto-end model to generate vague information to reflect user behavior and introduce long-range rollback and recovery procedures to simulate user error scenarios, enabling assessment of a model's integrated capabilities in task tracking and error recovery, and Gemini 2.5 pro achieves the best performance.

Fields of Study & Subjects

Speech and dialogue systems Natural Language Processing Techniques Topic Modeling

Publication details

Journal / source
Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP)
Pages
26653–26678
Publication date
1 January 2026
Language
EN