Using LLM-generated draft replies to support human experts in responding to stakeholder inquiries in maritime industry: a real-world case study of Industrial AI

This study investigated the utility of Large Language Models (LLMs) in supporting human experts in the maritime industry by generating draft replies to stakeholder inquiries. Using a mixed-methods approach, observations, interviews, a survey, and text similarity analysis, we examined how human experts perceive LLM drafts, how drafts align with final replies, and their potential impact on workflows and stakeholder trust. Findings show that while LLM drafts can improve efficiency, linguistic consistency, and time management, they frequently require manual adjustments to ensure accuracy and relevance. Case handlers’ attitudes were generally ‘skeptical but curious,’ reflecting cautious adoption and emphasizing the role of managerial support in fostering trust and engagement. Text similarity analyses indicated moderate alignment between drafts and final replies, reinforcing the need for human oversight. Overall, LLMs are not yet suitable for independent use in safety-critical settings but can serve as valuable augmentative tools. By leveraging human expertise alongside LLM capabilities, organizations can enhance workflow efficiency while maintaining quality, accuracy, and stakeholder trust. These findings provide actionable guidance for implementing AI responsibly in specialized industrial contexts and for designing human-AI collaboration that balances automation benefits with critical human judgment.