How Creative Is AI Writing? Generative and Collaborative AI in Japanese Fiction

This study examines the quality of generative and collaborative AI for fictional prose in Japanese. We rolled out an experiment followed by focus groups to elicit responses to 25 texts with the second half randomly presenting the original text, zero-shot-AI-generated text, AI-generated with prompt specification, and human-AI collaboration using multiple-shot and prompt specification. We found that human-AI collaboration had the highest perceived quality, followed by the original text, AI-generated with prompt specification, and finally zero-shot AI-generated content. While participants mentioned the relative absence of originality in human-AI-collaboration, this content was deemed more human-like than the original text due to the polished editing and selection resulting from inputting drafts, adding or replacing nouns, and rearranging sentences and dialogues. The results presented in this study offer a positive outlook on the use of Generative AI for creative writing, but they also highlight the limitations of these tools for developing highly original literary work.