https://mllog.dev/en/tags/evaluation/Recent content in Evaluation on MLLog.devhttps://mllog.dev/images/default_mllog.pnghttps://mllog.dev/images/default_mllog.pngHugo -- 0.147.9enMon, 09 Mar 2026 00:00:00 +0000https://mllog.dev/en/posts/constory-consistency-bugs-long-stories-llm/Mon, 09 Mar 2026 00:00:00 +0000https://mllog.dev/en/posts/constory-consistency-bugs-long-stories-llm/<p>Ask any language model to write a 10,000-word story. On page one, the hero has blue eyes. By page five — brown. In chapter three it&rsquo;s Thursday; in chapter six, the same day is suddenly Saturday. A character who died on page seven is chatting away on page ten.</p> <p>Sound familiar? The paper <strong>&ldquo;Lost in Stories: Consistency Bugs in Long Story Generation by LLMs&rdquo;</strong> systematically investigates this problem for the first time — and the results are sobering. Even the best models produce an average of <strong>one consistency error per 10,000 words</strong>, and human experts catch only 17% of them.</p>