Why Doesn’t AI Reduce the Workload as Much as Expected? : The Limits of Human-in-the-Loop and AI-Ready Workflow Design

Although AI was supposed to make the work easier, the workload has not decreased as much as expected.

In this session, we will share our work on an AI workflow that combines classification models and LLMs, using product patrol operations for Yahoo! JAPAN Auction and Yahoo! JAPAN Flea Market as a case study. We will also discuss the limits of Human-in-the-loop that became clear through this initiative.

By using classification models to narrow down review targets and LLMs to provide reasoning, AI-assisted review has improved in both accuracy and explainability. At the same time, we found that as long as the workflow continues to rely on human confirmation, there remains a gap between improving quality and actually reducing operational workload.

Instead of simply applying AI to existing operations as they are, we have been rethinking workflows, rules, and operations with AI as a premise. We will share our hands-on trial and error around designing rules that are easier for AI to judge and easier for operations teams to use, defining the roles of classification models and LLMs, and building workflows that can be improved through repeated testing in the field.

What does it take to move AI-driven process improvement beyond PoC and make it work in real operations? This session will offer practical perspectives for engineers, as well as those involved in operations, PM, planning, CS, and business process improvement, on how to make AI useful in day-to-day work.