Research paper

When Evaluators Cry Wolf: Lessons from Production LLM-as-Judge Evaluation in Educational AI

Published by Association for Computational LinguisticsOct 2026PDF

See what happened when automated evaluators raised incorrect alarms in a production educational AI service.

When Evaluators Cry Wolf: Lessons from Production LLM-as-Judge Evaluation in Educational AI cover

Resource access

Get the full resource

Enter your business details to read the full resource.

First nameLast nameWork emailCompanyJob titleCountry or region

Chris Rohlfs, Rodrigo Vergara Bosse, Priscilla Hopper, Keanon O’Keefe, Patrick J Russell. When Evaluators Cry Wolf: Lessons from Production LLM-as-Judge Evaluation in Educational AI. Proceedings of the 2026 Conference on Empirical Methods in Natural Language Processing: Industry Track. © 2026 Association for Computational Linguistics. CC BY 4.0. 10.18653/2026.emnlp-industry.193. Original PDF, unmodified.

CC BY 4.0

When Evaluators Cry Wolf: Lessons from Production LLM-as-Judge Evaluation in Educational AI

Related research.

Learning Selective LLM Autonomy from Copilot Feedback in Enterprise Customer Support Workflows

ECom-Bench: Can LLM Agent Resolve Real-World E-commerce Customer Support Issues?

Retrieval Enhancements for RAG: Insights from a Deployed Customer Support Chatbot