Expert-Directed LLM Annotation
Term
Expert-Directed LLM Annotation
Idea Level
Method
Definition
Expert-Directed LLM Annotation is a method introduced by Yuen Yuen Ang and her collaborators (2026) that uses large language models (LLMs) to scale expert human classification rather than replacing it. It combines expert-designed codebooks, structured annotator training, gold-standard human labels, and fine-tuned LLMs to annotate large corpora reliably. In the Adaptive Policy Communication (APC) project, they used this method to classify millions of Chinese policy directives according to a typology of five policy signals.
Connection with Directed Improvisation with AI
Expert-Directed LLM Annotation illustrates one method under Ang’s philosophy of Directed Improvisation with AI: humans structure the paradigm, categories, and training protocol, while LLMs scale classification across large datasets. It treats AI not as a passive tool or substitute for human expertise, but as an improvising agent whose outputs depend on how humans direct the task.
Sources
Method:
Sun, Chang, Ang et al. (2026). "CAPC-CG: A Large-Scale, Expert-Directed LLM-Annotated Corpus of Adaptive Policy Communication in China." Association for Computational Linguistics (ACL).
Human–AI theoretical foundation:
Ang, Y.Y. (2026). “Directed Improvisation with AI.” Course syllabus, Johns Hopkins University.
Genealogy
[Model] Directed Improvisation
→ [Application] Directed Improvisation with AI: human as director, AI as improvising agent
+
[Theory] Adaptive Policy Communication
→ [Typology] Five-signal typology, including ambiguous signals
→ [Method] Expert-Directed LLM Annotation
→ [Dataset] CAPC-CG: Chinese Adaptive Policy Communication–Central Government Corpus
Quotes
[Definition of method] Rather than relying on prompt-based annotation alone, our approach integrates expert-designed codebooks, rigorous annotator training, and fine-tuning on a gold-standard labeled set to capture varied and ambiguous policy signals.
[Using LLMs to scale human expert judgment, not replacing it] Unlike prior work that treats LLMs as substitutes for expert human annotators, our method treats expert annotation as the foundation and uses LLMs to scale this system across large corpora.
[Paradigm to operationalization] Our study demonstrates that APE, as a paradigm, can generate new lines of inquiry (how leaders communicate under complexity), theory (adaptive policy communication), typology (five policy signals), measurement (theory-driven, human-LLM annotation), and novel data (CAPC-CG), which can inspire more data collection.
— Sun, Chang, Ang et al., “CAPC-CG”
[Human–AI logic] In a world where AI will be ubiquitous, the task is not to eliminate AI, but to redefine human agency—from producing answers to directing how humans and AI produce them together.
[Direction as meta-skill] The meta-skill explored in the course is direction: the ability to shape how problem-solving or creativity unfolds by structuring its conditions, rather than producing answers directly.
— Ang, “Directed Improvisation with AI” (2026), p. 2
Concept Constellation
Across Ang’s work, Expert-Directed LLM Annotation consistently co-appears with the following concepts and analytic themes:
From Paradigm to theory and methodology
CAPC-CG