Session 13

Large Language Models for linguistic annotation

Author

Masaki EGUCHI, Ph.D.

Modified

August 7, 2025

One-liner

We will explore how powerful and limited Large Language Models (LLMs) can be at the same time.

🎯 Learning Objectives

By the end of this session, students will be able to:

Describe how LLMs are trained generally and what LLMs do to produce language.

Explain the benefits and drawbacks of using LLMs for linguistic annotation.

Demonstrate/discuss potential impacts of prompts on the LLMs performance on linguistic annotation.

Design an experiment to investigate LLMs output accuracy on a given annotation task.

🔑 Key Concepts

Large Language Models (LLMs) and Language Generation
Prompt engineering
Fine-tuning

📚 Required Readings

Mizumoto, A., Shintani, N., Sasaki, M., & Teng, M. F. (2024). Testing the viability of ChatGPT as a companion in L2 writing accuracy assessment. Research Methods in Applied Linguistics, 3(2), 100116. https://doi.org/10.1016/j.rmal.2024.100116
(Skim) Kim, M., & Lu, X. (2024). Exploring the potential of using ChatGPT for rhetorical move-step analysis: The impact of prompt refinement, few-shot learning, and fine-tuning. Journal of English for Academic Purposes, 71, 101422. https://doi.org/10.1016/j.jeap.2024.101422

🌊 Dive Deeper - Recommended Readings

Mizumoto, A. (2025). Automated analysis of common errors in L2 learner production: Prototype web application development. Studies in Second Language Acquisition, 1–18. https://doi.org/10.1017/S0272263125100934
Yu, D., Li, L., Su, H., & Fuoli, M. (2024). Assessing the potential of LLM-assisted annotation for corpus-based pragmatics and discourse analysis: The case of apology. International Journal of Corpus Linguistics, 29(4), 534–561. https://doi.org/10.1075/ijcl.23087.yu

Materials

Slides for the session

📊 View Interactive Slides

Other resources

Reflection

You can now:

--- title: "Session 13" subtitle: "Large Language Models for linguistic annotation" --- ## One-liner We will explore how powerful and limited Large Language Models (LLMs) can be at the same time. ## 🎯 Learning Objectives > By the end of this session, students will be able to: > > - Describe how LLMs are trained generally and what LLMs do to produce language. > - Explain the benefits and drawbacks of using LLMs for linguistic annotation. > - Demonstrate/discuss potential impacts of prompts on the LLMs performance on linguistic annotation. > - Design an experiment to investigate LLMs output accuracy on a given annotation task. ## 🔑 Key Concepts - Large Language Models (LLMs) and Language Generation - Prompt engineering - Fine-tuning ## 📚 Required Readings - Mizumoto, A., Shintani, N., Sasaki, M., & Teng, M. F. (2024). Testing the viability of ChatGPT as a companion in L2 writing accuracy assessment. Research Methods in Applied Linguistics, 3(2), 100116. https://doi.org/10.1016/j.rmal.2024.100116 - (**Skim**) Kim, M., & Lu, X. (2024). Exploring the potential of using ChatGPT for rhetorical move-step analysis: The impact of prompt refinement, few-shot learning, and fine-tuning. Journal of English for Academic Purposes, 71, 101422. https://doi.org/10.1016/j.jeap.2024.101422 ## 🌊 Dive Deeper - Recommended Readings - Mizumoto, A. (2025). Automated analysis of common errors in L2 learner production: Prototype web application development. Studies in Second Language Acquisition, 1–18. https://doi.org/10.1017/S0272263125100934 - Yu, D., Li, L., Su, H., & Fuoli, M. (2024). Assessing the potential of LLM-assisted annotation for corpus-based pragmatics and discourse analysis: The case of apology. International Journal of Corpus Linguistics, 29(4), 534–561. https://doi.org/10.1075/ijcl.23087.yu ## Materials ### Slides for the session <div class="d-flex gap-2 mb-3"> [📊 View Interactive Slides](../../slides/session-13.html){.btn .btn-primary .btn-lg target="_blank"} </div> ### Other resources - [Prompting engineering guide](https://www.promptingguide.ai/) - [Post by Mark Davies on how similar LLM's "introspections" are to corpus data](https://www.english-corpora.org/ai-llms/corpora-vs-llms.html) ## Reflection You can now: