Readings

Linguistic Data Analysis II

Readings are assigned by day and should be completed before the corresponding day of the intensive course. Each day lists a small required reading (marked Skim or Read); optional items are provided for those who want to go deeper.

Access

PDFs for readings not freely available online are posted on Google Classroom. Where a work is open access, a DOI link is provided below.

Day 1 · Introduction & First Experience

Skim

Abdurahman, S., Ziabari, A. S., Moore, A. K., Bartels, D. M., & Dehghani, M. (2025). A primer for evaluating large language models in social-science research. Advances in Methods and Practices in Psychological Science, 8(2). https://doi.org/10.1177/25152459251325174
- Get a feel for the questions the course tackles; don’t worry about the details yet.

Watch in class

Sanderson, G. (3Blue1Brown). (2024). Large Language Models explained briefly [Video]. https://www.3blue1brown.com/lessons/mini-llm/ (YouTube)
- What the model predicts, where its numbers come from, and what attention does.

Day 2 · Annotation, Gold Standards & Metrics

Read

Eguchi, M., & Kyle, K. (2024). Building custom NLP tools to annotate discourse-functional features for second language writing research: A tutorial. Research Methods in Applied Linguistics, 3(3), 100153. https://doi.org/10.1016/j.rmal.2024.100153

Skim §3.1–§3.2 before Session 5 — this is the corpus you annotate against all week, and Session 5 opens by asking you to extract two of its design decisions.

Arase, Y., Uchida, S., & Kajiwara, T. (2022). CEFR-based sentence difficulty annotation and assessment. Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing (pp. 6206–6219). https://doi.org/10.18653/v1/2022.emnlp-main.416
- Their Table 6 is the supervised benchmark your Day-3 prompt gets compared against in Session 9.

Optional / further reading

Fuoli, M. (2018). A stepwise method for annotating appraisal. Functions of Language, 25(2), 229–258. https://doi.org/10.1075/fol.15016.fuo
- The source of the three annotation principles used in Session 4; useful if you are building a scheme of your own.
Kyle, K., & Eguchi, M. (2024). Evaluating NLP models with written and spoken L2 samples. Research Methods in Applied Linguistics, 3(2), 100120. https://doi.org/10.1016/j.rmal.2024.100120
Mizumoto, A. (2025). Automated analysis of common errors in L2 learner production: Prototype web application development. Studies in Second Language Acquisition, 1–18. https://doi.org/10.1017/S0272263125100934
Yamashita, T. (2024). An application of many-facet Rasch measurement to evaluate automated essay scoring: A case of ChatGPT-4.0. Research Methods in Applied Linguistics, 3(3), 100133. https://doi.org/10.1016/j.rmal.2024.100133

Day 3 · Prompt Design & Iteration

Read

Huang, J., & Mizumoto, A. (2025). Prompt engineering: Enhancing AI-driven language learning and feedback. In L. McCallum & D. Tafazoli (Eds.), The Palgrave Encyclopedia of Computer-Assisted Language Learning (pp. 1–8). Springer Nature Switzerland. https://doi.org/10.1007/978-3-031-51447-0_103-1
Kim, M., & Lu, X. (2024). Exploring the potential of using ChatGPT for rhetorical move-step analysis: The impact of prompt refinement, few-shot learning, and fine-tuning. Journal of English for Academic Purposes, 71, 101422. https://doi.org/10.1016/j.jeap.2024.101422

Day 4 · Methodology & Pipeline Assembly

Read (in full) — revisit the Day 1 primer, this time closely, as the checklist behind reproducible, transparent LLM-based reporting; Ready for full discussion of the contents and implication to applied linguistics research.

Abdurahman, S., Ziabari, A. S., Moore, A. K., Bartels, D. M., & Dehghani, M. (2025). A primer for evaluating large language models in social-science research. Advances in Methods and Practices in Psychological Science, 8(2). https://doi.org/10.1177/25152459251325174

Day 5 · Project Finalization & Presentations

No new reading — project work only.

Citing the datasets

Your report must cite the corpus your track uses, not just the course. Full provenance, licences and the paper for every track — CEFR-SP, RAAMove, CaRS-50, AutoErrorAnalyzer, ICNALE GRA — is in resources/datasets/SOURCES.md.

Reference — used in class, not assigned

You do not need to read these; they are here so you can cite the source of a number you were shown.

Other

Landis, J. R., & Koch, G. G. (1977). The measurement of observer agreement for categorical data. Biometrics, 33(1), 159–174. https://doi.org/10.2307/2529310
McHugh, M. L. (2012). Interrater reliability: The kappa statistic. Biochemia Medica, 22(3), 276–282. https://doi.org/10.11613/BM.2012.031
- Both are behind the κ interpretation bands in Session 4. Treat the bands as conventions, not thresholds.
Prompt Engineering Guide — the technique map from Session 7, for self-study.

--- title: "Readings" subtitle: "Linguistic Data Analysis II" toc: true toc-depth: 2 --- Readings are assigned by day and should be completed **before** the corresponding day of the intensive course. Each day lists a small **required** reading (marked *Skim* or *Read*); **optional** items are provided for those who want to go deeper. ::: {.callout-note} ## Access PDFs for readings not freely available online are posted on **Google Classroom**. Where a work is open access, a DOI link is provided below. ::: ## Day 1 · Introduction & First Experience **Skim** - Abdurahman, S., Ziabari, A. S., Moore, A. K., Bartels, D. M., & Dehghani, M. (2025). A primer for evaluating large language models in social-science research. *Advances in Methods and Practices in Psychological Science, 8*(2). [https://doi.org/10.1177/25152459251325174](https://doi.org/10.1177/25152459251325174) - Get a feel for the questions the course tackles; don't worry about the details yet. **Watch** in class - Sanderson, G. (3Blue1Brown). (2024). *Large Language Models explained briefly* [Video]. [https://www.3blue1brown.com/lessons/mini-llm/](https://www.3blue1brown.com/lessons/mini-llm/) ([YouTube](https://www.youtube.com/watch?v=WMcwoIyK4DA)) - What the model predicts, where its numbers come from, and what attention does. ## Day 2 · Annotation, Gold Standards & Metrics **Read** - Eguchi, M., & Kyle, K. (2024). Building custom NLP tools to annotate discourse-functional features for second language writing research: A tutorial. *Research Methods in Applied Linguistics, 3*(3), 100153. [https://doi.org/10.1016/j.rmal.2024.100153](https://doi.org/10.1016/j.rmal.2024.100153) **Skim §3.1–§3.2 before Session 5** — this is the corpus you annotate against all week, and Session 5 opens by asking you to extract two of its design decisions. - Arase, Y., Uchida, S., & Kajiwara, T. (2022). CEFR-based sentence difficulty annotation and assessment. *Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing* (pp. 6206–6219). [https://doi.org/10.18653/v1/2022.emnlp-main.416](https://doi.org/10.18653/v1/2022.emnlp-main.416) - Their Table 6 is the supervised benchmark your Day-3 prompt gets compared against in Session 9. **Optional / further reading** - Fuoli, M. (2018). A stepwise method for annotating appraisal. *Functions of Language, 25*(2), 229–258. [https://doi.org/10.1075/fol.15016.fuo](https://doi.org/10.1075/fol.15016.fuo) - The source of the three annotation principles used in Session 4; useful if you are building a scheme of your own. - Kyle, K., & Eguchi, M. (2024). Evaluating NLP models with written and spoken L2 samples. *Research Methods in Applied Linguistics, 3*(2), 100120. [https://doi.org/10.1016/j.rmal.2024.100120](https://doi.org/10.1016/j.rmal.2024.100120) - Mizumoto, A. (2025). Automated analysis of common errors in L2 learner production: Prototype web application development. *Studies in Second Language Acquisition*, 1–18. [https://doi.org/10.1017/S0272263125100934](https://doi.org/10.1017/S0272263125100934) - Yamashita, T. (2024). An application of many-facet Rasch measurement to evaluate automated essay scoring: A case of ChatGPT-4.0. *Research Methods in Applied Linguistics, 3*(3), 100133. [https://doi.org/10.1016/j.rmal.2024.100133](https://doi.org/10.1016/j.rmal.2024.100133) ## Day 3 · Prompt Design & Iteration **Read** - Huang, J., & Mizumoto, A. (2025). Prompt engineering: Enhancing AI-driven language learning and feedback. In L. McCallum & D. Tafazoli (Eds.), *The Palgrave Encyclopedia of Computer-Assisted Language Learning* (pp. 1–8). Springer Nature Switzerland. [https://doi.org/10.1007/978-3-031-51447-0_103-1](https://doi.org/10.1007/978-3-031-51447-0_103-1) - Kim, M., & Lu, X. (2024). Exploring the potential of using ChatGPT for rhetorical move-step analysis: The impact of prompt refinement, few-shot learning, and fine-tuning. *Journal of English for Academic Purposes, 71*, 101422. [https://doi.org/10.1016/j.jeap.2024.101422](https://doi.org/10.1016/j.jeap.2024.101422) ## Day 4 · Methodology & Pipeline Assembly **Read (in full)** — revisit the Day 1 primer, this time closely, as the checklist behind reproducible, transparent LLM-based reporting; Ready for full discussion of the contents and implication to applied linguistics research. - Abdurahman, S., Ziabari, A. S., Moore, A. K., Bartels, D. M., & Dehghani, M. (2025). A primer for evaluating large language models in social-science research. *Advances in Methods and Practices in Psychological Science, 8*(2). [https://doi.org/10.1177/25152459251325174](https://doi.org/10.1177/25152459251325174) ## Day 5 · Project Finalization & Presentations No new reading — project work only. ## Citing the datasets Your report must cite the corpus your track uses, not just the course. Full provenance, licences and the paper for every track — CEFR-SP, RAAMove, CaRS-50, AutoErrorAnalyzer, ICNALE GRA — is in [`resources/datasets/SOURCES.md`](../resources/datasets/index.md). ## Reference — used in class, not assigned You do not need to read these; they are here so you can cite the source of a number you were shown. **Other** - Landis, J. R., & Koch, G. G. (1977). The measurement of observer agreement for categorical data. *Biometrics, 33*(1), 159–174. [https://doi.org/10.2307/2529310](https://doi.org/10.2307/2529310) - McHugh, M. L. (2012). Interrater reliability: The kappa statistic. *Biochemia Medica, 22*(3), 276–282. [https://doi.org/10.11613/BM.2012.031](https://doi.org/10.11613/BM.2012.031) - Both are behind the κ interpretation bands in Session 4. Treat the bands as conventions, not thresholds. - [Prompt Engineering Guide](https://www.promptingguide.ai/techniques) — the technique map from Session 7, for self-study.