AutoErrorAnalyzer — download & preprocess

L2 written-error annotation (and human-vs-tool comparison)

What it is. Sentences from Japanese-EFL essays, each with a human gold error label. The same file also stores a published tool’s predictions, so you can compare your LLM to both.

Difficulty of the labeling judgment: ★★★ — hard. Many error types; some sentences have several.

License: see the OSF project (osf.io/jyf3r)
Cite: Mizumoto (2025), Studies in Second Language Acquisition, 47(3), 867–884.

---

Every dataset in this course is reshaped into the same canonical schema so one notebook works for all of them:

[{"id": 1, "text": "...", "label": "..."}]

The raw data, though, looks different every time. That difference is the lesson — half of building a gold standard is getting messy real data into a clean, consistent shape.

Step 1 — Download the raw data

The annotations live on the OSF project for the paper. We download one CSV directly by its OSF link.

import urllib.request
urllib.request.urlretrieve("https://osf.io/download/gezat/", "data_category.csv")
print("downloaded data_category.csv")

Step 2 — Look at the raw format

It is a CSV. The columns we care about are Sentence, Human_ErrorCategories (the gold), and AEA_ErrorCategories (the tool’s prediction). A sentence can carry several comma-separated error codes, or NO_ERROR.

import csv
with open("data_category.csv", encoding="utf-8-sig", newline="") as f:
    reader = csv.DictReader(f)
    print("columns:", reader.fieldnames)
    for i, row in enumerate(reader):
        print(row["Sentence"], "|GOLD:", row["Human_ErrorCategories"],
              "|TOOL:", row["AEA_ErrorCategories"])
        if i == 3:
            break

Step 3 — Reshape into the canonical schema

The 23 fine error codes are a lot for one task, so we collapse them into 3 broad families (plus No error). Decisions:

1. Map each code to Grammatical / Lexical / Mechanical (COARSE below).
2. Use the sentence’s first error code as its label; NO_ERROR → No error.
3. Skip sentences whose errors span more than one family (keeps the task single-label).

COARSE = {}
for c in "ART PREP NUM TENSE VFORM WO AGR DET POSS MOD CONJ STRUCT".split():
    COARSE[c] = "Grammatical"
for c in "N ADJ ADV V REF EXPR".split():
    COARSE[c] = "Lexical"
for c in "SP MIS UNN CWS PUNC".split():
    COARSE[c] = "Mechanical"

def to_label(human_field):
    codes = [c.strip() for c in human_field.split(",") if c.strip()]
    if not codes:
        return None
    if codes[0] == "NO_ERROR":
        return "No error"
    families = {COARSE.get(c) for c in codes} - {None}
    return families.pop() if len(families) == 1 else None  # skip mixed sentences

rows = []
with open("data_category.csv", encoding="utf-8-sig", newline="") as f:
    for row in csv.DictReader(f):
        text = (row["Sentence"] or "").strip()
        label = to_label((row["Human_ErrorCategories"] or "").strip())
        if text and label:
            rows.append({"text": text, "label": label})

print("sentences:", len(rows))

Step 4 — Inspect the labels

from collections import Counter
counts = Counter(item["label"] for item in rows)
print("total items:", len(rows))
print("label counts:", dict(counts))
rows[:3]  # peek at the first three reshaped items

Step 5 — Build a balanced gold set

Four classes × 15 = 60 items.

# Build a small BALANCED gold set: an equal number of items per label.
# Balance matters so precision/recall/F1 and the confusion matrix are meaningful.
import random
from collections import defaultdict

PER_LABEL = 15          # how many items per label
random.seed(42)         # fixed seed = same sample every run (reproducible)

by_label = defaultdict(list)
for item in rows:
    by_label[item["label"]].append(item)

gold = []
for label in sorted(by_label):
    bucket = by_label[label]
    random.shuffle(bucket)
    gold.extend(bucket[:PER_LABEL])

random.shuffle(gold)
gold = [{"id": i + 1, "text": x["text"], "label": x["label"]} for i, x in enumerate(gold)]

from collections import Counter
print("items:", len(gold), "| per label:", dict(Counter(x["label"] for x in gold)))

Step 6 — Save it

OUT_FILE = "l2_errors.json"

import json
with open(OUT_FILE, "w", encoding="utf-8") as f:
    json.dump(gold, f, ensure_ascii=False, indent=2)
print(f"Saved {len(gold)} items to {OUT_FILE}")
gold[:3]  # preview the first three items

---

You built l2_errors.json. Because the raw file also has the tool’s predictions, a great mini-project is to compare your LLM vs. the human gold vs. the original tool.