CEFR-SP — download & preprocess

The on-ramp dataset: sentence proficiency level (A1–C2)

What it is. Single English sentences, each labeled with its CEFR level by language-education professionals. We use the openly-shipped Wiki-Auto portion.

Difficulty of the labeling judgment: ★☆☆ — easy. Levels are concrete and annotators usually agree.

License: CC BY-SA 3.0 (Wiki-Auto portion)
Cite: Arase, Uchida & Kajiwara (2022), EMNLP. github.com/yukiar/CEFR-SP

---

Every dataset in this course is reshaped into the same canonical schema so one notebook works for all of them:

[{"id": 1, "text": "...", "label": "..."}]

The raw data, though, looks different every time. That difference is the lesson — half of building a gold standard is getting messy real data into a clean, consistent shape.

Step 1 — Download the raw data

The corpus lives in a GitHub repository, so we just clone it. (! runs a shell command from inside the notebook.)

!git clone --depth 1 https://github.com/yukiar/CEFR-SP

Step 2 — Look at the raw format

The Wiki-Auto files are tab-separated text, one sentence per line:

sentence <TAB> label_by_annotator_A <TAB> label_by_annotator_B

Labels are numbers: 1=A1, 2=A2, … 6=C2. Let’s print the first few raw lines.

raw_path = "CEFR-SP/CEFR-SP/Wiki-Auto/CEFR-SP_Wikiauto_dev.txt"
with open(raw_path, encoding="utf-8") as f:
    sample = [next(f) for _ in range(5)]
for line in sample:
    print(repr(line))

Step 3 — Reshape into the canonical schema

Three decisions, each a real gold-standard-building choice:

1. Trust only agreement. We keep a sentence only when both annotators gave the same level — so every label is unambiguous (ideal for a first task).
2. Make labels human-readable. Convert 1→A1, …, 6→C2.
3. Fit the schema. Output {id, text, label}.

import glob

CEFR = {"1": "A1", "2": "A2", "3": "B1", "4": "B2", "5": "C1", "6": "C2"}

rows = []
for path in glob.glob("CEFR-SP/CEFR-SP/Wiki-Auto/*.txt"):
    with open(path, encoding="utf-8") as f:
        for line in f:
            parts = line.rstrip("\n").split("\t")
            if len(parts) < 3:
                continue
            text, a, b = parts[0].strip(), parts[1].strip(), parts[2].strip()
            if text and a == b and a in CEFR:      # keep only agreed labels
                rows.append({"text": text, "label": CEFR[a]})

print("kept", len(rows), "agreed sentences")

Step 4 — Inspect the labels

from collections import Counter
counts = Counter(item["label"] for item in rows)
print("total items:", len(rows))
print("label counts:", dict(counts))
rows[:3]  # peek at the first three reshaped items

Step 5 — Build a balanced gold set

# Build a small BALANCED gold set: an equal number of items per label.
# Balance matters so precision/recall/F1 and the confusion matrix are meaningful.
import random
from collections import defaultdict

PER_LABEL = 12          # how many items per label
random.seed(42)         # fixed seed = same sample every run (reproducible)

by_label = defaultdict(list)
for item in rows:
    by_label[item["label"]].append(item)

gold = []
for label in sorted(by_label):
    bucket = by_label[label]
    random.shuffle(bucket)
    gold.extend(bucket[:PER_LABEL])

random.shuffle(gold)
gold = [{"id": i + 1, "text": x["text"], "label": x["label"]} for i, x in enumerate(gold)]

from collections import Counter
print("items:", len(gold), "| per label:", dict(Counter(x["label"] for x in gold)))

Step 6 — Save it

OUT_FILE = "cefr_sentences.json"

import json
with open(OUT_FILE, "w", encoding="utf-8") as f:
    json.dump(gold, f, ensure_ascii=False, indent=2)
print(f"Saved {len(gold)} items to {OUT_FILE}")
gold[:3]  # preview the first three items

---

Done! You built cefr_sentences.json from raw research data. Next: use it in the Day 2 tutorial (annotation → evaluation).