CEFR-SP — download & preprocess

The on-ramp dataset: sentence proficiency level (A1–C2)

What it is. Single English sentences, each labeled with its CEFR level by language-education professionals. We use the openly-shipped Wiki-Auto portion.

Difficulty of the labeling judgment: ★☆☆ — easy. Levels are concrete and annotators usually agree.

License: CC BY-SA 3.0 (Wiki-Auto portion)
Cite: Arase, Uchida & Kajiwara (2022), EMNLP. github.com/yukiar/CEFR-SP

Every dataset in this course is reshaped into the same canonical schema so one notebook works for all of them:

[{"id": 1, "text": "...", "label": "..."}]

The raw data, though, looks different every time. That difference is the lesson — half of building a gold standard is getting messy real data into a clean, consistent shape.

Step 1 — Download the raw data

The corpus lives in a GitHub repository, so we just clone it. (! runs a shell command from inside the notebook.)

Show code 
!git clone --depth 1 https://github.com/yukiar/CEFR-SP

Step 2 — Look at the raw format

The Wiki-Auto files are tab-separated text, one sentence per line:

sentence <TAB> label_by_annotator_A <TAB> label_by_annotator_B

Labels are numbers: 1=A1, 2=A2, … 6=C2. Let’s print the first few raw lines.

Show code 
raw_path = "CEFR-SP/CEFR-SP/Wiki-Auto/CEFR-SP_Wikiauto_dev.txt"
with open(raw_path, encoding="utf-8") as f:
    sample = [next(f) for _ in range(5)]   # just the first 5 lines
for line in sample:
    print(repr(line))   # repr() makes the tabs and newlines visible

Step 3 — Reshape into the canonical schema

Three decisions, each a real gold-standard-building choice:

  1. Trust only agreement. We keep a sentence only when both annotators gave the same level — so every label is unambiguous (ideal for a first task).
  2. Make labels human-readable. Convert 1→A1, …, 6→C2.
  3. Fit the schema. Output {id, text, label}.
Show code 
import glob

### Step 3.1: translate the numeric codes into readable CEFR levels ###
CEFR = {"1": "A1", "2": "A2", "3": "B1", "4": "B2", "5": "C1", "6": "C2"}

### Step 3.2: read every file line by line, keeping only what we can trust ###
rows = []
for path in glob.glob("CEFR-SP/CEFR-SP/Wiki-Auto/*.txt"):   # every .txt in there
    with open(path, encoding="utf-8") as f:
        for line in f:
            parts = line.rstrip("\n").split("\t")   # one line -> its 3 columns
            if len(parts) < 3:
                continue                       # a short line — skip it
            text, a, b = parts[0].strip(), parts[1].strip(), parts[2].strip()
            if text and a == b and a in CEFR:      # keep only agreed labels
                rows.append({"text": text, "label": CEFR[a]})

print("kept", len(rows), "agreed sentences")   # the rest were disagreements

Step 4 — Inspect the labels

Show code 
from collections import Counter
counts = Counter(item["label"] for item in rows)   # tally how often each label appears
print("total items:", len(rows))
print("label counts:", dict(counts))   # watch for labels with very few items
rows[:3]  # peek at the first three reshaped items

Step 5 — Build a balanced gold set

Show code 
# Build a small BALANCED gold set: an equal number of items per label.
# Balance matters so precision/recall/F1 and the confusion matrix are meaningful.
import random
from collections import defaultdict

### Step 5.1: the two settings you control ###
PER_LABEL = 12          # how many items per label
random.seed(42)         # fixed seed = same sample every run (reproducible)

### Step 5.2: sort every reshaped row into a bucket named after its label ###
by_label = defaultdict(list)   # a dict that starts each new key at []
for item in rows:
    by_label[item["label"]].append(item)   # drop it in its label's bucket

### Step 5.3: shuffle each bucket, then take the first PER_LABEL from it ###
gold = []
for label in sorted(by_label):     # sorted() = same label order every run
    bucket = by_label[label]
    random.shuffle(bucket)         # mix, so we don't just take the first ones found
    gold.extend(bucket[:PER_LABEL])   # a rare label simply gives fewer

### Step 5.4: mix the labels together, renumber the ids from 1, and report ###
random.shuffle(gold)               # so the labels aren't grouped in blocks
gold = [{"id": i + 1, "text": x["text"], "label": x["label"]} for i, x in enumerate(gold)]

from collections import Counter
print("items:", len(gold), "| per label:", dict(Counter(x["label"] for x in gold)))

Step 6 — Save it

Show code 
OUT_FILE = "cefr_sentences.json"
Show code 
import json
with open(OUT_FILE, "w", encoding="utf-8") as f:   # open for writing
    json.dump(gold, f, ensure_ascii=False, indent=2)   # list of dicts -> JSON text
print(f"Saved {len(gold)} items to {OUT_FILE}")
gold[:3]  # preview the first three items

Step 7 — Also save the full pool

Step 5 threw most of the data away on purpose: 12 items per level is the right size for a tutorial you hand-check. But the Day 3 prompt-tuning work and the final mini-project both want the whole thing — a larger set to draw fresh samples from, and items to use as few-shot examples that are not in your gold set.

So save every agreed sentence too, in the same schema. Same rows as Step 3, just numbered from 1.

Show code 
POOL_FILE = "cefr_pool.json"

### Step 7.1: give every reshaped row an id, in order ###
pool = [{"id": i + 1, "text": x["text"], "label": x["label"]}
        for i, x in enumerate(rows)]

### Step 7.2: write it out ###
with open(POOL_FILE, "w", encoding="utf-8") as f:
    json.dump(pool, f, ensure_ascii=False, indent=2)

from collections import Counter
print(f"Saved {len(pool)} items to {POOL_FILE}")
print("per label:", dict(Counter(x["label"] for x in pool)))

Note how unbalanced the pool is compared with your gold set — B1 and B2 dominate, A1 and C2 are scarce. That is what the real data looks like, and it is why Step 5 sampled evenly instead of just taking the first 72 rows.

Done! You built two files from raw research data:

  • cefr_sentences.json — the balanced 72-item gold set
  • cefr_pool.json — every agreed sentence (~3,200), to sample from later

⚠️ Both live in the Colab session and disappear when the runtime disconnects — save them to your Google Drive before you close the tab.

Next: use them in the Day 2 tutorial (annotation → evaluation) and the Day 3 tutorial (prompt design).