Day 2 · S6 — Evaluation metrics

Day 2 — Linguistic Data Analysis II

Day 2 has two notebooks — S5 builds a gold standard by hand (day2-s5_gold_standard_construction.ipynb), S6 measures a model against one (this one). Submit both at the end of the day.

How to use this notebook

It has two parts:

Part A · Corpus Lab — build the metrics yourself on one yes/no question: TP/FP/FN/TN → confusion matrix → precision, recall, F1 → Cohen’s κ.
Part B · Tutorial — the same job with scikit-learn on the real six-level task, plus error analysis.

You only edit the cells marked ✏️ YOU EDIT. Run the 🔧 Library cells and leave them alone.

➡️ Work top to bottom. When you’re done, Runtime → Run all, then File → Download → Download .ipynb and submit that file.

Setup — run this first

Both parts run on the same data: the CEFR-SP gold set (72 sentences, 12 per level) and one model’s answer for each of them.

Today runs on frozen predictions — no API key, no live model

On Day 1 the live model’s answers changed from run to run, which would get in the way while you are learning to measure quality. So today’s predictions are pre-computed and committed to a file. Everyone’s precision, recall, F1 and κ come out identical every run: if your number differs from the slide, you have a bug, not a different model.

Show code

#@title 📦 Setup — run me first { display-mode: "form" }
# Helper — you don't need to read this. Run it and move on.
# Generated from _notebook_lib.py — edit there, not here; changes to this cell are replaced.
import json, urllib.request
from sklearn.metrics import (classification_report, confusion_matrix,
                             cohen_kappa_score)
import pandas as pd, seaborn as sns, matplotlib.pyplot as plt

# CEFR-SP gold set (72 sentences, 12 per level), fetched from the course repo.
GOLD_URL = "https://raw.githubusercontent.com/egumasa/linguistic-data-analysis-II-2026/main/sources/resources/datasets/gold/cefr_sentences.json"
LEVELS = ["A1", "A2", "B1", "B2", "C1", "C2"]
PREDICTIONS_URL = "https://raw.githubusercontent.com/egumasa/linguistic-data-analysis-II-2026/main/sources/resources/datasets/gold/predictions_day2.json"   # frozen model predictions

print("Setup done. scikit-learn ready.")

Show code

#@title 🔧 Library cell: load_gold, load_predictions { display-mode: "form" }
# Helper — you don't need to read this. Run it and move on.
# Generated from _notebook_lib.py — edit there, not here; changes to this cell are replaced.
#   load_gold(url_or_path) → gold
#   load_predictions(url_or_path) → predictions

def load_gold(url_or_path: str) -> list[dict[str, str]]:
    """Read the canonical gold JSON: [{'id','text','label'}, ...].

    Args:
        url_or_path: a web address, or the path to a file on this machine.

    Returns:
        The gold items, each a dict with "id", "text" and "label".

    Example:
        >>> gold = load_gold(GOLD_URL)
    """
    if str(url_or_path).startswith("http"):                 # a web address?
        raw = urllib.request.urlopen(url_or_path).read().decode("utf-8")  # download it
        gold = json.loads(raw)                              # JSON text -> list of dicts
    else:                                                   # otherwise a file on disk
        gold = json.loads(open(url_or_path, encoding="utf-8").read())
    print(f"Loaded {len(gold)} items. First one:", gold[0])  # proof it worked
    return gold


def load_predictions(url_or_path: str) -> list[str]:
    """Read a frozen predictions list — a committed URL or a local path.

    Args:
        url_or_path: a web address, or the path to a file on this machine.

    Returns:
        One predicted label per gold item, in gold order.

    Example:
        >>> predictions = load_predictions(PREDICTIONS_URL)
    """
    if str(url_or_path).startswith("http"):                 # a web address?
        raw = urllib.request.urlopen(url_or_path).read().decode("utf-8")  # download it
        predictions = json.loads(raw)                       # JSON text -> list
    else:                                                   # otherwise a file on disk
        predictions = json.loads(open(url_or_path, encoding="utf-8").read())
    print(f"Loaded {len(predictions)} frozen predictions.")
    return predictions

Load the data — the pipeline’s first step

Every evaluation starts by loading the two things it will compare: the gold standard and the model’s predictions. This is a step in its own right, not part of the setup — the confusion matrix you build later is made from exactly what you load here, nothing typed in by hand.

First, what these files are and how reading one works; then the two loads.

What a gold file actually is

A gold standard is stored as JSON text — the same {"id", "text", "label"} records you met on Day 1, written to a file. json.loads(...) turns that text into a Python list of dicts you can index into.

Show code

raw = '[{"id": 1, "text": "Hello.", "label": "A1"}, {"id": 2, "text": "Nevertheless, the findings were inconclusive.", "label": "C1"}]'
items = json.loads(raw)               # JSON text → list of dicts
print("number of items:", len(items))    # two records in that text
print("first record:", items[0])         # the whole first dict
print("its label:", items[0]["label"])   # index: list by position, dict by key

Files are read and written with with open(...) as f: — it opens the file, gives it to you as f, and closes it when the block ends. The 🔧 Library cells do this for you, but you will see it around, so here it is once:

Show code

with open("example_gold.json", "w", encoding="utf-8") as f:   # write
    json.dump(items, f)              # the list of dicts becomes JSON text on disk

with open("example_gold.json", encoding="utf-8") as f:        # read back
    reloaded = json.loads(f.read())  # ...and JSON text becomes a list again
print("read back", len(reloaded), "records — same shape:", reloaded[0])

Now load the real gold and predictions

load_gold(...) does the read you just saw, but from a URL. Every dataset this week has the same shape, {"id", "text", "label"}.

Below the gold is the prompt we sent the model, where {text} is the slot each sentence drops into. We ran it once over the gold set and committed the answers, so today you load that frozen file rather than call the model.

Show code

gold = load_gold(GOLD_URL)

# The prompt used to produce the frozen predictions (shown for reference — not run today):
PROMPT = """You are an expert rater of English sentence difficulty using the CEFR scale.
Classify the sentence into exactly one of: A1, A2, B1, B2, C1, C2.
Answer with the level only.

Sentence: {text}"""

# Load the pre-computed predictions (same order as `gold`):
predictions = load_predictions(PREDICTIONS_URL)

Part A · Corpus Lab — build the metrics yourself

In S5 calc_cohen_kappa() printed Cohen’s κ for you. Here you build precision, recall, F1 and κ yourself, before Part B runs the same job with the scikit-learn functions themselves.

Work in ten small steps. Steps 1–7 build the confusion matrix, and you can run them as they are. In steps 8 and 9 you write the formulas: each metric arrives with its lookups and its zero-guard already there, and a None where the arithmetic goes. Replace every None marked ✏️. Step 10 checks all four against scikit-learn. No imports — just for, if, and dictionaries.

If you get stuck on a formula

The maths is on the slide above each step, and every formula is one line. If a cell errors, read the line it names — the fix is almost always a + where a / belongs, or a count you have not looked up yet.

Step 1 · Collapse to one yes/no question

CEFR’s six levels mean 36 confusion-matrix cells — too many to learn on. So Part A asks one thing:

Is this sentence advanced — C1 or C2?

"yes" (C1 or C2) is our positive class; everything else is "no". Precision and recall are always about the positive class, so choosing it is a decision you state out loud.

The 12 rows below are built from the gold and predictions you just loaded — positions 17 to 28 — with each six-level label collapsed to yes/no. Nothing is typed in by hand; this is the pipeline that feeds the confusion matrix:

Show code

ADVANCED = ["C1", "C2"]      # the levels that count as "yes"

# Take twelve of the 72 you loaded (ids 17–28) and their predictions. Slicing keeps
# positions 16–27; predictions[16:28] are the answers for those same sentences.
gold_12 = gold[16:28]
pred_12 = predictions[16:28]

# Pair each gold sentence with its prediction, collapsing both to the yes/no question:
items = []
i = 0                          # position within the twelve, counted up by hand
for g in gold_12:              # g is one gold record: {"id", "text", "label"}
    p = pred_12[i]             # the model's answer for that same sentence
    if g["label"] in ADVANCED:   # collapse the gold level to yes/no
        gold_answer = "yes"
    else:
        gold_answer = "no"
    if p in ADVANCED:            # collapse the model's answer the same way
        pred_answer = "yes"
    else:
        pred_answer = "no"
    items.append({"id": g["id"], "text": g["text"],
                  "gold": gold_answer, "pred": pred_answer})
    i = i + 1                  # move to the next of the twelve

print(len(items), "items — id, gold, pred:")
for it in items:                # the twelve, to count by hand with your partner
    print(it["id"], it["gold"], it["pred"])

Before you write any code, count the four outcomes by hand — with your partner, off the twelve rows you just printed. How many TP (gold yes, model yes), FP (gold no, model yes), FN (gold yes, model no), TN (gold no, model no)?

Check your count (open after you’ve counted)

	model yes	model no
gold yes	TP = 3	FN = 2
gold no	FP = 1	TN = 6

TP: rows 18, 21, 27 · FN: rows 22, 24 (real C1s the model missed) · FP: row 28 (a B2 it called advanced) · TN: the other six.

Everything else in Part A is arithmetic on these four numbers.

Step 2 · Ask one item

A metric never looks at more than one item and two labels at a time. Start there.

Show code

item = items[0]                     # the first of the 12 rows
print(item["gold"], item["pred"])   # its gold answer, then the model's

Row 17: the gold says not advanced, and the model agrees.

Now ask the first of the four questions — is this a true positive? That means both labels are "yes":

Show code

# `and` means BOTH sides have to be true:
if item["gold"] == "yes" and item["pred"] == "yes":
    print("TP")     # only runs when both are "yes"

Nothing printed. That isn’t a bug — it’s the honest answer: row 17 is not a TP, and an if with no else stays silent when its condition is false.

One if can only ever answer one of the four questions. We need all four.

Step 3 · The four branches

Four cells in the table → four branches, in the same order. elif means “only if none of the above matched”, so the branches are checked top to bottom and exactly one runs — every item lands in exactly one cell. The final else needs no condition: if it isn’t TP, FP or FN, it can only be TN.

✏️ Change the index and re-run to see each branch fire.

Show code

item = items[0]      # ✏️ try 1 (a TP), 5 (an FN), 11 (an FP)
print(item["gold"], item["pred"])

# checked top to bottom; the FIRST match wins, so exactly one branch runs:
if item["gold"] == None and item["pred"] == None:
    print("TP")      # advanced, and the model agreed
elif item["gold"] == None and item["pred"] == None:
    print("FP")      # a false alarm
elif item["gold"] == None and item["pred"] == None:
    print("FN")      # a real one, missed
else:
    print("TN")      # not advanced, left alone

Step 4 · Make it a function

You are about to ask that same question of all 12 rows, so name it once. Two changes from step 3: it is wrapped in def, and every print became a return.

print puts a value on the screen and then it is gone; return hands the value back to whoever called the function, so you can keep it, store it, count it — which is what step 5 needs.

Show code

def outcome(gold_label, pred_label):
    """Which of the four cells does this one item land in?"""
    if gold_label == "yes" and pred_label == "yes":
        return "TP"      # `return` hands the answer back, instead of printing it
    elif gold_label == "no" and pred_label == "yes":
        return "FP"
    elif gold_label == "yes" and pred_label == "no":
        return "FN"
    else:
        return "TN"


print(outcome("yes", "no"))     # gold said advanced, model said no → a miss

Test it by hand before you trust it

outcome("yes", "no") → FN ✓. Try all four combinations against the table in step 1. A function you haven’t checked is a guess.

Step 5 · Loop, and store every decision

Run that function over all 12 rows and keep the answers — the same “build a list in a loop” pattern you wrote on Day 1.

Show code

decisions = []                     # start empty; the loop fills it
for item in items:                 # one pass per row of the table
    decisions.append(____)   # add its verdict

print(decisions)                   # 12 verdicts, in the table's order

Twelve items in, twelve verdicts out, in the same order as the table. Position 6 is 'FN' — row 22, the racing-bicycle sentence the model missed.

We could have counted as we went, but a metric is only a summary and decisions is the thing being summarised. Every 'FN' and 'FP' in it points at a specific sentence you can read and argue about.

Step 6 · Tally the verdicts

This is Day 1’s count_labels exercise, run on a new list. tally.get(d, 0) means “how many so far — and if you’ve never seen this one, start from 0”; without the .get, the very first 'TN' would crash, because tally["TN"] doesn’t exist yet.

Show code

tally = {}                         # a dict: verdict -> how many times
for d in decisions:                # d is "TP", "FP", "FN" or "TN"
    tally[d] = tally.get(d, 0) + 1   # count so far (0 if new), plus one

print(tally)

Compare with what you counted by hand in step 1: TP = 3, FP = 1, FN = 2, TN = 6. They match — your code and your eyes agree.

Step 7 · The confusion matrix

A confusion matrix is not a new calculation. It is those same four numbers put in a square, so you can see where the errors went.

show_2x2 only prints — the arithmetic was all yours.

Show code

#@title 🔧 Library cell: show_2x2 { display-mode: "form" }
# Helper — you don't need to read this. Run it and move on.
# Generated from _notebook_lib.py — edit there, not here; changes to this cell are replaced.
#   show_2x2(tally) → the four counts as a square

def show_2x2(tally: dict[str, int]) -> None:
    """Print a tally of TP/FP/FN/TN as a confusion matrix — rows are the gold
    label, columns are the prediction. No arithmetic: the same four numbers,
    arranged so you can see where the errors went.

    Args:
        tally: how many items fell into each outcome, e.g. {"TP": 3, "FP": 1}.
            A missing outcome counts as 0.

    Returns:
        Nothing. It prints the square.

    Example:
        >>> show_2x2(tally)
    """
    # .get(..., 0) so a missing outcome shows as 0 rather than crashing:
    tp = tally.get("TP", 0)
    fp = tally.get("FP", 0)
    fn = tally.get("FN", 0)
    tn = tally.get("TN", 0)
    # :<9 pads a label to 9 characters, :>9 right-aligns a number in 9 — that is all
    # the f-strings below are doing: lining the four counts up into a square.
    print(f"{'':<9}{'pred yes':>9}{'pred no':>9}")     # column headings
    print(f"{'gold yes':<9}{tp:>9}{fn:>9}")              # top row:    TP  FN
    print(f"{'gold no':<9}{fp:>9}{tn:>9}")               # bottom row: FP  TN

Show code

show_2x2(tally)

The diagonal (3 and 6) is everything the model got right — 9 of 12.
The off-diagonal (2 and 1) is everything it got wrong, split by direction: two misses and one false alarm.

Now the margins — you need them for κ in step 9:

	model yes	model no	total
gold yes	3	2	5
gold no	1	6	7
total	4	8	12

The gold set calls 5 sentences advanced; the model calls only 4. It uses the positive label slightly less often than it should — that pattern comes back in Part B.

Step 8 · Precision, recall, F1

Precision — of everything the model CALLED advanced, how much really was?

\[P = \frac{TP}{TP + FP}\]

Numbers first, so you can check it on paper — that one is filled in for you. Then ✏️ write the same thing read off the tally, so it survives a change of data. Both lines should print the same number.

Show code

print(3 / (3 + 1))    # TP / (TP + FP), by hand — done for you
print(None)           # ✏️ the same, read off `tally` instead of typed in

Of the 4 sentences the model called advanced, 3 really were. Precision = 0.75.

Now as a function, with a guard: if a model never predicts the positive class, TP + FP is 0 and Python raises ZeroDivisionError. Returning 0.0 says it earned no credit, which is the honest reading. Every metric you write today gets this guard.

✏️ The lookups and the guard are given. The formula is yours — replace None. You are aiming for 0.75, the number you just computed by hand.

Show code

def precision(tally):
    """Of everything CALLED advanced, how much really was?  TP / (TP + FP)"""
    tp = tally.get("TP", 0)      # .get(..., 0) so a missing count reads as 0
    fp = tally.get("FP", 0)
    if tp + fp == 0:             # the model never said "yes" at all
        return 0.0               # no credit earned — and no division by zero
    return None                  # ✏️ everything it got right, over everything it claimed


print(round(precision(tally), 3))   # round to 3 decimal places

Recall asks the other question — of everything that TRULY was advanced, how much did we catch?

\[R = \frac{TP}{TP + FN}\]

✏️ Same shape as precision, with one real change: FN instead of FP.

Show code

def recall(tally):
    """Of everything that TRULY was advanced, how much did we find?"""
    tp = tally.get("TP", 0)
    fn = tally.get("FN", 0)       # ← the only real change
    if tp + fn == 0:              # nothing was truly advanced — nothing to find
        return 0.0
    return None                   # ✏️ what it found, over everything there was to find


print(round(recall(tally), 3))

Three of the five genuinely advanced sentences were found. Recall = 0.60.

Precision and recall pull against each other: flag everything and recall hits 1.0 while precision collapses. F1 is their harmonic mean, so a high score cannot cover for a low one:

\[F_1 = 2 \cdot \frac{P \cdot R}{P + R}\]

✏️ p and r come from the two functions you just wrote. Write the harmonic mean of them.

Show code

def f1(tally):
    """Harmonic mean of precision and recall."""
    p = precision(tally)         # reuse the function you just wrote
    r = recall(tally)            # ...and the other one
    if p + r == 0:               # both zero — nothing to average
        return 0.0
    return None                  # ✏️ harmonic mean: a low score drags it down


print(round(precision(tally), 3), round(recall(tally), 3), round(f1(tally), 3))

The gap between P and R describes the model

Precision 0.75 > recall 0.60 → this model is conservative: when it commits it is usually right, but it labels real C1s as B2.

Which error can you live with? is a research design question. Screening for a C1 reading list, a miss is expensive — favour recall. Claiming which sentences are advanced, a false alarm is expensive — favour precision.

Step 9 · Cohen’s κ

Start with the easy number — observed agreement, the diagonal over the total. ✏️ n is given; write p_o. You should get 0.75, which is 9 of 12.

Show code

n = tally["TP"] + tally["FP"] + tally["FN"] + tally["TN"]   # all 12 items
p_o = None    # ✏️ the diagonal (both agreed: TP + TN), over the total
print(round(p_o, 3))

Gold and model agree on 9 of 12. But a model that answered “no” to everything would score 7/12 while knowing nothing — raw agreement flatters a rater who just uses the commonest label.

So subtract the agreement you would get by luck. \(p_e\) multiplies the two raters’ own rates, label by label — the row and column totals from step 7:

✏️ Three lines to write. Each rate is a row or column total over n.

Show code

# each line: how often GOLD says it × how often the MODEL says it = agreement by luck
p_yes = None    # ✏️ (gold says yes) × (model says yes)
p_no  = None    # ✏️ (gold says no)  × (model says no)
p_e   = None    # ✏️ luck on "yes" plus luck on "no"
print(round(p_yes, 3), round(p_no, 3), round(p_e, 3))

Check what you got against the margins:

p_yes: gold says yes 5/12 × model says yes 4/12 = 0.139
p_no: gold says no 7/12 × model says no 8/12 = 0.389

More than half the agreement we observed (0.75) was available by luck alone. κ asks how much of what luck couldn’t explain the two of them actually achieved:

\[\kappa = \frac{p_o - p_e}{1 - p_e}\]

Show code

# ✏️ what they beat luck by (p_o - p_e), over how much was left to beat (1 - p_e).
#    Round it to 3 decimal places, the way you did for the other metrics.
print(None)

75% agreement → κ = 0.47 — “moderate” (Landis & Koch) or “weak” (McHugh). The same gap S4 showed you (80% → κ ≈ 0.52), now computed by your own code.

Wrap it up so you can reuse it. ✏️ This is the four lines you just wrote, moved inside a function, plus the same guard shape as the other three metrics — so you can copy your own work down into it.

Show code

def kappa(tally):
    """Agreement corrected for chance: (p_o - p_e) / (1 - p_e)."""
    n = tally["TP"] + tally["FP"] + tally["FN"] + tally["TN"]   # every item
    p_o = None      # ✏️ agreement we actually observed
    p_yes = None    # ✏️ (gold says yes) × (model says yes)
    p_no = None     # ✏️ (gold says no)  × (model says no)
    p_e = None      # ✏️ agreement luck alone would give
    if 1 - p_e == 0:                          # luck already explains everything
        return 0.0
    return None                               # ✏️ how much of the rest they achieved


print(round(kappa(tally), 3))

This is yesterday’s function

Nothing in kappa() knows whether the second column came from your partner or from a model. That is why S5’s agreement number and today’s evaluation number are the same statistic: κ measures two label columns, whoever produced them.

Report both numbers

Two numbers on the same twelve items: raw agreement 0.75, κ 0.47. A κ of 0.47 under 75% agreement says something different from a κ of 0.47 under 95% — so report both.

Which κ joins the percentage follows from your design: unordered labels → cohen_kappa_score(a, b); labels on a scale → the weighted one as well; three or more coders → Fleiss’ κ.

Step 10 · Check yourself against scikit-learn

You built a confusion matrix and four metrics by hand. This cell checks them against scikit-learn on the same twelve items: the library builds the same 2×2 and computes the same numbers. Run it and read one line per metric — a ❌ means that formula is wrong, so go back to the cell it names and fix it, then run this again. This is the only place scikit-learn enters Part A — it grades work you already did.

Show code

#@title 🔎 Self-check against scikit-learn — run me { display-mode: "form" }
# Helper — you don't need to read this. Run it and move on.
from sklearn.metrics import precision_score, recall_score, f1_score, confusion_matrix

### Step 1: the same 12 items as two plain lists — the shape sklearn wants ###
y_gold = []
y_pred = []
for item in items:               # the twelve you built and tallied by hand
    y_gold.append(item["gold"])
    y_pred.append(item["pred"])

### Step 2: a small checker — is your number the same as sklearn's? ###
TOL, results = 1e-9, []      # TOL: how close counts as "the same"
def _chk(name: str, got: float, exp: float) -> None:
    """Print whether your number matches sklearn's, and remember the answer."""
    ok = abs(got - exp) < TOL   # compare sizes, not exact bits: floats wobble
    results.append(ok)
    print(("✅" if ok else "❌"), f"{name:<14} yours={got:.6f}  sklearn={exp:.6f}")

### Step 3: first the confusion matrix — do your four counts match? ###
cm = confusion_matrix(y_gold, y_pred, labels=["yes", "no"])   # rows = gold, cols = pred
sk = {"TP": cm[0][0], "FN": cm[0][1],    # name sklearn's four cells the way you did
      "FP": cm[1][0], "TN": cm[1][1]}
counts_ok = all(tally.get(k, 0) == sk[k] for k in ["TP", "FP", "FN", "TN"])
results.append(counts_ok)
yours = f"{tally.get('TP',0)}/{tally.get('FP',0)}/{tally.get('FN',0)}/{tally.get('TN',0)}"
theirs = f"{sk['TP']}/{sk['FP']}/{sk['FN']}/{sk['TN']}"
print(("✅" if counts_ok else "❌"), f"TP/FP/FN/TN     yours={yours}  sklearn={theirs}")

### Step 4: then the four metrics you wrote ###
_chk("precision", precision(tally),
     precision_score(y_gold, y_pred, pos_label="yes", zero_division=0))
_chk("recall", recall(tally),
     recall_score(y_gold, y_pred, pos_label="yes", zero_division=0))
_chk("f1", f1(tally),
     f1_score(y_gold, y_pred, pos_label="yes", zero_division=0))
_chk("cohen_kappa", kappa(tally), cohen_kappa_score(y_gold, y_pred))

### Step 5: one overall verdict ###
print("-" * 47)              # a divider line, 47 dashes long
print(f"All {len(results)} checks passed ✅  — your confusion matrix and metrics match scikit-learn."
      if all(results) else
      f"{results.count(False)} of {len(results)} checks FAILED — fix and re-run.")

This is the point of Part A

sklearn is not doing anything you cannot do. It is doing exactly what you just wrote — faster, and for every class at once. Twelve items is enough to learn the mechanics, never enough to judge a model — so in Part B you load the full ordinal CEFR labels and let scikit-learn score all 72. From here on, when a report prints 0.36, you know precisely which counts produced it.

The names you just checked yourself against

These are the real scikit-learn names, and you call them directly from here on:

The call	What it gives you	Where it comes back
`precision_score(y, p, pos_label=…)`	precision for one positive class	anywhere you have a yes/no question
`recall_score(y, p, pos_label=…)`	recall for that class	as above
`f1_score(y, p, average=…)`	one F1 number over all classes	your headline number
`classification_report(y, p, labels=…)`	precision, recall and F1 for every class	your per-class table
`confusion_matrix(y, p, labels=…)`	which classes get mixed up with which	coder vs coder, and gold vs model
`cohen_kappa_score(y, p)`	agreement corrected for chance	coder vs coder, and gold vs model
`cohen_kappa_score(y, p, weights="quadratic")`	the same, with a near miss counting as a smaller error	only when your labels are a scale

y is the gold labels as a plain list; p is the other column — your partner’s labels, or the model’s answers.

Part B · Tutorial — the same job, six classes, with scikit-learn

Drop the yes/no simplification. The real task is the full ordinal CEFR scale: A1 · A2 · B1 · B2 · C1 · C2.

The gold and predictions you loaded at the start already carry the full level. Part A only collapsed them to yes/no; here you use them as they are:

Show code

# The same items, now with their full CEFR level instead of yes/no:
print("full gold label:", gold[0]["label"], " full prediction:", predictions[0])

Show code

#@title 🔧 Library cell: load_your_gold, predictions_for { display-mode: "form" }
# Helper — you don't need to read this. Run it and move on.
# Generated from _notebook_lib.py — edit there, not here; changes to this cell are replaced.
#   load_your_gold(path, fallback_url) → your gold, or the published one
#   predictions_for(gold, published, predictions) → predictions in YOUR order

def load_your_gold(path: str, fallback_url: str) -> list[dict[str, str]]:
    """Load the gold set you saved in S5, and fall back to the published one.

    S5's last step writes your adjudicated labels to your Drive. If that file is
    there, this reads it and you score the model against your own labels. If it is
    not — you did not save it, or Drive is not mounted — you get the published
    CEFR-SP set instead, and the notebook still runs from top to bottom.

    Args:
        path: where S5 saved your gold set, e.g. the Drive path in the cell above.
        fallback_url: the published gold set, used when `path` is not readable.

    Returns:
        The gold items, each {"id", "text", "label"}.

    Example:
        >>> gold = load_your_gold(MY_GOLD_PATH, GOLD_URL)
    """
    try:
        gold = load_gold(path)
        print(f"→ scoring against YOUR gold set ({len(gold)} items).")
        return gold
    except OSError:                      # no such file, or Drive not mounted
        print(f"No file at {path} — falling back to the published gold set.")
        gold = load_gold(fallback_url)
        print(f"→ scoring against the PUBLISHED gold set ({len(gold)} items).")
        return gold


def predictions_for(gold: list[dict[str, str]],
                    published: list[dict[str, str]],
                    predictions: list[str]) -> tuple[list[dict[str, str]], list[str]]:
    """Line the frozen predictions up with YOUR gold items, matching on text.

    The frozen predictions are one label per *published* item, in published order.
    Your own gold set is a sample of those sentences, renumbered from 1, so
    position 3 of yours and position 3 of theirs are two unrelated sentences.
    Matching on the text pairs each of your items with the answer the model gave
    for that same sentence. Items whose text is not in the published set have no
    frozen prediction and are dropped, with a count.

    Args:
        gold: your gold items, from load_your_gold.
        published: the published gold items, from load_gold(GOLD_URL).
        predictions: the frozen predictions, in published order.

    Returns:
        Two lists of the same length: your matched items, and their predictions.

    Example:
        >>> gold, predictions = predictions_for(gold, published, predictions)
    """
    ### Step 1: which prediction belongs to which sentence? ###
    pred_by_text = {}
    for i, item in enumerate(published):
        if i < len(predictions):                  # published and predictions run together
            pred_by_text[str(item["text"])] = predictions[i]

    ### Step 2: keep the items we have a frozen answer for ###
    matched, matched_predictions = [], []
    for item in gold:
        text = str(item["text"])
        if text in pred_by_text:
            matched.append(item)
            matched_predictions.append(pred_by_text[text])

    dropped = len(gold) - len(matched)
    print(f"{len(matched)} of your {len(gold)} items have a frozen prediction.")
    if dropped:
        print(f"  {dropped} dropped — not in the published set, so no answer was frozen for them.")
    return matched, matched_predictions

Which gold set do you score against?

Part A ran on the published CEFR-SP set, so your hand-count and everyone else’s matched. Part B lets you choose.

At the end of S5 you saved your own adjudicated labels to your Drive. Load them here and the model is scored against the boundaries you and your partner argued about — which is what makes the error analysis at the end of this notebook yours rather than a demonstration. If you did not save that file, the cells below fall back to the published set and the notebook still runs from top to bottom.

Show code

# ✏️ Uncomment the mount if you are in Colab and saved a gold set in S5:
# from google.colab import drive; drive.mount("/content/drive")

MY_GOLD_PATH = "/content/drive/MyDrive/my_gold_day2.json"   # where S5 saved it

published = load_gold(GOLD_URL)                  # the published set, always needed below
gold = load_your_gold(MY_GOLD_PATH, GOLD_URL)    # yours if it is there, published if not

Now the predictions. The frozen file holds one answer per published item, in published order — but your gold set is a sample of those sentences, renumbered from 1. Position 3 of yours and position 3 of theirs are two unrelated sentences, so pairing them by position would score the model against the wrong labels.

predictions_for(...) pairs each of your items with the answer the model gave for that same sentence, matching on the text. It is the same reason S5’s compare_to_published matched on text rather than id.

Show code

predictions = load_predictions(PREDICTIONS_URL)   # one answer per published item

# Line them up with whichever gold set you just loaded:
gold, predictions = predictions_for(gold, published, predictions)
print(len(gold), "items to score ·", len(predictions), "predictions")

4 cells → 36. Counting by hand stops being reasonable.
Six classes give you six precisions, six recalls, six F1s.
To score C1, treat C1 as “yes” and the other five as “no” — Part A, run six times. That is one-vs-rest, and it is all classification_report does.

These are the scikit-learn functions from the table above, called by their own names. You built each of them by hand in Part A, so nothing below is new — only faster, and for six classes at once.

Show code

# The gold labels as a plain list — the shape every scikit-learn call wants:
y_true = []
for item in gold:
    y_true.append(item["label"])
y_pred = predictions              # the model's labels, already a plain list

print(classification_report(y_true, y_pred, labels=LEVELS, zero_division=0))

Now the two κ values. cohen_kappa_score corrects agreement for chance, exactly as it did coder-against-coder in S5 — the only change is that one of the two columns is now the model.

weights="quadratic" is the argument that says the labels sit on a scale, so a near miss counts as a smaller error than a far one. Run both and compare:

Show code

print("Cohen's kappa           ", round(cohen_kappa_score(y_true, y_pred), 3))

# weights="quadratic": A1 -> A2 is a smaller error than A1 -> C2.
weighted = cohen_kappa_score(y_true, y_pred, labels=LEVELS, weights="quadratic")
print("Cohen's kappa (weighted)", round(weighted, 3), "  <- labels are ordered")

And the confusion matrix — the same 36 numbers the report is built from, drawn so you can see where the errors went. seaborn does the drawing; you are not asked to write it.

Show code

matrix = confusion_matrix(y_true, y_pred, labels=LEVELS)   # counts per gold/pred pair

plt.figure(figsize=(5.5, 4.5))
sns.heatmap(matrix, annot=True, fmt="d", cmap="Blues",     # annot=True writes the counts
            xticklabels=LEVELS, yticklabels=LEVELS)
plt.xlabel("Predicted"); plt.ylabel("Gold"); plt.title("Confusion matrix")
plt.tight_layout(); plt.show()

Reading the report

Every row is one Part-A run: for C1, TP/FP/FN/TN with C1 as the positive class.
support is how many gold items that class has.
macro avg is the plain average of the six F1s: every class counts equally, however rare.

The figures quoted here and below are the ones the published 72-item set gives. If you loaded your own gold set, your numbers differ — read your own output and make the same kind of observation about it.

On the published set, overall accuracy is about 39% — but look at how it is wrong. Roughly 97% of its answers are within one level of the gold label. It has the right idea and imprecise thresholds, which no accuracy figure can tell apart from not understanding the task.

One number, three different questions

The report prints two other averages beside macro avg. On an uneven label set they disagree:

Show code

from sklearn.metrics import f1_score

# The same predictions, scored three ways.
y_gold = []                        # the gold column, as a plain list
for item in gold:
    y_gold.append(item["label"])

for how in ["macro", "micro", "weighted"]:
    score = f1_score(y_gold, predictions, average=how, zero_division=0)
    print(how, round(score, 3))

macro — every class counts the same, however rare.
micro — every item counts the same, so common classes dominate.
weighted — per-class F1 averaged by how common each class is: between the two.

The three land close together here because this gold set is balanced, 12 per level. On an unbalanced set they can differ by a lot.

Pick the question before you see the answers

Which one you report follows from what you are claiming, and you can settle it before any number exists. A reader cannot detect that you ran all three and reported the highest.

Read the matrix down the columns

Rows are gold, so rows tell you what happened to each true level. Read down the columns instead — how often the model says each level. The published gold set is balanced, 12 per level, so an unbiased rater would use each label about 12 times. (Your own set is smaller and need not be balanced, so work out what an even spread would look like for it before you judge the columns.)

On the published set it says A2 twenty times, A1 four times, and C2 exactly once in 72 chances: everything is squeezed toward the middle of the scale. It is your Part-A finding again — precision ran ahead of recall (0.75 vs 0.60 on your twelve), because it under-uses the top of the scale.

Two κ values, same predictions

On the published set, plain κ = 0.27, because plain κ treats A1 → A2 as exactly as wrong as A1 → C2. Quadratic weighted κ = 0.85, because CEFR levels are ordinal and a near miss should hurt less. Your own gold set will give two different values, and the gap between them will still be large.

Report the one that matches your labels

Same predictions: 0.27 or 0.85, depending on a single argument. Ordered labels → weighted κ. Unordered categories → plain κ. State which you used and why, and say which gold set it came from. (Arase et al. reported weighted κ = .628 on this task.)

Error analysis — the model’s fault, or the scheme’s?

There are 44 misses — too many to read one by one. Skim a dozen, then look at the rows where the gold label is C2 or A1, where this model disagrees most often.

For each miss, ask: is the gold defensible, or is this a genuinely borderline sentence? Would you and your partner have agreed on it? “Is the disagreement the model’s fault or the scheme’s?” is the central question of annotation work.

Show code

rows = []
for g, p in zip(gold, predictions):   # walk gold and predictions side by side
    if g["label"] != p:               # keep only the disagreements
        rows.append({"id": g["id"], "gold": g["label"], "pred": p, "text": g["text"]})
print(len(rows), "of", len(gold), "wrong.")

# Name the columns even when there are no rows. A table built from an empty list has
# no columns at all, and then errors["gold"] fails for a student who got everything right.
errors = pd.DataFrame(rows, columns=["id", "gold", "pred", "text"])
errors.head(15)     # ...or errors[errors["gold"] == "C2"] to see the hard end

Four words for four different findings

“The model got it wrong” covers four situations that call for four different responses. These are the words your final project asks for:

Word	What it means	What you would do about it
`model`	the label is clear, two coders would agree at once, and the model still missed it	nothing — this is the model’s limit
`scheme`	the item is genuinely borderline under your scheme, and you know which ones those are because you argued about them	rewrite the boundary rule
`wording`	the label name misleads. `Gap` may read to a model as “missing data”	one more prompt round could fix this
`ambiguous`	the item itself is unclear in a way no scheme would settle	say so, and move on

scheme and wording are the pair to be careful about: a prompt can reach one of them and not the other.

Now do it, out loud, with your partner. Pick two or three rows from the table above, read the actual sentence, and say which of the four words fits and why.

Give a reason, not a verdict. “model — wrong” is not worth saying; “model — this is about as plainly C1 as a sentence gets, and it said A2” is.

The cross-reference: where did you two disagree?

Here is the join that makes scheme an evidenced claim rather than an impression. You already have a list of items your S5 partner and you labelled differently. If the model’s errors land on those same items, what you have measured is a fuzzy boundary in your scheme — not a stupid model.

This join is only exact if you loaded your own gold set above, because then the ids here are the ids in your S5 sheet. On the published set the ids are the published ones, and the comparison is a demonstration rather than a finding about your scheme.

Type in a few ids from your own disagreements(rows) table in S5 and see:

Show code

# ✏️ ids from YOUR S5 disagreement table — the rows you two argued about.
DISAGREED_IDS = [17, 23, 41]

both = []
for row_id in errors["id"]:          # every item the model got wrong
    if row_id in DISAGREED_IDS:      # ...that you two also disagreed about
        both.append(row_id)

print(len(both), "of", len(errors), "model errors are items you argued about too:", both)

A high overlap says the scheme is the problem. A low one says the model is missing things two humans found easy — a different finding, and just as reportable. In the project this is one call, errors_on_disagreed(errors, disagreed).

The question you ask decides the error you can see

Look back at rows 19, 23 and 25 in step 1 — the three obituary sentences. Under Part A’s yes/no question the model got all three right. Scored on the published set in six classes, it called every one of them A2 instead of A1. Same predictions, same gold; a different question made a different error visible.

✅ Before you submit

Runtime → Run all and check every cell ran without error.
Tutorial outputs are visible (tables / charts / the model’s answers).
Every Corpus Lab self-check prints ✅ (or your TODO answers are filled in).
Part B says which gold set it scored against — your own S5 set, or the published one — and your reported numbers come from that same run.
File → Download → Download .ipynb and upload both of today’s Day-2 notebooks.