Day 2 · S5 — Build a gold standard

Day 2 — Linguistic Data Analysis II

Day 2 has two notebooks — S5 builds a gold standard by hand (this one), S6 measures a model against one (day2-s6_evaluation_metrics.ipynb). Submit both at the end of the day.

How to use this notebook

You only edit the cells marked ✏️ YOU EDIT. Run the 🔧 Library cells and leave them alone.

➡️ Work top to bottom. When you’re done, Runtime → Run all, then File → Download → Download .ipynb and submit that file.

🎯 Learning Objectives

By the end of this session, you will be able to:

Working in a pair, each annotate the same ~20 sentences by hand against a prepared scheme.
Read a function’s first line and its help(...) to work out what to pass in and what comes back.
Import the sheet into Colab and compute percent agreement, Cohen’s κ, and an annotator-vs-annotator confusion matrix.
Read the confusion matrix to see where you disagree, then refine the scheme and re-annotate — iterate until agreement is acceptable.
Adjudicate the remaining disagreements, compare to the published gold, and interpret the differences.
Export your annotations as a canonical {id, text, label} gold set.

Start here — what Arase et al. actually decided

S4 ended on the two decisions the summary table skipped — and they are the two you make today:

Which sentences go into the pool?
Who is allowed to annotate them?

In pairs, with the two excerpts (§3.2 and §3.1): fill the worksheet. You are scanning for specifics — a number, a criterion, a reason — not reading for comprehension. 7 minutes.

Build a gold standard yourself

In this session, we will learn how to create a Gold-Standard dataset.

A Google Sheet — where you and your partner annotate (C) and re-annotate (E).
This notebook — the sample (A) and the numbers (D–F).

Colab runs at step A, where you draw your sample and make your sheet. Steps B–C happen in that sheet, with no code. You come back here at step D. Find your place by the letter.

Today’s goal, as six steps (A–F)

Your goal: build a gold standard for your track, and know how far you can trust it. That is phase ③ (Eguchi & Kyle Step 6) — broken into six steps. The same A–F labels appear on every surface, so you always know where you are: find your place by the letter, not the slide number.

Step	Where you work	What happens
A · Sample	Sheet	concept: how to draw a representative ~20; you copy your track’s sheet
B · Apply the scheme	guidelines	restate the decidable rule you’ll annotate against
C · Annotate blind	Sheet	each partner labels the same items, independently
D · Measure agreement	Colab	% agreement · Cohen’s κ · annotator-vs-annotator matrix
E · Read → refine → re-annotate	Sheet	where do you diverge? fix the scheme, re-label
F · Adjudicate → gold	Colab	resolve, compare to published, export `{id,text,label}`

A · Draw your sample → make your sheet (E&K Step 3 · ①) ✏️ YOU EDIT

You cannot annotate a whole corpus, so you annotate a sample of it. Four things make that sample defensible, and you decide all four here:

Representative — drawn at random from the pool, not picked by hand.
Right-sized — big enough to measure agreement on, small enough to finish today.
Reproducible — a fixed seed, so anyone can draw the same sample again.
One fixed unit — here, one sentence gets one label.

The cells below do this in order: load the pool, fix the seed, draw the sample, then write it to a new Google Sheet with the columns ID · Text · CoderA · CoderB · Final · Note. ID and Text are filled in; the rest is what you and your partner fill by hand in steps C and F.

Show code

#@title 📦 Setup — run me first { display-mode: "form" }
# Helper — you don't need to read this. Run it and move on.
# Generated from _notebook_lib.py — edit there, not here; changes to this cell are replaced.
import json, random, urllib.request
from sklearn.metrics import confusion_matrix
import pandas as pd, seaborn as sns, matplotlib.pyplot as plt

# CEFR-SP gold set — the published labels you compare against in step F.
GOLD_URL = "https://raw.githubusercontent.com/egumasa/linguistic-data-analysis-II-2026/main/sources/resources/datasets/gold/cefr_sentences.json"
LEVELS = ["A1", "A2", "B1", "B2", "C1", "C2"]

print("Setup done. scikit-learn ready.")

Load the pool you will sample from.

Show code

#@title 🔧 Library cell: load_gold { display-mode: "form" }
# Helper — you don't need to read this. Run it and move on.
# Generated from _notebook_lib.py — edit there, not here; changes to this cell are replaced.
#   load_gold(url_or_path) → gold

def load_gold(url_or_path: str) -> list[dict[str, str]]:
    """Read the canonical gold JSON: [{'id','text','label'}, ...].

    Args:
        url_or_path: a web address, or the path to a file on this machine.

    Returns:
        The gold items, each a dict with "id", "text" and "label".

    Example:
        >>> gold = load_gold(GOLD_URL)
    """
    if str(url_or_path).startswith("http"):                 # a web address?
        raw = urllib.request.urlopen(url_or_path).read().decode("utf-8")  # download it
        gold = json.loads(raw)                              # JSON text -> list of dicts
    else:                                                   # otherwise a file on disk
        gold = json.loads(open(url_or_path, encoding="utf-8").read())
    print(f"Loaded {len(gold)} items. First one:", gold[0])  # proof it worked
    return gold

Show code

pool = load_gold(GOLD_URL)      # your track's labelled pool
print(len(pool), "sentences in the pool")

Show code

## Understand the strcuture of the pool
# Print the element of the pool

You should see a count, then three sentences, each with an id, a CEFR label and the text.

The pool already has labels. Yours will not: the sheet you make below gets id and text only, so that in step C you and your partner label without seeing anyone else’s answer.

Random sampling

We can randomly sample the data from the pool.

We import random module.

SEED is a number that fix randomization algorithm.

Show code

import random

SEED = 42                # ✏️ your group's seed — write it in your report

random.seed(SEED)
print(random.sample(pool, 3))   # run this cell twice — the same 3 items both times

Your turn. Make a list of 5 words, then draw 3 of them at random.

Run random.sample on your list without calling random.seed first. Run the cell a few times — the 3 words change each time.
Then call random.seed(SEED) on the line before the draw and run the cell a few times again — the same 3 words every time.

Write down which of the two you would use for a sample you report in a paper, and why.

Show code

words = ["apple", "bridge", "cloud", "desk", "engine"]   # ✏️ your 5 words

# ✏️ 1. draw 3 words without a seed

Show code

# ✏️ 2. draw 3 words with random.seed(SEED) on the line before

Now draw the sample you will actually annotate, using the same seed.

Show code

N_ITEMS = 20             # ✏️ how many sentences you will annotate

random.seed(SEED)        # start from the seed again, so this draw is the reproducible one
sample = random.sample(pool, N_ITEMS)
print(len(sample), "sentences drawn from a pool of", len(pool))

Look at what you drew before you build a sheet out of it.

Show code

for item in sample[:5]:          # the first five, to check the draw looks right
    print(item["id"], "—", item["text"])

Ids and sentences, and no labels — that is what goes into the sheet.

Creating Google Spreadsheet

You can actually create a Google Spreadsheet from colaboratory.

The three 🔧 cells below load the code that does it. Run them in order and read nothing:

connect to Google Sheets — signs you in.
read one tab of your annotation sheet — fixes the six column names ID · Text · CoderA · CoderB · Final · Note, and gives you load_annotation_sheet for step D.
create_annotation_sheet — makes the new spreadsheet and returns its URL.

Then the cell after them creates the sheet. It writes one row per sentence you drew, with ID and Text filled in and the other four columns blank — the pool’s own labels are deliberately left out, so that in step C you and your partner label without seeing them.

Show code

#@title 🔧 Library cell: connect to Google Sheets { display-mode: "form" }
# Helper — you don't need to read this. Run it and move on.
# Generated from _notebook_lib.py — edit there, not here; changes to this cell are replaced.
#   connect to Google Sheets

def _sheets_client():
    """Authorise gspread with your Google account (a pop-up asks for permission).

    Returns:
        A logged-in connection to Google Sheets.

    Raises:
        RuntimeError: when signing in from your own computer fails.
    """
    ### Step 1: in Colab, use the Google account you are already signed in with ###
    try:
        from google.colab import auth
        import google.auth, gspread
        auth.authenticate_user()           # the pop-up: "let Colab use your Sheets"
        creds, _ = google.auth.default()   # the permission slip that pop-up produced
        return gspread.authorize(creds)    # a logged-in connection to Google Sheets
    except ImportError:                    # `google.colab` only exists inside Colab
        pass

    ### Step 2: on your own computer, let gspread do its own sign-in ###
    import gspread
    try:
        return gspread.oauth()
    except Exception as error:
        raise RuntimeError(
            "Could not sign in to Google Sheets from this computer.\n"
            "This step is written for Google Colab, where your Google account is "
            "already available — open the notebook there and it will work with no "
            "setup.\n"
            "To run it here instead, gspread needs a credentials file first: "
            "https://docs.gspread.org/en/latest/oauth2.html\n"
            f"The error was: {error}") from error

Show code

#@title 🔧 Library cell: read one tab of your annotation sheet { display-mode: "form" }
# Helper — you don't need to read this. Run it and move on.
# Generated from _notebook_lib.py — edit there, not here; changes to this cell are replaced.
#   read one tab of your annotation sheet

# Sheet column headers (the annotation template uses these exact names):
COL_ID, COL_TEXT = "ID", "Text"
COL_A, COL_B = "CoderA", "CoderB"
COL_FINAL, COL_NOTES = "Final", "Note"
ANNOTATION_HEADER = [COL_ID, COL_TEXT, COL_A, COL_B, COL_FINAL, COL_NOTES]

def load_annotation_sheet(sheet_id: str,
                          worksheet: str = "round1") -> list[dict[str, str]]:
    """Read one TAB of your annotation sheet back as a list of row dicts.

    Opening by id or URL always opens the exact sheet, so two copies that share a
    name (\"Copy of ...\") are never confused. Each round lives in its own tab, so
    re-annotating in round2 never overwrites round1.

    Args:
        sheet_id: the long id in the sheet's URL
            (docs.google.com/spreadsheets/d/<THIS PART>/edit). The whole URL works too.
        worksheet: the TAB name — one tab per annotation round.

    Returns:
        One dict per row, keyed by the column headings (ID, Text, CoderA, ...).

    Raises:
        ValueError: when the sheet has no tab by that name. The message lists the
            tabs it does have.

    Example:
        >>> rows = load_annotation_sheet(SHEET_ID, worksheet="round1")
    """
    ### Step 1: open the sheet — a pasted URL and a bare id both work ###
    client = _sheets_client()
    if str(sheet_id).startswith("http"):
        sheet = client.open_by_url(sheet_id)
    else:
        sheet = client.open_by_key(sheet_id)

    ### Step 2: find the tab (the "round") — and say which tabs exist if it is missing ###
    try:
        ws = sheet.worksheet(worksheet)
    except Exception:
        tabs = [w.title for w in sheet.worksheets()]   # what IS in this sheet
        raise ValueError(f"No tab named {worksheet!r}. Tabs in this sheet: {tabs}")

    ### Step 3: read every row as a dict keyed by the header names ###
    rows = ws.get_all_records()        # [{"ID": 1, "Text": "...", "CoderA": "B1", ...}, ...]
    print(f"Read {len(rows)} rows from tab '{worksheet}'.")
    return rows

Show code

#@title 🔧 Library cell: create_annotation_sheet { display-mode: "form" }
# Helper — you don't need to read this. Run it and move on.
# Generated from _notebook_lib.py — edit there, not here; changes to this cell are replaced.
#   create_annotation_sheet(title, items, labels) → url

def create_annotation_sheet(title: str,
                            items: list[dict[str, str]],
                            labels: list[str]) -> str:
    """Create a Sheet in YOUR Drive: one row per item, blank columns to label.

    Any existing label on an item is deliberately NOT copied across, so you
    annotate blind.

    Args:
        title: the name to give the new spreadsheet.
        items: the items to annotate, each with "id" and "text".
        labels: the labels your scheme allows, printed as a reminder.

    Returns:
        The URL of the sheet it created.

    Example:
        >>> url = create_annotation_sheet("Group 1 gold", items, LEVELS)
    """
    ### Step 1: make an empty spreadsheet in your own Drive ###
    sheet = _sheets_client().create(title)
    worksheet = sheet.sheet1
    worksheet.update_title("round1")   # first round lives in the 'round1' tab

    ### Step 2: one row per item — id and text filled in, label columns left blank ###
    rows = []
    for item in items:
        #                id            text          CoderA CoderB Final Note
        rows.append([item["id"], item["text"], "", "", "", ""])

    ### Step 3: write it all in one go, then pin the header row ###
    worksheet.update([ANNOTATION_HEADER] + rows)   # header first, then the data
    worksheet.freeze(rows=1)                       # header stays put as you scroll
    print(f"Created '{title}' with {len(rows)} rows in tab 'round1'.")
    print("Allowed labels:", ", ".join(labels))
    print("Open it:", sheet.url)
    return sheet.url

Run the next cell once

It makes a new spreadsheet in your Drive every time it actually runs, so it is written to make one only if you have not made one yet. Once SHEET_URL exists, re-running the cell just prints it again — so Runtime → Run all at the end of the session will not hand you a second, empty sheet after you have annotated the first.

To deliberately start over, change SHEET_TITLE and run del SHEET_URL in a new cell first.

Show code

SHEET_TITLE = "lda2_day2_cefr"   # Sheet title

if "SHEET_URL" in globals():                   # you have already made your sheet
    print("You already made a sheet:", SHEET_URL)
else:
    SHEET_URL = create_annotation_sheet(SHEET_TITLE, sample, LEVELS)

It prints how many rows it wrote, the labels your scheme allows, and the sheet’s URL. Open that URL and check the CoderA, CoderB and Final columns are empty — you fill those by hand next.

Keep the sheet open. You paste its id into step D.

B · Apply the operationalized scheme (E&K Steps 4–5; Fuoli · ②)

Before you label, restate the decidable rule you’re annotating against — the scheme your team drafted in the earlier sessions — and skim the guidelines and per-level examples. One label per unit; know your label set cold. → interpret this on step B (slides).

CEFR descriptor

Arase instructed the annotator to assign the most approproate CEFR level to the sentence using the descriptor below.

Level	Overall reading comprehension descriptor
C2	Can understand virtually all types of texts including abstract, structurally complex, or highly colloquial literary and non-literary writings.
C1	Can understand in detail lengthy, complex texts, whether or not these relate to their own area of speciality, provided they can reread difficult sections.
B2	Can read with a large degree of independence, adapting style and speed of reading to different texts and purposes. Has a broad active reading vocabulary, but may experience some difficulty with low-frequency idioms.
B1	Can read straightforward factual texts on subjects related to their field of interest with a satisfactory level of comprehension.
A2	Can understand short, simple texts on familiar matters of a concrete type which consist of high frequency everyday or job-related language.
A1	Can understand very short, simple texts a single phrase at a time, picking up familiar names, words and basic phrases and rereading as required.

D · Measure agreement (E&K Step 6 · ③) ✏️ YOU EDIT

Back in Colab. Paste in the id of the sheet you made at step A, read the tab you both annotated, and measure how far apart you were — one step at a time: how often you matched, then the same figure with luck taken out, then which labels you disagreed on.

Show code

#@title 🔧 Library cell: labelled_pairs { display-mode: "form" }
# Helper — you don't need to read this. Run it and move on.
# Generated from _notebook_lib.py — edit there, not here; changes to this cell are replaced.
#   labelled_pairs(rows) → the labels you BOTH chose

def labelled_pairs(rows: list[dict[str, str]],
                   a: str = COL_A,
                   b: str = COL_B) -> tuple[list[str], list[str]]:
    """The two annotators' labels, keeping only the rows BOTH of them labelled.

    A row one of you has not reached yet is not a disagreement, so it is left out
    rather than counted. The three functions below all start by calling this, which
    is why each of them can be run on its own.

    Args:
        rows: the rows read back by load_annotation_sheet.
        a: the column holding the first annotator's labels.
        b: the column holding the second annotator's labels.

    Returns:
        Two lists of the same length: annotator A's labels, annotator B's labels.

    Example:
        >>> a_labels, b_labels = labelled_pairs(rows)
    """
    a_labels = []
    b_labels = []
    for row in rows:
        label_a = str(row[a]).strip()           # .strip() drops the spaces a sheet adds
        label_b = str(row[b]).strip()
        if label_a != "" and label_b != "":     # skip the rows only one of you reached
            a_labels.append(label_a)
            b_labels.append(label_b)
    return a_labels, b_labels

Read your sheet back into Python.

Show code

SHEET_ID = "1AbCdEf...paste_yours"   # ✏️ the id of the sheet you made in step A
                                     #    (…/spreadsheets/d/THIS/edit) — the whole URL works too
ROUND    = "round1"                  # ✏️ which round's tab to analyze

rows = load_annotation_sheet(SHEET_ID, ROUND)   # read that tab back into Python

Understand the data structure

rows is a list of dictionaries — one dictionary per row of your sheet, keyed by the column headings. Print the first one to see the six columns you made in step A.

Show code

print(rows[0])              # the first row, as a dictionary
print(list(rows[0]))        # just the column names

Show code

## extract labels
a_labels, b_labels = labelled_pairs(rows)

Show code

# print a_labels and b_labels separately

Show code

## Iterate over two labels at once
#  `zip(a_labels, b_labels)` walks them side by side, handing you one
#  pair of labels per turn:
#  for a_label, b_label in zip(a_labels, b_labels):

Calculating agreement

Percent agreement is how often the two of you chose the same label, out of the rows you both labelled. Rows only one of you reached are left out — those are not disagreements.

Show code

# ✏️ YOU EDIT — replace the NotImplementedError with your code.

def calc_percentage_agreement(rows, a=COL_A, b=COL_B):
    """How often the two of you chose the same label, out of the rows you both
    labelled. Return the proportion, and print it.

    It counts every match, including the ones two annotators would hit by luck
    alone — which is why it comes out higher than the kappa below it.
    Example: two coders who matched on 3 of the 5 rows they shared -> 0.6
    """
    a_labels, b_labels = labelled_pairs(rows, a, b)   # rows you BOTH labelled
    if len(a_labels) == 0:
        print("No rows where BOTH annotators have labelled. Nothing to compare yet.")
        return None

    # HINT: 
    #       Start `matches` at 0, add one every time the two labels are equal,
    #       then divide by len(a_labels) to get the proportion. Print it with
    #           print(f"{len(a_labels)} doubly-annotated · agreement {percent:.1%}")
    #       and return it.
    
    n_matches = 0

    ## Iterate over both the a_labels and b_labels at the same time.


    percent = None #replace this with the formula
    return percent

Now run it. If you filled the loop in correctly you will see a percentage.

Show code

agreement = calc_percentage_agreement(rows)

Calculating Cohen’s κ

Percent agreement flatters two coders who both lean on the same label: some of those matches are luck. Cohen’s κ subtracts the agreement you would expect by chance, so it is the number to trust — recall S4, where 80% raw agreement was only κ ≈ 0.52.

Show code


def calc_cohen_kappa(rows: list[dict[str, str]],
                     a: str = COL_A,
                     b: str = COL_B) -> float | None:
    """The same comparison, with agreement-by-luck subtracted.

    Two annotators who both lean on the same label agree often without the scheme
    doing any work. Cohen's κ takes that luck out, so it is the number to trust —
    recall S4, where 80% raw agreement was only κ ≈ 0.52.

    Args:
        rows: the rows read back by load_annotation_sheet.
        a: the column holding the first annotator's labels.
        b: the column holding the second annotator's labels.

    Returns:
        Cohen's κ, or None when no row has both annotators filled in.

    Example:
        >>> kappa = calc_cohen_kappa(rows)
    """
    from sklearn.metrics import cohen_kappa_score
    a_labels, b_labels = labelled_pairs(rows, a, b)   # rows you BOTH labelled
    
    if len(a_labels) == 0:
        print("No rows where BOTH annotators have labelled. Nothing to compare yet.")
        return None
    
    kappa = cohen_kappa_score(a_labels, b_labels)
    print(f"{len(a_labels)} doubly-annotated · Cohen's κ {kappa:.3f}")
    return kappa

kappa = calc_cohen_kappa(rows)

Plot confusion matrix

The two numbers say how much you disagreed. This says where. The diagonal is where you agreed; find the off-diagonal cell dragging κ down — that label pair is your worklist for step E.

Show code



def plot_confusion(rows: list[dict[str, str]],
                   a: str = COL_A,
                   b: str = COL_B) -> None:
    """Draw WHICH labels the two of you confuse, not just how often.

    The diagonal is where you agreed; an off-diagonal cell is a label pair whose
    boundary your scheme has not made decidable yet. That cell is your worklist for
    step E. Same kind of picture evaluate() draws for gold against a model.

    Args:
        rows: the rows read back by load_annotation_sheet.
        a: the column holding the first annotator's labels.
        b: the column holding the second annotator's labels.

    Returns:
        Nothing. It shows the matrix.

    Example:
        >>> plot_confusion(rows)
    """
    a_labels, b_labels = labelled_pairs(rows, a, b)   # rows you BOTH labelled
    if len(a_labels) == 0:
        print("No rows where BOTH annotators have labelled. Nothing to compare yet.")
        return None
    labels = sorted(set(a_labels) | set(b_labels))   # every label either of you used
    cm = confusion_matrix(a_labels, b_labels, labels=labels)
    plt.figure(figsize=(5.5, 4.5))
    sns.heatmap(cm, annot=True, fmt="d", cmap="Blues",
                xticklabels=labels, yticklabels=labels)
    plt.xlabel("Annotator B"); plt.ylabel("Annotator A")   # diagonal = you agreed
    plt.title("Annotator-vs-annotator confusion matrix")
    plt.tight_layout(); plt.show()

Now draw it.

Show code


plot_confusion(rows)

Which of those numbers you report is not a free choice

Which of those three belong in your report follows from your design:

Your design	Report
two coders, labels with no order	percent agreement and Cohen’s κ
two coders, labels on a scale	those two, and the weighted κ
three or more coders	percent agreement and Fleiss’ κ, plus Cohen’s κ per pair

Both numbers, not one. Percent agreement alone counts lucky agreement as earned; a κ alone is hard to read without the raw figure beside it.

Settle this before you run anything, so the choice does not depend on which number comes out higher.

E · Read the matrix → refine → re-annotate (E&K Step 6; Fuoli princ. 2 · ③)

A low κ is a diagnosis of your scheme, not just your annotating. The cell below lists every row the two of you saw differently.

What to do with this list

For the label pair the matrix flagged, refine the scheme until the ambiguity becomes decidable: add a rule, a boundary case, an example. Then re-annotate in a fresh round tab (below) and re-run step D to see κ move.

Let’s reveal the specific data you disagree with each other.

To do so, you will iterate the record, check if they disagree. If yes, append in the to_argue_about. If no, pass.

Show code

rows[0]

Show code

### The rule: a row is a disagreement when the two of you chose DIFFERENT labels ###
to_argue_about = []

### Here iterate rows and append row if two coders disagree.


print(len(to_argue_about), "rows to adjudicate")
pd.DataFrame(to_argue_about)     # the same table the helper printed

Two things in there are choices, not facts.

A blank cell is skipped, not counted as a disagreement. A row one of you has not reached yet is not two people disagreeing.

a != b is the obvious rule, not the only defensible one. If your labels sit on a scale (A1 < A2 < … < C2), you might count only a gap of two or more as worth an argument. Whichever you use, your report has to say which.

Re-annotate in a fresh round tab, then re-run step D

Don’t overwrite round1. In the Sheet, right-click the round1 tab → Duplicate, rename the copy round2, and re-label the confused items there. Then set ROUND = "round2" in step D and re-run it. Repeat (round3, …) until κ is acceptable, then move to step F.

F · Adjudicate → gold (E&K Step 6 → feeds ④⑤) ✏️ YOU EDIT

The last disagreements don’t refine away — you decide them. In your latest round tab, fill a single Final label for every row. Where you already agreed, Final is that agreed label. Then read it back and convert it to canonical form.

First — looking up what a helper expects

to_canonical is the first helper you pass more than one thing to. Two ways to find out what it wants.

1 — The first line of the function.

def to_canonical(rows: list[dict[str, str]],
                 labels: list[str],
                 column: str = COL_FINAL) -> list[dict[str, str]]:

Before each colon: what the argument is called.
After each colon: the kind of data it expects.
= COL_FINAL means that one already has a value, so you can leave it out.
After the ->: what you get back.

2 — help(to_canonical), or Shift+Tab after typing to_canonical(.

You read these; you never write them.

Show code

#@title 🔧 Library cell: to_canonical { display-mode: "form" }
# Helper — you don't need to read this. Run it and move on.
# Generated from _notebook_lib.py — edit there, not here; changes to this cell are replaced.
#   to_canonical(rows, labels) → gold

def to_canonical(rows: list[dict[str, str]],
                 labels: list[str],
                 column: str = COL_FINAL) -> list[dict[str, str]]:
    """Turn annotation rows into canonical gold: [{"id","text","label"}, ...].

    Blank rows are skipped; labels outside `labels` are reported, not silently kept.

    Args:
        rows: the rows read back by load_annotation_sheet.
        labels: the labels your scheme allows. Anything else is reported as invalid.
        column: which column holds the agreed label.

    Returns:
        The usable rows as gold items, each {"id", "text", "label"}.

    Example:
        >>> my_gold = to_canonical(rows, LEVELS)
    """
    ### Step 1: sort every row into one of three piles ###
    gold, blank, invalid = [], 0, []     # usable rows · not labelled yet · typos
    for row in rows:
        label = str(row.get(column, "")).strip()   # .strip() drops stray spaces
        if not label:
            blank += 1                    # nobody has filled this row in yet
        elif label not in labels:
            invalid.append((row.get(COL_ID), label))   # e.g. "b1" or "B11"
        else:
            gold.append({"id": int(row[COL_ID]), "text": str(row[COL_TEXT]), "label": label})

    ### Step 2: report all three counts, so nothing is dropped silently ###
    print(f"{len(gold)} usable · {blank} still blank · {len(invalid)} invalid")
    if invalid:
        print("  fix these in the sheet, then re-run:", invalid[:10])   # first 10
    return gold

Show code

help(to_canonical)   # ✏️ change the name to look up any other helper

Show code

rows = load_annotation_sheet(SHEET_ID, ROUND)   # re-read your latest round, `Final` filled in
my_gold = to_canonical(rows, LEVELS)            # reads the `Final` column
my_gold[:3]                                     # peek at the first three items

How does your gold compare with the published gold? The CEFR-SP labels came from language-education professionals, keeping only sentences where two of them agreed. Arase’s own experts agreed exactly only 37.6% of the time, so a difference is not simply an error — but each one needs a look and a reason. → interpret this on step F (slides).

Show code

#@title 🔧 Library cell: compare_to_published { display-mode: "form" }
# Helper — you don't need to read this. Run it and move on.
# Generated from _notebook_lib.py — edit there, not here; changes to this cell are replaced.
#   compare_to_published(gold, published) → how often you two agree

def compare_to_published(gold: list[dict[str, str]],
                         published: list[dict[str, str]]) -> pd.DataFrame | None:
    """How often does YOUR final label match the published gold, item by item?

    Items are matched by their TEXT, not their id, because a sampled set is often
    renumbered from 1 — and matching those ids against the original set would pair
    your item 7 with their item 7: two unrelated sentences, and a percentage that
    means nothing. (Ids are still used as a fallback, in case a text was edited.)

    Args:
        gold: your own gold items, from to_canonical.
        published: the published gold items, from load_gold.

    Returns:
        A table of the items where you and the published gold differ, or None
        when nothing could be matched.

    Example:
        >>> compare_to_published(my_gold, published)
    """
    ### Step 1: index the published labels by text, and by id as a fallback ###
    label_by_text = {}
    label_by_id = {}
    for item in published:
        label_by_text[str(item["text"])] = item["label"]
        label_by_id[item["id"]] = item["label"]

    ### Step 2: pair each of your items with its published label ###
    matched = []
    for item in gold:
        text = str(item["text"])
        if text in label_by_text:
            theirs = label_by_text[text]
        elif item["id"] in label_by_id:
            theirs = label_by_id[item["id"]]
        else:
            continue                       # not in the published set at all
        matched.append({"id": item["id"], "yours": item["label"],
                        "published": theirs, "text": item["text"]})
    if len(matched) == 0:
        print("None of your items could be matched to the published set.")
        return None

    ### Step 3: count the matches, then show only the rows where you differ ###
    agree = 0
    differences = []
    for row in matched:
        if row["yours"] == row["published"]:
            agree = agree + 1
        else:
            differences.append(row)
    print(f"{agree}/{len(matched)} match the published label "
          f"({agree / len(matched):.1%})")
    return pd.DataFrame(differences)

Show code

published = load_gold(GOLD_URL)      # the CEFR-SP labels, for comparison only
compare_to_published(my_gold, published)   # how often you two agree, item by item

Show code

## confusion matrix with your gold and published gold

Save your gold set to your Drive — it belongs in your Drive, not the course repo, and it becomes S6’s yardstick. See Housing your data in Google Drive.

Show code

# ✏️ Uncomment in Colab to save:
# from google.colab import drive; drive.mount("/content/drive")
# with open("/content/drive/MyDrive/my_gold_day2.json", "w", encoding="utf-8") as f:
#     json.dump(my_gold, f, ensure_ascii=False, indent=2)
# print("saved", len(my_gold), "items")

✅ Before you submit

Runtime → Run all and check every cell ran without error.
Your sample was drawn with a seed you can state, and the sheet you annotated in was made by the step A cells (step A).
Your agreement numbers and the annotator-vs-annotator matrix are visible (step D), for the last round you ran.
my_gold printed a list of {id, text, label} records, and step F’s comparison against the published gold ran (step F).
File → Download → Download .ipynb and upload both of today’s Day-2 notebooks.

How to use this notebook

🎯 Learning Objectives

Start here — what Arase et al. actually decided

Build a gold standard yourself

Today’s goal, as six steps (A–F)

A · Draw your sample → make your sheet (E&K Step 3 · ①) ✏️ YOU EDIT

Creating Google Spreadsheet

B · Apply the operationalized scheme (E&K Steps 4–5; Fuoli · ②)

CEFR descriptor

C · Annotate blind, in pairs (E&K Step 6 · ③)

D · Measure agreement (E&K Step 6 · ③) ✏️ YOU EDIT

Understand the data structure

Calculating agreement

Calculating Cohen’s κ

Plot confusion matrix

Which of those numbers you report is not a free choice

E · Read the matrix → refine → re-annotate (E&K Step 6; Fuoli princ. 2 · ③)

Let’s reveal the specific data you disagree with each other.

F · Adjudicate → gold (E&K Step 6 → feeds ④⑤) ✏️ YOU EDIT

First — looking up what a helper expects

✅ Before you submit