Day 3 — Linguistic Data Analysis II
This is your single submission for the day. It has two parts:
Every prompt you run gets its own id, and its score lands in one dict, f1_by_round — that dict is your results table.
You only edit the cells marked ✏️ YOU EDIT. Run the 🔧 Library cells and leave them alone. One cell — the evaluate skeleton — has five lines we fill in together in class; Run all only works after those five lines are filled in (the collapsed note under the skeleton has the answers).
➡️ Work top to bottom. A full run makes about 170 model calls and takes about 15 minutes. When you’re done, Runtime → Run all, then File → Download → Download .ipynb and submit that file.
This file is the completed version of day3_prompt_design.ipynb: the five evaluate lines are filled in, and every fill-in string carries one worked example. Work through the fill-in notebook first; open this one to check your work — your prompts, predictions and numbers do not have to match these.
Today only the prompt changes between rounds — the data, the run code and the scoring code are fixed once, at the top. Three rounds, three techniques:
| Id | Technique | Idea |
|---|---|---|
1 zero-shot |
zero-shot | just describe the task |
2 few-shot |
few-shot | add a few labeled examples |
3 chain-of-thought |
chain-of-thought | ask the model to reason before answering |
After each round, evaluate hands back the macro-F1 and you store it under the round’s id — so at the end the comparison table prints itself.
From Day 3 on you call the model live, so the notebook switches to the Gemini API. Get a free key and add it to Colab Secrets as GEMINI_API_KEY — one-time, ~2 minutes, no install. Full steps: Get a free Gemini API key.
When the setup cell prints LLM backend: Gemini API (...) you’re set. If it still says Colab Gemini, your secret isn’t set or its notebook-access toggle is off — and the JSON-reply cells below are not guaranteed to work, because the keyless demo backend ignores the JSON-mode request, so run_prompt logs ?? for any reply it cannot read. Rate limits are handled for you, and explained at the end of Part A.
Before any course code, make one call with nothing around it. The whole API is three parts:
reply.text is the model’s text out.from google import genai # the Gemini API library
from google.colab import userdata # reads your Colab Secrets
MODEL_ID = "gemini-3.1-flash-lite" # the model this course uses all week
client = genai.Client(api_key=userdata.get("GEMINI_API_KEY"))
reply = client.models.generate_content(
model=MODEL_ID,
contents="What CEFR level is this sentence? I like cats.")
print(reply.text)You should see a short answer — maybe just A1, maybe a sentence or two. That is the entire API: client, model name, text in, text out.
SecretNotFoundError or an authentication error means your key is not in Colab’s Secrets, or the secret’s notebook-access toggle is off — the callout above has the steps. Fix it now; everything below needs the key.
The minimal call works, but it is fragile in three ways, and each one matters once you score whole sets:
temperature=0, no seed, so the same question can get a different answer tomorrow.The Setup cell below takes this same call and adds exactly those protections. That is all it does.
The 📦 Setup cell is collapsed: you run it, you don’t edit it. It connects to the model and defines three functions, one per step:
| Function | What it does |
|---|---|
_resolve_gemini_key() |
finds your API key — Colab’s Secrets panel first, then the environment. Returns nothing if neither has one. |
_raw_generate_text(prompt, json_reply, schema) |
the actual model call. With a key: the Gemini API with temperature=0 and seed=42, so the same prompt gives the same answer; json_reply=True turns on JSON mode, which guarantees the reply is valid JSON — in whatever shape the prompt asks for. Without a key: Colab’s keyless demo model — non-reproducible, and it ignores the JSON-mode request. |
generate_text(prompt, json_reply=...) |
the one you call all day. It wraps _raw_generate_text with two protections: it waits between calls so you stay under the 15-per-minute limit, and when the server still says “too fast” it waits longer and tries again — unless the message names the per-day cap, where retrying cannot help. Both pieces are walked through at the end of Part A. |
It also sets three values: MODEL_ID (the pinned model), POOL_URL (the sentence pool this day draws from) and LEVELS (the six CEFR labels).
When the cell prints LLM backend: Gemini API (...) your key was found; Colab Gemini means it was not — go back to the key callout above.
Three more ready-made functions (load_gold, split_pool, show_errors) arrive later, each in a collapsed 🔧 Library cell right before the step that first calls it. The two functions you will call most — run_prompt and evaluate — are built in visible cells in Part A, so you see what is inside them.
#@title 📦 Setup — run me first { display-mode: "form" }
# Helper — you don't need to read this. Run it and move on.
# Generated from _notebook_lib.py — edit there, not here; changes to this cell are replaced.
import json, os, random, time, urllib.request
import pandas as pd, seaborn as sns, matplotlib.pyplot as plt
# --- LLM backend: Gemini API when a key is set, else colab.ai demo ------------
MODEL_ID = "gemini-3.1-flash-lite" # pinned model for the reproducible (API) backend
### Step 1: find an API key — Colab's Secrets panel first, then the environment ###
def _resolve_gemini_key() -> str | None:
"""Find a Gemini API key: Colab Secrets first (not auto-exported to env), then env.
Returns:
The key, or None when neither place has one.
"""
try:
from google.colab import userdata # only exists in Colab
key = userdata.get("GEMINI_API_KEY") # what you saved in the Secrets panel
if key:
return key # found one — use it
except Exception:
pass # not in Colab, or secret not set
return os.environ.get("GEMINI_API_KEY") # last resort: an environment variable
### Step 2: pick a backend — your API key if you have one, else Colab's demo model ###
_key = _resolve_gemini_key()
if _key:
from google import genai
from google.genai import types
_client = genai.Client(api_key=_key) # your own connection to the API
def _raw_generate_text(prompt: str, json_reply: bool = False,
schema: dict | None = None) -> str:
# temperature=0 + a fixed seed = the same prompt gives the same answer every
# run, which is what makes the autograded Corpus Labs reproducible.
# json_reply=True turns on JSON mode: the reply is valid JSON, in whatever
# shape the prompt asks for. A schema goes further and enforces one shape.
if schema is not None:
cfg = types.GenerateContentConfig(temperature=0, seed=42,
response_mime_type="application/json",
response_schema=schema)
elif json_reply:
cfg = types.GenerateContentConfig(temperature=0, seed=42,
response_mime_type="application/json")
else:
cfg = types.GenerateContentConfig(temperature=0, seed=42)
return _client.models.generate_content(model=MODEL_ID, contents=prompt,
config=cfg).text # prompt in, text out
_backend = f"Gemini API ({MODEL_ID}, temperature=0, seed=42)"
_min_interval = 4.4 # keeps us under gemini-3.1-flash-lite's 15-per-minute cap
else:
try:
from google.colab import ai # Colab's built-in Gemini — no key
def _raw_generate_text(prompt: str, json_reply: bool = False,
schema: dict | None = None) -> str:
# colab.ai has no JSON mode and no schemas — both requests are ignored
# here, so replies are not guaranteed JSON and run_prompt logs "??" for
# ones it cannot read. Another reason Day 3 asks for a key.
return ai.generate_text(prompt)
_backend = "Colab Gemini (demo, non-reproducible)"
_min_interval = 13.2 # colab.ai publishes no rate limit — pace conservatively
except ImportError: # no key AND not in Colab — nothing to call
raise RuntimeError(
"No LLM backend found. Run this notebook in Google Colab (free built-in "
"Gemini, no key needed), or set GEMINI_API_KEY — in Colab via the Secrets "
"panel, or as an environment variable when running locally. "
"See resources/tools/gemini-api-key.md.")
### Step 3: the one function you call all week — pace, ask, and retry if told to ###
_last_call_time = 0.0 # generate_text remembers & updates this with `global`
def generate_text(prompt: str, max_retries: int = 5, json_reply: bool = False,
schema: dict | None = None) -> str:
"""Send a prompt to the model and give back its reply.
It waits between calls so we stay under the free tier's speed limit, and tries
again if the server tells us to slow down.
Args:
prompt: the text to send to the model.
max_retries: how many times to try again after a rate-limit message.
json_reply: True asks for JSON mode — the reply is valid JSON; say the
shape you want in the prompt itself.
schema: a reply shape to enforce (the optional last section of Day 3).
None = nothing enforced.
Returns:
The model's reply, as text — JSON text when json_reply is True or a
schema was given.
Raises:
RuntimeError: when the daily quota is used up, or after the last retry.
Example:
>>> reply = generate_text("What CEFR level is this sentence? I like cats.")
"""
global _last_call_time # share the clock across every call
for attempt in range(max_retries + 1): # try, then retry up to max_retries times
wait = _min_interval - (time.monotonic() - _last_call_time) # still too soon?
if wait > 0:
time.sleep(wait) # pause so we stay under the speed limit
try:
_last_call_time = time.monotonic() # note the time of this attempt
return _raw_generate_text(prompt, json_reply, schema) # success — hand the reply back
except Exception as error:
text = str(error).lower() # the error message, as lowercase text
if not ("429" in text or "quota" in text or "rate limit" in text):
raise # a real bug — don't hide it
if "per day" in text: # the PER-DAY cap — waiting won't help
raise RuntimeError(
"Daily quota used up for today — waiting won't help until it "
"resets. Come back tomorrow, or ask your instructor.") from error
if attempt == max_retries:
raise # we've been patient enough
print(f" (rate limited — waiting before trying again, attempt {attempt+1})")
time.sleep(_min_interval * (attempt + 2)) # wait longer each time round
raise RuntimeError("Still rate-limited after several tries.")
# The CEFR-SP pool (335 sentences, natural level imbalance) — Day 3 draws its train/valid/test sets from it.
POOL_URL = "https://raw.githubusercontent.com/egumasa/linguistic-data-analysis-II-2026/main/sources/resources/datasets/gold/cefr_pool_demo.json"
LEVELS = ["A1", "A2", "B1", "B2", "C1", "C2"]
print(f"Setup done. LLM backend: {_backend}.")Day 2 handed you ready-made files. Today you draw your own three sets from one pool, because the split S7 introduced needs three sets with three different jobs:
| Set | Job |
|---|---|
train |
Take few-shot examples from here. The model is allowed to see these. |
valid |
Tune on this. Every prompt you try gets scored here. |
test |
Report on this. Scored once, at the very end of Part B. |
If you try five prompts on the test items and keep the highest score, that score no longer estimates how the prompt does on new sentences: you picked it because it suited those items. Tuning on valid and reporting once on test is what keeps the final figure honest.
First, load the pool:
#@title 🔧 Library cell: load_gold { display-mode: "form" }
# Helper — you don't need to read this. Run it and move on.
# Generated from _notebook_lib.py — edit there, not here; changes to this cell are replaced.
# load_gold(url_or_path) → gold
def load_gold(url_or_path: str) -> list[dict[str, str]]:
"""Read the canonical gold JSON: [{'id','text','label'}, ...].
Args:
url_or_path: a web address, or the path to a file on this machine.
Returns:
The gold items, each a dict with "id", "text" and "label".
Example:
>>> gold = load_gold(GOLD_URL)
"""
if str(url_or_path).startswith("http"): # a web address?
raw = urllib.request.urlopen(url_or_path).read().decode("utf-8") # download it
gold = json.loads(raw) # JSON text -> list of dicts
else: # otherwise a file on disk
gold = json.loads(open(url_or_path, encoding="utf-8").read())
print(f"Loaded {len(gold)} items. First one:", gold[0]) # proof it worked
return goldpool = load_gold(POOL_URL) # the CEFR-SP pool — 335 labelled sentencesYou should see Loaded 335 items. and the first item — the same {id, text, label} shape as every gold file this course uses.
Before drawing anything, look at how the 335 items spread over the six levels:
for level in LEVELS:
count = 0
for item in pool: # count the pool items with this level
if item["label"] == level:
count += 1
print(level, count)B1 and B2 dominate; A1 and C2 have 12 items each. That has two consequences:
train + valid + test cannot exceed 12.A balanced set is the right tool for comparing prompts — every level gets an equal say in macro-F1. Keep in mind that it no longer mirrors the pool’s natural proportions; Session 9 returns to what that does to a reported score.
The pool is one limit. The other is time and quota: every item you score is one model call. The free tier allows 15 calls per minute — the Setup cell paces calls about 4.4 seconds apart to stay under it — and 500 calls per day. Measure the real speed on five items:
start = time.time()
for item in pool[:5]: # five items are enough to measure
reply = generate_text("What CEFR level is this sentence? " + item["text"])
print(reply[:60]) # the first 60 characters of each reply
seconds_per_item = (time.time() - start) / 5
print("about", round(seconds_per_item, 1), "seconds per item")Now the arithmetic that decides the set sizes, with the defaults used below:
valid = 24 items × ~4.4 s ≈ 2 minutes;About 170 calls in total — comfortably inside the 500-per-day budget, with room left for extra rounds. Bigger sets give steadier scores but cost time and quota on every single round. That trade is yours to set in the next cell.
The three sizes are per level, and the sets are disjoint — no sentence appears in two of them. The seed makes the draw repeatable: everyone who keeps seed=42 gets the same three sets.
#@title 🔧 Library cell: split_pool { display-mode: "form" }
# Helper — you don't need to read this. Run it and move on.
# Generated from _notebook_lib.py — edit there, not here; changes to this cell are replaced.
# split_pool(pool, train_per_level, valid_per_level, test_per_level, seed) → train, valid, test
def split_pool(pool: list[dict[str, str]], train_per_level: int,
valid_per_level: int, test_per_level: int,
seed: int = 42) -> tuple[list[dict[str, str]],
list[dict[str, str]],
list[dict[str, str]]]:
"""Draw three disjoint, level-balanced sets from the pool.
Each set gets the same number of items per CEFR level, drawn without
replacement — no item appears in two sets. The seed makes the draw
repeatable: same pool, same numbers, same seed = same three sets.
Args:
pool: the items to draw from, each with a "label" key.
train_per_level: items per level for the train set (few-shot examples).
valid_per_level: items per level for the validation set (tune here).
test_per_level: items per level for the test set (scored once, at the end).
seed: fixes the randomness so the draw is repeatable.
Returns:
Three lists: train, valid, test.
Raises:
ValueError: when a level has fewer items than the three sizes add up to.
Example:
>>> train, valid, test = split_pool(pool, 2, 4, 6, seed=42)
"""
rng = random.Random(seed) # a private, seeded random-number source
need = train_per_level + valid_per_level + test_per_level
train, valid, test = [], [], []
for level in LEVELS:
stock = [] # every pool item with this level
for item in pool:
if item["label"] == level:
stock.append(item)
if len(stock) < need:
raise ValueError(
f"Not enough {level} items: the pool has {len(stock)}, but the three "
f"sizes add up to {need} per level. Make one of the sets smaller.")
drawn = rng.sample(stock, need) # `need` distinct items, in random order
train += drawn[:train_per_level]
valid += drawn[train_per_level:train_per_level + valid_per_level]
test += drawn[train_per_level + valid_per_level:]
print(f"train {len(train)} · valid {len(valid)} · test {len(test)} items "
f"({train_per_level}/{valid_per_level}/{test_per_level} per level)")
return train, valid, test# ✏️ the three sizes are per level — discuss them, then change them if you like.
# Per level, train + valid + test cannot exceed 12 (the smallest level's stock).
train, valid, test = split_pool(pool, train_per_level=2,
valid_per_level=4, test_per_level=6, seed=42)You should see train 12 · valid 24 · test 36 items. Three sets, three jobs — and from here on, test is not touched again until the last section of Part B.
Two short steps: JSON replies, and the loop that runs a prompt over a whole set.
You made the raw API call at the very top of the notebook — a client, a model name, text in, free text out. From here on, every call goes through generate_text (defined in the Setup cell): the same call, wrapped with the pacing and retry protections listed under it. The last section of Part A walks through both.
One problem is still open: the raw call gave back free text — maybe A1, maybe a paragraph — and your pipeline needs a label. That is step 1.
S7 named output formatting as one component of a prompt. Two things work together here:
Reply as JSON, like: {"label": "B1"};json_reply=True) guarantees the reply is valid JSON rather than a paragraph.The shape comes from your prompt; JSON mode only guarantees that the reply parses.
# First, call the LLM as we have been doing
# The prompt names the shape; json_reply=True guarantees the reply is valid JSON.
raw_reply = generate_text('What CEFR level is this sentence? I like cats.')
print(raw_reply) # the raw reply: JSON text# The prompt names the shape; json_reply=True guarantees the reply is valid JSON.
reply = generate_text('What CEFR level is this sentence? I like cats. '
'Reply as JSON, like: {"label": "B1"}', json_reply=True)
print(reply) # the raw reply: JSON text
answer = json.loads(reply) # JSON text -> a Python dict
print(answer["label"]) # just the levelYou should see something like {"label": "A1"} and then A1. json.loads turns JSON text into a Python dict — the same {...} shape as the gold items you have worked with since Day 1 — so the label comes out with a plain answer["label"]. No searching through a paragraph for the level.
run_prompt, the loop you will run all dayOne prompt, one set, one list of predicted labels back. Read it top to bottom — every line is something you have now seen: .format fills the {text} slot, generate_text makes the paced call with json_reply=True, json.loads unpacks the reply. A reply no label can be read out of becomes ?? instead of crashing a 24-item run.
def run_prompt(prompt: str, gold: list[dict[str, str]]) -> list[str]:
"""Send each item's `text` to the LLM via {text}, collect predicted labels.
Args:
prompt: your prompt, containing {text} where the sentence should go. It
should ask for a JSON reply with a "label" field — see Part A, step 1.
gold: the items to label, each with a "text" key.
Returns:
One predicted label per gold item, in the same order. "??" marks a reply
no label could be read out of.
Example:
>>> predictions = run_prompt(PROMPT, valid)
"""
predictions = [] # answers, in gold order
for i, item in enumerate(gold, 1): # i counts 1, 2, 3, ...
reply = generate_text(prompt.format(text=item["text"]), json_reply=True)
try:
answer = json.loads(reply) # JSON text -> a Python dict
except json.JSONDecodeError: # not JSON (keyless backend)
answer = {}
label = answer.get("label", "??") # "??" = no label in the reply
if label not in LEVELS: # not one of the six levels?
label = "??"
predictions.append(label)
if i % 12 == 0: # every 12th item...
print(f" ...{i}/{len(gold)} done") # ...show progress
print(f"Got {len(predictions)} predictions.")
return predictions
print("run_prompt defined.")This is the exact call form the final project uses — run_prompt(prompt, items) — so everything you do with it today transfers directly.
run_prompt gives predictions; now the function that scores them. You built every measurement inside it on Day 2 (S6): the per-class precision/recall/F1 table, Cohen’s κ, the confusion matrix. Today they become one function, evaluate, that prints all of that and hands back the macro-F1 as a number — so a round’s score can be stored instead of copied off the screen.
First the tools, by name:
from sklearn.metrics import (classification_report, confusion_matrix,
cohen_kappa_score, f1_score)The structure is in place; the five numbered measuring lines are blank. We fill them in together in class — each one is a function you used on Day 2.
# the five numbered lines are filled in — this is the completed version.
def evaluate(gold: list[dict[str, str]],
predictions: list[str],
ordered: bool = False) -> float:
"""Score predictions against gold: print the full report, return macro-F1.
ordered=True adds quadratic weighted kappa, for labels that sit on a scale.
Example:
>>> f1_by_round["1 zero-shot"] = evaluate(valid, predictions, ordered=True)
"""
### Step 1: line the two label lists up, gold first ###
y_true = [] # the correct labels, from the gold set
for item in gold:
y_true.append(item["label"])
y_pred = predictions # the model's labels, in the same order
### Step 2: per-class precision / recall / F1, as a text table ###
print(classification_report(y_true, y_pred, labels=LEVELS, zero_division=0)) # (1)
### Step 3: one overall number — agreement corrected for chance ###
kappa = cohen_kappa_score(y_true, y_pred) # (2)
print(f"Cohen's kappa {kappa:.3f}")
if ordered: # only when the labels sit on a scale
weighted = cohen_kappa_score(y_true, y_pred, labels=LEVELS,
weights="quadratic") # (3)
print(f"Cohen's kappa (weighted) {weighted:.3f} <- labels are ordered")
### Step 4: draw the same information as a picture ###
cm = confusion_matrix(y_true, y_pred, labels=LEVELS) # (4)
plt.figure(figsize=(5.5, 4.5))
sns.heatmap(cm, annot=True, fmt="d", cmap="Blues",
xticklabels=LEVELS, yticklabels=LEVELS)
plt.xlabel("Predicted"); plt.ylabel("Gold"); plt.title("Confusion matrix")
plt.tight_layout(); plt.show()
### Step 5: one number to keep — handed back to whoever called ###
macro_f1 = f1_score(y_true, y_pred, labels=LEVELS, average="macro",
zero_division=0) # (5)
print(f"F1 (macro) {macro_f1:.3f}")
return macro_f1evaluate (open after we have filled it in together)
The five lines, as written in class — each replaces one None:
print(classification_report(y_true, y_pred, labels=LEVELS, zero_division=0)) # (1)
kappa = cohen_kappa_score(y_true, y_pred) # (2)
weighted = cohen_kappa_score(y_true, y_pred, labels=LEVELS, weights="quadratic") # (3)
cm = confusion_matrix(y_true, y_pred, labels=LEVELS) # (4)
macro_f1 = f1_score(y_true, y_pred, labels=LEVELS, average="macro",
zero_division=0) # (5)Last piece of the pipeline: one dict that collects every round’s macro-F1 under the round’s id. Storing a score is one assignment, and you will meet the same pattern, unchanged, in the final project’s 04_develop.ipynb:
f1_by_round["1 zero-shot"] = evaluate(valid, predictions, ordered=True)f1_by_round = {} # one entry per prompt id
print("scores so far:", f1_by_round)Just describe the task. This is your baseline: every later round is judged against its score.
# ✏️ everything between the triple quotes is the prompt — edit it freely.
# `{text}` is the slot each sentence gets dropped into; keep it.
# The doubled braces {{ }} print as single ones — that is how a literal brace
# survives .format. Keep the reply-shape line: run_prompt reads the "label" field.
PROMPT_ZERO = """ YOUR PROMPT HERE """
print(PROMPT_ZERO.format(text="(each sentence lands here)"))Run it over the validation set — 24 items, about 2 minutes:
pred_zero = run_prompt(PROMPT_ZERO, valid)Score it and log it. evaluate prints the full report and hands back the macro-F1; the assignment stores that number under the round’s id:
f1_by_round["1 zero-shot"] = evaluate(valid, pred_zero, ordered=True)You should see the per-class table, the two κ lines, the confusion matrix, and last a line F1 (macro) 0.xxx — that number is now sitting in f1_by_round under "1 zero-shot".
The Setup cell connected to the model with these two arguments:
cfg = types.GenerateContentConfig(temperature=0, seed=42)temperature is how much the model is allowed to vary. At 0 it takes its most likely answer every time. Higher values are useful for writing and unhelpful when you are measuring something.seed fixes the randomness that is left, so a repeat of the same call starts from the same place.Run the same prompt over the same five items twice and see whether the answers match:
first = run_prompt(PROMPT_ZERO, valid[:5]) # five items, to keep this cheap
second = run_prompt(PROMPT_ZERO, valid[:5]) # the same five, the same prompt
print("first run: ", first)
print("second run:", second)
print("identical?", first == second) # two lists are == when every item matchesTrue is what temperature=0 is for. On the keyless Colab backend you may see fewer matches, because colab.ai exposes neither setting — that is the Day-1 behaviour, and why this course asks for a key from today.
Even at temperature=0 a hosted model can change its answer: the provider updates the model, or batches your request differently. So save the run to a file and report from the file, not from what is on screen.
Add a few labeled examples so the model can pattern-match. This is what the train set exists for: examples must come from train, and never from valid or test — an example the model has seen cannot also measure it.
Look at what train offers:
for item in train:
print(item["label"], "·", item["text"])Pick a few — the default prompt below uses four of them, one per level it covers. Copy the sentence text exactly, and write each example’s answer in the same JSON shape the prompt asks for — the examples demonstrate the output format as well as the labels.
# ✏️ same prompt as round 1, plus labelled examples. Add, remove or swap them —
# but only ever from `train`. Never use a sentence from `valid` or `test`.
PROMPT_FEWSHOT = """ YOUR PROMPT HERE """
print("examples in the prompt:", PROMPT_FEWSHOT.count("->"))Same 24 items, new prompt — run and log under the round’s id:
pred_few = run_prompt(PROMPT_FEWSHOT, valid)
f1_by_round["2 few-shot"] = evaluate(valid, pred_few, ordered=True)The four above are one choice out of many. Two strategies pull in opposite directions:
Predict first, then find out. Say which you think will help here, swap two examples for that kind (from train), and re-run.
Two rules either way: examples come from train only, and cover every label if you can, or the ones you left out get under-predicted.
Ask the model to reason first, then answer. Room to think often helps on borderline items.
With free-text replies this would create a parsing problem: the reasoning mentions several levels along the way — which one is the answer? The reply shape solves it — ask for a JSON object with two fields, reasoning written first, then label. The reasoning happens, and run_prompt still reads only the label field. ✏️ Edit the reasoning instruction if you like:
# ✏️ this prompt asks the model to reason before deciding.
PROMPT_COT = """ YOUR PROMPT HERE """
print(PROMPT_COT.format(text="(each sentence lands here)"))Run and log — the same call as every round; only the prompt changed. This round is slower than the others, because the model writes its reasoning for every item:
pred_cot = run_prompt(PROMPT_COT, valid)
f1_by_round["3 chain-of-thought"] = evaluate(valid, pred_cot, ordered=True)What did that reasoning actually look like? Read one full reply:
reply = generate_text(PROMPT_COT.format(text=valid[0]["text"]), json_reply=True)
print(reply)A JSON object with the model’s reasoning first and its label after — the order the prompt asked for. run_prompt read the label field and ignored the rest. Whether the reasoning helped is what the score says, not the reasoning itself.
The log has been collecting all along — print it:
for name in f1_by_round:
print(name, round(f1_by_round[name], 3))Three rounds, three numbers, all on valid. This table is what Part B iterates on — and the test set has still never been scored. It stays that way until the end of Part B.
The free tier limits you in two independent ways, on two different clocks:
A plain for loop over a few dozen sentences can pass the RPM limit in the first few seconds, long before it has used much of the day’s RPD budget. While building this course, a loop tripped a 15-per-minute cap after 16 calls, with only 126 of that day’s 500 used. The fix is to go slower, and to know which limit you hit.
The Setup cell’s guard does this, and the rest of this section walks through it.
Never call the model faster than the limit allows. At 15 calls per minute that is one call every 60 / 15 = 4 seconds, so before each call, check how long it has been since the last one and wait out the difference.
That means the function has to remember when the last call happened. The global keyword tells Python to keep one variable and share it across every call.
Try it below — no model, no internet, just pacing:
import time
### Step 1: two things to remember — when we last called, and how long to wait ###
_demo_last_call = 0.0 # remembered BETWEEN calls, thanks to `global`
DEMO_INTERVAL = 2 # seconds (the real guard uses 4.4s or 13.2s)
### Step 2: before each call, wait out whatever time is still owed ###
def wait_your_turn() -> None:
"""Wait out whatever time is still owed, then say we are calling."""
global _demo_last_call # "remember this one, and share it across calls"
wait = DEMO_INTERVAL - (time.monotonic() - _demo_last_call) # time still owed
if wait > 0: # too soon — sit it out
print(f" waiting {wait:.1f}s so we don't call too often...")
time.sleep(wait)
_demo_last_call = time.monotonic() # note when this call happened
print(" → calling now!")
### Step 3: three calls in a row — watch the gap appear between them ###
for i in range(3):
wait_your_turn()Pacing alone isn’t enough: the server can still say “too fast, try again later”. What to do depends on the kind of failure, which the error message tells you:
A demo — no model, just the try/except shape, retrying until it works:
### Step 1: a stand-in for the real model — it fails twice, then works ###
attempt_count = 0 # how many times we have called it so far
def unreliable_call() -> str:
"""Fails on the first two calls, then succeeds — like a real rate-limited API call."""
global attempt_count # keep the count between calls
attempt_count += 1
if attempt_count <= 2: # the first two tries
raise Exception("429 rate limit — please slow down") # ...blow up
return "success!" # the third works
### Step 2: try it, and if it breaks, go round again instead of crashing ###
for attempt in range(3): # up to three goes
try:
result = unreliable_call()
print("Got:", result)
break # it worked — stop looping
except Exception as error: # it broke — `error` holds the message
print(f" attempt {attempt+1} failed ({error}) — trying again...")Piece 1 plus piece 2 is what is inside generate_text in the Setup cell above, and it protects every loop you run from here on — including every Part B round.
A fuller version, which also remembers past answers so you never pay for the same prompt twice, is in resources/extra/handling-rate-limits.ipynb.
Part A handed you three logged prompts. From here you run your own rounds — as many as time and quota allow. Anyone can try ten prompts and keep the best; what makes it a study is that each change comes with a reason and a prediction, so that when the number moves you can say why.
Every round follows the same five steps:
PROMPT_ID, name the one thing you will change, and write down what you expect it to do.valid. Never on test.f1_by_round[PROMPT_ID] = evaluate(valid, ...).show_errors on what is still wrong; it feeds the next round.Round 4 below walks the cycle once, step by step. Round 5 is a ready-made template to copy for every round after it. The test set stays closed until the last section.
No new code needed — evaluate already printed it. Scroll back to your best round’s report and read down the F1 column: one or two levels will be far below the rest.
Then run the cell below and read three of the actual sentences for that level. The counts tell you where the misses are; only the sentences tell you why. Two cases to tell apart: the sentences are genuinely borderline between two levels · your prompt never described that level. The first is a property of the data; only the second is fixable by prompting.
#@title 🔧 Library cell: show_errors { display-mode: "form" }
# Helper — you don't need to read this. Run it and move on.
# Generated from _notebook_lib.py — edit there, not here; changes to this cell are replaced.
# show_errors(gold, predictions) → misclassified table
def show_errors(gold: list[dict[str, str]], predictions: list[str]) -> pd.DataFrame:
"""The items the model got wrong, as a table you can read and argue about.
Args:
gold: the gold items, each with "id", "text" and "label".
predictions: one predicted label per gold item, in the same order.
Returns:
A table with one row per mistake: id, gold, pred, text.
Example:
>>> show_errors(gold, predictions)
"""
rows = []
for g, p in zip(gold, predictions): # walk gold and predictions side by side
if g["label"] != p: # keep only the disagreements
rows.append({"id": g["id"], "gold": g["label"], "pred": p, "text": g["text"]})
print(f"{len(rows)} of {len(gold)} wrong.")
# Name the columns even when there are no rows. A table built from an empty list has
# no columns at all, and then errors["gold"] fails for a student whose prompt got
# everything right.
return pd.DataFrame(rows, columns=["id", "gold", "pred", "text"]) # Colab shows a tableerrors = show_errors(valid, pred_cot) # swap pred_cot for your best round's predictions
errors.head(15)Fill in all four strings before you touch the prompt. This is the cell that turns the next step from tinkering into an experiment.
# a worked example of the declare cell — one possible round 4.
PROMPT_ID = "4 describe C1 explicitly"
WORST_CLASS = "C1"
MY_CHANGE = "add one line describing what makes a sentence C1"
I_PREDICT = "C1 recall goes up, because the prompt now says what to look for; B2 and C2 may lose a little to it"
print(f"Round {PROMPT_ID} — targeting {WORST_CLASS}. Change: {MY_CHANGE}.")
print(f"Prediction: {I_PREDICT}")If you add examples and rewrite the instruction and ask for reasoning, and the score moves, you have learned nothing about which of the three did it. Change one thing, score it, keep or discard it, then change the next.
Ideas worth trying, roughly cheapest first: describe the weak level explicitly · add two train examples of that level · say what separates it from its neighbour · give the model an out (“if unsure between two adjacent levels, choose the lower”).
Start from your best Part-A prompt and make the one change you just declared.
# the worked round-4 prompt: the zero-shot prompt plus the ONE declared change.
PROMPT_MINE = """You are an expert rater of English sentence difficulty using the CEFR scale.
Classify the sentence into exactly one of: A1, A2, B1, B2, C1, C2.
C1 sentences use precise, lower-frequency vocabulary and complex noun phrases,
but still read as ordinary prose rather than technical writing.
Reply as JSON, like: {{"label": "B1"}}
Sentence: {text}"""
print(PROMPT_MINE.format(text="(each sentence lands here)"))Same call as every round — the score lands under the id you declared:
pred_mine = run_prompt(PROMPT_MINE, valid)
f1_by_round[PROMPT_ID] = evaluate(valid, pred_mine, ordered=True)Compare the F1 row for WORST_CLASS in the report you just printed against the same row in step 1’s report. Macro-F1 can rise while the level you aimed at gets worse, because another level carried the average — so check the row you predicted, not the headline number. Then read what is still wrong:
errors_mine = show_errors(valid, pred_mine)
errors_mine.head(15)Was your prediction right? A “no” costs you nothing — a wrong prediction you can explain teaches more than a right one you cannot.
This round is ready-made: it adds a tie-break rule to your round-4 prompt. Keep it, change it, or replace it — either way, give the round its own id and write your prediction before running. For round 6 and beyond, copy this round’s three code cells (and the markdown between them), change the id, and go again.
# the ready-made round, with the prediction written out as an example.
PROMPT_ID = "5 tie-break rule"
I_PREDICT = "fewer far misses upward: when the model hesitates between two adjacent levels it now takes the lower one, so weighted kappa should improve even if macro-F1 barely moves"
PROMPT_R5 = PROMPT_MINE + """
If you are unsure between two adjacent levels, choose the lower one."""
print(f"Round {PROMPT_ID}. Prediction: {I_PREDICT}")Run and log:
pred_r5 = run_prompt(PROMPT_R5, valid)
f1_by_round[PROMPT_ID] = evaluate(valid, pred_r5, ordered=True)And inspect, feeding the next round if you run one:
errors_r5 = show_errors(valid, pred_r5)
errors_r5.head(15)Copy round 5’s cells, give the new round its own id, and go again — each round costs one valid-sized run (24 calls ≈ 2 minutes with the default sizes). Stop while you still have quota for the test run. Two rounds with stated reasons and predictions are worth more than five undocumented ones.
Every round you ran, one line each:
for name in f1_by_round:
print(name, round(f1_by_round[name], 3))Pick the best prompt from the log and score it on test — the 36 sentences drawn at the top of Part A and never touched since. This is the number you report.
Expect it to be lower than the validation score. That gap is the cost of having chosen a prompt by looking at results, and reporting it honestly is the point of having a held-out set.
If you run this cell, then go back and edit a prompt, test has stopped being held out. In the final project this is a file boundary: 04_develop.ipynb cannot reach the test items and 05_test.ipynb opens them once.
# ✏️ swap in whichever prompt the log says is best (PROMPT_ZERO / PROMPT_FEWSHOT /
# PROMPT_COT / PROMPT_MINE / PROMPT_R5 / ...).
BEST_PROMPT = PROMPT_FEWSHOT
pred_test = run_prompt(BEST_PROMPT, test) # the 36 held-out sentences
f1_test = evaluate(test, pred_test, ordered=True) # <- report THIS macro-F1The numbers below are from one example run; yours will differ. What should match is the shape: every claim names a set, a number, and a reason.
valid was round 3 (chain-of-thought), with macro-F1 = 0.42.test set the same prompt scored macro-F1 = 0.36.0.06, which we read as the cost of tuning: the prompt was chosen because it suited the 24 validation items, and new sentences pay that back.(If we have time — this section is not part of the submission, and it costs two model calls.)
Everything above asked for the reply shape in the prompt and turned on JSON mode. JSON mode guarantees the reply is valid JSON — not that it has any particular fields. A model could answer {"level": "B1"} and run_prompt would log ??. The API can also enforce the fields: describe the shape you want as a schema and pass it as response_schema; the reply then has exactly those fields.
First, the call you have made all day:
# The way every round above worked: shape in the prompt, JSON mode on.
reply = generate_text('What CEFR level is this sentence? I like cats. '
'Reply as JSON, like: {"label": "B1"}', json_reply=True)
print(reply)Now the enforced version. The schema below describes the same shape the prompt was asking for — an object with one required string field, label — and goes to the API as response_schema. The prompt no longer needs the shape line:
# The enforced version: the same shape, described as a schema.
LABEL_SCHEMA = {"type": "OBJECT",
"properties": {"label": {"type": "STRING"}},
"required": ["label"]}
reply = generate_text("What CEFR level is this sentence? I like cats.",
schema=LABEL_SCHEMA)
print(reply)Same reply, two routes. When to prefer which:
label field is guaranteed, so a renamed field can never produce ??. The fix when a model keeps drifting away from the shape you asked for.The final project uses the prompt-side route, which is why this notebook does too. On the keyless demo backend both requests are ignored — one more thing the API key buys you.
evaluate are filled in (the collapsed note above it has the answers).f1_by_round has its own id, and every round’s prediction string is filled in..ipynb and upload that one file.