Day 3 · Prompt design & iteration

Day 3 — Linguistic Data Analysis II

How to use this notebook

This is your single submission for the day. It has two parts:

  • Part A · Tutorial — improve a prompt through zero-shot → few-shot → chain-of-thought on the SAME CEFR-SP task, comparing macro-F1 at each step.
  • Part B · Corpus Lab — run your own prompt-iteration study and error analysis (to be written).

You only edit the cells marked ✏️ YOU EDIT. Cells marked 🔧 Library cell are pre-written — run them, don’t change them.

➡️ Work top to bottom. When you’re done, Runtime → Run all, then File → Download → Download .ipynb and submit that file.

Part A · Tutorial — three ways to prompt

Same pipeline as Day 2, same CEFR-SP data — only the prompt changes. We compare three techniques and watch macro-F1 move:

Iteration Technique Idea
0 zero-shot just describe the task
1 few-shot add a few labeled examples
2 chain-of-thought ask the model to reason before answering

Record the macro-F1 (the macro avg row of evaluate) after each run so you can tell a story about what helped.

ImportantFrom today you run the model yourself — you need a free API key

Days 1–2 used Colab’s built-in Gemini (or a frozen file). From Day 3 on you call the model live and need your prompt runs to be reproducible (temperature=0 + a fixed seed), so the notebook switches to the Gemini API. Get a free key and add it to Colab Secrets as GEMINI_API_KEY — one-time, ~2 minutes, no install. Full steps: Get a free Gemini API key. When the setup cell prints LLM backend: Gemini API (...) you’re set; if it still says Colab Gemini, your secret isn’t set or its notebook-access toggle is off.

Setup — run this first

Show code 
#@title 📦 Setup — run me first { display-mode: "form" }
# Imports + the LLM backend. No pip install needed in Colab.
import json, re, urllib.request, os
from sklearn.metrics import classification_report, confusion_matrix
import pandas as pd, seaborn as sns, matplotlib.pyplot as plt

MODEL_ID = "gemini-2.5-flash"   # pinned model for the reproducible (API) backend

def _resolve_gemini_key():
    """Find a Gemini API key: Colab Secrets first (not auto-exported to env), then env."""
    try:
        from google.colab import userdata      # only exists in Colab
        key = userdata.get("GEMINI_API_KEY")
        if key:
            return key
    except Exception:
        pass                                    # not in Colab, or secret not set
    return os.environ.get("GEMINI_API_KEY")

def _make_api_backend(key):
    """Reproducible backend: Gemini API with temperature=0 + a fixed seed."""
    from google import genai
    from google.genai import types
    client = genai.Client(api_key=key)
    cfg = types.GenerateContentConfig(temperature=0, seed=42)
    return (lambda p: client.models.generate_content(
                model=MODEL_ID, contents=p, config=cfg).text,
            f"Gemini API ({MODEL_ID}, temperature=0, seed=42)")

# Prefer the API key when set (reproducible); else fall back to colab.ai (demo).
_key = _resolve_gemini_key()
if _key:
    generate_text, _backend = _make_api_backend(_key)
else:
    try:
        from google.colab import ai            # Colab's built-in Gemini — no key
        generate_text, _backend = (lambda p: ai.generate_text(p)), "Colab Gemini (demo, non-reproducible)"
    except ImportError:
        raise RuntimeError(
            "No LLM backend found. Run this notebook in Google Colab (free built-in "
            "Gemini, no key needed), or set GEMINI_API_KEY — in Colab via the Secrets "
            "panel, or as an environment variable when running locally. "
            "See resources/tools/gemini-api-key.md.")

# CEFR-SP gold set (72 sentences, 12 per level), fetched from the course repo.
GOLD_URL = "https://raw.githubusercontent.com/egumasa/linguistic-data-analysis-II-2026/main/sources/resources/datasets/gold/cefr_sentences.json"
LEVELS = ["A1", "A2", "B1", "B2", "C1", "C2"]
print(f"Setup done. LLM backend: {_backend}. scikit-learn ready.")
Show code 
#@title 🔧 Library cell: load_gold(url_or_path) → gold { display-mode: "form" }
def load_gold(url_or_path):
    """Read the canonical gold JSON: [{'id','text','label'}, ...]."""
    if str(url_or_path).startswith("http"):
        raw = urllib.request.urlopen(url_or_path).read().decode("utf-8")
        gold = json.loads(raw)
    else:
        gold = json.loads(open(url_or_path, encoding="utf-8").read())
    print(f"Loaded {len(gold)} items. First one:", gold[0])
    return gold
Show code 
#@title 🔧 Library cell: run_prompt(prompt, gold) → predictions { display-mode: "form" }
def _extract_level(text):
    """Pull the first A1/A2/B1/B2/C1/C2 out of the model's reply."""
    m = re.search(r"\b([ABC][12])\b", str(text).upper())
    return m.group(1) if m else "??"

def run_prompt(prompt, gold):
    """Send each item's `text` to the LLM via {text}, collect predicted labels."""
    predictions = []
    for i, item in enumerate(gold, 1):
        reply = generate_text(prompt.format(text=item["text"]))
        predictions.append(_extract_level(reply))
        if i % 12 == 0:
            print(f"  ...{i}/{len(gold)} done")
    print(f"Got {len(predictions)} predictions.")
    return predictions
Show code 
#@title 🔧 Library cell: evaluate(gold, predictions) → P/R/F1 + confusion matrix { display-mode: "form" }
def evaluate(gold, predictions):
    y_true = [item["label"] for item in gold]
    y_pred = predictions
    print(classification_report(y_true, y_pred, labels=LEVELS, zero_division=0))
    cm = confusion_matrix(y_true, y_pred, labels=LEVELS)
    plt.figure(figsize=(5.5, 4.5))
    sns.heatmap(cm, annot=True, fmt="d", cmap="Blues",
                xticklabels=LEVELS, yticklabels=LEVELS)
    plt.xlabel("Predicted"); plt.ylabel("Gold"); plt.title("Confusion matrix")
    plt.tight_layout(); plt.show()
Show code 
#@title 🔧 Library cell: show_errors(gold, predictions) → misclassified table { display-mode: "form" }
def show_errors(gold, predictions):
    rows = [{"id": g["id"], "gold": g["label"], "pred": p, "text": g["text"]}
            for g, p in zip(gold, predictions) if g["label"] != p]
    print(f"{len(rows)} of {len(gold)} wrong.")
    return pd.DataFrame(rows)
Show code 
#@title 🔧 Library cell: save_predictions / load_predictions { display-mode: "form" }
def save_predictions(predictions, path):
    """Freeze a list of predicted labels to JSON."""
    with open(path, "w", encoding="utf-8") as f:
        json.dump(predictions, f)
    print(f"Saved {len(predictions)} predictions to {path}")

def load_predictions(url_or_path):
    """Read a frozen predictions list — a committed URL or a local path."""
    if str(url_or_path).startswith("http"):
        raw = urllib.request.urlopen(url_or_path).read().decode("utf-8")
        predictions = json.loads(raw)
    else:
        predictions = json.loads(open(url_or_path, encoding="utf-8").read())
    print(f"Loaded {len(predictions)} frozen predictions.")
    return predictions
Show code 
gold = load_gold(GOLD_URL)

Iteration 0 — zero-shot ✏️ YOU EDIT

Just describe the task. This is your baseline; note its macro-F1.

Show code 
PROMPT_ZERO = """You are an expert rater of English sentence difficulty using the CEFR scale.
Classify the sentence into exactly one of: A1, A2, B1, B2, C1, C2.
Answer with the level only.

Sentence: {text}"""

pred_zero = run_prompt(PROMPT_ZERO, gold)
evaluate(gold, pred_zero)

Iteration 1 — few-shot ✏️ YOU EDIT

Add a few labeled examples so the model can pattern-match. The examples are hand-written here (feel free to draw more from cefr_pool.json — never from the gold set you’re scoring on, or you’d be testing on training data).

Show code 
PROMPT_FEWSHOT = """You are an expert rater of English sentence difficulty using the CEFR scale.
Classify the sentence into exactly one of: A1, A2, B1, B2, C1, C2. Answer with the level only.

Examples:
Sentence: "I have a cat." -> A1
Sentence: "She went to the shops because she needed some milk." -> A2
Sentence: "The results suggest a modest but consistent improvement." -> B2
Sentence: "Notwithstanding these caveats, the framework generalises well." -> C2

Sentence: {text}"""

pred_few = run_prompt(PROMPT_FEWSHOT, gold)
evaluate(gold, pred_few)

Iteration 2 — chain-of-thought (CoT) ✏️ YOU EDIT

Ask the model to reason first, then answer. Giving it room to think often helps on borderline items.

Show code 
PROMPT_COT = """You are an expert rater of English sentence difficulty using the CEFR scale.
Think step by step about the vocabulary and grammar, then decide the level.
Do NOT mention any other CEFR level while reasoning.
End your answer with the final level on its own, exactly one of: A1, A2, B1, B2, C1, C2.

Sentence: {text}"""

pred_cot = run_prompt(PROMPT_COT, gold)
evaluate(gold, pred_cot)
NoteA real limitation to notice

run_prompt grabs the first CEFR level it sees in the reply. With chain-of-thought the model may mention a level mid-reasoning, so the parser can pick the wrong one — which is exactly why the prompt says don’t mention other levels. If CoT scores worse than few-shot, check show_errors to see whether it’s the model or the parser.

Compare the three

Fill in the macro-F1 you saw at each step. This little table is your result.

Iteration macro-F1
0 · zero-shot …
1 · few-shot …
2 · chain-of-thought …

Part B · Corpus Lab — your own prompt-iteration study

Warning🚧 TODO — to be written

This lab is still being written. When it is, you will:

  1. Pick one class the model handles worst (use show_errors).
  2. Write one concrete prompt change you predict will help, and test it.
  3. Log each iteration’s macro-F1 in the table below and explain the change.
Iteration Prompt change macro-F1
… … …

✅ Before you submit

  1. Runtime → Run all and check every cell ran without error.
  2. Part A outputs are visible (tables / charts / the model’s answers).
  3. Part B self-check prints ✅ (or your TODO answers are filled in).
  4. File → Download → Download .ipynb and upload that one file.