Handling rate limits without crashing

Linguistic Data Analysis II — extra resource.

Sooner or later a cell that was working fine will stop with a wall of red text ending in something like:

ClientError: 429 RESOURCE_EXHAUSTED. ... Quota exceeded for metric:
generativelanguage.googleapis.com/generate_content_free_tier_requests,
limit: 15, model: gemini-3.1-flash-lite. Please retry in 52.24s.

This notebook explains what that means and builds, step by step, a wrapper that makes it stop happening. Everything here runs without an API key — the demos use a fake backend that imitates a rate-limited server, so you can see each strategy work (and watch the naive version fail) without spending any quota.

What you will build. One function, make_resilient(call), that takes a plain “prompt → text” function and returns a version that paces itself, backs off and retries when the server pushes back, refuses to retry when retrying cannot help, and remembers answers it already paid for.

1 · The single most common misunderstanding

“I’ve barely used it today. Why am I rate limited?”

Because there is more than one limit, and they run on different clocks. The free tier meters you three ways at once:

Limit	Meaning	Clock	What trips it
RPM	requests per minute	rolling 60 s	a `for` loop with no pause
TPM	input tokens per minute	rolling 60 s	long prompts, few-shot with big examples
RPD	requests per day	resets ~midnight Pacific	a long session, or many re-runs

Almost every 429 that surprises people is RPM, not RPD. Fifteen requests per minute sounds generous until you notice that an unthrottled loop issues them in about two seconds. You can be at 126 requests out of a 500/day allowance — 25% of your daily budget, nowhere near exhausted — and still be hard-blocked, because you spent 16 of them inside one minute.

The distinction matters because the two failures need opposite responses:

RPM → wait a moment and retry. The quota refills continuously.
RPD → do not retry. It will not refill until tomorrow, so a retry loop just burns time and makes noise. Switch models, or stop.

The rest of this notebook is essentially that table turned into code.

2 · Read the error, don’t guess

The 429 tells you exactly which limit you hit — people just don’t read that far. The useful part is the quotaId:

GenerateRequestsPerMinutePerProjectPerModel-FreeTier → per minute
GenerateRequestsPerDayPerProjectPerModel-FreeTier → per day

There is also a retryDelay ('Please retry in 52.24s'). That number is the server telling you precisely how long until it will accept you again. Honoring it beats guessing.

Below: two small functions that classify an error and pull out the suggested delay. They work on the error’s string form, which sounds crude but is deliberate — Colab’s built-in Gemini and the google-genai package raise different exception classes for the same condition, and matching on text handles both without importing either.

Show code

import re, time, json, random

def looks_like_rate_limit(error):
    """True if this exception is a throttling / quota error (either kind)."""
    text = str(error).lower()
    return any(signal in text for signal in
               ["429", "resource_exhausted", "rate limit",
                "quota", "too many requests"])

def looks_like_daily_quota(error):
    """True only for a PER-DAY cap — the kind waiting cannot fix."""
    text = str(error).lower()
    return ("perday" in text.replace(" ", "")
            or "per day" in text
            or "requests per day" in text)

def suggested_delay(error, fallback):
    """The server's own 'please retry in Ns' hint, if it gave one."""
    match = re.search(r"retry in ([0-9]+(?:\.[0-9]+)?)s", str(error).lower())
    return float(match.group(1)) if match else fallback

# Try them on the real error text from the top of this notebook.
REAL_429 = ("429 RESOURCE_EXHAUSTED. Quota exceeded for metric: "
            "generativelanguage.googleapis.com/generate_content_free_tier_requests, "
            "quotaId: GenerateRequestsPerMinutePerProjectPerModel-FreeTier, "
            "limit: 15. Please retry in 52.242615755s.")

print("rate limit? ", looks_like_rate_limit(REAL_429))
print("daily cap?  ", looks_like_daily_quota(REAL_429), " <- False, so waiting WILL help")
print("wait for:   ", suggested_delay(REAL_429, fallback=5), "seconds")

rate limit?  True
daily cap?   False  <- False, so waiting WILL help
wait for:    52.242615755 seconds

3 · A fake server, so we can fail safely

To compare strategies we need something that rate-limits us on demand. This FakeGemini enforces a real rolling-window RPM cap and a daily cap, and raises error messages shaped like the real ones — but it costs nothing and needs no key.

It also counts how many calls it actually served, which is how we will score each strategy.

One concession to your patience: the window parameter. A real RPM window is 60 seconds, so an honest demo at 5 RPM would take nine minutes to watch. The cells below shrink the server’s window and the client’s pacing by the same factor, which keeps the arithmetic faithful while letting each cell finish in about a second. Scaling only one of the two would quietly turn the demo into a lie — which is exactly the bug this notebook had in its first draft.

Show code

class RateLimitError(Exception):
    pass

class FakeGemini:
    """A stand-in for a rate-limited LLM endpoint. No network, no key, no cost.

    window is the RPM averaging period; it is 60s in reality, and smaller here
    only so the demos run quickly.
    """

    def __init__(self, rpm=15, rpd=500, latency=0.02, window=60.0):
        self.rpm, self.rpd, self.latency, self.window = rpm, rpd, latency, window
        self.call_times = []      # timestamps of accepted calls
        self.served = 0           # successful responses
        self.rejected = 0         # 429s raised

    def __call__(self, prompt):
        now = time.monotonic()
        # Drop timestamps that have aged out of the rolling window.
        self.call_times = [t for t in self.call_times if now - t < self.window]

        if len(self.call_times) >= self.rpm:
            self.rejected += 1
            wait = self.window - (now - self.call_times[0])
            raise RateLimitError(
                "429 RESOURCE_EXHAUSTED. Quota exceeded for metric: "
                "generate_content_free_tier_requests, quotaId: "
                f"GenerateRequestsPerMinutePerProjectPerModel-FreeTier, limit: {self.rpm}. "
                f"Please retry in {wait:.2f}s.")

        if self.served >= self.rpd:
            self.rejected += 1
            raise RateLimitError(
                "429 RESOURCE_EXHAUSTED. Quota exceeded for metric: "
                "generate_content_free_tier_requests, quotaId: "
                f"GenerateRequestsPerDayPerProjectPerModel-FreeTier, limit: {self.rpd}.")

        self.call_times.append(now)
        self.served += 1
        time.sleep(self.latency)          # pretend the model thought about it
        return f"B2"                      # a plausible CEFR reply

print("FakeGemini ready.")

FakeGemini ready.

The naive loop, for reference

This is what most first drafts look like — and what produced the error at the top of this notebook. We give it 40 sentences to label. Watch where it dies.

(We use a 5-per-minute cap, which is genuinely what gemini-2.5-flash allows on the free tier — not a rigged setup. DEMO_SCALE shrinks the clock so you are not waiting a real minute for the point to land.)

Show code

DEMO_SCALE = 0.01                      # 1/100th-speed clock, for demos only
DEMO_WINDOW = 60.0 * DEMO_SCALE        # the server's RPM window, shrunk to match

server = FakeGemini(rpm=5, latency=0.01, window=DEMO_WINDOW)
prompts = [f"Rate the CEFR level of sentence {i}." for i in range(40)]

try:
    answers = [server(p) for p in prompts]       # <- the naive version
    print("finished:", len(answers))
except Exception as error:
    print("💥 crashed after", server.served, "of", len(prompts), "prompts")
    print("  ", str(error)[:110], "...")
    print("\nNote what was lost: the", server.served, "answers already paid for are")
    print("gone too, because the list comprehension never returned.")

💥 crashed after 5 of 40 prompts
   429 RESOURCE_EXHAUSTED. Quota exceeded for metric: generate_content_free_tier_requests, quotaId: GenerateReque ...

Note what was lost: the 5 answers already paid for are
gone too, because the list comprehension never returned.

4 · Strategy 1 — Pace yourself (the one that actually matters)

Every other strategy in this notebook is damage control. This one is prevention, and it removes the great majority of 429s on its own.

The idea is trivial: if the limit is 15 requests per minute, never issue them faster than one every 60 / 15 = 4 seconds. Before each call, check how long ago the previous call went out and sleep off the difference.

Two details that are easy to get wrong:

Leave headroom. Pace to slightly below the cap (we use a 1.1× safety factor), because the server’s idea of “a minute” and yours will drift apart.
Measure with time.monotonic(), not time.time(). A wall clock can jump backwards when the machine syncs to a time server; a monotonic clock cannot. Elapsed-time arithmetic on time.time() is a classic source of rare, baffling bugs.

The free-tier limits differ sharply per model, so pacing has to be derived from the model you actually chose:

Show code

# Free-tier limits, Gemini API. Check https://ai.dev/rate-limit for current values --
# these change, and the dashboard is authoritative.
RATE_LIMITS = {
    "gemini-2.5-flash":      {"rpm": 5,  "rpd": 20},
    "gemini-3.1-flash-lite": {"rpm": 15, "rpd": 500},
}

def min_interval_for(model, safety=1.1):
    """Seconds to leave between calls to stay under that model's RPM cap."""
    rpm = RATE_LIMITS.get(model, {}).get("rpm", 5)   # unknown model -> be cautious
    return (60.0 / rpm) * safety

for model, limits in RATE_LIMITS.items():
    print(f"{model:24s} {limits['rpm']:>3} RPM, {limits['rpd']:>4} RPD"
          f"  -> pace at {min_interval_for(model):.1f}s between calls")

gemini-2.5-flash           5 RPM,   20 RPD  -> pace at 13.2s between calls
gemini-3.1-flash-lite     15 RPM,  500 RPD  -> pace at 4.4s between calls

Notice how large the gap is. gemini-2.5-flash allows 20 requests per day on the free tier — one lab exercise over a 25-sentence gold set exhausts it before lunch. gemini-3.1-flash-lite allows 500. Pacing cannot rescue you from picking the wrong model; check RPD before you start a long run.

Here is pacing on its own, applied to the same 40 prompts that just crashed:

Show code

def with_pacing(call, min_interval):
    """Wrap a call so consecutive invocations are at least min_interval apart."""
    last = [0.0]                       # a list, so the closure can mutate it

    def paced(prompt):
        wait = min_interval - (time.monotonic() - last[0])
        if wait > 0:
            time.sleep(wait)
        last[0] = time.monotonic()
        return call(prompt)

    return paced

# Same server as the crash above, but paced. Both the server's window and our
# pacing are scaled by DEMO_SCALE, so the ratio between them is the real one.
server = FakeGemini(rpm=5, latency=0.01, window=DEMO_WINDOW)
paced_call = with_pacing(server,
                         min_interval=min_interval_for("gemini-2.5-flash") * DEMO_SCALE)

start = time.monotonic()
answers = [paced_call(p) for p in prompts]
print(f"✅ {len(answers)}/{len(prompts)} prompts completed, {server.rejected} rejections")
print(f"   (took {time.monotonic()-start:.1f}s at {DEMO_SCALE}x scale; "
      f"~{(time.monotonic()-start)/DEMO_SCALE/60:.0f} min for real)")

✅ 40/40 prompts completed, 0 rejections
   (took 5.4s at 0.01x scale; ~9 min for real)

Zero rejections. The trade is honest and unavoidable: staying inside a 5 RPM cap means 40 sentences takes about nine minutes of wall time. That is not the wrapper being slow — it is the actual cost of the free tier, previously hidden behind a crash.

This is why you keep gold sets small. Twenty-five sentences is a pedagogical choice, but it is also a quota choice.

5 · Strategy 2 — Back off and retry when you’re pushed back anyway

Pacing handles your traffic. It cannot handle everything: the limit is per project, so another notebook of yours sharing the key eats the same budget, and transient server-side congestion happens regardless.

So the second layer catches a 429 and retries. Three rules make the difference between a retry loop that helps and one that makes things worse:

Honor the server’s retryDelay when it gives one. It knows when your window reopens; you are guessing.
Back off exponentially otherwise — 4 s, 8 s, 16 s. A fixed-interval retry against a still-closed window is just more rejected traffic.
Add jitter (a small random offset). If you and three classmates hit the limit simultaneously and all retry after exactly 8 s, you collide again in lockstep. Randomness breaks the synchronization. This is the “thundering herd” problem, and jitter is the standard fix.

Give up after a handful of attempts rather than looping forever — a wrapper that retries indefinitely turns a clear error into a hung notebook, which is worse.

Show code

def with_retry(call, max_retries=5, base_delay=4.0):
    """Retry rate-limit errors with exponential backoff + jitter."""

    def retrying(prompt):
        for attempt in range(max_retries + 1):
            try:
                return call(prompt)
            except Exception as error:
                # Real bugs (bad prompt, no network) must surface immediately,
                # not be retried 5 times behind a misleading "rate limited" message.
                if not looks_like_rate_limit(error):
                    raise
                # A daily cap will not clear today. Retrying is pure waste.
                if looks_like_daily_quota(error):
                    raise RuntimeError(
                        "Daily free-tier quota exhausted for this model. Retrying "
                        "will not help until it resets (~midnight Pacific). Switch to "
                        "gemini-3.1-flash-lite (500/day), run in Colab, or continue "
                        "tomorrow."
                    ) from error
                if attempt == max_retries:
                    raise
                delay = suggested_delay(error, base_delay * (2 ** attempt))
                delay += random.uniform(0, 0.3 * delay)      # jitter
                print(f"   rate limited — waiting {delay:.1f}s (attempt {attempt+1})")
                time.sleep(delay)
        raise RuntimeError("unreachable")

    return retrying

print("with_retry defined.")

with_retry defined.

Note the two raise statements before the backoff. They are the point of the function as much as the retrying is:

A typo in your prompt code should fail in one second, not after five backoffs totaling a minute, disguised as a quota problem.
A daily cap should say “come back tomorrow” immediately. This is exactly the case where a naive retry loop wastes the most time — it will patiently sleep, retry, sleep, retry, and still fail, because nothing it does can refill a daily bucket.

Worth seeing rather than taking on faith. Here is a server whose daily allowance is already spent — the wrapper gives up immediately, with an error that tells you what to do next:

Show code

exhausted = FakeGemini(rpm=5, rpd=3, latency=0.01, window=DEMO_WINDOW)
guarded = with_retry(exhausted, max_retries=5, base_delay=4.0 * DEMO_SCALE)

start = time.monotonic()
try:
    for i in range(10):
        guarded(f"prompt {i}")
except RuntimeError as error:
    print(f"stopped after {exhausted.served} calls "
          f"in {time.monotonic()-start:.2f}s — no pointless retrying")
    print("message:", error)

stopped after 3 calls in 0.04s — no pointless retrying
message: Daily free-tier quota exhausted for this model. Retrying will not help until it resets (~midnight Pacific). Switch to gemini-3.1-flash-lite (500/day), run in Colab, or continue tomorrow.

6 · Strategy 3 — Never pay for the same answer twice

The naive loop lost every answer it had already earned. On a metered API that is the expensive mistake, not the crash itself.

A cache fixes it: keep results keyed by prompt, and skip anything already answered. Re-running a cell after a failure then costs only the remaining calls. Because the key is the prompt text, editing your prompt correctly invalidates the cache — changed prompt, changed key, fresh call.

Writing the cache to disk (in Colab, to your mounted Drive) carries it across runtime restarts, which is worth doing for anything longer than a few minutes.

Show code

def with_cache(call, store=None):
    """Skip the network entirely for prompts already answered."""
    store = {} if store is None else store

    def cached(prompt):
        if prompt not in store:
            store[prompt] = call(prompt)
        return store[prompt]

    cached.store = store          # exposed so you can save/inspect it
    return cached

def save_cache(store, path="llm_cache.json"):
    with open(path, "w", encoding="utf-8") as f:
        json.dump(store, f, ensure_ascii=False, indent=1)

def load_cache(path="llm_cache.json"):
    try:
        with open(path, encoding="utf-8") as f:
            return json.load(f)
    except FileNotFoundError:
        return {}

print("with_cache defined.")

with_cache defined.

7 · Putting it together

The three wrappers compose. Order matters, and it is worth reasoning through rather than memorizing — read the nesting from the inside out:

cache  ⟶  pacing  ⟶  retry  ⟶  the actual API call

Cache outermost, so a cached hit skips the pacing sleep as well as the network. Put it inside the pacing and every cache hit would still sleep 4 seconds for no reason.
Retry innermost, wrapping the raw call, so a retry re-enters only the call and not the pacing bookkeeping.

Show code

def make_resilient(call, model="gemini-3.1-flash-lite", store=None,
                   max_retries=5, scale=1.0):
    """Wrap a raw prompt->text function so free-tier limits stop being your problem.

    scale is for demos only — it shrinks the sleeps so examples run quickly.
    Leave it at 1.0 for real work.
    """
    interval = min_interval_for(model) * scale
    guarded = with_retry(call, max_retries=max_retries, base_delay=4.0 * scale)
    guarded = with_pacing(guarded, min_interval=interval)
    guarded = with_cache(guarded, store=store)
    return guarded

# Full test: 40 prompts, a stingy server, and a re-run to show the cache working.
server = FakeGemini(rpm=5, latency=0.01, window=DEMO_WINDOW)
generate = make_resilient(server, model="gemini-2.5-flash", scale=DEMO_SCALE)

start = time.monotonic()
answers = [generate(p) for p in prompts]
first_pass = time.monotonic() - start
print(f"✅ pass 1: {len(answers)} answers, {server.served} API calls, "
      f"{server.rejected} rejections, {first_pass:.2f}s")

start = time.monotonic()
answers = [generate(p) for p in prompts]
print(f"✅ pass 2: {len(answers)} answers, {server.served} API calls total "
      f"(unchanged — all cached), {time.monotonic()-start:.3f}s")

✅ pass 1: 40 answers, 40 API calls, 0 rejections, 5.45s
✅ pass 2: 40 answers, 40 API calls total (unchanged — all cached), 0.000s

The second pass costs zero quota and returns instantly. That is what makes iterating on downstream analysis cheap: you pay for the model’s answers once, then re-run the scoring, the confusion matrix, and the plots as often as you like.

Using it with a real backend

Drop-in, in any course notebook. Everything downstream keeps calling generate_text(prompt) and never learns the difference:

from google import genai
from google.genai import types

MODEL_ID = "gemini-3.1-flash-lite"      # 500/day, not 20 — check before long runs
client = genai.Client(api_key=_resolve_gemini_key())
cfg = types.GenerateContentConfig(temperature=0, seed=42)

raw = lambda p: client.models.generate_content(
    model=MODEL_ID, contents=p, config=cfg).text

generate_text = make_resilient(raw, model=MODEL_ID, store=load_cache())

Then save the cache when you are done, so a runtime restart does not cost you the calls again:

save_cache(generate_text.store)

Colab’s built-in google.colab.ai backend has its own separate, undocumented throttling, but the same wrapper applies — pass model="gemini-2.5-flash" to get conservative pacing.

8 · When the wrapper is the wrong tool

Retry logic makes a program resilient to transient limits. It cannot manufacture quota, and reaching for it in the following situations wastes your time:

Situation	What to do instead
Daily cap (RPD) hit	Switch model (`gemini-3.1-flash-lite`, 500/day), or resume tomorrow. No amount of waiting helps today.
Long runs, repeatedly	Shrink the job: fewer items, or `n_per_class` sampling. Most course tasks need 25 items, not 250.
Iterating on a prompt	Test on 3–5 sentences. Run the full gold set only when the prompt has stopped changing.
Need speed and volume	Enable billing. The paid tier’s limits are far higher; the free tier is not meant for bulk work.
TPM (token) limit, not RPM	Shorten the prompt — trim few-shot examples or truncate long texts. Pacing does not help if a single call is too large.

The general principle: a rate limit is a budget, not an obstacle. The wrapper spends the budget smoothly instead of all at once, and it stops you losing work you already paid for. Deciding how much to spend is still your job, and the cheapest call remains the one you never make.

1 · The single most common misunderstanding

2 · Read the error, don’t guess

3 · A fake server, so we can fail safely

The naive loop, for reference

4 · Strategy 1 — Pace yourself (the one that actually matters)

5 · Strategy 2 — Back off and retry when you’re pushed back anyway

6 · Strategy 3 — Never pay for the same answer twice

7 · Putting it together

Using it with a real backend

8 · When the wrapper is the wrong tool

See also