week02_bpe.ipynb
Download .ipynb ·
Deliverables hub
Open in Jupyter from repo: jupyter notebook mmc/notebooks/week02_bpe.ipynb
Cell 1 · markdown
# Week 2 — BPE tokenizer + Data Card
**MMC / DeepMind track** — show-work notebook.
### How to run
- **Local (Anaconda):** from repo root
`conda activate controlsift-mmc` then `powershell -File scripts/start_mmc_jupyter.ps1`
or open this file in JupyterLab and use kernel **Python (controlsift-mmc)**.
- **GitHub Pages cannot execute notebooks** (static hosting only). After the repo is on GitHub, use **Open in Colab** / Binder links on the deliverables hub, or clone and run locally.
## Official DeepMind labs (checkbox)
- `vendor/ai-foundations/course_2/gdm_lab_2_3_tokenize_texts_into_subword_tokens.ipynb`
- `vendor/ai-foundations/course_2/gdm_lab_2_4_implement_a_bpe_tokenizer.ipynb`
Full map: `mmc/GDM_LAB_MAP.md`. Upstream clone: `vendor/ai-foundations/`.
## ControlSift exceed path
Run the cells below to regenerate **this repo's** graded artifacts under `mmc/deliverables/outputs/`.
Do not invent Gemma metrics; leave GPU rows null until a real HF/Kaggle run.
Cell 2 · code
from pathlib import Path
import os
import sys
# Resolve repo root whether Jupyter cwd is repo root or mmc/notebooks/
here = Path.cwd().resolve()
root = None
for candidate in [here, *here.parents]:
if (candidate / "mmc" / "labs").is_dir() and (candidate / "mmc" / "notebooks").is_dir():
root = candidate
break
if root is None:
raise RuntimeError("Could not find ControlSift repo root (expected mmc/labs/). Open Jupyter from the repo.")
os.chdir(root)
if str(root) not in sys.path:
sys.path.insert(0, str(root))
print("cwd:", Path.cwd())
print("python:", sys.executable)
Cell 3 · code
from mmc.labs.week02_bpe import run
import json
result = run()
show = {k: result.get(k) for k in ['num_merges', 'vocab_size', 'sample_line_tokens', 'data_card']} if ['num_merges', 'vocab_size', 'sample_line_tokens', 'data_card'] else result
print(json.dumps(show, indent=2, default=str))
result