Warmer

Daily word games like Wordle are hugely popular, but they're English-only and letter-based: a guess is either right or wrong. Warmer asks a different question, how close is your guess in meaning? You guess a secret word, and an AI language model scores how semantically near you are, from Freezing to Scorching.

It's built for word-game players in India in particular: Warmer is one of very few daily games with a native Hinglish (romanized Hindi-English) mode, which turned out to be the hardest engineering problem in the project.

The key design decision is that no AI runs while you play. Each day's puzzle is fully precomputed offline: a Python generator picks the secret word, embeds the entire vocabulary with a sentence-embedding model (a model that turns words into lists of numbers, where similar meanings land near each other), ranks every word by similarity to the secret, and writes one small compressed file per day per language.

The Flutter app, Android and web, just downloads that day's file and looks your guess up in the precomputed ranking. That makes the marginal serving cost per player effectively zero, and it means the generator can afford a heavier, more accurate model, because it only runs once at build time.

The payload is obfuscated (hashed keys, an encoded secret) so opening the file in a browser doesn't spoil the puzzle, and the app bundles a week of puzzles offline as a fallback.

Two halves that never run at the same time: an offline generator that does all the ML work, and a runtime that is deliberately ML-free.

Deployed with flutter build web --wasm on Vercel; uptime-checked every 15 minutes via GitHub Actions.

1. Precompute everything offline instead of running a model per guess

   The game only needs a fixed daily vocabulary ranked once, so the ML moves to build time. Runtime cost per player drops to ~$0, and the generator can use a more accurate model than a live API could afford.

2. Swap the Hinglish model for one trained on romanized text

   The original multilingual model was trained on Hindi in Devanagari script. Romanized words like "baarish" or "jeet" were out-of-distribution, so it clustered them near unrelated Latin-script languages by surface form. The fix was an eval-gated bake-off across nine candidates; a model trained natively on code-mixed romanized text won.

3. Compile Dart to WebAssembly, decided by measurement

   The --wasm target was adopted only after an isolated preview measured the difference: total blocking time 221ms → 13.5ms and 417 KB less page weight versus the default JavaScript build.

   self-reported (private repo)

4. When fine-tuning failed twice, change the method, not the data

   Two full-parameter fine-tuning attempts both regressed held-out accuracy, the second got worse after adding more training data (543 pairs, then 756/1028 pairs), falsifying the "more data fixes it" hypothesis. The eventual win held the data fixed and swept the method instead: every full-parameter and frozen-layer config stayed within noise, every LoRA config beat the baseline, three of them CI-significantly. The shipped model is that LoRA fine-tune, published openly on Hugging Face with the benchmark that judges it.

   self-reported (private repo)

0.813, show source for Hinglish semantic accuracy after the fine-tune (Spearman vs. human judgments)

Verified source

Both endpoints pinned in `content/metrics.json` against the same file/line at two commits: pre-fix `mindmeld/generator/evals/reports/baseline_report.md:24` @ `675994d` (2026-07-10, first commit of this eval harness — no earlier history exists), post-fix same file/line @ `16f35d1` (2026-07-18, after the script-mismatch fix and a later LoRA fine-tune). Root cause narrative in `mindmeld/spec-hinglish-fix.md:4,6-7`. Corrected 2026-08-12: this row previously cited an intermediate 0.639 (the value right after the script-mismatch fix alone, before the LoRA fine-tune pushed it to 0.813 — see case-studies/warmer.ts's result detail) via a stale `PLAN.md:265-273` reference that no longer resolves at HEAD; 0.813 is the number displayed everywhere on the site (metrics.json, products.ts, the case study, resume content, the gh-profile banner) and is what this row now matches.

• evals/warmer/baseline_report.md:24 ↗

commit 1f4fc87 · measured 2026-07-18

Hinglish semantic accuracy after the fine-tune (Spearman vs. human judgments) · the script-mismatch fix reached 0.639; a LoRA fine-tune plus an eval-harness RNG fix took it to 0.813 on the production set (English holds at 0.842). The stricter held-out generalization number is 0.435 → 0.704, both are reported, neither hidden

−0.003, show source for The same metric before the fine-tune

Verified source

Hinglish embedding eval (Spearman correlation), pre-fix baseline

• evals/warmer/baseline_report_pre-fix.md:24 ↗

commit 1f4fc87 · measured 2026-07-10

The same metric before the fine-tune · literally no correlation, same eval, same file and line, at the pre-fix commit

0.0370 base vs 0.0247 fine-tuned, show source for Cluster separation, base vs fine-tuned (mean silhouette, raw embeddings)

Verified source

Embedding cluster separation (mean silhouette, raw embeddings)

• evals/warmer/embedding_separation_report.md:17 ↗

commit 1f4fc87 · measured 2026-08-13

Cluster separation, base vs fine-tuned (mean silhouette, raw embeddings) · each model scored on its OWN k=7 partition, the control. On the fine-tune's own partition the gap looks 9.3× the other way; that version favours the fine-tune by construction, so both are published and the viewer states the reversal

5.1% → 78.0%, show source for Cross-language consistency (translation pairs landing in the right band)

Verified source

Both endpoints pinned in `content/metrics.json` against the same file/line at two commits: pre-fix `mindmeld/generator/evals/reports/baseline_report.md:24` @ `675994d` (2026-07-10, first commit of this eval harness — no earlier history exists), post-fix same file/line @ `16f35d1` (2026-07-18, after the script-mismatch fix and a later LoRA fine-tune). Root cause narrative in `mindmeld/spec-hinglish-fix.md:4,6-7`. Corrected 2026-08-12: this row previously cited an intermediate 0.639 (the value right after the script-mismatch fix alone, before the LoRA fine-tune pushed it to 0.813 — see case-studies/warmer.ts's result detail) via a stale `PLAN.md:265-273` reference that no longer resolves at HEAD; 0.813 is the number displayed everywhere on the site (metrics.json, products.ts, the case study, resume content, the gh-profile banner) and is what this row now matches.

• evals/warmer/baseline_report.md:24 ↗

commit 1f4fc87 · measured 2026-07-18

Cross-language consistency (translation pairs landing in the right band)

0.675 Spearman [95% CI 0.601, 0.741], 0.740 cross-language pass-rate [0.62, 0.86], show source for Public Hinglish relatedness benchmark vs. 7 off-the-shelf alternatives

Cited in provenance.md

Model card `README.md:50-56,82` ("Evaluation" section, field-leading claim) — fetched 2026-09-23 (verified: 2026-09-23)

measured 2026-09-23

Public Hinglish relatedness benchmark vs. 7 off-the-shelf alternatives · leads all 7 tested alternatives (including 2 released after the original bake-off) on both metrics, CI-significant in all 14 comparisons, per the model card's own reported eval set; the companion dataset published alongside the model describes its own contents as 34 secrets and reports a close but distinct 0.657 for the same metric, a source discrepancy noted rather than resolved (see provenance.md)

160/160 generator · 94/94 app · 2/2 emulator integration, show source for Test suite

Cited in provenance.md

`mindmeld/README.md:175-176`

verified against source 2026-07-31

Test suite · self-reported (private repo)

FCP 992ms (≤1800ms) · TBT 26ms (≤200ms) · CLS 0, show source for Web perf (tracked budget)

Cited in provenance.md

**Wave 19 correction (2026-07-31):** `mindmeld/README.md:178-190`, commit `e8d03c7` (2026-07-30, "fix(docs): re-measure and correct stale performance budgets in README"). Superseded the "LCP 3,082ms, fails its own 3,000ms ceiling by 82ms" framing, which was accurate when originally sourced but replaced 5 days later. **Known cross-repo inconsistency, not fixed here (out of scope — a different repo):** `mindmeld/docs/known-limitations.md:383-402` still asserts the old LCP-ceiling framing as of its own last edit (commit `62b3ce9`, 2026-07-30 19:41 — *after* `e8d03c7`, but that commit touched an unrelated section of the file and never reconciled this one). `README.md` is treated as the more authoritative source here since it was the explicit, deliberate subject of the fix commit; `known-limitations.md` needs a matching correction in the `mindmeld` repo itself.

measured 2026-07-31

Web perf (tracked budget) · LCP is deliberately not budgeted, this CanvasKit-rendered app fires zero LCP entries by design (no traditional largest-contentful DOM element for the Paint Timing API to key off), confirmed via a live PerformanceObserver check; FCP is the tracked load-speed proxy instead · self-reported (private repo)

At launch, the Hinglish mode's semantic ranking measured a Spearman correlation of −0.003 against human similarity judgments, literally no correlation. The root cause was empirical, not speculative: the multilingual embedding model had learned Hindi in Devanagari script, so romanized Hinglish words were gibberish to it. "Dil" landed near Turkish words; "dua" near Italian, clustered by how they're spelled, not what they mean.

The fix was a nine-candidate bake-off under a fixed evaluation harness, transliteration pipelines, code-mixed BERT variants, MuRIL, decided purely on measured metrics. A model trained natively on romanized code-mixed text won, taking Spearman to 0.639.

Then came the more interesting result: two full fine-tuning cycles tried to push past that ceiling with curated and corpus-mined Hinglish word pairs, and both made the model worse, the second attempt, with more data, regressed further than the first. The project documented that as a falsified hypothesis and shipped the off-the-shelf model while the question sat open.

The eventual breakthrough came from changing the method, not the data: holding the training set fixed and sweeping capacity-control configurations showed every full-parameter variant stuck at noise while every LoRA variant improved, a small-data overfitting signature the earlier attempts had misread as a data problem. The winning LoRA model now ships in production and is published on Hugging Face (hinglish-relatedness-sbert), alongside the public benchmark used to judge it against seven alternatives.

If you need a model that has to work across code-mixed or low-resource languages, not just English demos, this is the debugging discipline that gets it there: falsify the easy hypothesis, measure the alternative, and don't ship a fix until it's beaten the baseline on held-out data.