Multimodal Fashion Recommender

Fashion e-commerce sites need a good "you might also like", but the catalogue is huge, customers almost never leave ratings, and what makes two items similar depends on both how they look and what they actually are. A plain-text search misses the visual half; a plain-image search misses the descriptive half.

This project targets small and mid-size Indian D2C fashion brands specifically: businesses that want real recommendations without hiring a machine-learning team or standing up their own infrastructure to get them.

The model is trained on the H&M Kaggle dataset, 1.37 million customers, 105,000 articles, 31 million past purchases, and it's a "two-tower" design: one tower turns an item into a single vector of numbers, the other turns a shopper into one, and training pulls a shopper's vector close to the items they actually bought.

The item tower doesn't learn to see or read from scratch. It takes a frozen CLIP model's image embedding (CLIP already knows how to describe what's in a photo) and a frozen SBERT model's text embedding (SBERT already knows what a product description means), concatenates them, and passes the combination through a small neural network down to one 256-number vector. The user tower runs a small Transformer over a shopper's last 20 purchases and pools that into the same 256-number space.

Training uses a contrastive loss (InfoNCE): for each shopper in a batch, their real next purchase should sit closer in that 256-dimensional space than every other item in the batch. At serving time, a FAISS index (a fast nearest-neighbor search library) finds the closest items to a shopper's vector, and a Groq-hosted LLM writes a one-sentence, cached explanation for why each item was picked.

The same pipeline was extended to ingest real Indian brand catalogues: Snitch, Fashor, Powerlook, Virgio, from CSV, so a brand can get "more like this" recommendations from day one, before it has any purchase history of its own on the platform.

Two towers trained together in one shared 256-dimensional space, served through a per-brand FAISS index behind one shared process.

H&M's own 2M-row transaction history is deliberately excluded from live serving via a BRANDS_ENABLED flag, see the FAISS scaling decision below.

1. Frozen CLIP + SBERT fusion, not end-to-end fine-tuning

   The image and text encoders are frozen and only the fusion MLP on top of them is trained. That means the model transfers to any fashion catalogue, including Indian ethnic wear it never saw during training, without retraining the encoders themselves.

2. Temperature 0.1, learning rate 3e-4, and a 500-step warmup, after the standard config collapsed

   The textbook contrastive-learning config (temperature 0.07, learning rate 1e-3, no warmup) caused total representation collapse: every item and user vector converged to the same point in space, loss looked deceptively low, and the model learned nothing. Raising the temperature, lowering the learning rate, and, the single most important fix, adding a 500-step warmup period, resolved it.

3. FAISS, not a managed vector database, for up to 4 brands, a written trade-off, not an oversight

   For a handful of brands, a plain FAISS index that lives in local process memory needs zero extra infrastructure and behaves identically in development and production. An architecture decision record spells out the math on when that stops being true and lays out a 4-phase migration path (today's single shared process, then lazy-loading with an LRU cache, then a real vector database, Qdrant, then per-brand service instances) for when it does.

4. Evaluated on exactly the 10,556-item pool the production index actually serves

   Recall is measured against the same active item pool FAISS serves at inference time, not a larger offline-only pool, so the reported number is what a real user would actually experience, not an optimistic upper bound.

3.06× (0.0328), show source for Recall@10 vs. popularity baseline

Verified source

Recall@10 lift vs. popularity baseline

• README.md:124 ↗

commit 7b5d84b · measured 2026-07-12

Recall@10 vs. popularity baseline · also 2.12× vs. a co-purchase baseline (0.0155) and ahead of text-only SBERT (0.0248); temporal held-out test, 110,390 users

0.0107, show source for Popularity baseline, Recall@10

Verified source

Recall@10, popularity baseline

• README.md:121 ↗

commit 7b5d84b · measured 2026-07-12

Popularity baseline, Recall@10 · the baseline the 3.06× lift is measured against, same README table, same commit

0.0208 / 0.0172, show source for NDCG@10 / MRR

Cited in provenance.md

`README.md:113-124`

verified against source 2026-10-10

NDCG@10 / MRR

~$0.001–$0.004 per 1,000 recommendations, show source for Serving cost

Cited in provenance.md

`COST.md:26-27,64`

verified against source 2026-10-10

Serving cost · with a warm explanation cache

Illustrative only, not yet validated, show source for Honest caveat: new-brand personalization

Cited in provenance.md

`README.md:178-190`

verified against source 2026-10-10

Honest caveat: new-brand personalization · for a brand new to the platform, the personalized /recommend endpoint runs on synthetic users because the user tower never saw that brand's real purchase history; only the content-based /similar endpoint is a validated day-one capability

Training with the paper-standard contrastive-learning recipe, temperature 0.07, learning rate 1e-3, no warmup, produced a suspiciously clean-looking loss curve and a completely useless model: every item and user embedding had converged to the same point in the 256-dimensional space. Cosine similarity between any two items was effectively 1.0. Nothing had been learned; the loss function had just found the cheapest way to look small.

The fix came from empirical iteration rather than a known formula: raising the temperature to 0.1, dropping the learning rate to 3e-4, and, the change that mattered most, adding a 500-step warmup before the learning rate reached its full value. Warmup gives the towers a chance to spread out into distinct regions of the space before the optimizer starts taking large steps, which is exactly what a cold, aggressive learning rate short-circuits.

Once the model worked, a second question came up: how many brands can one shared FAISS process actually hold? An architecture decision record ran the numbers rather than guessing: three real brand catalogues fit comfortably in roughly 250–400MB, but H&M's own 2-million-row transaction history alone would push the current 4GiB Cloud Run instance over budget, scaling to include it is estimated to need 8GiB, and even that is a tight fit, not comfortable headroom. Rather than ship a serving path that would eventually fall over, H&M is deliberately excluded from live serving behind a BRANDS_ENABLED flag, with a 4-phase migration path, lazy-loading with an LRU cache, then a real vector database like Qdrant, then per-brand service instances, written down for when the catalogue count grows past what one process can hold.

If you're building contrastive or two-tower retrieval and your loss curve looks suspiciously clean, this is worth checking first: a collapsed representation can look like convergence. And if you're deciding how far a single-process architecture can scale, write the math down before you find out the hard way in production.