Samidha Reviews

Indian D2C (direct-to-consumer) sellers get customer reviews in English and Hinglish (romanized, code-mixed Hindi-English, "yeh product bahut accha hai" written in Latin letters), and reading all of them manually to catch the urgent or safety-related ones doesn't scale. Most review-analysis tools don't natively handle Hinglish at all.

Samidha Reviews (built and shipped under the name ReviewIQ, live-rebranded since) turns unstructured review text into structured data: sentiment, pros/cons, and an urgency flag, so a seller can triage what actually needs attention instead of reading everything.

A review first passes through a sanitizer that strips personally identifiable information, then a language detector classifies it as English or Hinglish, which routes it to a language-branched prompt (a prompt template tuned for that language's phrasing).

From there, a tiered router sends the easy cases to a small, fast model (openai/gpt-oss-20b) by default, and escalates to a larger model (openai/gpt-oss-120b) only when the small model's output fails schema validation or comes back with low confidence, most reviews never need the expensive model.

The model's response is validated against a strict JSON schema before it's accepted, and results are stored behind API-key authentication (using argon2id, a memory-hard password-hashing algorithm) with Postgres row-level security enforcing that one seller's data is never visible to another in the shared multi-tenant database.

Sanitize, detect language, route by difficulty to the cheapest model that can do the job, and never let an unvalidated response reach storage.

Cross-cutting: structlog structured logging + Prometheus metrics, deployed on Cloud Run in asia-south1.

1. Groq only on the client-data path, Gemini is dev-only

   Gemini's free tier trains on submitted data, which is unacceptable for real customer reviews. The client-facing path is hard-restricted to Groq-hosted models, enforced in code by an assert_privacy_safe() check rather than left as a policy someone could forget to follow.

2. Tiered routing: try the small model first, escalate on failure

   Sending every review to the large model would be simpler, but wasteful. Routing simple cases to the small model first and escalating only on validation failure or low confidence cut token cost 27.9% in v0.5.0. The same historical table lists 84.4% overall for that routed release and 85.8% for the direct-LLM v0.4.0 before it (46 fixtures each), a 1.4-point gap derived here by subtraction, not a published figure.

3. Cassette-replay CI, zero live API calls in CI

   Every CI run replays previously-recorded LLM responses keyed on a hash of the model name and prompt (sha256 of model+prompts), rather than calling a live LLM. That makes CI runs free, deterministic, and reproducible, and, as the debugging story below shows, it's also how a real prompt bug got diagnosed without spending a single new API call.

4. Urgency rubric rewritten from tone-based to signal-based

   The original urgency rubric leaned on the review's tone, an angry-sounding review scored higher urgency than a polite one. That's backwards for safety: a physical-harm report should be HIGH urgency regardless of star rating or how politely it's phrased. The rubric was rewritten to key on the presence of a harm signal, not the emotional register of the writing.

78.6% (95% CI 73–83%, n=43, eval 2026-09-19), show source for Overall extraction accuracy

Verified source

`review-iq/eval/report.md:5,11-12`, `review-iq/eval/results.json`, generated 2026-09-19, commit `e7186c9`

• eval/report.md:5 ↗

commit e7186c9 · measured 2026-09-19

Overall extraction accuracy · PASS on review-iq's own CI regression gate (a regression detector, not a real-world accuracy claim)

No published accuracy figure; an earlier 40-item fixture result (21 positive) was in-sample and is not cited as a measurement, show source for Honest caveat: authenticity scoring has no published accuracy figure

Cited in provenance.md

review-iq docs/compliance.md:76 ("The authenticity scorer has no published accuracy figure."; "An earlier 40-item fixture set (21 positive) was in-sample and its result is not reproducible ... is not cited as a measurement.") and README.md:424-427 ("Removed:" the /v2/authenticity routes "because the flag is unmeasurable"; "Not measured: no authenticity labels exist for the held-out set"), at review-iq main sha 48b00a6e122bcaf1adef45002c2bc34625f25b5a (read 2026-10-18)

measured 2026-10-18

Honest caveat: authenticity scoring has no published accuracy figure · the authenticity routes were removed from the product because the flag is unmeasurable: no authenticity labels exist for the held-out set

Devanagari Hindi is not supported (retired scope, ADR 0022); the Hinglish gold labels are LLM-generated, called "not published-credible" in the spec, show source for Honest caveat: Devanagari Hindi is retired scope

Cited in provenance.md

`spec.md:11-13`, `PROMPTS.md:38-43`

verified against source 2026-10-05

Honest caveat: Devanagari Hindi is retired scope · a follow-up investigation attributed most of the apparent hi-en gap to benchmark-label noise, not model failure

A review that praised the product overall but mentioned a safety issue in passing came back scored as LOW urgency, because the original rubric keyed off tone: the review read as positive, so it scored as low-priority, regardless of the safety mention buried inside it. v2.2 rewrote the rubric to be signal-based instead of tone-based, so a harm signal drives urgency no matter how the rest of the review reads.

v2.3 then dug into a related failure using cassette replay, re-running recorded LLM responses rather than calling the API again, and found the small model wasn't actually reasoning about the harm clause at all. It was pattern-matching the literal phrase "poor fit," which happened to appear in one of the prompt's own few-shot examples, and copying that example's classification regardless of context. The fix was to remove that example from the prompt and add a regression fixture so the same shortcut couldn't silently return.

Separately, the team investigated a suspected Hinglish-specific accuracy gap, traced it mostly to noise in the LLM-generated gold labels used for evaluation rather than a real model weakness, and logged the finding as "NOT PURSUED" instead of chasing a metric that wasn't measuring what it claimed to measure.

If your product has to catch safety-critical signals buried in ordinary-sounding text, this is the discipline worth copying: score on the signal, not the tone, and build CI so a real production bug can be diagnosed for free, from recordings, without spending a live API call to reproduce it.