AetherArt

State-of-the-art image generators like Stable Diffusion XL (SDXL) normally need a 16GB+ GPU, which puts them out of reach on a consumer laptop. AetherArt fine-tunes SDXL into a Japanese ukiyo-e woodblock-print style and engineers the whole pipeline to run inside an 8GB VRAM budget, with a live public demo.

It's also a small research project in its own right: the field's default way of scoring whether a generated image is good, CLIP score, turns out to silently miss real quality changes in some situations, and AetherArt set out to measure exactly when that's true rather than assume it.

The style comes from a LoRA adapter, a small ~45MB add-on file, not a full retrained model, trained on 80 ukiyo-e images from WikiArt in 4 hours 26 minutes on a rented GCP L4 GPU, for about $3.50. Adding the trigger word "ukyowood" to a prompt turns the style on.

Fitting everything into 8GB of VRAM took three things working together: running the model at half precision (FP16, which halves memory use), swapping in a corrected VAE (the component that turns the model's internal representation back into a viewable image) because the stock SDXL VAE silently produces black or corrupted output in FP16, and an optional 4-bit quantization mode (NF4) for even tighter budgets. A separate Hyper-SD 8-step LoRA cuts generation from the usual 25-50 steps down to 8, for a fast demo mode.

Training checkpoints at 500, 1000, and 1500 steps were compared using two quality scorers, HPSv2.1 and ImageReward, instead of the more common CLIP score, because the project's own experiments found CLIP score can be blind to real quality differences (more on that below). The live demo runs on Cloud Run with an L4 GPU, adds ControlNet for composition control (steering the layout of the generated image from a reference sketch or depth map), and an NSFW safety filter.

A base SDXL model composed with a style LoRA, a speed LoRA, and optional composition control, all fitted inside 8GB of VRAM.

Offline training path (separate from serving): 80 WikiArt ukiyo-e images → diffusers LoRA training → checkpoint sweep at 500/1000/1500 steps, scored with HPSv2.1 + ImageReward on a GCP L4 → checkpoint 1000 selected → published to HuggingFace Hub.

1. FP16 plus a corrected VAE, with NF4 quantization as an optional extra step

   SDXL simply won't load in an 8GB budget without FP16 precision. But FP16 alone isn't enough, the stock SDXL VAE fails silently in FP16, producing black or corrupted images with no error, so the corrected VAE (madebyollin/sdxl-vae-fp16-fix) is treated as a required component, not an optional optimization.

2. HPSv2.1 and ImageReward as the real quality metric, CLIP demoted to comparison-only

   Nine experiments on SD 2.1, and a seven-experiment replication on SDXL, found that CLIP score is structurally blind to rendering-level changes, things like quantization mode, sampling scheduler, or the presence of the style trigger word, while it stays responsive to semantic changes like what's actually depicted in the prompt. That made CLIP unsafe to use as the sole quality gate for engineering decisions.

3. Checkpoint 1000 selected after quality scoring confirmed what the loss curve already flagged

   Training loss jumped roughly 10× between checkpoint 1000 and checkpoint 1500 (0.008 → ~0.08), a warning sign on its own, not by itself proof the model had gotten worse. HPSv2.1/ImageReward scoring plus human review confirmed a real, visible quality regression at checkpoint 1500, so checkpoint 1000 was selected instead, the independent quality check corroborated the loss signal rather than overriding it.

4. Revised its own headline finding downward once a stricter statistical bar was applied

   The project's early claim was that CLIP score was blind to quality changes in 9 out of 9 experiments. Recomputing that claim under a stricter 1-standard-error significance threshold dropped the honest count to 4 out of 9, and the corrected, smaller number is what the project reports, not the more dramatic original one.

5. The validated LoRA recipe: curate, train, pre-register an A/B, then sweep the adapter weight, the sweep is not optional

   A second style adapter (Pattachitra, an Odishan scroll-painting style) followed the same recipe that produced the ukiyo-e adapter: a curated training set, a rank-8 LoRA, and a pre-registered A/B against the un-adapted base model before any publish decision. Evaluating only at the library-default adapter weight (1.0) would have ended the project there, the adapter loses to base on both axes at that weight. Running the same pre-registered comparison across a range of adapter weights instead is what actually found the adapter's usable region.

6. Publish the Pattachitra LoRA at weight 0.3–0.5, and say plainly that the library default (1.0) is the failure mode

   At adapter_weight=1.0, both trained checkpoints significantly regress style adherence and figure preservation against the un-adapted base model, visually confirmed cases where a requested human figure is dropped from the image entirely. At weight 0.3–0.5, the same two checkpoints show a significant style-adherence lift with no figure-preservation regression (a modest improvement, in fact). The model card states both facts side by side: this adapter is good at a documented low weight and actively worse than not using it at the default weight most tooling assumes.

6.2GB, show source for Peak VRAM, full production pipeline (SDXL FP16 + Ukiyo-e LoRA)

Verified source

`AetherArt/README.md:40-46,87`, GCP L4 eval run 2026-06-01 per `docs/lab_notebook.md` line 266

• docs/lab_notebook.md:266 ↗

commit dbde03d · measured 2026-06-01

Peak VRAM, full production pipeline (SDXL FP16 + Ukiyo-e LoRA) · inside the 8GB consumer-GPU budget; optional NF4 4-bit path uses 2.6GB for a single pipeline

HPS 0.239 · ImageReward 1.479, show source for Ukiyo-e LoRA quality (GCP L4, seed 42, 30 steps, 1024×1024, avg. of 4 prompts)

Cited in provenance.md

`docs/lab_notebook.md:258-266`

verified against source 2026-07-31

Ukiyo-e LoRA quality (GCP L4, seed 42, 30 steps, 1024×1024, avg. of 4 prompts) · CLIP score 0.359 shown for comparison only, not as the quality bar

Prompt choice moves CLIP score 18× more than scheduler choice, show source for Scheduler vs. prompt sensitivity (360-run benchmark)

Cited in provenance.md

`README.md:248-252`

verified against source 2026-07-31

Scheduler vs. prompt sensitivity (360-run benchmark) · range 0.130 (prompt) vs. 0.007 (scheduler)

229 tests, ~60 seconds, show source for Test suite

Cited in provenance.md

`README.md:283`

verified against source 2026-07-31

Test suite · heavy GPU dependencies mocked, no GPU required to run the suite

An underfitting paradox, plus an unresolved artifact, show source for Honest caveats

Cited in provenance.md

`README.md:256-260`, `reports/what_didnt_work.md:93-95`

verified against source 2026-07-31

Honest caveats · a rank-4 LoRA scores higher on CLIP than rank-8, and 20 training images beat 80, read as CLIP rewarding literal keyword matching, not real quality; a calligraphy-cartouche artifact baked in from WikiArt caption text remains unresolved, only mitigated by negative prompting

38 · 23 · 0, show source for HF LoRA adapters, 3 live, downloads last 30 days, as of 2026-10-10 (HF API)

Cited in provenance.md

`https://huggingface.co/api/models?author=gauravgandhi2411` → `.downloads` per model (rolling 30 days), generated weekly into `content/generated/hf-downloads.json` by `scripts/refresh-hf-downloads.mjs`; "measured" below is that snapshot's fetch date

measured 2026-10-10

HF LoRA adapters, 3 live, downloads last 30 days, as of 2026-10-10 (HF API) · ukiyo-e SDXL · ukiyo-e SD 2.1 · Pattachitra SDXL (published most recently, hence 0 so far), links below

Withdrawn under corrected scoring, show source for Ukiyo-e curated-retrain promotion

Cited in provenance.md

`AetherArt/docs/MODEL_VERDICT.md` §4.6 (443-541, table at 487-491), §4.7 (542-576, MDE table at 549-555)

verified against source 2026-07-31

Ukiyo-e curated-retrain promotion · 3.18×SEM under single-call judging fell to 0.58×SEM, below the promotion bar, once independent per-axis scoring removed a halo-effect risk; the published checkpoint was not re-promoted.

5, show source for Measurement-defect methodology finding

Cited in provenance.md

`AetherArt/docs/MODEL_VERDICT.md` §7.7 (1419-1483); `AetherArt/docs/paper/measurement_defects.md` (full draft, commit `eeb5f3a`)

verified against source 2026-07-31

Measurement-defect methodology finding · silent evaluation-validity defects found and fixed: context-window truncation, a phantom VRAM counter, CUDA context corruption, a mis-domained judge question, and a stale reused reference score, full writeup linked below.

The natural assumption is that lower-precision weights use less memory. So it was surprising when switching from FP16 to INT8 quantization, with CPU offload active, increased peak VRAM rather than decreasing it, 2,210MB versus 1,803MB, a 407MB increase in the wrong direction.

The mechanism traced back to how the quantization library (bitsandbytes) actually works: it allocates a full FP16 compute buffer to dequantize the INT8 weights on every single forward pass. That dequantization buffer is a real, recurring memory cost, and under CPU offload it outweighs whatever the smaller stored weights save, only NF4's larger 4-bit compression margin is wide enough to still come out ahead in that specific regime.

The project documents this plainly as "not a bug, but a wrong assumption that cost real investigation time," and states the boundary condition rather than a blanket claim: on a GPU with 12GB or more and no CPU offload, INT8 does recover its expected savings; under 8GB with offload active, it doesn't. A related bug surfaced in the same area, a pipeline cache dictionary that kept accumulating multiple loaded quantized pipelines with no eviction policy, caused out-of-memory crashes that looked random until they were traced back to the cache and fixed with a single-slot cache instead.

If you're relying on CLIP score, or any single automated metric, as your quality gate, this is worth testing first: run the experiment that checks whether the metric actually moves with the changes you care about, and hold your own claims to a statistical bar strict enough to survive being wrong.

The same discipline applied to a second style adapter turned up a further lesson: evaluating an adapter at only one operating point (the library default) produced a true but incomplete verdict, and evaluating the evaluation itself, repeatedly, is what caught five separate silent ways the measurements could have been wrong without ever throwing an error.