Rendered at 19:12:06 GMT+0000 (Coordinated Universal Time) with Cloudflare Workers.
tensor 1 days ago [-]
Amazing. I think this is an understudied areas. The big labs if anything are disincentivized to make efficient models as currently access to compute is one of their moats. That's not the same for Chinese labs, which is why you see groundbreaking efficiency work mostly coming from China.
Really happy to see a small shop like yours doing this sort of work!
blueblazin 16 hours ago [-]
That makes no sense. The big labs don’t reveal their efficiency research but that doesn’t mean they aren’t doing it. They can charge the same prices while making models more efficient internally and increase margins…
darkwater 12 hours ago [-]
I bet they do that as well but I also think that if they have to choose between inference speed or "intelligence" they would choose the latter.
yorwba 9 hours ago [-]
They don't have to choose, though, since they can release multiple models with different tradeoffs.
darkwater 3 hours ago [-]
This would imply they have infinite development capacity
yorwba 48 minutes ago [-]
Not infinite, just enough to make more than one model. OpenAI and Anthropic both have four different size categories...
andai 12 hours ago [-]
Well done. $20k? That's insane. I remember tinyllama back in the day (2023) was trained on $40k and could barely form coherent sentences.
Are pictures easier than English, or did the methods advance so much?
Edit: I guess I should have said, are pictures plus English easier than English!
>The text representation comes from a frozen Qwen3.5-4B model, which we read at layers 7, 15 and 27 (its three full-attention layers). We concatenate and project these representations into the text refiner.
Still, as a layperson I would have expected text to be simpler than images, so I'm still amazed by this.
wongarsu 7 hours ago [-]
Over the last 3 years we made a lot of progress in efficient training methods. On text models, you can now train GPT-2 levels of coherent English from scratch for about $50 (e.g. using Karpathy's nanochat [1])
That's insane. How does that work? Does it take so much less data now, or smaller models, or what? Apparently GPT-2 cost ~$40,000 to train when it came out.
I'm sure Moore's Law for GPUs helped a little, but I'm really curious about the savings on the methodology side. (The repo mentions "hyperparameters")
mkl 21 hours ago [-]
It looks really good! Except almost every hand is wrong. I'm guessing that's a training data thing? Can you tell us more about the 660M training images and prompts? Where did they come from and what do they cover? (I've seen the README say text was excluded.)
fzysingularity 21 hours ago [-]
Congrats guys, this is super cool. I think if you built an arena where you can drive the cost even lower ($1K), it'd really get developers stoked.
Have you thought about setting up auto-research on this for smaller experiments? What's the minimum-viable budget for these types of workloads?
What was the most difficult challenge for you guys with this project?
Also I'm a little intrigued on looking into efficiency techniques for video gen. What are current research directions in that field and promising directions you are particularly interested in? Any pointers for beginners?
yhjc2692 22 hours ago [-]
I think there's a whole plethora of research on efficient attention implementations (similar to how DeepSeek & Kimi has made strides with theirs in LLMs) [1], [2]. And various techniques to reduce the flow-matching / diffusion iterations ("reflow") [3].
CVPR 2026 paper list (after filtering on video generation) might be a good start - asking LLMs to take a look and filter relevant papers sound even better.
Thanks! And sorry, I was just curious to get some opinions on it from people who have worked with that kind of stuff. I feel like someone who has worked on it in a serious capacity will be a better filter than LLM (I don't know enough to provide the right keywords that would give me high signal papers). However, I realized I should probably just read a survey to get some overview on the state if I can't get any answers.
jjcm 15 hours ago [-]
Very nice all. I've been playing around having poor results of training a pixel space image model of my own, though in a space very opposite from what you're doing (UI design, very heavy on text).
Any chance you'll release your training recipe as well? I know the model code is public, but I'd love to dive into the full process.
yhjc2692 22 hours ago [-]
Wow this is really impressive. Throwing away the VAE feels counterintuitive - I'm shocked it works so well! Appreciate that you shared your training insights on the blog post!
Reminds me of this great paper where they trained a LLM that beats LLaMA2-13B from scratch with < $0.1M: https://arxiv.org/abs/2404.07413
Tarq0n 4 hours ago [-]
Does it? Text generation models have been decoder-only for a while, with great benefits to their training. It's surprising to me that more attempts weren't made for other modalities.
jcattle 5 hours ago [-]
$100k seems like a lot. I guess it isn't in the grand scheme of things but still
gunalx 1 days ago [-]
Interesting. It does however seem to struggle with a lot of the typical t2i models struggle with.
Over smoothing, lots of uncanney shine, mangled limbs and such.
dvh 1 days ago [-]
How many ladybugs are in training dataset?
Robin_Message 23 hours ago [-]
What are you using as your training data out of interest? Very impressive generation.
jcattle 5 hours ago [-]
I'm always wondering that. Where do people get 660 million labelled images from?
robotswantdata 23 hours ago [-]
What open source datasets did you use? Or did you pirate like the big labs ?
micw 23 hours ago [-]
Impressive. Even the average number of fingers seems to be correct ;-)
Looking at the failures, I wonder how much cherry picking went into the showcase.
sebion7125 20 hours ago [-]
This is awesome. Thanks for sharing weights on HF :-)
pruthvishetty 10 hours ago [-]
Bravo!
gigel82 1 days ago [-]
I wonder how this behaves when quantized, and how far are we from being able to run something like this purely in the browser with WebGPU.
If you swap the text encoder for something like T5-small or CLIP, this could be <2.5Gb at INT4; doable on a lot of machines.
apishare_api 19 hours ago [-]
[flagged]
yajan2 6 hours ago [-]
[dead]
vancekai 8 hours ago [-]
[flagged]
Atlas-Compiler 14 hours ago [-]
[flagged]
looplilt 16 hours ago [-]
[flagged]
Upbiddr 20 hours ago [-]
The focus on iteration speed rather than just impressive individual outputs is refreshing. For animation, being able to revise one part without changing everything else seems especially important. How are you thinking about that kind of control as you move from this image model toward video?
Really happy to see a small shop like yours doing this sort of work!
Are pictures easier than English, or did the methods advance so much?
Edit: I guess I should have said, are pictures plus English easier than English!
>The text representation comes from a frozen Qwen3.5-4B model, which we read at layers 7, 15 and 27 (its three full-attention layers). We concatenate and project these representations into the text refiner.
Still, as a layperson I would have expected text to be simpler than images, so I'm still amazed by this.
1: https://github.com/karpathy/nanochat
I'm sure Moore's Law for GPUs helped a little, but I'm really curious about the savings on the methodology side. (The repo mentions "hyperparameters")
Have you thought about setting up auto-research on this for smaller experiments? What's the minimum-viable budget for these types of workloads?
Also I'm a little intrigued on looking into efficiency techniques for video gen. What are current research directions in that field and promising directions you are particularly interested in? Any pointers for beginners?
CVPR 2026 paper list (after filtering on video generation) might be a good start - asking LLMs to take a look and filter relevant papers sound even better.
[1] https://arxiv.org/abs/2609.15810 [2] https://arxiv.org/abs/2602.13515 [3] https://arxiv.org/abs/2605.23381
Any chance you'll release your training recipe as well? I know the model code is public, but I'd love to dive into the full process.
Reminds me of this great paper where they trained a LLM that beats LLaMA2-13B from scratch with < $0.1M: https://arxiv.org/abs/2404.07413
If you swap the text encoder for something like T5-small or CLIP, this could be <2.5Gb at INT4; doable on a lot of machines.