Writing · Comparisons
How I Test AI Video Generators (and Why Most Benchmarks Lie)
Cherry-picked demo reels sell subscriptions. A repeatable test sheet tells you what you're actually paying for.
Every AI video tool launches with a reel. Thirty seconds of drone shots over misty mountains, a woman turning toward the camera with perfect hair physics, a dog running through a field in slow motion. The reels are gorgeous. They are also the least useful thing you can look at when deciding which tool to pay for.
A launch reel is the best five clips out of a few hundred attempts, picked by people who know exactly what the model is good at. It tells you the ceiling. What you need to know is the floor: what you get on an ordinary Tuesday, with an ordinary prompt, on your third try.
So here's the method I use for every video comparison I write. Steal it.
1. The same ten prompts, every time
I keep a fixed sheet of ten prompts and run every tool against all of them. They're deliberately boring, because boring prompts are what people actually type:
- A person walking toward the camera and stopping to talk
- Two people having a conversation at a café table
- Hands doing something precise (tying shoelaces, typing, chopping vegetables)
- A car turning a corner in the rain
- A pet doing something mildly chaotic
- Text on a sign in the background
- A slow pan across a crowded room
- A character from a reference image, in a new location
- A single continuous action longer than five seconds
- A shot that cuts to a reaction close-up
Notice what's on that list: hands, text, conversation, continuity. Those are the things that break.
2. Three generations per prompt, no rerolls beyond that
I run each prompt three times and score all three. Not the best one. All three. The average tells you what your workflow will feel like. If a tool needs eight attempts to land a usable clip, that's a cost, whether it's billed in credits or in your evening.
3. Score what matters, not what impresses
Each clip gets a score from 1 to 5 on five axes:
| Axis | What I'm checking |
|---|---|
| Prompt adherence | Did it do what I asked, in the order I asked? |
| Motion | Do bodies move like bodies? Any limb soup? |
| Identity | Does the character look like the same person start to finish? |
| Physics | Do objects have weight? Does liquid behave? |
| Usability | Could I put this in an actual edit without apologizing? |
"Usability" is the one most reviews skip, and it's the one I weight most heavily. A clip can score well on everything else and still have a two-frame glitch in the middle that makes it unusable.
4. Count the real cost per usable second
This is the number I wish every review led with. Take what you paid, divide by the seconds of footage you'd actually keep. A tool with a cheaper plan and a 30% keep rate is often more expensive than a pricier tool that lands 70% of the time.
5. Test the boring features
Can you extend a clip? Can you set the first and last frame? Can you upload a reference character and keep it? Can you export without a watermark at the tier you're on? What's the queue time at 9pm on a Sunday? None of this shows up in a reel. All of it shows up in your week.
Why benchmarks lie
Public leaderboards aren't useless, but they usually measure preference on short clips with short prompts, judged in isolation. That rewards visual polish. It doesn't reward consistency across a sequence, which is what anyone telling a story actually needs.
The best tool for a single beautiful shot and the best tool for a forty-second scene with the same two characters are frequently not the same tool. Any review that doesn't tell you which one it's testing for is giving you half an answer.
The short version
Ignore the reel. Run your own ten prompts. Score every generation, not the best one. Divide money by usable seconds. You'll be surprised how often the leaderboard winner isn't your winner.
— Marc
American tech writer covering AI tool comparisons and AI entertainment. Disagree, or tested something I haven't? Tell me.
