Accepted episodes. Not hours worked.
Recorded human demonstration is becoming a commodity sold by the hour, and an hour of unusable footage costs a lab more than it saves. This is what benned accepts as an episode, what it rejects, and what it refuses to charge for — written down before the first engagement rather than after it.
Eight things, all checkable: synchronised streams within 10 milliseconds, valid calibration, hands and workpiece visible for at least 90% of the episode, an outcome judged against a written tolerance by someone who does the trade, a plain-language instruction, complete phase segmentation, a privacy pass, and start and end conditions that match the task. benned treats these as acceptance criteria rather than good intentions: an episode that fails any one of them is not delivered and not billed.
| Criterion | What it means |
|---|---|
| All required streams present | Full duration, fewer than 1% dropped frames. |
| Synchronised within 10 ms | Demonstrable from the start signal, not asserted. |
| Valid calibration | Taken at the start and end of the session, agreeing within 2 mm. |
| The work is visible | Hands and workpiece in view in at least one stream for 90% of the episode. |
| Outcome judged against a tolerance | Measured against the trade’s own standard by someone who does the trade. |
| Instruction and phases complete | One plain-language instruction, every phase bounded. |
| Privacy pass | No identifiable faces, badges, client papers, number plates or addresses. |
| Start and end conditions match | The episode is the one the task family describes. |
A rate per hour is not a price.
The real cost of an hour is the quoted rate divided by one minus the rejection rate, and the rejection rate is the number a supplier has least reason to tell you. Quoting per hour therefore moves the quality risk onto the buyer and calls it a discount.
Published estimates put fully loaded teleoperation data at roughly $118 to $136 an hour in 2026, down from around $340 in early 2024 — figures from vendors and consultancies that conflict with one another, so read them as direction rather than price. The direction is clear enough: anyone whose only product is hours is in a race they will lose.
- Rejected is not invoicedAn episode that fails one of the eight does not appear in the delivery or on the bill.
- Every batch reports its rejectionsOffered, accepted, rejected and the reason for each, with the variation matrix filled in.
- Raw files always includedA buyer is never locked into our conversion choices or our annotation of them.
Three hundred identical episodes is one episode, billed three hundred times.
Variation is agreed before capture starts and demonstrated at delivery. Per task family, at minimum:
| Dimension | Minimum |
|---|---|
| Material | At least 4 distinct materials |
| Size and weight | At least 3 classes |
| Tool | At least 2 variants for the same action |
| Operator | At least 3 people, one of them left-handed |
| Posture | Standing, bent, and above shoulder height |
| Light | Daylight, artificial and mixed |
| Substrate | At least 3 surfaces |
| Outcome | 15 to 25% failed or recovered |
Recovery is the part nobody collects.
Nearly every dataset on the market is made of successful attempts, because suppliers are paid for successful attempts. The result is models that have never seen a mistake, and mistakes are precisely what a robot meets on its first real day.
A skilled worker who sets a hole wrong and recovers it demonstrates something that appears in no clean dataset anywhere. We go looking for those instead of filtering them out, and the worker is paid the same for them — otherwise they quietly disappear from the data again.
Where this sits in what benned does- A failure is a valid episodeCorrectly captured and annotated as failed, it is delivered and paid for like any other.
- The trade judges the outcomeNot an annotator who has never done the work. At the end of the day, on their own footage.
- Never to the employerThe footage cannot reach the employer or be used to assess anyone. People who are judged perform instead of work.
A licence from the business, consent from each person.
European regulators hold that an employee cannot give freely-chosen consent to their own employer, because of the power imbalance. benned asks for that consent in writing anyway, per person, but does not lean on it as the legal basis — that sits with benned as controller, with a documented assessment. Faces are made unrecognisable before material leaves the site, not afterwards in a cloud. Audio is off by default.
A dataset assembled this way is an entry a model provider can cite rather than a gap it has to explain, which is the same argument that applies to a licensed game under the EU AI Act. Provenance is not paperwork around the product. On this side of the business it is most of the product.
About the standard
What makes robot demonstration data usable?
Synchronised streams within 10 milliseconds, valid camera calibration, the hands and the workpiece visible for at least 90% of the episode, an outcome judged against a written tolerance by someone who does the trade, a plain-language instruction, complete phase segmentation, and a privacy pass. benned treats those as acceptance criteria rather than aspirations: an episode that fails any one of them is not delivered and not billed.
Why bill accepted episodes instead of hours?
An hourly rate tells a buyer nothing until they know the rejection rate, because the real cost is the quoted rate divided by one minus that rate. Billing accepted episodes moves that risk to the supplier, where it belongs, and gives the buyer a number they can plan against.
Should a dataset contain failures?
Yes, and benned targets 15 to 25% failed or recovered episodes. Almost every commercial dataset is all successes, because suppliers are paid for successes. A model that has only seen things go right has no idea what to do when they go wrong, which is exactly where robots fail in the field. Recovery is the scarce part.
Who owns the recording, and does the employer see it?
No. The footage never goes to the employer in any form and can never be used to assess, compare or discipline anyone. That clause is in the site licence and in each worker’s consent. It is as much a quality measure as an ethical one: people who are being judged perform rather than work, and performed work is worthless as training data.
Has benned collected data to this standard yet?
No. There is no signed capture site. This is the standard benned commits to and will be held to, published before the first engagement rather than after it, so that a buyer and a site owner can both read it in advance.