Reward hacking is a procurement question, not a research one
Test the scoring, not the graphics. An environment is only as useful as its tasks, and tasks fail in one specific way: the agent finds a way to score well that does not do the thing the score was meant to measure. Ask the provider to show you an agent that games one of their own tasks. A provider who has that example has tested their scoring adversarially; one who treats scoring as a configuration step is selling you a game with an API in front of it.
Everyone evaluating an environment asks about throughput, observation spaces and uptime. Those are checkable and they are rarely where a deal goes wrong. What goes wrong is the scoring, because a task that can be gamed produces training signal that looks excellent and teaches the wrong thing.
An agent asked to deliver loads quickly learns to clip through scenery. An agent scored on profit finds the pricing bug. The behaviour is correct by the rules it was given and useless by the intent behind them. Reward hacking is the known weakness of this whole product category, which means it belongs on the purchase checklist and not only in the research plan.
Four things to ask before signing
- Show me an agent gaming one of your tasks. A provider who has tested their own scoring adversarially will have the example ready, probably with some pride. A provider who has not will say it has not come up.
- What does the task measure that the score does not? Every scored task is a proxy. The useful answer names the gap rather than denying it.
- Can we write our own tasks, and against what interface? If scoring is sealed, every change to your research plan becomes a change request to someone else.
- What happens when we find a hack six months in? The answer should be a process, not goodwill.
Why this is a procurement question
Because the scoring is most of what the money buys. Making a commercial game run headless is engineering with a known shape; designing tasks that reward the behaviour you actually want, and that survive an agent looking for the cheapest path, is judgement. It is the part that takes the longest and the part a provider is most tempted to under-sell, because it does not demo well.
So a provider who treats scoring as a configuration step has told you what they are selling: a game with an API in front of it. That is a real product and it is cheaper. Just do not buy it expecting the other one. The questions to ask an environment provider cover the rest.