Rendered at 12:05:42 GMT+0000 (Coordinated Universal Time) with Cloudflare Workers.
Good4boothee 4 hours ago [-]
I think ambiguously worded questions have their place, but you have to give up automatic scoring on such tasks.
vintermann 3 hours ago [-]
You don't have to give it up entirely, you could grade on the interpretation they chose. Personally I'm a fan of models which seek clarification on critical ambiguities rather than barging ahead "autonomously" though.
But wow, they found a lot of issues with the benchmark.
scotty79 2 hours ago [-]
I really like the benchmarks where frontier models score each other's solutions to a problem.
But wow, they found a lot of issues with the benchmark.