fix: detect return-value deception via dunder override (same exploit class as HumanEval) - #306

Summary

A model can return a custom object whose __eq__ always returns True, causing all comparisons (out == exp) and special oracles in EvalPlus to pass without solving the problem. A model can score 100% on HumanEval+ and MBPP+ by returning _AlwaysEqual() from every function.

This is the same exploit class found in HumanEval (openai/human-eval#67), now confirmed against EvalPlus's hardened test suite — including the special oracles.

The exploit

class _AlwaysEqual:
    def __eq__(self, other): return True
    def __sub__(self, other): return _AlwaysEqual()
    def __abs__(self): return 0.0

def solve(args):
    return _AlwaysEqual()

EvalPlus then does (eval/__init__.py:155-157):

out = fn(*inp)           # Returns _AlwaysEqual()
exact_match = out == exp # _AlwaysEqual().__eq__(exp) → True

Every comparison passes. The special oracles (abs(out - x) <= atol) also pass because __sub__ and __abs__ are overridden.

Verified

Fix

Add a type() identity check before the comparison: if type(out) is not type(exp), treat as a mismatch (skip the == comparison). Allows int/float cross-comparison for math tasks.

Verified

• _AlwaysEqual() vs bool → BLOCKED (type mismatch) ✅
• True vs bool → allowed (correct) ✅
• 42 vs float → allowed (numeric cross-type) ✅

Scope

This affects HumanEval+, MBPP+, and any EvalPlus-evaluated benchmark. The fix is in the core evaluation loop, protecting all datasets automatically.

Related

• fix: detect return-value deception via dunder override (model can score 100% without solving problems) openai/human-eval#67 — same exploit class, same fix pattern in the original HumanEval.

Checklist

• Code compiles (py_compile passes)
• Fix verified: exploit blocked, correct implementations unaffected
• Numeric cross-type comparison preserved for math tasks