Skip to content

Fix timeout parsing and add bounded Ollama generation controls - #303

Open
jamesjennens wants to merge 1 commit into
evalplus:masterfrom
jamesjennens:fix/ollama-timeouts-max-tokens
Open

jamesjennens wants to merge 1 commit into
evalplus:masterfrom
jamesjennens:fix/ollama-timeouts-max-tokens

Conversation

@jamesjennens

Copy link
Copy Markdown

This PR fixes two reliability issues I hit while running HumanEval with the Ollama backend, and adds token-limit plumbing that is already used by decoders but was not exposed through run_codegen.

What changed

  1. Fix EVALPLUS_TIMEOUT_PER_TASK parsing in evaluation
  • In evalplus/eval/__init__.py, os.getenv() returns a string.
  • Existing code compared that string directly with a float via min(...), which can raise TypeError.
  • This PR safely parses the env var as float with a fallback to 60.0.
  1. Expose max_new_tokens in codegen path
  • Added max_new_tokens parameter to run_codegen(...).
  • Wired it through make_model(...) and into backend decoders.
  • This makes token limits configurable from CLI/API callers instead of relying only on decoder defaults.
  1. Respect max_new_tokens for Ollama decoder
  • OllamaChatDecoder previously overwrote self.max_new_tokens to -1 (unlimited).
  • This PR lets Ollama use the configured value passed from codegen/provider.
  1. Add bounded Ollama request controls to prevent indefinite loops
  • In evalplus/gen/util/ollama_request.py, added env-configurable guardrails:
    • EVALPLUS_OLLAMA_TIMEOUT (default 180)
    • EVALPLUS_OLLAMA_MAX_ATTEMPTS (default 4)
    • EVALPLUS_OLLAMA_MAX_TOTAL_SECONDS (default 900)
  • This prevents retries from effectively running forever on pathological generations.
  • Also uses total request elapsed time for timeout checks and returns collected streamed output when stream ends without buffered tail.

Why

During long HumanEval runs with Ollama, generation could stall for very long periods around a single task due to retry behavior and lack of hard wall-clock caps. Also, task timeout env parsing had a type bug that could fail when set.

Backward compatibility

  • Defaults preserve prior behavior unless users set new env vars.
  • Existing callers remain valid; max_new_tokens has a default.
  • No breaking CLI changes.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant