Skip to content

Build Lambda functions for their deployed runtime; check Python syntax; skip unchanged functions at dev start - #342

Merged
matuscongrady merged 4 commits into
mainfrom
fix/python-runtime-packaging
Oct 27, 2026
Merged

matuscongrady merged 4 commits into
mainfrom
fix/python-runtime-packaging

Conversation

@matuscongrady

@matuscongrady matuscongrady commented Oct 26, 2026 •

Copy link
Copy Markdown
Contributor

Fixes findings 1, 2 and 8 of the third dev-mode dogfood run (r2p).

Python, Java, Ruby and .NET Lambda functions now build for the runtime they are deployed with. A function without runtime got its runtime from the first entry of its language's list in the template, but the packager received undefined and used its own default. A Python function then ran on python3.14 with wheels built for 3.12, and every request failed with Runtime.ImportModuleError as soon as a native package such as psycopg, pydantic-core or numpy was imported. Ruby (ruby4.0 runtime, 3.3 build, so gems sit in vendor/bundle/ruby/3.3.0), Java (java25/11) and .NET (dotnet10/8) had the same split. The packager now resolves the runtime with getLambdaRuntime, the function the template uses. The four Lambda buildpacks require a version and no longer have defaults of their own. fn-deployment now passes the configured runtime as well; before, dev rebuilds and deployment scripts built a pinned function with the default version too.

Python now defaults to python3.13, not python3.14. Native wheels are built per Python minor version, and many pins from before late 2025 have 3.13 wheels but no 3.14 ones: psycopg-binary==3.2.3 (the version the dogfood agent wrote, and it has no source distribution either), pydantic-core 2.27.2, numpy 2.2.6. A 3.14 default would turn those runtime import errors into failed installs. 3.13 is also the v3 default runtime, so v3 stacks keep their runtime when they move to v4. Java, Ruby and .NET keep their newest defaults (java25, ruby4.0, dotnet10); their dependencies do not depend on the minor version the way wheels do. The buildpack page and the v3 upgrade guide state the defaults.

What existing stacks do on their next deploy: a v4-preview stack with an unpinned Python function changes it from python3.14 to python3.13 in place through CloudFormation, and the code is rebuilt for 3.13, because the resolved runtime is part of the digest. Verified live: the stack deployed by origin/main failed every invoke; the redeploy with this branch took 28 s, updated 2 resources in place, and the invoke returned {"python": "3.13", "pq": "binary"}. Pinned functions keep their digest. Unpinned Java and Ruby functions on v3 stacks move to java25/ruby4.0; that change already came with the v4 runtime list, and now the build follows the runtime.

Python builds now check the packaged .py files with the function's Python version. A small checker runs in the build image before dependencies are installed (bundlers/py/compile-check.ts). It follows the entry file's imports through the project's own files: absolute imports from the build root and from the entry file's directory, and relative imports, including package __init__.py files. It compiles every .py file outside virtual environments and hidden directories in memory, so no bytecode is written, and records the result. Only a file the entry file imports, directly or through other project files, fails the build. The error names the file and line, for example api/main.py, line 103: SyntaxError: invalid syntax, so deploy stops early and dev answers REBUILD_FAILED. Live: 0.6 s, and the function kept serving its previous code (r2p got ok: true after 5.7 s, then 500s). Any other file that does not compile, such as a script, a test fixture, a template or vendored Python 2 code, produces a warning and the build continues. Files left out with excludeFiles are ignored. Imports made at run time (importlib) are not followed, so errors there stay warnings. I did not import the handler in the build image to catch missing modules: importing runs module-level code (database connections, environment reads), which would fail good builds.

Dev starts:

  • A function whose deployed artifact digest (its artifactDigest tag) matches the build is skipped. The build stops after computing the digest, and nothing is uploaded or updated. On a cold start the Python function was skipped 8 ms after the dev stack deploy; Node functions are still redeployed once there because the stack split-bundles them. Warm starts took 5.6 s with all three functions skipped (r2p: 22–31 s).
  • A function that fails to build or deploy at start is marked error with the reason, and the session keeps running, because the dev stack still holds the function's previous code. The start is not reported as a success. dev --agent exits with DEV_STARTED_WITH_FAILURES, the errors in message, and the session in data.result with ready: false and failedWorkloads also at the top of data; it prints no AGENT_READY. A dev --agent that finds a running session with a failed workload returns the same code, and DEV_SESSION_NOT_READY while that session still starts. CliError can now carry data into the JSONL result for this. The MCP start keeps the session, returns ok: false with that code and message, data.session, data.ready: false, data.failedWorkloads, and nextActions (fix, then rebuild each failed workload with this session ID, test after it returns ok). Containers and other workloads that fail still stop the session with CLI_DEV_WORKLOAD_FAILED, as before. The agent-mode docs describe both cases.
  • After such a failure, and after a failed rebuild (each attempt of Make dev mode robust: daemon output, Lambda log reads and levels, no dev alarms, MCP take-back, stop/start #343's retry), the watcher follows the function's full source set. The language builds (Python, Java, Ruby, .NET, Go, Rust) now report their source files before they build (reportSourceFiles). Dev mode records them per job, because the function starts run in parallel and each clears the packaging jobs when it finishes. A JavaScript bundle that fails before it knows its files still leaves only the entry file.
  • The Python Lambda build image goes through the same daily pull and local digest pin as other generated Dockerfiles. Before, it was only in BuildKit's cache, so every build asked public.ecr.aws for the tag, and a short registry timeout failed starts with DOCKER_COMMAND_FAILED. I did not hard-code a digest in source: that would freeze the image's security updates and need a bump for each release. Java, Ruby and .NET build images do not use the pin yet.

Checks:

  • New apps/cli/scripts/packaging-archives/python-lambda-acceptance.ts (test:python-lambda, in test:runtime-acceptances, registered under S21). It packages an unpinned and a python3.12 function with psycopg[binary]==3.2.3 through the source CLI, reads the runtime from the template, checks the .so ABI tags and invokes both ZIPs in the official Lambda images. Then a syntax error in a module the handler imports must stop package with its file and line. After the fix, a broken script the handler does not import must only be reported. All 9 checks pass (27 s warm). On origin/main, the original 8-check version failed 4 checks with the r2p error.
  • language-lambdas: 17 pass, 0 fail (58 s). New test, with the checker on Python 3.14: a syntax error in a module reached through a package __init__.py stops packaging with file and line, and the failed build still reports that module among its source files. After the fix, a Python 2 script, a tab error in a test fixture and a Jinja template give one warning; the function runs in lambda/python:3.14; the artifact has no bytecode or report file; with those files in excludeFiles there is no warning.
  • packaging-images (Docker smoke): 23 real Docker builds passed (176 s).
  • Project qualification, import and package lanes, for the two Python Lambda projects in the corpus (serverless-python-http-dynamodb, serverless-python-sqs-worker): 2/2 passed.
  • rebuild-results.test.ts (real agent server): a running session with a failed build is reported as DEV_STARTED_WITH_FAILURES, and the error's data reaches the JSONL result. dev-stop.test.ts (real CLI process against a real agent): a session that still starts gives exit 1, DEV_SESSION_NOT_READY with the session in data.result and no AGENT_READY; once it is ready, the same command gives ok: true. dev-tool.spec.ts: the MCP start returns the code, ready: false, the failed workloads and next actions, and the session takes a rebuild and a stop.
  • A one-off probe: Maven and Gradle reactors built for Java 25 run in lambda/java:25 (2/2).
  • @stacktape/packaging test 278/278, CLI test:src 185 files with 0 failures, CLI and packaging typecheck, oxlint, fmt:check, generate:check (LLM docs regenerated), check:docs, check:workspace, check:patterns, check:deadcode, typecheck:tools, pnpm test:tools 85/85. check:duplicates reports one clone in the private Console checkout (console/ui/e2e/ui-capture), which this change does not touch.
  • Live in eu-central-1 (stacktape-e2e): the deploy and redeploy above; dev cold start, warm starts, a syntax-error rebuild and a start with a syntax error on a copy of the r2p app. Second round through the real MCP server (stacktape mcp with the app as its root):
    • A warm start with a syntax error in api/helpers.py, which api/main.py imports, returned ok: false, DEV_STARTED_WITH_FAILURES, ready: false, the file and line, and next actions.
    • A second start that found the running session returned the same.
    • With --watch, fixing only api/helpers.py rebuilt api (2.9 s), and status became ready: true.
    • A cold start with the same error fails with BUILD_CODE_FAILED, because the dev stack itself has to package the function and there is no previous code.
  • Everything I created was deleted and checked both times: stacks, dev network, instances, ENIs, security groups, SSM parameters, log groups, functions, secret, roles, rules.

Uncertain:

  • The import walk is static: a module loaded with importlib, or found only through a sys.path change, is not followed, so its syntax error is a warning, not a build failure.
  • .NET: the project's target framework still has to match the runtime. I believe a net8.0 project does not run on dotnet10 without roll-forward, with or without this change; I did not test it.

matuscongrady and others added 4 commits October 26, 2026 01:23
…[skip ci]

A function without `runtime` was deployed with the first runtime of its language but built with a separate packaging
default, so a Python function ran python3.14 with wheels for 3.12 and failed every request on a native import. The
packager now resolves the runtime with the template's `getLambdaRuntime` and the Lambda buildpacks require a version.
Python defaults to python3.13: many pinned native packages have no 3.14 wheels, and v3 already used it.

Python builds compile the packaged sources, so a syntax error fails packaging with its file and line. Their build image
uses the daily pull and local digest pin other generated Dockerfiles use.

Dev starts skip functions whose deployed digest matches the build, and a function that fails to build at start is
reported as `error` while the session keeps running.

Co-Authored-By: Claude Opus 5.5 <[email protected]>
… ci]

A dev start whose function failed to build kept the session but returned `ok: true`, so an MCP client tested the
function's previous code. `dev --agent` now fails with `DEV_STARTED_WITH_FAILURES` (or `DEV_SESSION_NOT_READY` while a
found session still starts), carries the session with `ready: false` in its result data and prints no `AGENT_READY`.
The MCP start keeps the session, reports the code, the failed workloads and next actions.

The Python build check no longer fails on every packaged file: it follows the entry file's imports through the
project, and only those files stop the build. Other files that do not compile produce a warning, and files left out
with `excludeFiles` are ignored. A failed language build reports its source files, so dev mode rebuilds the function
when any of them changes.

Co-Authored-By: Claude Opus 5.5 <[email protected]>
…rallel [skip ci]

Each function start clears the packaging manager's jobs when it finishes. A function that finished first removed the
source files a failed one had reported, so the watcher followed only its entry file. The files are now taken per job.

Co-Authored-By: Claude Opus 5.5 <[email protected]>
Combines #343 (dev runtime robustness) with this branch. A failed function rebuild now also hands the files it read to
the watcher, through each attempt of #343's retry, and the not-ready MCP start result works with session selection by
stage.

Co-Authored-By: Claude Opus 5.5 <[email protected]>
@matuscongrady
matuscongrady merged commit 3b3aba6 into main Oct 27, 2026
@matuscongrady
matuscongrady deleted the fix/python-runtime-packaging branch October 27, 2026 12:12
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant