lambda-25/stacktape

This branch is one test of the deploy benchmarks: the Lambda backend with all 25 functions, deployed with Stacktape. It holds only that application and the tool's configuration, so it deploys as it is.

node branches.mjs on main generates this branch from main's fixtures and tool modules. Don't edit it here: change main and generate the branches again.

The Stacktape CLI built from source of stacktape/stacktape at commit 028ef1a7, as described in the main README. Each result records the commit (stacktapeCommit).

• AWS_PROFILE: a profile for the disposable AWS account. The harness uses it to call the deployed code and read stack outputs.
• STACKTAPE_API_KEY, or run stacktape login once.
• STACKTAPE_AWS_ACCOUNT: the name of the AWS account connection in the Stacktape Console. Stacktape deploys with credentials from that connection, not with AWS_PROFILE, so check with stacktape info:whoami that both point to the same account.
• STACKTAPE_BIN: the path of the Stacktape binary to measure.

You need Node.js 22 or newer, npm, Git, Docker with BuildKit, zip, the AWS CLI, a disposable AWS account and a Stacktape account with the AWS account connected. The AWS resources are created in eu-west-1. Run the harness from main, with the credentials above:

git clone https://github.com/stacktape/deploy-benchmarks && cd deploy-benchmarks
npm ci
node bench.mjs setup   stacktape lambda-25                             # write work/bmk-stacktape-lambda-25/ and deploy it once (not measured)
node bench.mjs package stacktape lambda-25 --runs 10                   # package from an empty build cache
node bench.mjs package stacktape lambda-25 --runs 10 --warm            # package after a source change
node bench.mjs deploy  stacktape lambda-25 --runs 10                   # deploy a code change from this machine
node bench.mjs deploy  stacktape lambda-25 --runs 10 --change shared   # change the shared service package instead
node bench.mjs deploy  stacktape lambda-25 --runs 10 --change mixed    # change handlers and their shared helper together
node report.mjs                                                        # median, fastest and slowest run
node bench.mjs remove  stacktape lambda-25                             # delete everything setup created

setup writes the files of this branch to work/bmk-stacktape-lambda-25/ and commits them as the same commit that this branch has.

Each run is a pair: a first push with an empty build cache, then a second push 15 minutes after the first one finished. The clock starts when git push returns.

1. Fork the repository with all branches: untick "Copy the main branch only".
2. Deploy this branch once with stacktape deploy from a terminal and answer yes to "Set up automatic deployments on push". This connects the repository and this branch to the project with the stage bmk in eu-west-1. Stacktape's GitHub app must have access to the repository.
3. Every push then deploys on the project's runner in your AWS account: an EC2 instance with a cache disk, launched for the first push and hibernated after 15 idle minutes. The runner installs the Stacktape version set for the project in the Console from npm unless its image already contains that version. A build from source is not on npm; the main README describes how we measured one.
4. To empty the runner's caches before each first push, remove the project's runner (its instance, cache disk and record) and pass --cold-reset manual. The Console can't do this yet; see the main README.
5. Run the commands below. The project in work/ has the same commit as this branch, so its pushes continue your fork's branch.

export BENCH_FORK=https://github.com/<you>/deploy-benchmarks   # setup adds it as the project's origin
node bench.mjs setup stacktape lambda-25                       # skip if you already ran it with BENCH_FORK set
node bench.mjs push stacktape lambda-25 --runs 10 --cold-reset manual
node report.mjs

From a checkout of this branch, without the harness:

npm ci
stacktape deploy --stage bmk --region eu-west-1 --awsAccount <connection-name> --autoConfirmOperation
# delete everything:
stacktape delete --stage bmk --region eu-west-1 --awsAccount <connection-name> --autoConfirmOperation

Each run appends one JSON line to results/package.jsonl or results/deploy.jsonl. Times are in milliseconds and sizes in bytes. Every line also records the tool version, the machine and the command log.

{ "app": "lambda-25", "tool": "stacktape", "cache": "clean", "run": 1, "ms": <packaging time>, "bytes": <upload size>, "artifacts": [{ "file": "...", "bytes": ... }] }
{ "app": "lambda-25", "tool": "stacktape", "journey": "cli", "run": 1, "change": "handlers", "changedTargets": 25, "readyMs": <time until the new code answers>, "commandMs": <deploy command duration>, "businessOk": true, "unchangedOk": true }
{ "app": "lambda-25", "tool": "stacktape", "journey": "git-cold", "run": 1, "change": "handlers", "changedTargets": 25, "readyMs": <time until the new code answers>, "coldReset": "manual", "build": <build records from the platform>, "businessOk": true, "unchangedOk": true }
{ "app": "lambda-25", "tool": "stacktape", "journey": "git-warm", "run": 1, "change": "handlers", "changedTargets": 25, "readyMs": <time until the new code answers>, "build": <build records from the platform>, "businessOk": true, "unchangedOk": true }

node report.mjs prints the median with the fastest and slowest run of each series. Runs whose business check failed are listed and left out.

Pending. The results will be published in the main README after the measurement runs on a dedicated EC2 machine.