Fix podman tests - #1312

Summary

Remove the manual installation of the Podman 5.5.2 static bundle from the Podman test job and use the Podman stack provided by the GitHub-hosted Ubuntu runner.

This resolves the failed to reexec: Permission denied failure observed in the Podman test job for #1303.

Background

The Podman test job manually installed a static Podman 5.5.2 bundle on ubuntu-latest. Although this replaced the Podman executable and some related files, the tests continued to depend on other components and configuration supplied by the runner image.

Podman is not an isolated executable. Rootless container builds depend on a compatible stack that includes:

• Podman
• Buildah
• crun or runc
• conmon
• Netavark and aardvark-dns
• Rootless networking helpers
• Storage and container configuration
• Runtime state directories
• Kernel behavior and AppArmor policy

Installing only a static Podman bundle over the runner-provided environment resulted in a mixed runtime stack that was not guaranteed to be compatible.

Failure

The affected job failed while running the Podman integration tests with:

failed to reexec: Permission denied

The tests invoke the Dev Containers CLI with:

--docker-path podman

The failure occurred in the Podman runtime environment rather than in the Dev Containers CLI logic.

Why It Started Failing

The workflow relied on implementation details of the moving ubuntu-latest runner image. A runner-image or host-policy update appears to have exposed an existing incompatibility between the downloaded static Podman bundle and the runtime components, paths, and confinement policy supplied by Ubuntu.

The exact runner-image revision that triggered the regression was not isolated. However, the observed behavior is consistent with the following sequence:

1. The workflow installed a static Podman bundle outside the runner’s distro-managed runtime stack.
2. The remaining runtime components and configuration continued to come from the runner image.
3. Podman attempted a rootless re-exec using this mixed environment.
4. The runner’s runtime or AppArmor policy rejected the operation with Permission denied.

This indicates an environment-integration problem rather than a regression in the Dev Containers CLI.

Investigation

Several approaches for retaining a manually pinned Podman version were evaluated:

• Adjusting the Podman binary location to align with Ubuntu’s expected AppArmor paths.
• Installing a matching version of crun.
• Installing the complete Podman 6.1.2 static bundle.
• Supplying matching storage, networking, and runtime helpers.
• Running Buildah with BUILDAH_ISOLATION=chroot.

None produced a reliable configuration on the GitHub-hosted runner:

• The original setup failed during Podman re-exec.
• Combining static Podman with the runner-provided crun produced runtime-version compatibility errors.
• Replacing crun led to rootless runtime-state permission errors under /run/user.
• Installing the complete static bundle continued to conflict with runner confinement.
• Using chroot isolation avoided the immediate runtime error but caused builds to hang and eventually time out.

These results demonstrate that replacing individual components is insufficient. A pinned Podman version requires a complete, internally compatible runtime environment.

Resolution

Remove the manual Podman 5.5.2 installation step and use the Podman installation supplied by the GitHub-hosted Ubuntu runner.

This keeps Podman aligned with the runner’s:

• OCI runtime
• Buildah installation
• conmon
• Networking helpers
• Storage configuration
• AppArmor profiles
• Kernel configuration
• Rootless-user setup
• Runtime state directories

The existing Tools Info step remains in place and reports the actual Podman and Docker Buildx versions used by CI, preserving visibility into the test environment.

Test Coverage

The existing Podman integration tests remain unchanged and continue to exercise:

• devcontainer up --docker-path podman
• Image-based dev container configurations
• Dockerfile-based dev container configurations
• Feature installation for image-based configurations
• Feature installation for Dockerfile-based configurations
• Successful container creation
• Container inspection and cleanup

No test assertions, behavior, or timeout values are changed by this PR.

Tradeoff

This change means CI no longer pins Podman to version 5.5.2. Instead, the Podman tests run against the distro-integrated version available on ubuntu-latest.

This trades exact version pinning for a runtime stack that is internally consistent and supported by the runner environment.

If coverage for an exact Podman version becomes mandatory, it should be implemented using a dedicated runner or VM image containing a complete and validated Podman stack. Replacing Podman binaries within a GitHub-hosted runner is not a reliable version-pinning mechanism because the runtime also depends on host configuration and multiple tightly coupled components.

The current ubuntu-latest GH runner is ubuntu-24.04 which has the default podman engine version 4.9.3 which works for devcontainer build. In future if the ubuntu-latest runner is updated to ubuntu-26.04 and continues with similar AppArmor restriction as the current one, the podman test pipeline will get blocked with the issue fixed in #1280 because the in-built podman version is expected for that is > 5.5.2 and < 6.1.0 unless a fix received from upstream in GH runners.

Test Plan

• Confirm the Tools Info step reports the runner-provided Podman version.
• Confirm podman info completes successfully.
• Confirm the image-based Podman integration test passes.
• Confirm the Dockerfile-based Podman integration test passes.
• Confirm image-based Feature installation succeeds.
• Confirm Dockerfile-based Feature installation succeeds.
• Confirm containers are created, inspected, and cleaned up successfully.
• Confirm the Podman test matrix completes without re-exec permission errors.
• Confirm the test matrix completes without runtime hangs or Mocha timeouts.

Scope

This PR only removes the manual Podman installation from the test workflow. It does not change:

• Dev Containers CLI behavior
• Podman integration test coverage
• Test assertions
• Test timeout values
• The --docker-path podman execution path
• Production dependencies or runtime behavior