Motivation

We continuously run into issues with the pipeline that are caused by an updated dependency. Dependencies currently can just update on the fly in an uncontrolled manner and we don't know when a dependency receives an upgrade.
This can obscure other issues with the pipeline or otherwise reduce the development velocity.

Changes

This PR and its companion in the Ext repository introduce dependency pinning in LocalStack Ext and Core.

To this end, new make targets are added which allow for the creation of lock files with the naming scheme requirements-<extra>.txt. These make targets can either be executed in CI or locally by devs:

• In the case of CI, this PR introduces a new workflow upgrade-python-dependencies which periodically (Mondays at 5AM - same as the ASF updates) and in a controlled manner update all the dependencies in the lock file to latest allowable versions defined in setup.cfg

• In the case of devs, this PR introduces a new pre-commit hook, referenced from the https://github.com/localstack/pre-commit-hooks repository, which checks whether a change in setup.cfg would make the lock files incompatible with the setup.cfg. If that is the case, the hook would fail and the dev will be prompted to run the make target for upgrading all dependency lock files. As an extra precaution, the run of this pre-commit hook was additionally added to the CI pipeline in the linting step.

Testing

To test out how the pinning works you can execute the upgrade-pinned-dependencies make target.
You can add any package you wish to the setup.cfg and stage it and try to commit it to check if the pre-commit hook works.
You can also manually run the pre-commit hook with the staged changes in the setup.cfg:

pre-commit run check-pinned-deps-for-needed-upgrade --files $(git diff --name-only --cached)

TODO

What's left to do:

• Make sure all CI pipelines are green in the PR
• Make sure the release process is solid
• Decide on what to do with the eager updating in the Docker build
• Run the dependency upgrade before merge to get off to a good start ;)