Skip to content

Update deployment code and dependencies#351

Merged
Bill-Becker merged 4 commits into
developfrom
werf-1-2
Aug 18, 2022
Merged

Update deployment code and dependencies#351
Bill-Becker merged 4 commits into
developfrom
werf-1-2

Conversation

@GUI
Copy link
Copy Markdown
Member

@GUI GUI commented Aug 9, 2022

  • Update deployments to use Werf 1.2 which brings various improvements and simplifies some things.
  • This upgrade to the underlying deployment image also brings a newer version of kubectl which I think is why some recent deployments have been failing against upgraded kubernetes clusters (since the old deployment code wasn't using a new enough version of kubectl tor rancher's cli).
  • Store the deployed images in ECR so that we can also implement a new image cleanup job. This helps cleanup unused images from old deploys from the image repository so the storage needs don't grow indefinitely.
  • Address some issues with ingress version and annotations needing to be upgraded for new versions of kubernetes. But note that we currently need slightly different approaches for dev and staging/prod due to different Kubernetes versions these clusters are running. But once staging/prod are on the same version of Kubernetes as dev, then this will need to be updated so they all use the same approach.

I think these changes will need to be merged into any branches you want to fix the deploys on. So not totally sure how you're branching strategy is working and whether you want this in developer or master or how that might propagate to other branches.

@GUI
Update deployment code and dependencies.
6e4fe9a 
- Update deployments to use Werf 1.2 which brings various improvements
  and simplifies some things.
- This upgrade to the underlying deployment image also brings a newer
  version of kubectl which I think is why some recent deployments have
  been failing against upgraded kubernetes clusters (since the old
  deployment code wasn't using a new enough version of kubectl tor
  rancher's cli).
- Store the deployed images in ECR so that we can also implement a new
  image cleanup job. This helps cleanup unused images from old deploys
  from the image repository so the storage needs don't grow
  indefinitely.
- Address some issues with ingress version and annotations needing to be
  upgraded for new versions of kubernetes. But note that we currently
  need slightly different approaches for dev and staging/prod due to
  different Kubernetes versions these clusters are running. But once
  staging/prod are on the same version of Kubernetes as dev, then this
  will need to be updated so they all use the same approach.
@CLAassistant
Copy link
Copy Markdown

CLAassistant commented Aug 9, 2022

CLA assistant check
Thank you for your submission! We really appreciate it. Like many open source projects, we ask that you sign our Contributor License Agreement before we can accept your contribution.
You have signed the CLA already but the status is still pending? Let us recheck it.

@adfarth adfarth requested a review from Bill-Becker August 11, 2022 15:17
Bill-Becker
Bill-Becker requested changes Aug 16, 2022
View reviewed changes
Copy link
Copy Markdown
Collaborator

@Bill-Becker Bill-Becker left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@GUI @adfarth I scrolled through all the code changes, but it's outside of my understanding in general. I'm trusting Nick on those. The one thing I tried to do was undeploy the PR 351 (should be equivalent to werf-1-2, I think) because werf-1-2 was already deployed to development. The undeploy failed with "Error: unable to load werf config: template: werfConfig:8:26: executing "werfConfig" at : error calling env: the environment variable "XPRESS_LICENSE_HOST" "

I'm not sure how to debug from here, but seems like XPRESS_LICENSE_HOST env var is not getting set, or something like that.

@adfarth
Copy link
Copy Markdown
Collaborator

adfarth commented Aug 16, 2022

@Bill-Becker, @GUI made changes in the nova-analysis2 branch that address this undeploy issue. Nick mentioned he would pull those changes into here while making the resource limit changes as well.

@GUI
Add resource requests/limits. Add topology spread constraints.
e673ac7 
- Add resource requests and limits to try and better guard against
  resource exhaustion and deal with potential memory growth over time.
- Add a topology spread constraint to try and make sure pods are
  deployed more evenly across available nodes for better redundancy.
@GUI
Copy link
Copy Markdown
Member Author

GUI commented Aug 16, 2022

Yes, the environment variable issue should be fixed by this commit I pulled back into this branch: b6acb51

I've also added configuration to setup resource requests/limits as we had talked about to try and help with potential memory spikes: e673ac7

@Bill-Becker Bill-Becker merged commit 51be5f3 into develop Aug 18, 2022
@Bill-Becker Bill-Becker deleted the werf-1-2 branch August 18, 2022 16:28
@Bill-Becker Bill-Becker mentioned this pull request Aug 18, 2022
Merged
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@Bill-Becker Bill-Becker Bill-Becker approved these changes

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

4 participants

@GUI @CLAassistant @adfarth @Bill-Becker