Skip to content

Transitions to GH action and resolves fork PR issue - #43

Merged
JBAhire merged 83 commits into
mainfrom
github-ci-test
Dec 18, 2020
Merged

JBAhire merged 83 commits into
mainfrom
github-ci-test

Conversation

@JBAhire

@JBAhire JBAhire commented Dec 7, 2020 •

Copy link
Copy Markdown
Member

Description

resolves hypertrace/hypertrace#132 and POC for CI with GitHub action.

Testing

e2e flow is working.

Documentation

@codecov

codecov Bot commented Dec 7, 2020 •

Copy link
Copy Markdown

Codecov Report

Merging #43 (16efd37) into main (b540503) will not change coverage.
The diff coverage is n/a.

Impacted file tree graph

@@            Coverage Diff            @@
##               main      #43   +/-   ##
=========================================
  Coverage     80.71%   80.71%           
  Complexity      245      245           
=========================================
  Files            27       27           
  Lines           752      752           
  Branches         56       56           
=========================================
  Hits            607      607           
  Misses           98       98           
  Partials         47       47           
Flag Coverage Δ Complexity Δ
integration 69.51% <ø> (ø) 0.00 <ø> (ø)
unit 67.77% <ø> (ø) 0.00 <ø> (ø)

Flags with carried forward coverage won't be shown. Click here to find out more.


Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update b540503...16efd37. Read the comment docs.

Comment thread .github/workflows/build.yml Outdated
fail_ci_if_error: true
verbose: true
flags: integration
- name: push docker image

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This should be build docker image -> ./gradlew dockerBuildImages

Comment thread .github/workflows/build.yml Outdated
uses: actions/setup-java@v1
with:
java-version: 14
- name: Cache Gradle

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think, we are missing downloading deps, right?

populate_and_save_cache:
    description: 'Downloads all gradle dependencies and uploads cache for later use'
    steps:
      - gradle:
          args: downloadDependencies
      - save_cache:
          paths:
            - ~/.gradle
          key: v1-dependencies-{{ checksum "/tmp/checksum.txt" }}

Comment thread .github/workflows/build.yml Outdated
- '**/*.md'
pull_request:
branches:
- main

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This should be for all the branches including forks.

Comment thread .github/workflows/build.yml Outdated
- main
paths-ignore:
- '**/*.md'
release:

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This will only trigger whenever we cut the release from their UI. How about if someone pushes the tag directly.

Comment thread .github/workflows/build.yml Outdated
# In GH action, we can setup remote docker but then we have to host our own docker setup somewhere and SSH into that.
# Installing docker every time eats a bit of a time at the moment but
- name: Install Docker
uses: docker-practice/actions-setup-docker@master

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

can we have a fixed version for this action docker-practice/actions-setup-docker@master ?

Comment thread .github/workflows/build.yml Outdated
strategy:
matrix:
docker-version: [19.03]
runs-on: ubuntu-latest

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Here as well, we should use the fixed version for OS

Comment thread .github/workflows/build.yml Outdated
# GitHub will remove any cache entries that have not been accessed in over 7 days.
# There is no limit on the number of caches you can store, but the total size of all caches in a repository is limited to 5 GB.
# Note: GitHub Actions does not have an equivalent of CircleCI’s Docker Layer Caching (or DLC).
- name: Cache docker

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

What is the value of this cache key?

Comment thread .github/workflows/master-build.yml Outdated
- name: Build with Gradle
run: ./gradlew build dockerBuildImages

- name: push docker image

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We should not run dockerPushImages in context of pull_request_target.

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

done

Comment thread .github/workflows/pr-build.yml Outdated
verbose: true
flags: integration

build:

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can we make this job as build-and-test? And move both unit-test and integration steps as part of this job only?

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

reverted to this

- main
paths-ignore:
- '**/*.md'
pull_request:

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We will need pull_request_target here, right?

Comment thread .github/workflows/master-build.yml Outdated
@@ -0,0 +1,44 @@
name: Build and merge-publish

@kotharironak kotharironak Dec 16, 2020 •

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Curious about the difference between master-build and pr-build?
As pr-build is also getting run on trigger - on:push, right?

The only difference I see here is that dockerPushImages. can we have if condition in pr-build that it happens only for merged to main. So, we don't need this entire workflow which is the same. Does this make sense?

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

That is how it was earlier

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

reverting it back to earlier state. We discussed having separate merge-publish so this was done.

Comment thread .github/workflows/publish.yml Outdated
publish-helm-charts:
runs-on: ubuntu-20.04
container: hypertrace/helm-gcs-packager:0.3.0
needs: publish-images

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Also needs validate-helm-charts right?

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

yup

Comment thread .github/workflows/pr-build.yml Outdated
paths-ignore:
- '**/*.md'
pull_request:
branches:

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I assume this is the target rather than source branch - any reason to restrict it to main? Running on any pull request seems sufficient.

Further, even though it's unnecessary, is there really any significant benefit to excluding markdown files as triggers? It only takes (or at least should) a few minutes to run, and the added complexity of having filter rules doesn't seem worth it to me. Particularly because this would mean we can no longer require status checks on protected branches, as a md-only change wouldn't meet that requirement.

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Okay. I agree. This will be required check so it doesn't make sense to have it here.

Comment thread .github/workflows/pr-build.yml Outdated
- main
paths-ignore:
- '**/*.md'
create:

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is on any branch or tag creation - we don't need that, do we?

paths-ignore:
- '**/*.md'
create:
pull_request_target:

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Does this replace pull_request or is it in addition to? if it's in addition to, it seems like it should have the same filter rules

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I removed filters from both. And yes, it's in addition to it. So pull_request will fail on forks but pull_request_target will pass

Comment thread .github/workflows/publish.yml Outdated
- name: Publish docker image
run: ./gradlew publish dockerPushImages

validate-helm-charts:

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This has already been run, right? For publish, we're not re-testing - we're assuming the code is valid to have made it to this point.

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

yup

Comment thread .github/workflows/publish.yml Outdated
uses: actions/cache@v2
with:
path: ~/.gradle
key: gradle-packages-${{ runner.os }}-${{ hashFiles('**/checksum.txt') }}

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

missing restore keys

Comment thread .github/workflows/publish.yml Outdated
HELM_GCS_CREDENTIALS: ${{ secrets.HELM_GCS_CREDENTIALS }}
HELM_GCS_REPOSITORY: ${{ secrets.HELM_GCS_REPOSITORY }}
# GITHUB_REF will be refs/tags/docker-MAJOR.MINOR.PATCH
CHART_VERSION: $(echo ${GITHUB_REF} | cut -d/ -f 3)

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can this execute bash code? I assume this needs to be a constant value since we're outside the run context


- name: package and release charts
env:
HELM_GCS_CREDENTIALS: ${{ secrets.HELM_GCS_CREDENTIALS }}

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We can easily avoid putting secrets in the env here - is it any safer? If there were a script in the below run block, the secret would be compromised. Can we inline them instead to lower the possibly exposures (assuming GH doesn't expose them via the commands being run)?

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Changed this because of https://docs.github.com/en/free-pro-team@latest/actions/reference/encrypted-secrets#using-encrypted-secrets-in-a-workflow .
here they mentioned

Avoid passing secrets between processes from the command line, whenever possible. Command-line processes may be visible to other users (using the ps command) or captured by security audit events. To help protect secrets, consider using environment variables, STDIN, or other mechanisms supported by the target process.

@aaron-steinfeld aaron-steinfeld Dec 18, 2020 •

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is interesting to me - you're right, they say to do it this way, and they're right, that traditionally you don't want to pass sensitive data via CLI. The part that I'm caught on though, is that this is running in an isolated container (so we don't have to worry about other processes or users seeing via ps, at least I think?), but on the other hand, environment variables are available to everything executing in this step so if a script were executing, it could be modified to read the environment variable.

I'm certainly open to be convinced, but my suspicion is that those docs aren't considering the full picture here. If we can't find any other info on it, my recommendation would be to use the STDIN approach and pipe the secrets (for example, our old docker login command was echo $DOCKERHUB_PASSWORD | docker login --username $DOCKERHUB_USERNAME --password-stdin) - that way neither concern applies.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@JBAhire - you're right, because this is not an external workflow, the secret protection can be a little looser - this is fine.

- main
paths-ignore:
- '**/*.md'
pull_request_target:

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

See pr build comments about triggers

@@ -0,0 +1,30 @@
name: validate charts
on:
push:

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

See pr build comments about triggers

Comment thread .github/workflows/validate_charts.yml Outdated
jobs:
validate-helm-charts:
runs-on: ubuntu-20.04
container: hypertrace/helm-gcs-packager:0.3.0

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Should we be providing pull credentials for dockerhub images to avoid rate limiting, or does GH have some work around for that?

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

yeah

@JBAhire

JBAhire commented Dec 18, 2020 via email •

Copy link
Copy Markdown
Member Author

Comment thread .github/workflows/pr-build.yml Outdated
flags: integration

- name: push docker image
if: github.event.pull_request.merged == true

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nit: assuming github.event.pull_request.merged is already a boolean, the == true part isn't needed.

push:
branches:
- main
pull_request:

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

validate charts should run in pull_request_target right?

Comment thread .github/workflows/publish.yml Outdated

publish-helm-charts:
runs-on: ubuntu-20.04
container: hypertrace/helm-gcs-packager:0.3.0

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

missing creds like in validate-helm-charts

flags: integration

- name: push docker image
if: github.event.pull_request.merged

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Will this work if forked PR merged to main?

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

yup. It will work. pull_request_target also has a same webhook pull_request.

ref: https://docs.github.com/en/free-pro-team@latest/actions/reference/events-that-trigger-workflows#pull_request_target

- name: Run Snyk to check for vulnerabilities
uses: snyk/actions/[email protected]
env:
SNYK_TOKEN: ${{ secrets.SNYK_TOKEN }}

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can you move this form env to with as this runs as part of pull_request_target?

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nope. Doesn't work in that way.

with:
fetch-depth: 0

- name: Login to Docker Hub

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Do we have to login here?

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Will remove this in separate PR as we want to test it and owner's approval is require.

@JBAhire
JBAhire merged commit fc6f400 into main Dec 18, 2020
@JBAhire
JBAhire deleted the github-ci-test branch December 18, 2020 08:11
# ref: https://github.com/snyk/actions/tree/master/gradle-jdk11
name: snyk for gradle
on:
push:

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The snyk and pr-build workflow are having the same condition. Can we move this job as part of pr-build.yml workflow?

As an example, below is the current scenario.
Screenshot 2020-12-18 at 1 59 00 PM

May less workflow makes it easy to look for

  1. pr-build (build-and-test)
  2. publish (release)

@@ -0,0 +1,29 @@
name: validate charts

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The validate-charts and pr-build workflow are having the same trigger condition. Can we move this job as part of pr-build.yml workflow?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Fix ENV variable issue for snyk on forked branch PR by non-member

5 participants