Transitions to GH action and resolves fork PR issue - #43
Conversation
Codecov Report
@@ Coverage Diff @@
## main #43 +/- ##
=========================================
Coverage 80.71% 80.71%
Complexity 245 245
=========================================
Files 27 27
Lines 752 752
Branches 56 56
=========================================
Hits 607 607
Misses 98 98
Partials 47 47
Flags with carried forward coverage won't be shown. Click here to find out more. Continue to review full report at Codecov.
|
| fail_ci_if_error: true | ||
| verbose: true | ||
| flags: integration | ||
| - name: push docker image |
There was a problem hiding this comment.
This should be build docker image -> ./gradlew dockerBuildImages
| uses: actions/setup-java@v1 | ||
| with: | ||
| java-version: 14 | ||
| - name: Cache Gradle |
There was a problem hiding this comment.
I think, we are missing downloading deps, right?
populate_and_save_cache:
description: 'Downloads all gradle dependencies and uploads cache for later use'
steps:
- gradle:
args: downloadDependencies
- save_cache:
paths:
- ~/.gradle
key: v1-dependencies-{{ checksum "/tmp/checksum.txt" }}
| - '**/*.md' | ||
| pull_request: | ||
| branches: | ||
| - main |
There was a problem hiding this comment.
This should be for all the branches including forks.
| - main | ||
| paths-ignore: | ||
| - '**/*.md' | ||
| release: |
There was a problem hiding this comment.
This will only trigger whenever we cut the release from their UI. How about if someone pushes the tag directly.
| # In GH action, we can setup remote docker but then we have to host our own docker setup somewhere and SSH into that. | ||
| # Installing docker every time eats a bit of a time at the moment but | ||
| - name: Install Docker | ||
| uses: docker-practice/actions-setup-docker@master |
There was a problem hiding this comment.
can we have a fixed version for this action docker-practice/actions-setup-docker@master ?
| strategy: | ||
| matrix: | ||
| docker-version: [19.03] | ||
| runs-on: ubuntu-latest |
There was a problem hiding this comment.
Here as well, we should use the fixed version for OS
| # GitHub will remove any cache entries that have not been accessed in over 7 days. | ||
| # There is no limit on the number of caches you can store, but the total size of all caches in a repository is limited to 5 GB. | ||
| # Note: GitHub Actions does not have an equivalent of CircleCI’s Docker Layer Caching (or DLC). | ||
| - name: Cache docker |
There was a problem hiding this comment.
What is the value of this cache key?
| - name: Build with Gradle | ||
| run: ./gradlew build dockerBuildImages | ||
|
|
||
| - name: push docker image |
There was a problem hiding this comment.
We should not run dockerPushImages in context of pull_request_target.
| verbose: true | ||
| flags: integration | ||
|
|
||
| build: |
There was a problem hiding this comment.
Can we make this job as build-and-test? And move both unit-test and integration steps as part of this job only?
| - main | ||
| paths-ignore: | ||
| - '**/*.md' | ||
| pull_request: |
There was a problem hiding this comment.
We will need pull_request_target here, right?
| @@ -0,0 +1,44 @@ | |||
| name: Build and merge-publish | |||
There was a problem hiding this comment.
Curious about the difference between master-build and pr-build?
As pr-build is also getting run on trigger - on:push, right?
The only difference I see here is that dockerPushImages. can we have if condition in pr-build that it happens only for merged to main. So, we don't need this entire workflow which is the same. Does this make sense?
There was a problem hiding this comment.
That is how it was earlier
There was a problem hiding this comment.
reverting it back to earlier state. We discussed having separate merge-publish so this was done.
| publish-helm-charts: | ||
| runs-on: ubuntu-20.04 | ||
| container: hypertrace/helm-gcs-packager:0.3.0 | ||
| needs: publish-images |
There was a problem hiding this comment.
Also needs validate-helm-charts right?
| paths-ignore: | ||
| - '**/*.md' | ||
| pull_request: | ||
| branches: |
There was a problem hiding this comment.
I assume this is the target rather than source branch - any reason to restrict it to main? Running on any pull request seems sufficient.
Further, even though it's unnecessary, is there really any significant benefit to excluding markdown files as triggers? It only takes (or at least should) a few minutes to run, and the added complexity of having filter rules doesn't seem worth it to me. Particularly because this would mean we can no longer require status checks on protected branches, as a md-only change wouldn't meet that requirement.
There was a problem hiding this comment.
Okay. I agree. This will be required check so it doesn't make sense to have it here.
| - main | ||
| paths-ignore: | ||
| - '**/*.md' | ||
| create: |
There was a problem hiding this comment.
This is on any branch or tag creation - we don't need that, do we?
| paths-ignore: | ||
| - '**/*.md' | ||
| create: | ||
| pull_request_target: |
There was a problem hiding this comment.
Does this replace pull_request or is it in addition to? if it's in addition to, it seems like it should have the same filter rules
There was a problem hiding this comment.
I removed filters from both. And yes, it's in addition to it. So pull_request will fail on forks but pull_request_target will pass
| - name: Publish docker image | ||
| run: ./gradlew publish dockerPushImages | ||
|
|
||
| validate-helm-charts: |
There was a problem hiding this comment.
This has already been run, right? For publish, we're not re-testing - we're assuming the code is valid to have made it to this point.
| uses: actions/cache@v2 | ||
| with: | ||
| path: ~/.gradle | ||
| key: gradle-packages-${{ runner.os }}-${{ hashFiles('**/checksum.txt') }} |
There was a problem hiding this comment.
missing restore keys
| HELM_GCS_CREDENTIALS: ${{ secrets.HELM_GCS_CREDENTIALS }} | ||
| HELM_GCS_REPOSITORY: ${{ secrets.HELM_GCS_REPOSITORY }} | ||
| # GITHUB_REF will be refs/tags/docker-MAJOR.MINOR.PATCH | ||
| CHART_VERSION: $(echo ${GITHUB_REF} | cut -d/ -f 3) |
There was a problem hiding this comment.
Can this execute bash code? I assume this needs to be a constant value since we're outside the run context
|
|
||
| - name: package and release charts | ||
| env: | ||
| HELM_GCS_CREDENTIALS: ${{ secrets.HELM_GCS_CREDENTIALS }} |
There was a problem hiding this comment.
We can easily avoid putting secrets in the env here - is it any safer? If there were a script in the below run block, the secret would be compromised. Can we inline them instead to lower the possibly exposures (assuming GH doesn't expose them via the commands being run)?
There was a problem hiding this comment.
Changed this because of https://docs.github.com/en/free-pro-team@latest/actions/reference/encrypted-secrets#using-encrypted-secrets-in-a-workflow .
here they mentioned
Avoid passing secrets between processes from the command line, whenever possible. Command-line processes may be visible to other users (using the ps command) or captured by security audit events. To help protect secrets, consider using environment variables, STDIN, or other mechanisms supported by the target process.
There was a problem hiding this comment.
This is interesting to me - you're right, they say to do it this way, and they're right, that traditionally you don't want to pass sensitive data via CLI. The part that I'm caught on though, is that this is running in an isolated container (so we don't have to worry about other processes or users seeing via ps, at least I think?), but on the other hand, environment variables are available to everything executing in this step so if a script were executing, it could be modified to read the environment variable.
I'm certainly open to be convinced, but my suspicion is that those docs aren't considering the full picture here. If we can't find any other info on it, my recommendation would be to use the STDIN approach and pipe the secrets (for example, our old docker login command was echo $DOCKERHUB_PASSWORD | docker login --username $DOCKERHUB_USERNAME --password-stdin) - that way neither concern applies.
There was a problem hiding this comment.
@JBAhire - you're right, because this is not an external workflow, the secret protection can be a little looser - this is fine.
| - main | ||
| paths-ignore: | ||
| - '**/*.md' | ||
| pull_request_target: |
There was a problem hiding this comment.
See pr build comments about triggers
| @@ -0,0 +1,30 @@ | |||
| name: validate charts | |||
| on: | |||
| push: | |||
There was a problem hiding this comment.
See pr build comments about triggers
| jobs: | ||
| validate-helm-charts: | ||
| runs-on: ubuntu-20.04 | ||
| container: hypertrace/helm-gcs-packager:0.3.0 |
There was a problem hiding this comment.
Should we be providing pull credentials for dockerhub images to avoid rate limiting, or does GH have some work around for that?
|
But one more thing here is this is a publish workflow so it will run only
when we release and not on forked PRs or any external PR for that matter.
That’s why I felt it’s fine.
We can pass them in run as you want and as it’s running in container it
won’t most probably cause any issue. I was just bit skeptical about this
after reading docs.
…On Fri, 18 Dec 2020 at 7:18 AM, Aaron Steinfeld ***@***.***> wrote:
***@***.**** commented on this pull request.
------------------------------
In .github/workflows/publish.yml
<#43 (comment)>
:
> + steps:
+ # Set fetch-depth: 0 to fetch commit history and tags for use in version calculation
+ - name: Checkout Repository
+ uses: ***@***.***
+ with:
+ fetch-depth: 0
+
+ - name: Login to Docker Hub
+ uses: ***@***.***
+ with:
+ username: ${{ secrets.DOCKERHUB_USERNAME }}
+ password: ${{ secrets.DOCKERHUB_TOKEN }}
+
+ - name: package and release charts
+ env:
+ HELM_GCS_CREDENTIALS: ${{ secrets.HELM_GCS_CREDENTIALS }}
This is interesting to me - you're right, they say to do it this way, and
they're right, that traditionally you don't want to pass sensitive data via
CLI. The part that I'm caught on though, is that this is running in an
isolated container (so we don't have to worry about other processes or
users seeing via ps, at least I think?), but on the other hand,
environment variables *are* available to everything executing in this
step so if a script were executing, it could be modified to read the
environment variable.
I'm certainly open to be convinced, but my suspicion is that those docs
aren't considering the full picture here. Maybe as a middle ground, we use
the STDIN approach and pipe the secrets (for example, our old docker login
command was echo $DOCKERHUB_PASSWORD | docker login --username
$DOCKERHUB_USERNAME --password-stdin
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
<#43 (comment)>,
or unsubscribe
<https://github.com/notifications/unsubscribe-auth/AGKW2PGRLDJH27EZW67RPFLSVKYFVANCNFSM4UQGYLUQ>
.
|
| flags: integration | ||
|
|
||
| - name: push docker image | ||
| if: github.event.pull_request.merged == true |
There was a problem hiding this comment.
nit: assuming github.event.pull_request.merged is already a boolean, the == true part isn't needed.
| push: | ||
| branches: | ||
| - main | ||
| pull_request: |
There was a problem hiding this comment.
validate charts should run in pull_request_target right?
|
|
||
| publish-helm-charts: | ||
| runs-on: ubuntu-20.04 | ||
| container: hypertrace/helm-gcs-packager:0.3.0 |
There was a problem hiding this comment.
missing creds like in validate-helm-charts
| flags: integration | ||
|
|
||
| - name: push docker image | ||
| if: github.event.pull_request.merged |
There was a problem hiding this comment.
Will this work if forked PR merged to main?
There was a problem hiding this comment.
yup. It will work. pull_request_target also has a same webhook pull_request.
| - name: Run Snyk to check for vulnerabilities | ||
| uses: snyk/actions/[email protected] | ||
| env: | ||
| SNYK_TOKEN: ${{ secrets.SNYK_TOKEN }} |
There was a problem hiding this comment.
Can you move this form env to with as this runs as part of pull_request_target?
There was a problem hiding this comment.
Nope. Doesn't work in that way.
| with: | ||
| fetch-depth: 0 | ||
|
|
||
| - name: Login to Docker Hub |
There was a problem hiding this comment.
Do we have to login here?
There was a problem hiding this comment.
Will remove this in separate PR as we want to test it and owner's approval is require.
| # ref: https://github.com/snyk/actions/tree/master/gradle-jdk11 | ||
| name: snyk for gradle | ||
| on: | ||
| push: |
| @@ -0,0 +1,29 @@ | |||
| name: validate charts | |||
There was a problem hiding this comment.
The validate-charts and pr-build workflow are having the same trigger condition. Can we move this job as part of pr-build.yml workflow?

Description
resolves hypertrace/hypertrace#132 and POC for CI with GitHub action.
Testing
e2e flow is working.
Documentation