Skip to content

Upgraded amass to v4#1864

Merged
Ilyesbdlala merged 33 commits intomainfrom
bugfix/amass-parser
Sep 15, 2023
Merged

Upgraded amass to v4#1864
Ilyesbdlala merged 33 commits intomainfrom
bugfix/amass-parser

Conversation

@Ilyesbdlala
Copy link
Member

@Ilyesbdlala Ilyesbdlala commented Aug 18, 2023 •
edited
Loading

Description

Closes #1833

Checklist

  • Test your changes as thoroughly as possible before you commit them. Preferably, automate your test by unit/integration tests.
  • Make sure that all your commits are signed-off and that you are added to the Contributors file.
  • Make sure that all CI finish successfully.
  • Optional (but appreciated): Make sure that all commits are Verified.

@Ilyesbdlala Ilyesbdlala added the scanner Implement or update a security scanner label Aug 18, 2023
@Ilyesbdlala Ilyesbdlala self-assigned this Aug 18, 2023
@github-actions
Copy link

github-actions bot commented Aug 18, 2023 •
edited
Loading

🦙 MegaLinter status: ⚠️ WARNING

Descriptor Linter Files Fixed Errors Elapsed time
⚠️ ACTION actionlint 1 3 0.07s
⚠️ DOCKERFILE hadolint 2 1 0.07s
⚠️ GO golangci-lint 2 1 0.89s
⚠️ JAVASCRIPT eslint 4 1 0.72s
✅ JSON eslint-plugin-jsonc 3 0 1.35s
⚠️ YAML prettier 7 1 2.12s

See detailed report in MegaLinter reports
Set VALIDATE_ALL_CODEBASE: true in mega-linter.yml to validate all sources, not only the diff

MegaLinter is graciously provided by OX Security

@Zero3141 Zero3141 mentioned this pull request Aug 23, 2023
Closed
@Ilyesbdlala Ilyesbdlala force-pushed the bugfix/amass-parser branch 2 times, most recently from 5e5e030 to 9d95f66 Compare August 25, 2023 14:55
@Ilyesbdlala Ilyesbdlala marked this pull request as ready for review August 25, 2023 15:10
J12934
J12934 reviewed Aug 29, 2023
View reviewed changes
Copy link
Member

@J12934 J12934 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nice 👍🥳
some small questions / notes

@Ilyesbdlala Ilyesbdlala requested a review from J12934 September 1, 2023 14:10
@Zero3141 Zero3141 mentioned this pull request Sep 4, 2023
Closed
This was referenced Sep 8, 2023
Merged
Closed
@netlify
Copy link

netlify bot commented Sep 12, 2023 •
edited
Loading

✅ Deploy Preview for docs-securecodebox ready!

Name Link
🔨 Latest commit 36f0614
🔍 Latest deploy log https://app.netlify.com/sites/docs-securecodebox/deploys/6504235efee6290007af17a9
😎 Deploy Preview https://deploy-preview-1864--docs-securecodebox.netlify.app
📱 Preview on mobile
Toggle QR Code...

QR Code

Use your smartphone camera to open QR code link.

To edit notification comments on pull requests, go to your Netlify site configuration.

Ilyesbdlala and others added 17 commits September 12, 2023 15:05
@Ilyesbdlala
#1833 Reimplemented amass parser to use the sqlite database instead …
4a2cbba 
…of the JSON output

This is done since the JSON output is no longer supported

Signed-off-by: Ilyes Ben Dlala <[email protected]>
@Ilyesbdlala
#1833 Updated amass parser tests and snapshot
62426ae 
It now uses sqlite databases as test-files

Signed-off-by: Ilyes Ben Dlala <[email protected]>
@Ilyesbdlala
#1833 Added sqlite3 dependency in new package.json for amass parser
b067cfa 
Signed-off-by: Ilyes Ben Dlala <[email protected]>
@Ilyesbdlala
#1833 Modified Amass Parser Dockerfile to include copying installed …
e37e3d9 
…dependencies

This is done because sqlite3 package is now required

Signed-off-by: Ilyes Ben Dlala <[email protected]>
@Ilyesbdlala
#1833 Added Dockerfile to build custom amass scanner image
9718172 
Signed-off-by: Ilyes Ben Dlala <[email protected]>
@Ilyesbdlala
#1833 Added custom_scanner flag to amass makefile
e903511 
This triggers building custom scanner image in ./scanner/Dockerfile

Signed-off-by: Ilyes Ben Dlala <[email protected]>
@Ilyesbdlala
#1833 Updated amass version to v4.1.0
f8803d3 
Signed-off-by: Ilyes Ben Dlala <[email protected]>
@Ilyesbdlala
#1833 changed amass scanner docker image to use our own custom image
56fc3fa 
Signed-off-by: Ilyes Ben Dlala <[email protected]>
@Ilyesbdlala
#1833 Upgraded amass scantype template to use amass.sqlite results file
9440941 
Signed-off-by: Ilyes Ben Dlala <[email protected]>
@Ilyesbdlala
#1833 Added amass to custom scanner release build matrix
580d732 
Signed-off-by: Ilyes Ben Dlala <[email protected]>
@Ilyesbdlala
#1833 Updated amass integration test to no longer use -noalts
20baa8b 
It is replaced with -alts to enable (instead of disabling) generation of altered names.
ref: https://github.com/owasp-amass/amass/blob/master/doc/user_guide.md#the-enum-subcommand

Signed-off-by: Ilyes Ben Dlala <[email protected]>
@Ilyesbdlala
#1833 Allow identified_at attribute to be null
ae5319b 
It is not always available and is not required

Signed-off-by: Ilyes Ben Dlala <[email protected]>
@Ilyesbdlala
#1833 Optimized Amass Scanner Dockerfile
e21d016 
Less Run commands and no usage of `latest`
Signed-off-by: Ilyes Ben Dlala <[email protected]>
@Ilyesbdlala
#1833 Renamed and added volume mount to amass parse definition
def10dc 
This is done since the parser requires creating a temp file and reading from it. But the parser environemnt is ready only by default

Signed-off-by: Ilyes Ben Dlala <[email protected]>
@Ilyesbdlala
#1833 Changed the openDatabase() function to expect the content of a…
49cadd6 
… database instead of the path

This is what the lurker expects. A temp file is created to the reading of the database easier

Signed-off-by: Ilyes Ben Dlala <[email protected]>
@Ilyesbdlala
#1833 Updated amass parser test and snapshot to fit the change to ex…
d887b76 
…pect filecontent instead of path of database

Signed-off-by: Ilyes Ben Dlala <[email protected]>
@Ilyesbdlala
Adjust db query to output subdomains even if no relations exist
8e60035 
Signed-off-by: Heiko Kiesel <[email protected]>
@Ilyesbdlala
#1833 Added check for empty collumn in the sql query for amass parser
3f867bc 
Signed-off-by: Ilyes Ben Dlala <[email protected]>
@Ilyesbdlala
#1833 Updated amass parser tests and included a new test
70e70ea 
new test, tests for when -passive arg is used

Signed-off-by: Ilyes Ben Dlala <[email protected]>
@Ilyesbdlala
#1833 Added a check in parser-wrapper.js to parse according to the s…
419d8c1 
…canType

Amass parser requires that the database remains in binary format

Signed-off-by: Ilyes Ben Dlala <[email protected]>
@Ilyesbdlala
#1833 DRAFT Added EncodingType Attribute to ParseDefinition CRD
2d19699 
Signed-off-by: Ilyes Ben Dlala <[email protected]>
@Ilyesbdlala
#1833 Changed parser-wrapper to check parseDefinition encodingType i…
5bd8243 
…nstead of ScanType

Signed-off-by: Ilyes Ben Dlala <[email protected]>
@Ilyesbdlala
#1833 Renamed ParseDefinition CRD attribute from encodingType to con…
3395516 
…tentType

It's a more fitting name for types such as "Binary"

Signed-off-by: Ilyes Ben Dlala <[email protected]>
@Ilyesbdlala
#1833 Changed amass Dockerfile to be based on the official amass image
308daad 
This is done to avoid the shorter lifecycle of amass docker images, and to also avoid the platform dependency

Signed-off-by: Ilyes Ben Dlala <[email protected]>
@Ilyesbdlala
#1833 Updated amass scan examples by removing nolonger supported par…
3ebae4b 
…ameters

Signed-off-by: Ilyes Ben Dlala <[email protected]>
@Ilyesbdlala
#715 Set Amass Docker Image in a numeric way to allow runAsNonRoot
a16f5ba 
Signed-off-by: Ilyes Ben Dlala <[email protected]>
@Ilyesbdlala
#1833 Added a notice to the DockerHub webpage for Amass image copyright
0b7c01d 
Signed-off-by: Ilyes Ben Dlala <[email protected]>
@Ilyesbdlala
secureCodeBox/documentation#157 Updated and corrected Amass docs to …
03d71b2 
…fit v4 changes

Signed-off-by: Ilyes Ben Dlala <[email protected]>
@Ilyesbdlala
#1833 Included amass upgrade to v4.2.0
52ad968 
Signed-off-by: Ilyes Ben Dlala <[email protected]>
@Ilyesbdlala
#1833 Renamed the unit test to emptyRelations table, since the term …
f08fcb4 
…"passive" is deprecated

Signed-off-by: Ilyes Ben Dlala <[email protected]>
@Ilyesbdlala
#1833 Added a timeout to the amass integration-test since the amass …
8b4dbe5 
…docker image does not exit directly after enum

Signed-off-by: Ilyes Ben Dlala <[email protected]>
Ilyesbdlala and others added 2 commits September 12, 2023 16:51
@Ilyesbdlala
#1833 updated unit tests snapshot
5284432 
Signed-off-by: Ilyes Ben Dlala <[email protected]>
@J12934
#1833 Refactor parser async handling
36f0614 
Signed-off-by: Jannik Hollenbach <[email protected]>
@J12934 J12934 force-pushed the bugfix/amass-parser branch from 0778430 to 36f0614 Compare September 15, 2023 09:26
@Ilyesbdlala Ilyesbdlala enabled auto-merge (rebase) September 15, 2023 09:33
@Ilyesbdlala Ilyesbdlala merged commit 59eaf68 into main Sep 15, 2023
@Ilyesbdlala Ilyesbdlala deleted the bugfix/amass-parser branch September 15, 2023 10:08
@Ilyesbdlala Ilyesbdlala mentioned this pull request Oct 17, 2023
Closed
5 tasks
@Ilyesbdlala Ilyesbdlala mentioned this pull request Oct 31, 2023
Merged
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@J12934 J12934 J12934 approved these changes

Assignees

@Ilyesbdlala Ilyesbdlala

Labels

scanner Implement or update a security scanner

Projects

secureCodeBox
Archived in project

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

Upgrade amass to v4.x

2 participants

@Ilyesbdlala @J12934