Skip to content

Handle .srcjar inputs in Bazel aspect - #754

Merged
antonsviridov-src merged 6 commits into
mainfrom
antonsviridov-graph-828-scip-java-for-bazel-does-not-handle-srcjar-inputs
Oct 10, 2024
Merged

Handle .srcjar inputs in Bazel aspect#754
antonsviridov-src merged 6 commits into
mainfrom
antonsviridov-graph-828-scip-java-for-bazel-does-not-handle-srcjar-inputs

Conversation

@antonsviridov-src

@antonsviridov-src antonsviridov-src commented Oct 10, 2024 •
edited
Loading

Copy link
Copy Markdown
Contributor

The aspect will add a new entry to sourceFiles:

  "sourceFiles": [
    "Foo.java",
    "bazel-out/darwin_arm64-fastbuild/bin/extracted_srcjar/sources.srcjar"
  ]

and ScipBuildTool will be able to traverse folders and lookup all source files in them.

Together this allows us to avoid unpacking jars in scip-java (which would entail tracking their location and caching, which would be annoying), instead delegating all that to Bazel.

Test plan

  • New example that is used as part of the CI

@antonsviridov-src
antonsviridov-src marked this pull request as ready for review October 10, 2024 10:34
olafurpg
olafurpg approved these changes Oct 10, 2024
View reviewed changes

@olafurpg olafurpg left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM 👍🏻 Very nice to see more interest in the Bazel aspects, a lot of work went into making those

@antonsviridov-src
antonsviridov-src merged commit a4eb361 into main Oct 10, 2024
@antonsviridov-src
antonsviridov-src deleted the antonsviridov-graph-828-scip-java-for-bazel-does-not-handle-srcjar-inputs branch October 10, 2024 12:54
@JohnnyMorganz

JohnnyMorganz commented Oct 11, 2024 •
edited
Loading

Copy link
Copy Markdown

Thank you very much for taking a look at this, very excited! I decided to build from source and seems to work for most cases.

However, it seems to fail when there are 2 java libraries in the same BUILD.bazel that use the same srcjar.
I was curious if this might be due to the fact scip-java doesn't use sandboxing (--spawn_strategy=local), but bazel build --spawn_strategy=local //test/... works for me.

Here's an extended example:

genrule(
    name = "generated-srcjar",
    outs = ["sources.srcjar"],
    cmd = "echo 'package com.testing; public class Bar {};' > Bar.java && jar cf $(@) Bar.java",
)

java_library(
    name = "other_library",
    srcs = [
        "Baz.java", # create a new file in source at test/Baz.java, alongside test/Foo.java
        ":generated-srcjar",
    ],
)

java_library(
    name = "testing",
    srcs = [
        "Foo.java",
        ":generated-srcjar",
    ],
)
$ /home/code/scip-java/out/bin/scip-java index --build-tool=bazel --bazel-scip-java-binary="/home/code/scip-java/out/bin/scip-java" --output=index-java.scip -- -- //test/...
...
ERROR: file 'test/extracted_srcjar/test/sources.srcjar' is generated by these conflicting actions:
Label: //test:testing, //test:other_library
Aspects: [//aspects:scip_java.bzl%scip_java_aspect]
RuleClass: java_library rule
JavaActionClass: class com.google.devtools.build.lib.analysis.actions.StarlarkAction
Configuration: 3ba97bf385bd693a2d11797e9a10970d8a18bd9a3c65cca1cc949fcdaf192b6a
Mnemonic: ExtractSourceJars
Action key: 4d908d0c4683f4e763487790dac25b2e8b0deb9f1d629ba30b7749749bdb03c0
Progress message: Extracting source jar bazel-out/k8-dbg--cd/bin/test/sources.srcjar
Action describeKey: Extracting source jar bazel-out/k8-dbg--cd/bin/test/sources.srcjar
  Argument: /bin/bash
  Argument: -c
  Argument: '
                unzip bazel-out/k8-dbg--cd/bin/test/sources.srcjar -d bazel-out/k8-dbg--cd/bin/test/extracted_srcjar/test/sources.srcjar
            '
  Output paths mode: OFF

PrimaryInput: File:[[source]]bin/jar
PrimaryOutput: File:[[<execution_root>]bazel-out/k8-dbg--cd/bin]test/extracted_srcjar/test/sources.srcjar
Owner information: //test:testing#//aspects:scip_java.bzl%scip_java_aspect ConfiguredTargetKey{label=//test:testing, config=BuildConfigurationKey[3ba97bf385bd693a2d11797e9a10970d8a18bd9a3c65cca1cc949fcdaf192b6a]} {}, //test:other_library#//aspects:scip_java.bzl%scip_java_aspect ConfiguredTargetKey{label=//test:other_library, config=BuildConfigurationKey[3ba97bf385bd693a2d11797e9a10970d8a18bd9a3c65cca1cc949fcdaf192b6a]} {}
MandatoryInputs: Attempted action contains artifacts not in previous action (first 5): 
        test/Foo.java
Previous action contains artifacts not in attempted action (first 5): 
        test/Bar.java
Outputs: are equal
ERROR: com.google.devtools.build.lib.actions.MutableActionGraph$ActionConflictException: for test/extracted_srcjar/test/sources.srcjar, previous action: action 'Extracting source jar bazel-out/k8-dbg--cd/bin/test/sources.srcjar', attempted action: action 'Extracting source jar bazel-out/k8-dbg--cd/bin/test/sources.srcjar'

@JohnnyMorganz

JohnnyMorganz commented Oct 11, 2024

Copy link
Copy Markdown

Also, somewhat of an edge case, but we have some srcjars that end up becoming empty due to Reasons, causing the following error:

genrule(
    name = "empty-srcjar",
    outs = ["sources.srcjar"],
    cmd = "touch test.txt && zip $(@) test.txt && zip -d $(@) test.txt",
)

java_library(
    name = "testing",
    srcs = [
        "Foo.java",
        ":empty-srcjar",
    ],
)
ERROR: /home/code/repository/test/BUILD.bazel:7:13: Extracting source jar bazel-out/k8-dbg--cd/bin/test/sources.srcjar failed: (Exit 1): bash failed: error executing ExtractSourceJars command (from target //test:testing) 
  (cd /home/.cache/bazel/.../execroot/_main && \
  exec env - \
  /bin/bash -c '
                unzip bazel-out/k8-dbg--cd/bin/test/sources.srcjar -d bazel-out/k8-dbg--cd/bin/test/extracted_srcjar/test/sources.srcjar
            ')
# Configuration: 3ba97bf385bd693a2d11797e9a10970d8a18bd9a3c65cca1cc949fcdaf192b6a
warning [bazel-out/k8-dbg--cd/bin/test/sources.srcjar]:  zipfile is empty
Aspect //aspects:scip_java.bzl%scip_java_aspect of //test:testing failed to build

But other than these 2 things, it is looking very promising. Thank you!!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

1 more reviewer
@olafurpg olafurpg olafurpg approved these changes
Reviewers whose approvals may not affect merge requirements

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

3 participants

@antonsviridov-src @JohnnyMorganz @olafurpg