Skip to content

ci: testing bad behavior change - #1364

Closed
suztomo wants to merge 4 commits into
googleapis:1.39.2-spfrom
suztomo:install_lts-library-check_bad-change
Closed

suztomo wants to merge 4 commits into
googleapis:1.39.2-spfrom
suztomo:install_lts-library-check_bad-change

Conversation

@suztomo

@suztomo suztomo commented May 5, 2021

Copy link
Copy Markdown
Member

No need to review.

This PR contains a behavior change that would break other libraries.

@google-cla google-cla Bot added the cla: yes This human has signed the Contributor License Agreement. label May 5, 2021
Comment on lines +879 to 880
if (executeInterceptor == null) { // This code is wrong. Just for testing.
executeInterceptor.intercept(this);

@suztomo suztomo May 5, 2021 •

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

google-cloud-storage's test threw NullPointerException correctly. Good.

Screen Shot 2021-05-05 at 17 00 57

@suztomo suztomo May 6, 2021 •

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

However, this approach does not work for Gradle projects such as "LTS Library Test / lts-library-test (beam)".

Comment thread .github/workflows/lts-test.yaml Outdated
with:
repository: ${{ matrix.repository }}
version_override_key: http.version
version_override_value: 1.39.2

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This should not be manually set.

java-version: 8
- name: Install this repository's artifacts to local Maven repository
run: |
mvn --batch-mode --show-version --no-transfer-progress install \

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is it possible to specify some strange version number (that would not conflict) when installing this artifact?

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Does {x-version-update:google-http-client-parent:current} help?

@suztomo

suztomo commented May 6, 2021 •

Copy link
Copy Markdown
Member Author

Why does "LTS Library Test / lts-library-test (beam)" succeed?

Experiment 1

In the install_lts-library-check_bad-change branch of google-http-java-client, update the version to 1.39.2-sp.2

 mvn versions:set -DnewVersion=1.39.2-sp.2

Then update the http.version property in the LTS BOM to use the version:

-    <http.version>1.39.2</http.version>
+    <http.version>1.39.2-sp.2</http.version>

Install the BOM to local Maven repository.

mvn install
...
[INFO] Installing /Users/suztomo/cloud-opensource-java/boms/cloud-lts-bom/pom.xml to /Users/suztomo/.m2/repository/com/google/cloud/gcp-lts-bom/0.1.0-SNAPSHOT/gcp-lts-bom-0.1.0-SNAPSHOT.pom

Running the test with the modification fails the Beam test as expected. Good.

suztomo-macbookpro44% ./gradlew --scan --console=plain -PdisableSpotlessCheck=true --continue \
          :sdks:java:extensions:google-cloud-platform-core:test --tests=org.apache.beam.sdk.extensions.gcp.util.GcsUtilTest
...
> Task :sdks:java:extensions:google-cloud-platform-core:test

org.apache.beam.sdk.extensions.gcp.util.GcsUtilTest > testNonExistentObjectReturnsEmptyResult FAILED
    java.lang.NullPointerException at GcsUtilTest.java:810
...

Experiment 2

What if we don't change the version? Google-http-client 1.39.2 is in Maven already.

In the install_lts-library-check_bad-change branch of google-http-java-client, keeping the version 1.39.2, I installed it to local Maven repository.

I installed the LTS BOM to local Maven repository.

[INFO] Installing /Users/suztomo/cloud-opensource-java/boms/cloud-lts-bom/pom.xml to /Users/suztomo/.m2/repository/com/google/cloud/gcp-lts-bom/0.1.0-SNAPSHOT/gcp-lts-bom-0.1.0-SNAPSHOT.pom

Then I ran the test again

$ ./gradlew  -PdisableSpotlessCheck=true cleanTest
$ ./gradlew  -PdisableSpotlessCheck=true\         
      :sdks:java:extensions:google-cloud-platform-core:test --tests=org.apache.beam.sdk.extensions.gcp.util.GcsUtilTest
$ open sdks/java/extensions/google-cloud-platform-core/build/reports/tests/test/index.html

It passed. Bad.

From this experiment, the artifacts stored in local Maven repository have lower priority than the artifacts in Maven central (or Gradle cache?).

@suztomo

suztomo commented May 7, 2021

Copy link
Copy Markdown
Member Author

Maybe the Gradle is not reading the local Maven repository first?

@suztomo

suztomo commented May 7, 2021 •

Copy link
Copy Markdown
Member Author

Version 1.39.2-sp.2 didn't work for the assembly module in GitHub Actions (it works in my MacBook):

[INFO] --- maven-assembly-plugin:3.3.0:single (make-assembly) @ google-http-client-assembly ---
[INFO] Reading assembly descriptor: assembly.xml
[INFO] ------------------------------------------------------------------------
[INFO] Reactor Summary:
[INFO] 
[INFO] Parent for the Google HTTP Client Library for Java 1.39.2-sp.2 SUCCESS [ 10.980 s]
[INFO] Google HTTP Client Library for Java 1.39.2-sp.2 .... SUCCESS [ 19.736 s]
[INFO] Protocol Buffer extensions to the Google HTTP Client Library for Java. 1.39.2-sp.2 SUCCESS [  6.046 s]
[INFO] Assembly for the Google HTTP Client Library for Java 1.39.2-sp.2 FAILURE [  6.234 s]
[INFO] Google App Engine extensions to the Google HTTP Client Library for Java. 1.39.2-sp.2 SKIPPED
[INFO] Android Platform Extensions to the Google HTTP Client Library for Java. 1.39.2-sp.2 SKIPPED
[INFO] Apache HTTP transport v2 for the Google HTTP Client Library for Java. 1.39.2-sp.2 SKIPPED
[INFO] GSON extensions to the Google HTTP Client Library for Java. 1.39.2-sp.2 SKIPPED
[INFO] Jackson 2 extensions to the Google HTTP Client Library for Java. 1.39.2-sp.2 SKIPPED
[INFO] XML extensions to the Google HTTP Client Library for Java. 1.39.2-sp.2 SKIPPED
[INFO] Google APIs Client Library Findbugs custom plugin. 1.39.2-sp.2 SKIPPED
[INFO] Shared classes used for testing of artifacts in the Google HTTP Client Library for Java. 1.39.2-sp.2 SKIPPED
[INFO] Simple example for the Dailymotion API. 1.39.2-sp.2  SKIPPED
[INFO] Google HTTP Client Library for Java BOM 1.39.2 ..... SKIPPED
[INFO] ------------------------------------------------------------------------
[INFO] BUILD FAILURE
[INFO] ------------------------------------------------------------------------
[INFO] Total time:  44.027 s
[INFO] Finished at: 2021-05-07T14:27:48Z
[INFO] ------------------------------------------------------------------------
Error:  Failed to execute goal org.apache.maven.plugins:maven-assembly-plugin:3.3.0:single (make-assembly) on project google-http-client-assembly: Failed to create assembly: Error adding file to archive: /home/runner/work/google-http-java-client/google-http-java-client/google-http-client-assembly/../google-http-client-android/target/site/dependencies.html -> [Help 1]

Some of the configuration files were not updated by mvn set:version command and broke the build in the assembly module.

@suztomo
suztomo force-pushed the install_lts-library-check_bad-change branch from b89e3a8 to 8c0db1c Compare May 7, 2021 15:25
@suztomo

suztomo commented May 7, 2021

Copy link
Copy Markdown
Member Author

Now Beam test fails as expected due to the null check. Good.

Screen Shot 2021-05-07 at 12 20 13

@suztomo
suztomo force-pushed the install_lts-library-check_bad-change branch 2 times, most recently from 828daed to 6406919 Compare May 7, 2021 17:51
@suztomo
suztomo force-pushed the install_lts-library-check_bad-change branch from 6406919 to 6750e91 Compare May 7, 2021 18:00
Comment on lines +32 to +37
- id: find_library_version
name: Find the library version in this branch
run: |
sudo apt-get install libxml2-utils
VERSION=`sed -e 's/xmlns="[^\s\\]*"//' pom.xml | xmllint --xpath '/project/version/text()' -`
echo "::set-output name=version::${VERSION}"

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@elharo (a followup from yesterday's discussion) This command extracts the version from the root pom.xml. This configuration file no longer needs to specify a hard-coded version. The value is then used at the end of this file as version_override_value parameter.

(I tweaked the command from https://github.com/googleapis/java-cloud-bom/blob/8a45813a6590d3a25dcaad4fa5d6beaeb341b3d3/.kokoro/bom-upper-bounds-check.sh#L43)

This sed+xmllint approach should work for Yoshi repositories. Other projects, such as grpc-java and Beam, will have other regular expression to extract their versions.

@elharo

elharo commented Jun 30, 2021

Copy link
Copy Markdown
Contributor

can this be closed now?

@suztomo

suztomo commented Jun 30, 2021

Copy link
Copy Markdown
Member Author

Yes. Closing. Thank you for followup.

@suztomo suztomo closed this Jun 30, 2021
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

cla: yes This human has signed the Contributor License Agreement.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants