Bump SLF4J from 1.7.30 to 2.0.16. by shunping · Pull Request #33574 · apache/beam

shunping · 2025-01-13T21:05:57Z

Upgrade SLF4J from 1.7.30 to 2.0.16.

Additional key changes:

Default Spark version in Beam upgraded to 3.5.0.
Workaround implemented for continued support of Spark 3.2.x.
SparkReceiverIO migrated and updated for full Spark 3.x compatibility.
CdapIO updated to leverage Spark 3.x and Scala 2.12.

The class path of slf4j has been changed from org/slf4j/impl to org/slf4j/jul. Two failed tests: - :runners:google-cloud-dataflow-java:worker:validateShadedJarContainsSlf4jJdk14 - :runners:google-cloud-dataflow-java:worker:validateShadedJarDoesntLeakNonProjectClasses

The failed tests are under org.apache.beam.runners.dataflow.worker.HotKeyLoggerTest

The previous version has a compile dependency on slf4j 1.x binding, which would no longer work with slf4j 2.x.

Abacn · 2025-02-04T21:31:49Z

triggered a performance test: https://github.com/apache/beam/actions/runs/13145020125

shunping · 2025-02-05T05:52:36Z

r: @Abacn

github-actions · 2025-02-05T05:53:52Z

Stopping reviewer notifications for this pull request: review requested by someone other than the bot, ceding control. If you'd like to restart, comment assign set of reviewers

Abacn

Thanks! Had a few comments.

when it is ready to merge good to announce it in CHANGES.md

Abacn · 2025-02-05T16:20:24Z

runners/google-cloud-dataflow-java/worker/build.gradle

    }
 }

+project.tasks.withType(Test).configureEach {


why are they needed? tests currently using Java11 should be fine as https://openjdk.org/jeps/403 is enforced in Java16+ only.

Did you run tests on Java 17+ locally? If so did the test experience similar issue on master branch?

I just took them out and triggered the tests again on github. Will see.

Looks like the test is fine. However, still good to confirm

Did you run tests on Java 17+ locally? If so did the test experience similar issue on master branch?

Abacn · 2025-02-05T16:23:13Z

sdks/java/io/sparkreceiver/2/build.gradle

    implementation library.java.slf4j_api
-    implementation library.java.spark_streaming
-    implementation library.java.spark_core
+    implementation library.java.spark3_streaming


let's move sparkreceiver/2 to sparkreceiver/3 or simply sparkreceiver/. The package name is unchanged so that release is unchanged.

I sent an email to dev list regarding this change. I will wait a couple days to see if there is any objection or other suggestion.

shunping · 2025-02-05T20:54:52Z

Run Java PreCommit

shunping · 2025-02-05T20:55:19Z

Run Java_Examples_Dataflow_Java21 PreCommit

Abacn · 2025-02-05T22:35:29Z

Java PreCommit testOnNewWorkerMetadata_redistributesBudget (org.apache.beam.runners.dataflow.worker.streaming.harness.FanOutStreamingEngineWorkerHarnessTest) failed

due to #28957 not related to this change

Abacn

Thank you!

shunping · 2025-02-12T21:21:20Z

Run Java_GCP_IO_Direct PreCommit

shunping · 2025-02-12T21:22:37Z

Thanks @Abacn. The perf test runs successfully: https://github.com/apache/beam/actions/runs/13294385116/job/37122641048.

codecov · 2025-02-12T22:04:47Z

Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 59.08%. Comparing base (fc43c12) to head (935a637).
Report is 59 commits behind head on master.

Additional details and impacted files

@@             Coverage Diff              @@
##             master   #33574      +/-   ##
============================================
- Coverage     59.08%   59.08%   -0.01%     
- Complexity     3238     3239       +1     
============================================
  Files          1156     1155       -1     
  Lines        176924   176929       +5     
  Branches       3391     3392       +1     
============================================
- Hits         104543   104540       -3     
- Misses        69015    69023       +8     
  Partials       3366     3366

Flag	Coverage Δ
java	`70.30% <ø> (-0.03%)`	⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

Abacn · 2025-02-24T16:45:55Z

Java PreCommit testOnNewWorkerMetadata_redistributesBudget failed not related to this change

Abacn · 2025-02-27T03:46:57Z

It breaks beam_PostCommit_Java_ValidatesRunner_SparkStructuredStreaming and beam_PostCommit_Java_ValidatesRunner_Spark: https://github.com/apache/beam/actions/workflows/beam_PostCommit_Java_ValidatesRunner_SparkStructuredStreaming.yml and https://github.com/apache/beam/commits/ea40d9e4db5092b96d4247c3f800267eee3debe8

Abacn · 2025-03-03T22:30:11Z

This likely also broke PostCommit Java Hadoop Versions : https://github.com/apache/beam/actions/workflows/beam_PostCommit_Java_Hadoop_Versions.yml?page=2&query=

Abacn · 2025-03-03T22:31:10Z

same fix as #34123 should fix it

* Update slf4j version * Remove slf4j from arrow dependency exclusion. * Fix two test failures due to upgrade slf4j-jdk14 to 2.x The class path of slf4j has been changed from org/slf4j/impl to org/slf4j/jul. Two failed tests: - :runners:google-cloud-dataflow-java:worker:validateShadedJarContainsSlf4jJdk14 - :runners:google-cloud-dataflow-java:worker:validateShadedJarDoesntLeakNonProjectClasses * Fixed another four failed tests. The failed tests are under org.apache.beam.runners.dataflow.worker.HotKeyLoggerTest * Bump the default spark version from 3.2.2 to 3.5.0. The previous version has a compile dependency on slf4j 1.x binding, which would no longer work with slf4j 2.x. * Add used but not declared deps for spark 3.5.0 * Temporary modify spark version to 3.x in sparkreceiver. * Fix failed spark tests. * A better workaround for Spark 3.2.x * Take out the add-opens for tests as they were only run in java 8 and 11. * Mention changes in CHANGES.md * Update comments * Move sparkReceiver/2 to sparkreceiver/3 that supports Spark 3.x. * Minor fix on cdap spark dependency

Update slf4j version

4b124cb

github-actions bot added the build label Jan 13, 2025

Remove slf4j from arrow dependency exclusion.

f3636ff

github-actions bot added java io extensions gcp labels Jan 13, 2025

github-actions bot added runners dataflow labels Jan 14, 2025

shunping added 5 commits January 14, 2025 15:02

Fixed another four failed tests.

0f4e5f1

The failed tests are under org.apache.beam.runners.dataflow.worker.HotKeyLoggerTest

Merge branch 'master' into upgrade-slf4j

fd01768

Bump the default spark version from 3.2.2 to 3.5.0.

fd16148

The previous version has a compile dependency on slf4j 1.x binding, which would no longer work with slf4j 2.x.

Add used but not declared deps for spark 3.5.0

bcc3465

Temporary modify spark version to 3.x in sparkreceiver.

43ddc02

github-actions bot added the spark label Feb 4, 2025

shunping added 2 commits February 4, 2025 21:44

Fix failed spark tests.

f452f05

A better workaround for Spark 3.2.x

40cdc9d

shunping marked this pull request as ready for review February 5, 2025 05:52

Abacn reviewed Feb 5, 2025

View reviewed changes

shunping added 5 commits February 5, 2025 12:22

Take out the add-opens for tests as they were only run in java 8 and 11.

b810034

Merge branch 'master' into upgrade-slf4j

6ac9b97

Merge branch 'master' into upgrade-slf4j

cc21ecd

Mention changes in CHANGES.md

bdbb72e

Update comments

1710eff

Abacn approved these changes Feb 6, 2025

View reviewed changes

shunping changed the title ~~Update slf4j version~~ [Do not merge] Update slf4j version Feb 6, 2025

shunping force-pushed the upgrade-slf4j branch 2 times, most recently from b50f03f to 7ab25e8 Compare February 12, 2025 01:58

Move sparkReceiver/2 to sparkreceiver/3 that supports Spark 3.x.

d6b56c6

shunping force-pushed the upgrade-slf4j branch 2 times, most recently from dad38fb to c836281 Compare February 12, 2025 20:06

Minor fix on cdap spark dependency

935a637

shunping force-pushed the upgrade-slf4j branch from c836281 to 935a637 Compare February 12, 2025 20:16

shunping changed the title ~~[Do not merge] Update slf4j version~~ Bump slf4j from 1.7.30 to 2.0.16. Feb 12, 2025

shunping changed the title ~~Bump slf4j from 1.7.30 to 2.0.16.~~ Breaking Change: Bump slf4j from 1.7.30 to 2.0.16. Feb 12, 2025

shunping changed the title ~~Breaking Change: Bump slf4j from 1.7.30 to 2.0.16.~~ [Breaking Change] Bump slf4j from 1.7.30 to 2.0.16. Feb 12, 2025

shunping changed the title ~~[Breaking Change] Bump slf4j from 1.7.30 to 2.0.16.~~ Bump SLF4J from 1.7.30 to 2.0.16. Feb 13, 2025

Merge branch 'master' into upgrade-slf4j

cb62acb

Abacn merged commit 42bbc1e into apache:master Feb 24, 2025
27 of 28 checks passed

Abacn mentioned this pull request Feb 27, 2025

Spark Runner : Replace queueStream with custom DStream in Spark streaming Flatten transform #34080

Merged

3 tasks

pinfrederic mentioned this pull request Feb 27, 2025

[Feature Request]: Upgrade slf4j to v2.0.17 #34107

Closed

17 tasks

shunping mentioned this pull request Feb 28, 2025

Add log4j2 deps for spark validatesRunner tasks. #34123

Merged

Abacn mentioned this pull request Mar 3, 2025

The PostCommit Java Hadoop Versions job is flaky #33252

Closed

shunping mentioned this pull request Mar 4, 2025

Fix hadoop version tests. #34155

Merged

shunping added this to the 2.64.0 Release milestone Mar 4, 2025

Conversation

shunping commented Jan 13, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Abacn commented Feb 4, 2025

Uh oh!

shunping commented Feb 5, 2025

Uh oh!

github-actions bot commented Feb 5, 2025

Uh oh!

Abacn left a comment

Choose a reason for hiding this comment

Uh oh!

Abacn Feb 5, 2025

Choose a reason for hiding this comment

Uh oh!

shunping Feb 5, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Abacn Feb 5, 2025

Choose a reason for hiding this comment

Uh oh!

Abacn Feb 5, 2025

Choose a reason for hiding this comment

Uh oh!

shunping Feb 5, 2025

Choose a reason for hiding this comment

Uh oh!

shunping commented Feb 5, 2025

Uh oh!

shunping commented Feb 5, 2025

Uh oh!

Abacn commented Feb 5, 2025

Uh oh!

Abacn left a comment

Choose a reason for hiding this comment

Uh oh!

shunping commented Feb 12, 2025

Uh oh!

shunping commented Feb 12, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

codecov bot commented Feb 12, 2025

Codecov Report

Uh oh!

Abacn commented Feb 24, 2025

Uh oh!

Uh oh!

Abacn commented Feb 27, 2025

Uh oh!

Abacn commented Mar 3, 2025

Uh oh!

Abacn commented Mar 3, 2025

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

shunping commented Jan 13, 2025 •

edited

Loading

shunping Feb 5, 2025 •

edited

Loading

shunping commented Feb 12, 2025 •

edited

Loading