Gradle Build System in Apache Beam
Overview
Apache Beam is a mono-repo using Gradle as its build system. The entire project (Java, Python, Go, website) is managed as a single Gradle project.
Key Files
-
build.gradle.kts
-
Root build configuration
-
settings.gradle.kts
-
Project structure and module definitions
-
gradle.properties
-
Global properties and versions
-
buildSrc/
-
Custom Gradle plugins including BeamModulePlugin
BeamModulePlugin
Located at buildSrc/src/main/groovy/org/apache/beam/gradle/BeamModulePlugin.groovy
Purpose
-
Manages Java dependencies centrally
-
Configures project types (Java, Python, Go, Proto, Docker, etc.)
-
Defines common custom tasks
Java Project Configuration
apply plugin: 'org.apache.beam.module' applyJavaNature( automaticModuleName: 'org.apache.beam.sdk.io.kafka' )
Common Commands
Build
Build entire project
./gradlew build
Build specific project
./gradlew :sdks:java:core:build ./gradlew -p sdks/java/core build
Compile only (no tests)
./gradlew :sdks:java:core:compileJava
Test
Run tests
./gradlew :sdks:java:core:test
Run specific test
./gradlew :sdks:java:core:test --tests *MyTest
Skip tests
./gradlew build -x test
Clean
Clean specific project
./gradlew :sdks:java:core:clean
Clean everything
./gradlew clean
Formatting
Java formatting (Spotless)
./gradlew spotlessApply
Check formatting
./gradlew spotlessCheck
Format CHANGES.md
./gradlew formatChanges
Publishing
Publish to Maven Local
./gradlew -Ppublishing :sdks:java:core:publishToMavenLocal
Publish all Java artifacts
./gradlew -Ppublishing publishToMavenLocal
Pre-commit Tasks
Java
./gradlew javaPreCommit
Python
./gradlew pythonPreCommit
Combined
./gradlew :checkSetup # Validates Go, Java, Python environments
Useful Flags
Flag Description
-p <path>
Run task in specific project directory
-x <task>
Exclude task
--tests <pattern>
Filter tests
-Ppublishing
Enable publishing tasks
-PdisableSpotlessCheck=true
Disable formatting check
-PdisableCheckStyle=true
Disable checkstyle
-PskipCheckerFramework
Skip Checker Framework
--continue
Continue after failures
--info
Verbose output
--debug
Debug output
--scan
Generate build scan
--parallel
Parallel execution
GCP-related Properties
-PgcpProject=my-project -PgcpRegion=us-central1 -PgcpTempRoot=gs://bucket/temp -PgcsTempRoot=gs://bucket/temp
Docker Tasks
Build Java SDK container
./gradlew :sdks:java:container:java11:docker
Build Python SDK container
./gradlew :sdks:python:container:py312:docker
Build and push a container into a custom repository
./gradlew :sdks:java:container:java11:docker
-Pdocker-repository-root=gcr.io/project
-Pdocker-tag=custom
-Ppush-containers
If a :docker task produces logs that contain the following:
WARNING: No output specified with docker-container driver. Build result will only remain in the build cache.
then you must use -PuseDockerBuildx when running :docker tasks in this environment. For example:
Build and push a go container into a custom repository
./gradlew :sdks:go:container:docker
-Pdocker-repository-root=gcr.io/project
-Pdocker-tag=custom
-Ppush-containers
-PuseDockerBuildx
Dependency Management
View Dependencies
./gradlew :sdks:java:core:dependencies ./gradlew :sdks:java:core:dependencies --configuration runtimeClasspath
Force Dependency Version
In build.gradle :
configurations.all { resolutionStrategy.force 'com.google.guava:guava:32.0.0-jre' }
Troubleshooting
Clean Gradle Cache
rm -rf ~/.gradle/caches rm -rf .gradle rm -rf build
Common Errors
NoClassDefFoundError
-
Run ./gradlew clean
-
Delete gradle cache
Proto-related Errors
- Regenerate protos: ./gradlew generateProtos
Dependency Conflicts
-
Check dependencies: ./gradlew dependencies
-
Use --scan for detailed analysis
Useful Tasks
List all tasks
./gradlew tasks
List tasks for a project
./gradlew :sdks:java:core:tasks
Show project structure
./gradlew projects
IDE Integration
IntelliJ
-
Open repository root as Gradle project
-
Wait for indexing
-
Gradle tool window shows all tasks
VS Code
Install Gradle extension for task discovery