Repository navigation
Conversation
CI and the new local script call the same runner scripts: - runner/cluster.json: image, BigQuery connector, extra jars per Spark release, the jars and Cloud SQL metadata every cluster gets (were the workflow's case, metadata and get_openlineage_jar.sh defaults) - runner/create_cluster.sh: stages the connector if missing, installs the jars with dev/dataproc/install_jars.sh (JARS on every node, each OpenLineage spark jar in /opt/openlineage/<version> on the master), init actions per cluster, removed once it's created - runner/run_scenarios.sh: one OpenLineage version, the body of the workflow's loop (transport jar, properties, run-scenarios, RUNNER_FAILED fallback, log upload); --events-dir downloads the events for local runs - run_spark_dataproc_tests.sh: replaces the unfinished FIXME script; one command for a local run, with --spark-jar for a local build dev/lib/dataproc_venv.sh is shared by the Hive and Spark local scripts; the Hive init action is now also removed after the cluster is created. Removes runner/get_openlineage_jar.sh. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: tnazarew <tomasz.nazarewicz@getindata.com>
Dataproc 2.1+ images replace their built-in BigQuery connector with the one this key names. Without it the built-in connector stayed next to ours and every BigQuery scenario on Spark 3.5.1 failed with "Multiple sources found for bigquery (Spark35BigQueryTableProvider, Spark34BigQueryTableProvider)". Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: tnazarew <tomasz.nazarewicz@getindata.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Stacked on #361 (and through it on #358, #356, #353 and #352). Retarget it once those are merged.
The Spark Dataproc workflow and a local run now call the same scripts. You can check a scenario, or your own
openlineage-sparkbuild, on your own cluster before committing. This replaces the unfinishedrun_spark_dataproc_tests.sh(# FIXME THIS IS STILL IN PROGRESS, NOT TESTED).What changes
producer/spark_dataproc/runner/cluster.json: the cluster settings that used to be spread over the workflow'scase, its metadata andget_openlineage_jar.sh's defaults:spark_short), and the Iceberg/Delta jars (3.5.1 only)The values are the ones CI uses today. For example, Bigtable stays
0.5.0-SNAPSHOT, not the init action's stale0.3.0default.runner/create_cluster.sh: stages the BigQuery connector if it's missing, then installs the jars with the shareddev/dataproc/install_jars.sh. The connector and the other jars go on every node. Each OpenLineage version's spark jar goes to/opt/openlineage/<version>/on the master. The init actions are uploaded per cluster and removed once the cluster is up, as the workflow did before.runner/run_scenarios.sh: one OpenLineage version on a cluster, the body of the workflow's per-version loop:run-scenariosRUNNER_FAILEDif the runner itself fails, and uploads the logsThe Postgres credentials now come from environment variables instead of being written into the step's script.
--events-dirdownloads the events for local runs.Workflow:
create-commandis now one call tocreate_cluster.sh, and the loop callsrun_scenarios.shonce per version. The "Stage Spark BigQuery connector", "Upload initialization actions" and "Remove initialization actions" steps are gone;create_cluster.shdoes that work. The jobs, the matrix, the GCS layout andvalidateare unchanged.run_spark_dataproc_tests.sh: one command for a local run.--spark-jartests a local build.dataproc-producer-test-because the expected events match that name.Shared code:
dev/lib/dataproc_venv.shis now used by both the Hive and Spark local scripts.delete_cluster.sh --list-mineuses the same user name. The Hive init action is now also removed once the cluster is up.Removed
runner/get_openlineage_jar.sh.Tests
scripts/tests/test_spark_runner.py, with stubpython,gcloudandcurl:create_cluster.shfor 3.5.1 and 3.1.3: the exact metadata, connector staging, and the init actions uploaded and removed, plus bad inputrun_scenarios.sh: the arguments and properties, the log upload, local event download,RUNNER_FAILEDwhen the runner crashes, and a missing transport jartest_run_scripts.py:--help, unknown options and missing arguments for the three new scripts.pytest scripts/tests: 122 passed. shellcheck is clean, and actionlint finds nothing new.🤖 Generated with Claude Code