Skip to content

spark: run the scenarios locally the way CI does - #362

Draft
tnazarew wants to merge 2 commits into
local-runs/hivefrom
local-runs/spark
Draft

tnazarew wants to merge 2 commits into
local-runs/hivefrom
local-runs/spark

Conversation

@tnazarew

Copy link
Copy Markdown
Collaborator

Stacked on #361 (and through it on #358, #356, #353 and #352). Retarget it once those are merged.

The Spark Dataproc workflow and a local run now call the same scripts. You can check a scenario, or your own openlineage-spark build, on your own cluster before committing. This replaces the unfinished run_spark_dataproc_tests.sh (# FIXME THIS IS STILL IN PROGRESS, NOT TESTED).

What changes

  • producer/spark_dataproc/runner/cluster.json: the cluster settings that used to be spread over the workflow's case, its metadata and get_openlineage_jar.sh's defaults:

    • per Spark release: the Dataproc image, the BigQuery connector line (spark_short), and the Iceberg/Delta jars (3.5.1 only)
    • for every cluster: the Spanner, Bigtable and PostgreSQL jars, the Cloud SQL metadata, and the master machine type

    The values are the ones CI uses today. For example, Bigtable stays 0.5.0-SNAPSHOT, not the init action's stale 0.3.0 default.

  • runner/create_cluster.sh: stages the BigQuery connector if it's missing, then installs the jars with the shared dev/dataproc/install_jars.sh. The connector and the other jars go on every node. Each OpenLineage version's spark jar goes to /opt/openlineage/<version>/ on the master. The init actions are uploaded per cluster and removed once the cluster is up, as the workflow did before.

  • runner/run_scenarios.sh: one OpenLineage version on a cluster, the body of the workflow's per-version loop:

    • picks the GCS transport jar and sets the job properties, then runs run-scenarios
    • records RUNNER_FAILED if the runner itself fails, and uploads the logs

    The Postgres credentials now come from environment variables instead of being written into the step's script. --events-dir downloads the events for local runs.

  • Workflow: create-command is now one call to create_cluster.sh, and the loop calls run_scenarios.sh once per version. The "Stage Spark BigQuery connector", "Upload initialization actions" and "Remove initialization actions" steps are gone; create_cluster.sh does that work. The jobs, the matrix, the GCS layout and validate are unchanged.

  • run_spark_dataproc_tests.sh: one command for a local run.

    • It stages the jars, creates a cluster and deletes it at the end, runs the scenarios and validates them. --spark-jar tests a local build.
    • The cluster name starts with dataproc-producer-test- because the expected events match that name.
    • Each run has its own GCS directory and table suffix, so it doesn't mix with CI.
  • Shared code: dev/lib/dataproc_venv.sh is now used by both the Hive and Spark local scripts. delete_cluster.sh --list-mine uses the same user name. The Hive init action is now also removed once the cluster is up.

  • Removed runner/get_openlineage_jar.sh.

Tests

  • scripts/tests/test_spark_runner.py, with stub python, gcloud and curl:
    • create_cluster.sh for 3.5.1 and 3.1.3: the exact metadata, connector staging, and the init actions uploaded and removed, plus bad input
    • run_scenarios.sh: the arguments and properties, the log upload, local event download, RUNNER_FAILED when the runner crashes, and a missing transport jar
  • test_run_scripts.py: --help, unknown options and missing arguments for the three new scripts.
  • pytest scripts/tests: 122 passed. shellcheck is clean, and actionlint finds nothing new.

🤖 Generated with Claude Code

tnazarew and others added 2 commits November 20, 2026 18:24
CI and the new local script call the same runner scripts:
- runner/cluster.json: image, BigQuery connector, extra jars per Spark release, the jars and Cloud SQL metadata every
  cluster gets (were the workflow's case, metadata and get_openlineage_jar.sh defaults)
- runner/create_cluster.sh: stages the connector if missing, installs the jars with dev/dataproc/install_jars.sh
  (JARS on every node, each OpenLineage spark jar in /opt/openlineage/<version> on the master), init actions per
  cluster, removed once it's created
- runner/run_scenarios.sh: one OpenLineage version, the body of the workflow's loop (transport jar, properties,
  run-scenarios, RUNNER_FAILED fallback, log upload); --events-dir downloads the events for local runs
- run_spark_dataproc_tests.sh: replaces the unfinished FIXME script; one command for a local run, with --spark-jar
  for a local build

dev/lib/dataproc_venv.sh is shared by the Hive and Spark local scripts; the Hive init action is now also removed
after the cluster is created. Removes runner/get_openlineage_jar.sh.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: tnazarew <tomasz.nazarewicz@getindata.com>
Dataproc 2.1+ images replace their built-in BigQuery connector with the one this key names. Without it the
built-in connector stayed next to ours and every BigQuery scenario on Spark 3.5.1 failed with "Multiple sources
found for bigquery (Spark35BigQueryTableProvider, Spark34BigQueryTableProvider)".

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: tnazarew <tomasz.nazarewicz@getindata.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant