Add CI that tests each Snowflake sample on Snowflake Next

[HarshCasper]

The repo had no CI. This adds a workflow that runs all 10 samples in a matrix against the Snowflake Next emulator (localstack/snowflake-next), as requested in review and tracked in LAV-2294. Each sample gets a test.sh behind make test, plus make install and a start target.

How the samples start

• Snowflake only (citi-bike, credit-card-fraud, credit-scoring, predicting-customer-spend, soda, streamlit): lstk start --type snowflake --image localstack/snowflake-next:latest (lstk 1.3.0+).
• Snowflake + AWS (airflow-dbt, glue, lambda, multi-container): Next has no AWS services and lstk can't run both emulators at once. These samples therefore get a docker-compose.yml with LocalStack AWS on 4566 and Snowflake Next on a second port, on a shared network.
  • Containers that LocalStack starts (MWAA, the Glue job, the Lambda function) reach Next by its service name over plain http.
  • SF_HOSTNAMES lists that service name.
• Fresh instance: Next starts with only the SNOWFLAKE database, as a new Snowflake account does. Each sample creates its database (init script or setup step) instead of relying on the old emulator creating it on connect.

What each test checks

Sample	Test
airflow-dbt-transformation	Creates the MWAA environment and checks the Airflow container can reach Next. Deploys the DAG and triggers a run: it succeeds and transform.PREPPED_DATA has 14 rows.
citi-bike-data-app	Loads the full trips file (799,414 rows) and the weather data from the public S3 bucket. Starts the web server and checks that / and all three chart endpoints return data.
credit-card-fraud-detection-with-snowpark	Runs the sample's Snowflake Cloud Snowpark scripts unchanged. Loads 11,033 transactions (590 frauds), 50 customers and 50 terminals, runs feature engineering, and checks the 10,927 feature rows and their 23 columns.
credit-scoring-with-snowpark	Executes every notebook cell and checks that both CSVs were written to Snowflake in full.
glue-snowflake-integration	Runs the Glue job. It opens a session with Next, reads 5 rows from src_glue and writes the 3 active rows to tgt_glue, which the test reads back.
lambda-snowpark-connector	Deploys and invokes the function. It returns 200, its logs show the 3 rows and the timestamp, and the test reads the rows back from Next.
multi-container	The AWS container logs in to Next by container name, and Next loads a CSV from LocalStack S3 through a stage.
predicting-customer-spend	Executes every notebook cell, checks the customers in Snowflake and an R-squared above 0.9.
soda-data-quality-checks	The scan reports the row count check failing (as written), and the NULL and schema checks passing.
streamlit-snowpark-dynamic-filters	Starts the app and runs its filter logic and its sample(n=5) preview against Snowflake, checking the result against the same filter in SQL.

Sample fixes

• airflow-dbt-transformation:
  • Ported to the Airflow 3 flow from the "Run Airflow and dbt pipelines on Snowflake locally" blog post.
  • dbt_project.yml configured models under my_project, but the project is test_dbt, so its schemas were never applied.
  • The DAG connects to snowflake:4566 over http.
• citi-bike-data-app:
  • Seeds into test.public, and the Node connection passes the account and schema.
  • date_trunc('day', ...) instead of a quoted identifier.
• credit-card-fraud-detection-with-snowpark:
  • The emulator-only scripts and checkpoint files are replaced by the Cloud scripts, run against Next.
  • The 10-row CSV, a single day that produced zero features, is replaced by a seeded, scaled-down simulation from the fraud detection handbook. Its generator is committed.
  • The image moves to Python 3.11, because Next rejects the decommissioned 3.8/3.9 UDF runtimes.
  • Model training is not in CI (see LAV-4520 below). The README says so.
• glue-snowflake-integration:
  • The job connects to snowflake:4566.
  • Next advertises stage uploads on a dedicated host that is aliased to the Snowflake service, so the Spark connector's PUTs reach Next while the Glue script download still goes to LocalStack S3.
  • deploy.sh fails when the job fails.
• lambda-snowpark-connector:
  • The zip never included handler.py, and the libraries are now built for the function's runtime.
  • The handler reads the Snowflake host, port and protocol from env vars.
• multi-container:
  • The health check uses bash /dev/tcp, because the Next image has no curl or python.
  • SF_S3_ENDPOINT_EXTERNAL is gone: Next serves internal stages itself.
• predicting-customer-spend:
  • The notebook used test.test, which does not exist. It now creates test and uses test.public.
  • corr(numeric_only=True) for pandas 2.
• soda-data-quality-checks:
  • The schema-change check needs Soda Cloud, so it is replaced with a required-columns check.
  • snowflake-cli-labs was renamed to snowflake-cli.
• streamlit-snowpark-dynamic-filters:
  • setup.sql used a PostgreSQL ::bit(32) cast.
  • pyarrow is pinned to 8.0.0, the version snowflake-connector-python 2.7.11 requires.
• READMEs: every sample documents Next (lstk --image or Docker Compose) and make test. The top-level README explains the two setups.

Emulator bugs

Found on Next while porting, and filed:

• LAV-4517: CTAS over staged files fails with 002043. Breaks Snowpark save_as_table from a stage. The fraud sample loads with copy_into_table.
• LAV-4518: a windowed SUM() / n is described as NUMBER(38,0), so Snowpark save_as_table silently truncates the values.
• LAV-4519: a scalar aggregate with ORDER BY on another column is rejected (002024). Snowpark 1.55 generates this by default. Needs a check against real Snowflake.
• LAV-4520: Snowpark add_packages with pinned versions fails on a fresh emulator, so the fraud sample's model UDF can't be deployed.

The earlier bugs from the Python emulator don't apply on Next:

• SAMPLE on a subquery (LAV-4420) works on Next.
• A negative ROUND scale (LAV-4421) works on Next.
• The samples no longer write into a missing schema (LAV-4422).

https://claude.ai/code/session_01DZfDmzUBPqNAu3yFKyfCNM

[hovaesco]

All sample apps should be updated to use the new Snowflake Next emulator https://linear.app/localstack/issue/LAV-2294/update-sample-apps-to-use-next-emulator

[hovaesco]

Plus we won't address regressions reported against Snowflake Python emualtor.

[HarshCasper]

@hovaesco thanks, all 10 samples now run on localstack/snowflake-next:latest and pass: https://github.com/localstack-samples/localstack-snowflake-samples/actions/runs/38553101332

• The 6 Snowflake-only samples start Next with lstk start --type snowflake --image localstack/snowflake-next:latest.
• airflow-dbt, glue, lambda and multi-container need AWS too, so they use Docker Compose with LocalStack AWS and Next side by side. The PR description explains how the MWAA, Glue and Lambda containers reach Next.
• The Python emulator workarounds are gone. SAMPLE on a subquery and a negative ROUND scale both work on Next.
• I filed four Next issues found along the way: LAV-4517 (CTAS from a stage), LAV-4518 (windowed division scale in describe), LAV-4519 (scalar aggregate with ORDER BY, needs a Cloud check) and LAV-4520 (pinned Snowpark packages). LAV-4520 is why the fraud sample's model training isn't in CI yet.

https://claude.ai/code/session_01DZfDmzUBPqNAu3yFKyfCNM