LocalStack for Snowflake Samples

This repository contains sample projects that can be deployed on LocalStack for Snowflake to demonstrate local development and testing workflows for Snowflake data applications.

The samples run on the next-generation Snowflake emulator, localstack/snowflake-next. It emulates Snowflake only, so the samples that also use AWS services run LocalStack AWS next to it with Docker Compose.

Prerequisites

• A valid LocalStack for Snowflake license with access to the Snowflake Next preview. Your license provides a LOCALSTACK_AUTH_TOKEN.
• Docker
• lstk 1.3.0 or later, which starts the emulator with lstk start --type snowflake --image localstack/snowflake-next:latest
• Docker Compose and the awslocal CLI for the samples that also use AWS services: airflow-dbt-transformation, glue-snowflake-integration, lambda-snowpark-connector and multi-container. Each of them has a docker-compose.yml that starts LocalStack AWS on port 4566 and Snowflake Next on a second port.
• Snowflake CLI (snow) with a local and a localstack connection, see below

Configuration

Set your auth token before running any sample:

export LOCALSTACK_AUTH_TOKEN=<your-auth-token>

Alternatively, log in with lstk login.

Several samples use the Snowflake CLI with a connection named local or localstack. Create both, pointing at the emulator:

for name in local localstack; do
  snow connection add --connection-name $name --user test --password test \
    --account test --host snowflake.localhost.localstack.cloud --no-interactive
done

Outline

Project Name	Description
airflow-dbt-transformation	Run local data transformation pipelines with Airflow, dbt, and Snowpark.
citi-bike-data-app	Seed Citibike data into Snowflake and query it from a web application.
credit-card-fraud-detection-with-snowpark	Feature engineering and ML-oriented fraud workflows with Snowpark.
credit-scoring-with-snowpark	Exploratory data analysis for credit scoring using Snowpark.
glue-snowflake-integration	Integrate AWS Glue ETL jobs with Snowflake in a local environment.
lambda-snowpark-connector	Use Snowpark from an AWS Lambda function running on LocalStack.
multi-container	Run LocalStack for Snowflake and LocalStack AWS emulation in separate containers.
predicting-customer-spend	Predict customer spend using Snowpark and Python ML tooling.
soda-data-quality-checks	Execute Soda data quality checks against local Snowflake tables.
streamlit-snowpark-dynamic-filters	Streamlit application with dynamic Snowpark-powered filters.

Testing

A fresh Snowflake Next instance only has the SNOWFLAKE database, as in a new Snowflake account, so each sample creates the database it uses before loading data.

Every sample has make start (or make start-localstack where make start runs the app) to start LocalStack, and make test to check that the sample works. The CI workflow runs make install, the start target and make test for each sample on every pull request.

Checking Out A Single Sample

To check out only one sample directory:

mkdir localstack-snowflake-samples && cd localstack-snowflake-samples
git init
git remote add origin -f git@github.com:localstack-samples/localstack-snowflake-samples.git
git config core.sparseCheckout true
echo <LOCALSTACK_SAMPLE_DIRECTORY_NAME> >> .git/info/sparse-checkout
git pull origin master

The commands above use sparse checkout to pull only the sample you need.