[balit-raibot]

Summary

Migrates the Opportunity Insights: The Opportunity Atlas Outcomes (OpportunityInsightsOutcomes) import from the legacy google3 Borg script (//depot/google3/datacommons/mcf/oi_v3/opportunity_atlas_outcomes_v2.py) to an automated stat_var_processor.py-based pipeline under statvar_imports/opportunity_insights_outcomes/, and expands coverage to include the 2024 Opportunity Atlas data refresh (1984–1989 late-cohort tract outcomes and 1978–1992 single-year birth cohort county/commuting-zone estimates).

Bug: b/542866823

What's Added

• Data Fetcher (download.py): Scrapes the Opportunity Insights catalog (https://opportunityinsights.org/data/) with canonical fallback URLs, downloads raw CSV/ZIP archives (cz_outcomes.csv, county_outcomes.zip, tract_outcomes.zip, tract_outcomes_late_simple.csv, Table_3_County_by_Cohort_Estimates.csv, Table_4_cz_by_cohort_estimates.csv), and extracts them into raw_data/.
• Preprocessor (preprocess.py): Normalizes multi-word column headers, formats geoId DCIDs (geoId/cz{cz:05d}, geoId/{state:02d}{county:03d}, geoId/{state:02d}{county:03d}{tract:06d}), writes compact seed CSVs into input_files/*_cleaned.csv for stat_var_processor.py header resolution, and shards the 283.56M observations in parallel into output/output_part_*.csv.
• StatVar Processor Config (pvmap.csv, metadata.csv, manifest.json):
  • Maps normalized column tokens into OpportunityInsightsCohort StatisticalVariable and StatVarObservation properties across demographic dimensions (race, gender, parentIncome) and statistical types (measuredValue, stdError, meanValue, meanStdError, sampleSize).
  • Configures automated monthly refresh (0 8 1 * *) and resource limits (32 CPU, 512 GB RAM, 400 GB disk).
• Tests & Sample Data (download_test.py, test_data/): Unit and end-to-end integration tests covering URL discovery, archive extraction, header normalization, wide-CSV sharding, and stat_var_processor.py output equivalence.
• Documentation (README.md): Documents source files, pipeline steps, schema/cohort coverage, and dc-import (genmcf) validation results.

Validation & Metrics (dc-import genmcf)

• Status: Succeeded with 0 errors / 0 fatals
• Total CSV Rows Processed (NumRowSuccesses): 283,560,361
• Total MCF Nodes Generated (NumNodeSuccesses): 283,658,483
• Unique StatisticalVariables: 10,596 across 741 commuting zones, ~3,221 counties, and ~73,438 census tracts

References

• Data Source: Opportunity Insights Data Library
• Piper Schema & Manifest CL: cl/990390911
• Import Doc: Document
• Summary Report: CNS Summary Report
• Report JSON: CNS report.json
• Differ Files: CNS Dataflow Diff
• CRA Report: CRA Report

[gemini-code-assist]

Code Review

This pull request implements a new data import pipeline for the Opportunity Insights Opportunity Atlas Outcomes dataset, including scraping, downloading, preprocessing, and testing scripts. The feedback highlights several opportunities to make the pipeline more robust: explicitly filtering out common missing value placeholders (such as 'NA' or '.'), handling float-like strings when formatting FIPS and CZ codes, raising exceptions instead of warnings when expected files are missing to ensure early pipeline failure, and improving the macOS metadata exclusion logic when extracting ZIP files.