fix(scd2): preserve NULL values in historical rows

[Arthur031221]

Description

Users running SCD_TYPE_2_BY_TIME or SCD_TYPE_2_BY_COLUMN models lost historical NULL values when a later source row supplied a value for the same key. updated_rows used COALESCE(target, source), so a NULL target fell through to the new source value. It now uses the target row's existence flag to keep its values, including NULL. The existence check also preserves target payloads when a key or validity timestamp is NULL. Its alias is chosen to avoid source and prefixed target column names.

This builds on the code @ionnich posted in issue #5332. Fixes #5332.

Test Plan

make fast-test, make style, make doc-test, and make benchmark-ci passed on Linux with Python 3.12.3. The four history regression cases cover both SCD2 kinds with SQL and Pandas inputs. They fail against upstream production source and pass with the fix. Four additional cases verify that removing a source key preserves the key and payload of an existing target row whose validity timestamp is NULL. Sixteen additional cases cover TEXT and BOOLEAN columns that collide with internal flag names, including case-insensitive names. The collision cases pass against upstream, fail with a constant flag name, and pass with the fix. All 24 added SCD2 regression cases pass with the fix.

Checklist

• I have run make style and fixed any issues
• I have added tests for my changes (if applicable)
• All existing tests pass (make fast-test)
• My commits are signed off (git commit -s) per the DCO

Co-authored-by: Nicholas 71755971+ionnich@users.noreply.github.com

[Arthur031221]

Anything else you need from me on this one?

[Vamsi-klu]

The explicit target-existence marker preserves historical NULLs without using a nullable key or validity timestamp as the existence test. The case-insensitive collision handling also covers both source columns and prefixed target columns.

Validation at bf1d818a passed 36 targeted SCD tests, including the DuckDB executions. All four new NULL-history regression cases fail against the base production code and pass with this change. No blocking issue found. GitHub currently reports 29 successful checks and 5 skipped checks.

A release note would help clarify that this prevents future history corruption; it does not reconstruct historical values already overwritten by earlier runs.