You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
fix(stages): Prevent silent data loss during currency parsing in stage_dtypes #533
If 70% of a column's sample values match currency (e.g. "", ""), but the remaining 30% contain valid non-currency strings (such as product identifiers, comments, tracking IDs, or "N/A"), pd.to_numeric(cleaned, errors="coerce")silently converts all remaining 30% of rows to NaN.
This results in irreversible silent data loss without warning the user or recording the destroyed rows in ActionRecord.
Proposed Solution
Check for unparseable non-null entries before applying errors="coerce".
If non-currency non-null values are present in the column:
Either: Do not coerce the column to float if it contains non-trivial textual data (e.g., if coerced unparseable non-null count exceeds an allowable threshold).
Or: Record an explicit ActionRecord indicating how many non-conforming rows were affected, or preserve original values in a separate column.
Add a test in tests/test_stages.py verifying that a mixed column (e.g. ["$10.00", "$20.00", "REF-9921"]) does not silently wipe "REF-9921" into NaN.
Acceptance Criteria
Non-numeric strings in mixed columns are not silently destroyed without tracking.
CleanReport and ActionRecord report accurate rows_affected and metadata.
All 43 test suite assertions pass.
👋 Want to work on this issue?
Comment /assign below to claim it! Check out the First Contribution Guide for setup instructions.
Activity
alibro005 commentedon Nov 19, 2026
/assign
github-actions commentedon Nov 19, 2026
Hey @alibro005! 🎉 You have claimed this issue.
Quick Onboarding Checklist
pytest tests/<test_file>.py --no-cov(bypasses CI 93% global gate during local development).ruff check .(Need to unassign yourself? Simply reply with
/unassign.)