Skip to content

fix(stages): Prevent silent data loss during currency parsing in stage_dtypes #533

Open

Description

@JohnnyWilson16

Summary

In src/freshdata/_core/stages.py:82-85, currency parsing uses heuristic thresholding to coerce object/string columns into floats:

matches_pattern = sample.apply(lambda v: bool(CURRENCY_REGEX.match(v)))
if has_symbol and (matches_pattern.sum() / len(sample)) >= 0.7:
    cleaned = out[col].astype(str).str.replace(r"[^\d.-]", "", regex=True)
    out[col] = pd.to_numeric(cleaned, errors="coerce")
    actions.append(ActionRecord("dtypes", str(col), "parse_currency", {"target_dtype": "float64"}, len(out)))

The Problem

If 70% of a column's sample values match currency (e.g. "", ""), but the remaining 30% contain valid non-currency strings (such as product identifiers, comments, tracking IDs, or "N/A"), pd.to_numeric(cleaned, errors="coerce") silently converts all remaining 30% of rows to NaN.

This results in irreversible silent data loss without warning the user or recording the destroyed rows in ActionRecord.

Proposed Solution

  1. Check for unparseable non-null entries before applying errors="coerce".
  2. If non-currency non-null values are present in the column:
    • Either: Do not coerce the column to float if it contains non-trivial textual data (e.g., if coerced unparseable non-null count exceeds an allowable threshold).
    • Or: Record an explicit ActionRecord indicating how many non-conforming rows were affected, or preserve original values in a separate column.
  3. Add a test in tests/test_stages.py verifying that a mixed column (e.g. ["$10.00", "$20.00", "REF-9921"]) does not silently wipe "REF-9921" into NaN.

Acceptance Criteria

  • Non-numeric strings in mixed columns are not silently destroyed without tracking.
  • CleanReport and ActionRecord report accurate rows_affected and metadata.
  • All 43 test suite assertions pass.

👋 Want to work on this issue?
Comment /assign below to claim it! Check out the First Contribution Guide for setup instructions.

Activity

  1. alibro005 commented on Nov 19, 2026

    @alibro005
    Contributor

    /assign

  2. github-actions commented on Nov 19, 2026

    @github-actions

    Hey @alibro005! 🎉 You have claimed this issue.

    Quick Onboarding Checklist

    • 📖 Read the First Contribution Guide.
    • ⚡ Fast local test iteration: pytest tests/<test_file>.py --no-cov (bypasses CI 93% global gate during local development).
    • 🔍 Lint & style checks: ruff check .
    • 💬 If you have questions or need clarification, ask anytime here in the comments!

    (Need to unassign yourself? Simply reply with /unassign.)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Labels

bugSomething isn't workinghelp wantedExtra attention is neededin-progressAn open issue currently being worked on

Type

No type

Projects

No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions