Skip to content

feat(sources): let the CSV reader accept larger cells - #789

Merged
karhu merged 6 commits into
mainfrom
feature/csv-reader-max-field-chars
Nov 20, 2026
Merged

karhu merged 6 commits into
mainfrom
feature/csv-reader-max-field-chars

Conversation

@karhu

@karhu karhu commented Nov 17, 2026 •

Copy link
Copy Markdown
Contributor

The csv module of Python stops at a cell of 131,072 characters by default. The M4-Caption release of OpenTSLM stores a whole series in one cell, and its largest cell has 169,126 characters.

CsvDataReader now lifts that limit before it reads, so a cell can have any length and no connector has to set a limit. The csv module keeps one limit for the whole process. The change also applies to other code in the process that uses the csv module. The reader uses min(sys.maxsize, 2**31 - 1), because Windows rejects a larger value.

The limit mainly protects against an unclosed quote that pulls the rest of a file into one cell. Such a file still fails, because the row then has too few cells. Every source has a pinned SHA-256. If a source file is broken, the build of its connector fails.

The csv module of Python stops at a cell of 131,072 characters by
default. The M4-Caption release of OpenTSLM stores a whole series in
one cell, and its largest cell has 169,126 characters.

CsvDataReader now takes max_field_chars. Before the reader parses, it
raises the limit of the csv module to that value. The limit applies to
the whole process, so the reader never lowers it. A lower limit could
stop another reader in the middle of a file.
@karhu
karhu added this pull request to stack #791 November 17, 2026 01:25
@karhu karhu added the area:connectors Changes packages/timenet-connectors label Nov 17, 2026
@karhu karhu changed the title feature/csv reader max field chars feat(sources): let the CSV reader accept larger cells Nov 17, 2026
The cell limit of the csv module applies to the whole process. The
max_field_chars option looked like a setting of one reader, but it
changed the limit for every later reader too.

Replace the option with CsvDataReader.raise_field_size_limit. The
caller now raises the limit in plain sight before it reads. The method
still never lowers the limit.
Rename CsvDataReader.raise_field_size_limit to set_field_size_limit.
It now sets the limit to the given value, also when the value is lower.
No caller reads two CSV files with different limits at the same time.
@karhu
karhu requested a review from TimoStoff November 17, 2026 17:29
The csv module accepts zero or a negative limit, and then rejects every
nonempty cell. set_field_size_limit now raises TimeFValidationError for
a limit below 1 and leaves the limit as it was.
@karhu
karhu marked this pull request as ready for review November 17, 2026 23:30
@karhu
karhu requested a review from RealLast November 18, 2026 00:17

@RealLast RealLast left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM but please check comment

CsvDataReader now lifts the cell limit of the csv module before it reads, so no connector has to set it. The set_field_size_limit method goes away.
@github-actions

github-actions Bot commented Nov 18, 2026 •

Copy link
Copy Markdown

✅ CI checks passed

Check Status
Quick checks ✅ Passed
Tests (Python 3.13) ✅ Passed
Tests (Python 3.11) ✅ Passed

Updated for b382ede · run #8481

@karhu
karhu merged commit ccde489 into main Nov 20, 2026
4 checks passed
@karhu
karhu deleted the feature/csv-reader-max-field-chars branch November 20, 2026 17:30
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

area:connectors Changes packages/timenet-connectors

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants