Repository navigation
feat(sources): let the CSV reader accept larger cells - #789
Merged
Merged
Conversation
The csv module of Python stops at a cell of 131,072 characters by default. The M4-Caption release of OpenTSLM stores a whole series in one cell, and its largest cell has 169,126 characters. CsvDataReader now takes max_field_chars. Before the reader parses, it raises the limit of the csv module to that value. The limit applies to the whole process, so the reader never lowers it. A lower limit could stop another reader in the middle of a file.
The cell limit of the csv module applies to the whole process. The max_field_chars option looked like a setting of one reader, but it changed the limit for every later reader too. Replace the option with CsvDataReader.raise_field_size_limit. The caller now raises the limit in plain sight before it reads. The method still never lowers the limit.
Rename CsvDataReader.raise_field_size_limit to set_field_size_limit. It now sets the limit to the given value, also when the value is lower. No caller reads two CSV files with different limits at the same time.
The csv module accepts zero or a negative limit, and then rejects every nonempty cell. set_field_size_limit now raises TimeFValidationError for a limit below 1 and leaves the limit as it was.
karhu
marked this pull request as ready for review
November 17, 2026 23:30
RealLast
approved these changes
Nov 18, 2026
RealLast
left a comment
Contributor
There was a problem hiding this comment.
LGTM but please check comment
CsvDataReader now lifts the cell limit of the csv module before it reads, so no connector has to set it. The set_field_size_limit method goes away.
✅ CI checks passed
Updated for |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The csv module of Python stops at a cell of 131,072 characters by default. The M4-Caption release of OpenTSLM stores a whole series in one cell, and its largest cell has 169,126 characters.
CsvDataReadernow lifts that limit before it reads, so a cell can have any length and no connector has to set a limit. The csv module keeps one limit for the whole process. The change also applies to other code in the process that uses the csv module. The reader usesmin(sys.maxsize, 2**31 - 1), because Windows rejects a larger value.The limit mainly protects against an unclosed quote that pulls the rest of a file into one cell. Such a file still fails, because the row then has too few cells. Every source has a pinned SHA-256. If a source file is broken, the build of its connector fails.