Repository navigation
Conversation
Hakaou
force-pushed
the
nested-as-array-of-tuple
branch
4 times, most recently
from
October 18, 2026 14:34
1ef19aa to
feb0f8e
Compare
Hakaou
force-pushed
the
nested-as-array-of-tuple
branch
from
October 19, 2026 15:18
feb0f8e to
10aa714
Compare
Hakaou
commented
Oct 19, 2026
| return std::make_shared<ColumnTuple>(columns, std::move(names)); | ||
| auto tuple = any_named ? std::make_shared<ColumnTuple>(columns, std::move(names)) | ||
| : std::make_shared<ColumnTuple>(columns); | ||
| if (ast.meta == TypeAst::Nested) { |
Author
There was a problem hiding this comment.
Nested takes the Tuple path and is wrapped here, so its elements are read from the AST by reference. An earlier version of this PR copied the node into a Tuple first, which costs the subtree at every level: about 180 MiB of peak RSS for 200 levels over 8 000 fields, the pattern #588 removes from LowCardinality. DeepNestedDoesNotCopyAst holds it.
The type parser had no entry for Nested, so a column typed
Array(Nested(key String, value String)) was created as Array(Void)
and the native block stream fell out of sync ("Can't cast from Void",
"some data was not read"). A top-level Nested column, which the server
sends with flatten_nested = 0, was rejected as unsupported.
The server defines Nested as a custom name over Array(Tuple(...))
(DataTypeNested.cpp). The parser now recognises Nested as its own
node, like Map and SimpleAggregateFunction, and the column factory
builds it in the Tuple case, wrapped in an Array. Sharing that case
keeps the AST read by reference, so a deeply nested type is not
copied at every level.
Closes ClickHouse#571
Hakaou
force-pushed
the
nested-as-array-of-tuple
branch
from
October 19, 2026 15:50
10aa714 to
b414b18
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Closes #571.
Summary
TypeParserhad no entry forNested, so the name became aVoidterminal. At the rootValidateASTrejects it ("unsupported column type"), but inside a container nothing checks it:Array(Nested(key String, value String))was created asArray(Void),ColumnNothingreads one byte per row and the block stream desynchronises.Nestedis now its own AST node, next toMapandSimpleAggregateFunction, and the column factory builds it in theTuplecase, wrapped in anArray. That is how the server defines it:DataTypeNested.cppbuildsDataTypeArray(DataTypeTuple(types, names))under a custom name. Sharing theTuplecase keeps the AST read by reference, so a deeply nested type is not copied at every level, the pattern #588 removes fromLowCardinality.Behavior changes
Array(Nested(...))andMap(K, Nested(...))columns decode, with their field names kept on theTuple.Nestedcolumn, which the server sends withflatten_nested = 0, decodes instead of failing with "unsupported column type".Array(Tuple(key String, value UInt8))rather thanNested(...). The server accepts that name onINSERT: a block selected from such a column goes back in as is (ArrayOfNested_ReinsertSelectedRows), andclickhouse local26.11 also accepts unnamed tuples. A dedicatedNestedcolumn type would add public API for no behavioural gain, so I did not go that way. Happy to discuss.TypeAst::Metagains theNestedvalue.Tests
type_parser_ut: theNestednode with its named elements, and a field itself namedNested.CreateColumnByType_ut:Nested,Array(Nested),Map(String, Nested)andNested(a Nested(...))give the expected column types.DeepNestedDoesNotCopyAstbuilds 200 levels ofNestedover 8 000 fields and checks that peak RSS grows by less than 16 MiB, about 180 MiB when the node is copied at each level. The measure runs in a child process (EXPECT_EXIT, hence theDeathTestsuite) with its own peak counter, so a peak reached earlier in the test binary cannot hide a regression. It readsgetrusage, so it runs on Linux and macOS and skips on Windows.client_ut, both fixtures (no compression and LZ4): select from a server-typedArray(Nested(...))column with empty inner and outer arrays, insert a selected block back as is and read it back withtoString, and a top-levelNestedcolumn withflatten_nested = 0.Each new test fails when the
Nestedentry inGetTypeMetais removed. Before this change, onmasterfc5d98f:CreateColumnByType("Array(Nested(key String, value UInt8))")Array(Void)Array(Array(Tuple(key String, value UInt8)))Selecton that column, no compressionCan't cast from VoidSelecton that column, LZ4some data was not readSelecton aNestedcolumn,flatten_nested = 0unsupported column type: Nested(key String, value UInt8)Select, then insert the block backSelectFull suite green locally (macOS arm64, Apple clang 21) against ClickHouse 26.11.1 master build, the new client tests also against 25.12 and 26.3. Under AddressSanitizer and UBSan, the parser and factory suites pass on this branch, and the client and roundtrip suites with #590 on top.
Out of scope
ValidateASTonly checks the root, so any other unknown type nested in a container still decodes asVoidsilently, the second point of the suggested fix in Array(Nested(...)) columns silently decode as Array(Nothing) and desynchronize the native block stream #571. Tracked separately in Unknown type names inside a container decode silently as wrong data #589, with a draft in Reject unknown type names inside containers #590.InsertDatatest reads pastTEST_DATAunder UBSan and depends on block order. Test code only, fixed separately in Make InsertData independent of block boundaries and order #576.