Skip to content

Create Nested(...) columns as Array(Tuple(...)) - #575

Open
Hakaou wants to merge 1 commit into
ClickHouse:masterfrom
Hakaou:nested-as-array-of-tuple
Open

Hakaou wants to merge 1 commit into
ClickHouse:masterfrom
Hakaou:nested-as-array-of-tuple

Conversation

@Hakaou

@Hakaou Hakaou commented Oct 17, 2026 •

Copy link
Copy Markdown

Closes #571.

Summary

TypeParser had no entry for Nested, so the name became a Void terminal. At the root ValidateAST rejects it ("unsupported column type"), but inside a container nothing checks it: Array(Nested(key String, value String)) was created as Array(Void), ColumnNothing reads one byte per row and the block stream desynchronises.

Nested is now its own AST node, next to Map and SimpleAggregateFunction, and the column factory builds it in the Tuple case, wrapped in an Array. That is how the server defines it: DataTypeNested.cpp builds DataTypeArray(DataTypeTuple(types, names)) under a custom name. Sharing the Tuple case keeps the AST read by reference, so a deeply nested type is not copied at every level, the pattern #588 removes from LowCardinality.

Behavior changes

  • Array(Nested(...)) and Map(K, Nested(...)) columns decode, with their field names kept on the Tuple.
  • A top-level Nested column, which the server sends with flatten_nested = 0, decodes instead of failing with "unsupported column type".
  • The created column reports the structural type name Array(Tuple(key String, value UInt8)) rather than Nested(...). The server accepts that name on INSERT: a block selected from such a column goes back in as is (ArrayOfNested_ReinsertSelectedRows), and clickhouse local 26.11 also accepts unnamed tuples. A dedicated Nested column type would add public API for no behavioural gain, so I did not go that way. Happy to discuss.
  • TypeAst::Meta gains the Nested value.

Tests

  • type_parser_ut: the Nested node with its named elements, and a field itself named Nested.
  • CreateColumnByType_ut: Nested, Array(Nested), Map(String, Nested) and Nested(a Nested(...)) give the expected column types. DeepNestedDoesNotCopyAst builds 200 levels of Nested over 8 000 fields and checks that peak RSS grows by less than 16 MiB, about 180 MiB when the node is copied at each level. The measure runs in a child process (EXPECT_EXIT, hence the DeathTest suite) with its own peak counter, so a peak reached earlier in the test binary cannot hide a regression. It reads getrusage, so it runs on Linux and macOS and skips on Windows.
  • client_ut, both fixtures (no compression and LZ4): select from a server-typed Array(Nested(...)) column with empty inner and outer arrays, insert a selected block back as is and read it back with toString, and a top-level Nested column with flatten_nested = 0.

Each new test fails when the Nested entry in GetTypeMeta is removed. Before this change, on master fc5d98f:

Case Before After
CreateColumnByType("Array(Nested(key String, value UInt8))") Array(Void) Array(Array(Tuple(key String, value UInt8)))
Select on that column, no compression Can't cast from Void 3 rows, values match
Select on that column, LZ4 some data was not read 3 rows, values match
Select on a Nested column, flatten_nested = 0 unsupported column type: Nested(key String, value UInt8) 2 rows, values match
Select, then insert the block back fails at the Select both rows read back identically

Full suite green locally (macOS arm64, Apple clang 21) against ClickHouse 26.11.1 master build, the new client tests also against 25.12 and 26.3. Under AddressSanitizer and UBSan, the parser and factory suites pass on this branch, and the client and roundtrip suites with #590 on top.

Out of scope

@CLAassistant

CLAassistant commented Oct 17, 2026 •

Copy link
Copy Markdown

CLA assistant check
All committers have signed the CLA.

return std::make_shared<ColumnTuple>(columns, std::move(names));
auto tuple = any_named ? std::make_shared<ColumnTuple>(columns, std::move(names))
: std::make_shared<ColumnTuple>(columns);
if (ast.meta == TypeAst::Nested) {

@Hakaou Hakaou Oct 19, 2026 •

Copy link
Copy Markdown
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nested takes the Tuple path and is wrapped here, so its elements are read from the AST by reference. An earlier version of this PR copied the node into a Tuple first, which costs the subtree at every level: about 180 MiB of peak RSS for 200 levels over 8 000 fields, the pattern #588 removes from LowCardinality. DeepNestedDoesNotCopyAst holds it.

The type parser had no entry for Nested, so a column typed
Array(Nested(key String, value String)) was created as Array(Void)
and the native block stream fell out of sync ("Can't cast from Void",
"some data was not read"). A top-level Nested column, which the server
sends with flatten_nested = 0, was rejected as unsupported.

The server defines Nested as a custom name over Array(Tuple(...))
(DataTypeNested.cpp). The parser now recognises Nested as its own
node, like Map and SimpleAggregateFunction, and the column factory
builds it in the Tuple case, wrapped in an Array. Sharing that case
keeps the AST read by reference, so a deeply nested type is not
copied at every level.

Closes ClickHouse#571
@Hakaou
Hakaou force-pushed the nested-as-array-of-tuple branch from 10aa714 to b414b18 Compare October 19, 2026 15:50
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Array(Nested(...)) columns silently decode as Array(Nothing) and desynchronize the native block stream

2 participants