Repository navigation
Tree traversal fixes and design notes - #3261
Merged
Merged
Conversation
harendra-kumar
commented
Oct 18, 2026
harendra-kumar
commented
Member
- altBfsConcatIterate is renamed to wideDfsConcatIterateRev, added wideDfsConcatIterate.
- The unfold BFS traversals no longer keep a stream open for every pending element.
- unfoldIterate is 5-17x faster. Leaves are marked with b -> Maybe a, so they are neither held nor unfolded.
- Breaking: parConcatIterate now takes a -> Maybe (Stream m a), like concatIterate.
- Moved the tree combinators to Streamly.Internal.Data.Stream.Tree.
- Added design notes in docs/Developer/tree-traversal.md.
…catIterate altBfsConcatIterate was documented as a breadth first traversal emitting each level in reverse order, but it is a depth first traversal that emits all the children of a node together, then expands each of them, the last child first. For the tree 1->[2,3], 2->[4], 3->[5], 4->[7], 5->[6] it emits [1,2,3,5,6,4,7]. Each stream is run to completion before the next one is started, so only one stream is open at a time: a directory tree traversal keeps one directory open, whereas concatIterate keeps one open for each level of the current path. Rename it to wideDfsConcatIterateRev, keeping altBfsConcatIterate and concatIterateBfsRev as deprecated aliases. Add wideDfsConcatIterate, which expands the children in order, [1,2,3,4,7,5,6], the order of ls -R. Document the defining characteristic and the resource use of concatIterate, bfsConcatIterate and wideDfsConcatIterate. Run the concat based BFS directory benchmarks on the big tree and add benchmarks and tests for wideDfsConcatIterate. Rename the DirIO ConcatBfsRev, ChunkBfsRev and ChunkFoldBfsRev benchmarks and tests to WideDfsRev.
Hide the stack of suspended unfold states from SpecConstr using a NoSpecConstr newtype. SpecConstr specialized the loop on every stack shape: with -fspec-constr-recursive=16 compilation ran out of memory, with the default limit the loop fell back to boxed state beyond a stack depth of two. On an in-memory tree unfoldIterate is now 5-17x faster than concatIterate. bfsUnfoldIterate and altBfsUnfoldIterate injected every element as it was emitted, keeping a stream (e.g. a directory) open for every pending element. They now keep the elements and inject them when their turn comes. Rename altBfsUnfoldIterate to wideDfsUnfoldIterateRev and add wideDfsUnfoldIterate. Enable the iterate tests that were disabled for the compile blowup, run the unfold BFS DirIO benchmarks and tests on the big tree, and add in-memory tree traversal benchmarks for all the iterate combinators. Rename the DirIO UnfoldBfsRev benchmark and test to UnfoldWideDfsRev.
These BFS variants start the streams of a level in the reverse of the order in which their parents were emitted, avoiding the reverse of the pending list at each level. bfsUnfoldIterateRev is about 2x faster than bfsUnfoldIterate on an in-memory tree; bfsConcatIterateRev is 30% faster on a binary tree of 10^6 nodes and within noise on smaller trees.
…ules Stream.Type.Iterate was at the 500 MB GHC heap limit, so a new benchmark could not be added to it. Move the concatIterate traversals to Stream.Type.ConcatIterate and the unfoldIterate traversals to Stream.Type.UnfoldIterate. The benchmark names are unchanged.
The unfold traversals took an Unfold m a a and could not tell a leaf
from a node with children, so the BFS and wide DFS traversals held every
pending element, e.g. the paths of the files as well as those of the
subdirectories of a directory tree.
All unfold traversals now take a function b -> Maybe a and an
Unfold m a b. An element for which the function returns Just a is
expanded by unfolding the seed a, an element for which it returns
Nothing is a leaf: it is not held, and no unfold is started for it.
Add unfoldIterateMaybe and bfsUnfoldIterateMaybe, and deprecate the
released unfoldIterate, bfsUnfoldIterate and altBfsUnfoldIterate in
favour of unfoldIterateMaybe, bfsUnfoldIterateMaybe and
wideDfsUnfoldIterateRev, implementing them as xMaybe Just. The
unreleased wideDfsUnfoldIterate, wideDfsUnfoldIterateRev and
bfsUnfoldIterateRev change their signature.
Summing the nodes of complete trees of 10^6 nodes held in memory,
bfsUnfoldIterate vs bfsUnfoldIterateMaybe:
tree time peak memory
binary 95 ms -> 46 ms 57 MB -> 33 MB
10-ary 145 ms -> 8.8 ms 104 MB -> 12 MB
1000-ary 134 ms -> 4.2 ms 129 MB -> 6 MB
and the other traversals, old vs new (ms):
binary 10-ary 1000-ary
unfoldIterate(Maybe) 9.00 -> 5.76 8.67 -> 2.86 8.40 -> 2.01
wideDfsUnfoldIterate 12.9 -> 9.37 12.3 -> 4.99 13.6 -> 3.78
wideDfsUnfoldIterateRev 5.64 -> 5.78 6.68 -> 4.66 6.96 -> 4.06
bfsUnfoldIterateRev 61.5 -> 31.5 94.0 -> 7.06 78.8 -> 4.32
The notes in docs/Developer/tree-traversal.md start from the primitive iterate form and cover the traversal orders and their resource use, marking the leaves with Maybe or Either, the concat and unfold forms, the StreamK forms, chunked elements, stateful and graph traversal, concurrent traversal and bottom-up reduction. They also list the current API, its naming, the gaps in what it can express, implementation notes and benchmark results.
parConcatIterate took a -> Stream m a, while concatIterate takes a -> Maybe (Stream m a). It now takes the same function as concatIterate: Nothing marks a leaf, which is emitted with K.fromPure instead of iterating an empty stream through chanConcatMapK. This is a breaking change; parConcatIterate cfg (Just . f) keeps the old behaviour. In-memory trees of 100,001 Int nodes, -N1, old code path vs Nothing: binary, default: 35.2 ms, 119 MB -> 34.3 ms, 111 MB binary, ordered True: 40.9 ms, 148 MB -> 37.4 ms, 135 MB 10-ary, default: 31.5 ms, 112 MB -> 30.3 ms, 94 MB 10-ary, ordered True: 37.5 ms, 138 MB -> 33.1 ms, 113 MB The FileSystem.DirIO benchmark shows no difference, its tree has no files and hence no leaves. Update the DirIO bench/test helpers, the tutorial example, the changelog and the design notes.
Move the unfoldIterate, concatIterate and foldIterate combinators and their tests from Stream.Type to a new Streamly.Internal.Data.Stream.Tree module, re-exported from Streamly.Internal.Data.Stream.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.