Repository navigation
Combine involved user and group identity link checks into a single EXISTS - #4310
Merged
Merged
Conversation
…ISTS When a query restricts by a user or any of their groups, the identity link checks were rendered as EXISTS(user link) OR EXISTS(group link). Most databases (Postgres, MySQL, CockroachDB, SQL Server, DB2) can only turn an EXISTS into a semi-join when it is not inside an OR, so they ran both subqueries for every candidate row. On Postgres the resulting cost estimate also triggered JIT compilation on every call. The user and group conditions now go into one EXISTS with the OR inside it. The result is the same, and when nothing else is in the OR the EXISTS can be executed as a semi-join. This applies to: * the or() blocks of the process, historic process, case and historic case instance queries, for involvedUser, involvedGroups and the typed involvedUser(userId, type) / involvedGroup(groupId, type) * startableByUser / startableByUserOrGroups for process and case definitions * forUserOrGroups for external worker job queries and acquisition * the or() blocks of the task and historic task queries
ACT_RU_IDENTITYLINK has an index on GROUP_ID_, but ACT_HI_IDENTITYLINK only has one on USER_ID_. Historic process, case and task queries that restrict by involved groups therefore had to scan the whole historic identity link table instead of looking up the matching links.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
When a query restricts by a user or any of their groups, the identity link checks were rendered as
EXISTS(user link) OR EXISTS(group link). Most databases can only turn anEXISTSinto a semi-join when it is not inside anOR, so they ran both subqueries for every candidate row. On Postgres the resulting cost estimate also lands around the JIT thresholds, so every call paid for JIT compilation.The user and group conditions now go into one
EXISTSwith theORinside it. The rows returned are the same, and when nothing else is in theORthe database can execute it as a semi-join. This applies to:or()blocks of the process, historic process, case and historic case instance queries, forinvolvedUser,involvedGroupsand the typedinvolvedUser(userId, type)/involvedGroup(groupId, type)startableByUser/startableByUserOrGroupsfor process and case definitionsforUserOrGroupsfor external worker job queries and acquisitionor()blocks of the task and historic task queries (there the assignee and owner checks stay next to theEXISTS, so this only saves one subquery per task)The second commit adds an index on
GROUP_ID_ofACT_HI_IDENTITYLINK(ACT_RU_IDENTITYLINKalready has one), so historic queries by involved groups no longer have to scan the whole historic identity link table. This bumps the schema version to 8.1.0.3.Measurements
Count query of
or().involvedUser(user).involvedGroups(groups).endOr(), on a dataset with about 15k running process instances, 6k running case instances, 50k historic process instances, 90k runtime and 520k historic identity links. Median time in ms for a user with 500 links (and one with 10k links), before and after:ORofEXISTScannot be pulled up into a semi-join. The cost estimate drops from about 530k to about 5k, below the JIT thresholds. With JIT off both shapes take about the same time.ORofEXISTSruns as two dependent subqueries per row. The singleEXISTSuses an index merge over the user and group indexes and a semi-join.ORofEXISTSinto aUNION ALLview, so it is neutral (same buffer gets).GROUP_ID_index.Check List: