Skip to content

Disambiguate abbreviated custom slice indices in SingleSliceSpec. - #657

Merged
copybara-service[bot] merged 1 commit into
masterfrom
cl/1013668176
Nov 5, 2026
Merged

copybara-service[bot] merged 1 commit into
masterfrom
cl/1013668176

Conversation

@copybara-service

Copy link
Copy Markdown

Disambiguate abbreviated custom slice indices in SingleSliceSpec.

SingleSliceSpec.str formats unnamed custom slices by interpolating
the train and test index arrays directly. NumPy abbreviates arrays above
its print threshold (1000 elements by default) to their first and last
three values, so two custom groupings over more than 1000 examples that
agree on those six values and share a group value produce the same
string. AttackResults._group_results_by_slice keys per-slice results by
that string, so summary(by_slices=True) merges the colliding slices into
one section.

This is a report-correctness issue: an audit over several custom
subgroups can report the best attack of two subgroups under one label
and drop the other subgroup's section. Callers can avoid it today by
giving each grouping distinct group values and naming them via
SlicingSpec.custom_slices_names.

When NumPy would abbreviate an index array, the slice string now appends
its shape, dtype and a truncated SHA-256 digest of its contents; smaller
arrays format exactly as before. Slice strings (and summary keys) for
large unnamed custom slices therefore change. Tests cover int and bool
groupings of 2000 examples that differ only mid-array, determinism,
unchanged output at the threshold, and separate per-slice summary
sections.

@copybara-service
copybara-service Bot force-pushed the cl/1013668176 branch 2 times, most recently from cb023f9 to 831b7b3 Compare November 5, 2026 15:33
SingleSliceSpec.__str__ formats unnamed custom slices by interpolating
the train and test index arrays directly. NumPy abbreviates arrays above
its print threshold (1000 elements by default) to their first and last
three values, so two custom groupings over more than 1000 examples that
agree on those six values and share a group value produce the same
string. AttackResults._group_results_by_slice keys per-slice results by
that string, so summary(by_slices=True) merges the colliding slices into
one section.

This is a report-correctness issue: an audit over several custom
subgroups can report the best attack of two subgroups under one label
and drop the other subgroup's section. Callers can avoid it today by
giving each grouping distinct group values and naming them via
SlicingSpec.custom_slices_names.

When NumPy would abbreviate an index array, the slice string now appends
its shape, dtype and a truncated SHA-256 digest of its contents; smaller
arrays format exactly as before. Slice strings (and summary keys) for
large unnamed custom slices therefore change. Tests cover int and bool
groupings of 2000 examples that differ only mid-array, determinism,
unchanged output at the threshold, and separate per-slice summary
sections.

PiperOrigin-RevId: 1014685339
@copybara-service
copybara-service Bot merged commit 8450e16 into master Nov 5, 2026
8 checks passed
@copybara-service
copybara-service Bot deleted the cl/1013668176 branch November 5, 2026 15:35
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant