Skip to content

Ignore empty epsilon bounds when selecting the max-epsilon MIA result. - #655

Open
copybara-service[bot] wants to merge 1 commit into
masterfrom
cl/1013659039
Open

copybara-service[bot] wants to merge 1 commit into
masterfrom
cl/1013659039

Conversation

@copybara-service

Copy link
Copy Markdown

Ignore empty epsilon bounds when selecting the max-epsilon MIA result.

AttackResults.get_result_with_max_epsilon averages each result's epsilon lower bounds and takes np.argmax. EpsilonLowerBound.compute_epsilon_lower_bound keeps only positive ratio bounds, so on small slices or slices with little measurable leakage every method can return an empty array, and EpsilonLowerBoundValue.get_max_epsilon_bounds then returns an empty array too. The mean of an empty array is NaN, and np.argmax always returns the index of the first NaN, so one empty-bounds slice wins the selection over slices with large finite epsilon. Calling the method on an empty AttackResults also raised ValueError instead of returning None like the sibling AUC, advantage and PPV selectors.

The overall and per-slice "top epsilon lower bounds" lines of AttackResults.summary then report an empty bound on a zero-leakage slice and hide the slice with the strongest empirical privacy-loss evidence, so audits can understate measured epsilon. AUC, attacker advantage and PPV selection are unaffected.

Empty and NaN average bounds now rank as negative infinity, each result's bounds are computed once, and empty results return None, matching the return type. Tests cover empty results, an empty-bounds slice placed ahead of a high-epsilon slice (selection and summary), negative finite bounds beating empty ones, and an all-empty input that no longer crashes; a synthetic NaN-bounds case is included defensively, since the library itself never emits NaN bounds.

`AttackResults.get_result_with_max_epsilon` averages each result's epsilon lower bounds and takes `np.argmax`. `EpsilonLowerBound.compute_epsilon_lower_bound` keeps only positive ratio bounds, so on small slices or slices with little measurable leakage every method can return an empty array, and `EpsilonLowerBoundValue.get_max_epsilon_bounds` then returns an empty array too. The mean of an empty array is NaN, and `np.argmax` always returns the index of the first NaN, so one empty-bounds slice wins the selection over slices with large finite epsilon. Calling the method on an empty `AttackResults` also raised ValueError instead of returning None like the sibling AUC, advantage and PPV selectors.

The overall and per-slice "top epsilon lower bounds" lines of `AttackResults.summary` then report an empty bound on a zero-leakage slice and hide the slice with the strongest empirical privacy-loss evidence, so audits can understate measured epsilon. AUC, attacker advantage and PPV selection are unaffected.

Empty and NaN average bounds now rank as negative infinity, each result's bounds are computed once, and empty results return None, matching the return type. Tests cover empty results, an empty-bounds slice placed ahead of a high-epsilon slice (selection and summary), negative finite bounds beating empty ones, and an all-empty input that no longer crashes; a synthetic NaN-bounds case is included defensively, since the library itself never emits NaN bounds.

PiperOrigin-RevId: 1013659039
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants