Skip to content

Add user group and member organization filters to workspace performance reports #2081

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
manuaero merged 1 commit into from
Oct 14, 2026
Merged

Add user group and member organization filters to workspace performance reports #2081

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion docs/labelbox/performance.rst
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -2,5 +2,5 @@ Performance
===============================================================================================

.. automodule:: labelbox.schema.performance
:members: PerformanceMetric, PerformanceMetricSeries, PerformanceTimeBucket, PerformanceScoreBucket, PerformanceHandlingTime, ProjectLabelerPerformance, ProjectReviewerPerformance, WorkspaceLabelerPerformance, PerformanceReportDownload, PerformanceRows, PerformanceInterval, PerformanceDeletedLabels, PerformanceMemberType
:members: PerformanceMetric, PerformanceMetricSeries, PerformanceTimeBucket, PerformanceScoreBucket, PerformanceHandlingTime, ProjectLabelerPerformance, ProjectReviewerPerformance, WorkspaceLabelerPerformance, PerformanceReportDownload, PerformanceRows, PerformanceTableChangedError, PerformanceInterval, PerformanceDeletedLabels, PerformanceMemberType
:show-inheritance:
1 change: 1 addition & 0 deletions libs/labelbox/src/labelbox/__init__.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -111,6 +111,7 @@
PerformanceReportDownload,
PerformanceRows,
PerformanceScoreBucket,
PerformanceTableChangedError,
PerformanceTimeBucket,
ProjectLabelerPerformance,
ProjectReviewerPerformance,
Expand Down
78 changes: 67 additions & 11 deletions libs/labelbox/src/labelbox/client.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -2365,6 +2365,8 @@ def get_workspace_performance_metric(
Union[_performance.PerformanceMemberType, str]
] = None,
owner_organization_ids: Optional[List[str]] = None,
member_organization_ids: Optional[List[str]] = None,
user_group_ids: Optional[List[str]] = None,
) -> _performance.PerformanceMetricSeries:
"""Returns one performance metric across the workspace.

Expand Down Expand Up @@ -2398,10 +2400,24 @@ def get_workspace_performance_metric(
owner_organization_ids (Optional[List[str]]): Count only projects
owned by these organizations. Use it to separate the projects
you own from the ones shared with you.

``member_type`` and ``owner_organization_ids`` report on other
member_organization_ids (Optional[List[str]]): Count only work by
members whose account belongs to one of these organizations,
for example a workforce provider's. Up to 100.
user_group_ids (Optional[List[str]]): Count only work by the
current members of these user groups of your organization. Up
to 25 groups, with at most 5,000 members between them. The
groups count only if you have the organization-wide
permission to view user groups, and a restricted API key has
it too. Without it no group adds anyone, and every value is
zero; no error is raised. A group of another organization
adds no one either.

``member_type``, ``owner_organization_ids``,
``member_organization_ids`` and ``user_group_ids`` report on other
members, so they need permission to view all performance in every
project the report covers.
project the report covers. Filters narrow each other: only work that
matches all of them is counted, and if none does, every value is
zero. It is never the unfiltered report.

Returns:
PerformanceMetricSeries: The metric as a value per interval.
Expand All @@ -2412,7 +2428,8 @@ def get_workspace_performance_metric(
lbox.exceptions.AuthorizationError: You may not view performance
across the organization, or not everyone's in a project the
report covers.
lbox.exceptions.InvalidQueryError: A parameter is not valid.
lbox.exceptions.InvalidQueryError: A parameter is not valid, or
the user groups have more than 5,000 members between them.
ValueError: A filter was given as an empty list. Pass ``None``
to leave a filter off; an empty list is not read as one.

Expand All @@ -2426,6 +2443,11 @@ def get_workspace_performance_metric(
>>> series = client.get_workspace_performance_metric(
>>> "labels_created", "2026-01-01", "2026-01-31",
>>> member_type="external")
>>>
>>> # The same, for the members of one user group.
>>> series = client.get_workspace_performance_metric(
>>> "labels_created", "2026-01-01", "2026-01-31",
>>> user_group_ids=[group.id])
"""
return _performance.get_workspace_metric(
self,
Expand All @@ -2440,6 +2462,8 @@ def get_workspace_performance_metric(
use_cache=use_cache,
member_type=member_type,
owner_organization_ids=owner_organization_ids,
member_organization_ids=member_organization_ids,
user_group_ids=user_group_ids,
)

def get_workspace_labeler_performance(
Expand All @@ -2460,15 +2484,18 @@ def get_workspace_labeler_performance(
descending: bool = False,
page_size: int = 50,
use_cache: bool = True,
member_organization_ids: Optional[List[str]] = None,
user_group_ids: Optional[List[str]] = None,
) -> _performance.PerformanceRows[_performance.WorkspaceLabelerPerformance]:
"""Returns per-labeler totals across the workspace.

This is the workspace Monitor's member table: one row for each
labeler on each project they worked on in the period. It covers the
projects your organization owns and the ones other organizations
share with it. Each row says which organization owns the project
(``project_organization_id``) and whether the labeler is one of your
own members (``member_type``).
(``project_organization_id``), whether the labeler is one of your
own members (``member_type``), and which organization the labeler's
account belongs to (``member_organization_id``).

Args:
start_date (date, datetime or str): First day of the period, in
Expand All @@ -2490,14 +2517,31 @@ def get_workspace_labeler_performance(
projects owned by these organizations.
sort_by (Optional[str]): A column to sort by, named as the row's
attribute, for example ``"labels_created"``. Without it, rows
are ordered by project, then labeler. Sorting by email is not
offered.
are ordered by project, then labeler. Sorting by email or by
member organization is not offered.
descending (bool): Sort from the highest value down.
page_size (int): Rows fetched per request, at most 100.
use_cache (bool): Pass False to read the report again instead of
using rows read within the last few minutes. Leave it on
while reading a large table, so that every page comes from
the same rows.
while reading a large table, so that the pages come from the
same rows while those are kept; see :class:`PerformanceRows`.
member_organization_ids (Optional[List[str]]): Return only
members whose account belongs to one of these organizations,
for example a workforce provider's. Up to 100.
user_group_ids (Optional[List[str]]): Return only the current
members of these user groups of your organization. Up to 25
groups, with at most 5,000 members between them. The groups
count only if you have the organization-wide permission to
view user groups, and a restricted API key has it too.
Without it no group adds anyone, and no rows are returned; no
error is raised. A group of another organization adds no one
either. Who is in the groups is worked out for each page; see
:class:`PerformanceRows`.

Filters narrow each other: a row is returned only if it matches all
of them. If you may view only your own performance, you get your own
rows, and with ``user_group_ids`` only if you are in one of the
groups.

Returns:
PerformanceRows[WorkspaceLabelerPerformance]: One row per labeler
Expand All @@ -2509,7 +2553,8 @@ def get_workspace_labeler_performance(
you can see.
lbox.exceptions.AuthorizationError: You may not view performance
across the organization.
lbox.exceptions.InvalidQueryError: A parameter is not valid.
lbox.exceptions.InvalidQueryError: A parameter is not valid, or
the user groups have more than 5,000 members between them.
ValueError: A filter was given as an empty list. Pass ``None``
to leave a filter off; an empty list is not read as one.

Expand All @@ -2519,6 +2564,15 @@ def get_workspace_labeler_performance(
>>> print(rows.total)
>>> for row in rows:
>>> print(row.project_name, row.email, row.labels_created)
>>>
>>> # Totals by vendor organization. Names are not unique and
>>> # can be masked, so the totals are kept by ID.
>>> from collections import Counter
>>> labels, names = Counter(), {}
>>> for row in client.get_workspace_labeler_performance(
>>> "2026-01-01", "2026-01-31", member_type="external"):
>>> labels[row.member_organization_id] += row.labels_created
>>> names[row.member_organization_id] = row.member_organization_name
"""
return _performance.get_workspace_labelers(
self,
Expand All @@ -2534,6 +2588,8 @@ def get_workspace_labeler_performance(
descending=descending,
page_size=page_size,
use_cache=use_cache,
member_organization_ids=member_organization_ids,
user_group_ids=user_group_ids,
)

def get_task_by_id(self, task_id: str) -> Union[Task, DataUpsertTask]:
Expand Down
86 changes: 83 additions & 3 deletions libs/labelbox/src/labelbox/schema/performance.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -188,6 +188,10 @@ class PerformanceMetricSeries(_CamelCaseMixin):
narrowed by owning organization, the projects of those owners that
are not counted because you may not view everyone's performance
in them. Empty when none were left out.
member_organization_ids (Optional[List[str]]): The member
organizations a workspace series was narrowed to, if any.
user_group_ids (Optional[List[str]]): The user groups a workspace
series was narrowed to, if any, as they were sent.
"""

metric: str
Expand All @@ -204,6 +208,8 @@ class PerformanceMetricSeries(_CamelCaseMixin):
owner_organization_ids: Optional[List[str]] = None
member_type: Optional[str] = None
skipped_project_ids: Optional[List[str]] = None
member_organization_ids: Optional[List[str]] = None
user_group_ids: Optional[List[str]] = None


class ProjectLabelerPerformance(_CamelCaseMixin):
Expand All @@ -221,6 +227,14 @@ class ProjectLabelerPerformance(_CamelCaseMixin):
organization, ``"external"`` for a member of another one (see
:class:`PerformanceMemberType`). ``None`` if the account no
longer exists.
member_organization_id (Optional[str]): The organization the
member's account belongs to: yours, or another one, such as a
workforce provider's or, if you are a workforce provider, a
customer's. It is the same whoever asks. ``None`` if the account
no longer exists.
member_organization_name (Optional[str]): That organization's name,
or ``org.name.<id>`` if it is masked for you. ``None`` if the
organization can no longer be found.
labels_created (int)
labels_skipped (int)
member_total_time (float)
Expand All @@ -244,6 +258,8 @@ class ProjectLabelerPerformance(_CamelCaseMixin):
project_id: str
email: str
member_type: Optional[str] = None
member_organization_id: Optional[str] = None
member_organization_name: Optional[str] = None
labels_created: int
labels_skipped: int
member_total_time: float
Expand Down Expand Up @@ -278,6 +294,14 @@ class ProjectReviewerPerformance(_CamelCaseMixin):
organization, ``"external"`` for a member of another one (see
:class:`PerformanceMemberType`). ``None`` if the account no
longer exists.
member_organization_id (Optional[str]): The organization the
member's account belongs to: yours, or another one, such as a
workforce provider's or, if you are a workforce provider, a
customer's. It is the same whoever asks. ``None`` if the account
no longer exists.
member_organization_name (Optional[str]): That organization's name,
or ``org.name.<id>`` if it is masked for you. ``None`` if the
organization can no longer be found.
labels_reviewed (int)
labels_reworked (int)
member_total_time (float)
Expand All @@ -294,6 +318,8 @@ class ProjectReviewerPerformance(_CamelCaseMixin):
project_id: str
email: str
member_type: Optional[str] = None
member_organization_id: Optional[str] = None
member_organization_name: Optional[str] = None
labels_reviewed: int
labels_reworked: int
member_total_time: float
Expand Down Expand Up @@ -324,6 +350,14 @@ class WorkspaceLabelerPerformance(_CamelCaseMixin):
organization, ``"external"`` for a member of another one (see
:class:`PerformanceMemberType`). ``None`` if the account no
longer exists.
member_organization_id (Optional[str]): The organization the
member's account belongs to: yours, or another one, such as a
workforce provider's or, if you are a workforce provider, a
customer's. It is the same whoever asks. ``None`` if the account
no longer exists.
member_organization_name (Optional[str]): That organization's name,
or ``org.name.<id>`` if it is masked for you. ``None`` if the
organization can no longer be found.
project_id (str)
project_name (Optional[str]): ``None`` for a project that can no
longer be found. Its figures are still counted.
Expand Down Expand Up @@ -353,6 +387,8 @@ class WorkspaceLabelerPerformance(_CamelCaseMixin):
user_id: str
email: str
member_type: Optional[str] = None
member_organization_id: Optional[str] = None
member_organization_name: Optional[str] = None
project_id: str
project_name: Optional[str] = None
project_organization_id: Optional[str] = None
Expand Down Expand Up @@ -397,6 +433,14 @@ class PerformanceReportDownload(_CamelCaseMixin):
generated_at: datetime


class PerformanceTableChangedError(LabelboxError):
"""A table changed while its pages were being read.

Raised while iterating :class:`PerformanceRows` when a page reports a
different ``total`` from the first page. Iterate the table again.
"""


Row = TypeVar("Row", bound=_CamelCaseMixin)


Expand All @@ -405,8 +449,23 @@ class PerformanceRows(Generic[Row]):

Iterating fetches the table a page at a time, so a large table is never
held in one response. Nothing is requested until the rows, ``total`` or
``cached_at`` are read. The rows come in a fixed order, so no row is
repeated or missed between pages.
``cached_at`` are read. The rows come in a fixed order.

Each page is a separate request, so the pages fit together only if the
table does not change between them. A workspace table read with the
cache on is cut from a reading of the report that is kept for five
minutes from when it was taken, possibly by an earlier request, so its
pages agree while that reading lasts. A very large table is not kept,
and an iteration that outlasts the reading continues from a new one. A
project table is read a page at a time, and on any table, who is in the
groups of a ``user_group_ids`` filter is worked out again for every
page. If the figures or a group's members change between pages, a row
can be missed or repeated.

When a page reports a different ``total`` from the first page, iterating
raises :class:`PerformanceTableChangedError`: read the table again. A
change that leaves the number of rows as it was cannot be seen this way,
such as one member joining a group as another with as many rows leaves.

Iterating a second time reads the table again from its first page, and
``total`` and ``cached_at`` then describe that reading.
Expand Down Expand Up @@ -456,13 +515,22 @@ def __iter__(self) -> Iterator[Row]:

number = 1
seen = 0
total: Optional[int] = None
while True:
page = self._page(number)
if total is None:
total = int(page["total"])
elif int(page["total"]) != total:
raise PerformanceTableChangedError(
f"The table changed while it was read: page 1 counted "
f"{total} rows and page {number} counts {page['total']}. "
"Read it again."
)
rows = page["data"]
for row in rows:
yield self._row_type.model_validate(row)
seen += len(rows)
if not rows or seen >= int(page["total"]):
if not rows or seen >= total:
return
number += 1

Expand Down Expand Up @@ -799,6 +867,8 @@ def get_workspace_metric(
use_cache: bool = True,
member_type: Optional[Union[PerformanceMemberType, str]] = None,
owner_organization_ids: Optional[Iterable[Any]] = None,
member_organization_ids: Optional[Iterable[Any]] = None,
user_group_ids: Optional[Iterable[Any]] = None,
) -> PerformanceMetricSeries:
body = _get(
client,
Expand All @@ -813,6 +883,10 @@ def get_workspace_metric(
ownerOrganizationIds=_to_ids(
"owner_organization_ids", owner_organization_ids
),
memberOrganizationIds=_to_ids(
"member_organization_ids", member_organization_ids
),
userGroupIds=_to_ids("user_group_ids", user_group_ids),
useCache=None if use_cache else False,
)
return PerformanceMetricSeries.model_validate(body)
Expand All @@ -834,6 +908,8 @@ def get_workspace_labelers(
descending: bool = False,
page_size: int = _DEFAULT_PAGE_SIZE,
use_cache: bool = True,
member_organization_ids: Optional[Iterable[Any]] = None,
user_group_ids: Optional[Iterable[Any]] = None,
) -> PerformanceRows[WorkspaceLabelerPerformance]:
return _table(
client,
Expand All @@ -852,5 +928,9 @@ def get_workspace_labelers(
ownerOrganizationIds=_to_ids(
"owner_organization_ids", owner_organization_ids
),
memberOrganizationIds=_to_ids(
"member_organization_ids", member_organization_ids
),
userGroupIds=_to_ids("user_group_ids", user_group_ids),
useCache=None if use_cache else False,
)
6 changes: 4 additions & 2 deletions libs/labelbox/src/labelbox/schema/project.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -1949,7 +1949,8 @@ def get_labeler_performance(
request can cover at most 366 days.
user_ids (Optional[List[str]]): Return only these users.
user_group_ids (Optional[List[str]]): Return only the current
members of these user groups.
members of these user groups. Who is in the groups is worked
out for each page; see :class:`PerformanceRows`.
batch_ids (Optional[List[str]]): Count only these batches.
deleted_labels (PerformanceDeletedLabels or str): Whether labels
deleted since are counted. Included by default.
Expand Down Expand Up @@ -2023,7 +2024,8 @@ def get_reviewer_performance(
request can cover at most 366 days.
user_ids (Optional[List[str]]): Return only these users.
user_group_ids (Optional[List[str]]): Return only the current
members of these user groups.
members of these user groups. Who is in the groups is worked
out for each page; see :class:`PerformanceRows`.
batch_ids (Optional[List[str]]): Count only these batches.
deleted_labels (PerformanceDeletedLabels or str): Whether labels
deleted since are counted. Included by default.
Expand Down
Loading
Loading