Skip to content

feat: EXPKITS-1576 add audio-window metadata for keyword results - #292

Open
zmolnar wants to merge 8 commits into
mainfrom
feature/EXPKITS-1576-audio-window-metadata
Open

zmolnar wants to merge 8 commits into
mainfrom
feature/EXPKITS-1576-audio-window-metadata

Conversation

@zmolnar

@zmolnar zmolnar commented Nov 17, 2026 •

Copy link
Copy Markdown
Contributor

Goal

EXPKITS-1576: Complete the deferred audio/keyword metadata contract so results identify their analyzed window and consumers can compare corresponding assessments.

EXPKITS-1575 / #290 already restored subjectless keyword comparison. This follow-up adds explicit temporal context while preserving that working unwindowed path.

The branch also implements seven accepted metadata-review follow-ups, each in its own signed commit.

Change

Before

Keyword classifications had no analyzed-window context. Their original input buffer could differ from the inference window, and matching keyword identities could conceal mismatched audio intervals. Suppressed/below-threshold windows emitted no assessment.

After

  • Append optional Classification.audio_window with logical source, opaque binding session, continuity segment, processing rate, half-open sample bounds, optional media PTS and discontinuity. It is self-contained and mutually exclusive with a spatial subject_id.
  • Stamp context in the audio scheduler. Repeated resets without execution coalesce; original buffer timing and model-only context/padding remain distinct from the scheduled PCM bounds.
  • Publish one assessment per successfully decoded keyword window, including empty candidates. Invalid input fails rather than becoming empty success; console events still require an accepted candidate.
  • Compare window identity/timing in Plumber, including reset segments, timestamp availability, ambiguous sessions and duplicate windows. Preserve categorical identity and existing unwindowed comparison. Packet/frame alignment remains unchanged.
  • Expose timing and invalid context in WebUI/native inspection. Clear retained audio results on session/segment changes even if the first reset-marked assessment was missed.
  • Regenerate the C++, Python, Rust and TypeScript SDKs, native Python bridge and browser bundle through their existing owners; update documentation and regressions. Existing payload IDs remain unchanged.

Accepted review follow-ups

Commit Change
7f50da24 Correct the BYOM model field list to version 2.0.0.
2d0cbda3 Package the complete descriptor-schema tree and validate pipeline-v1 plus model/OpChain-v2 entry points, including source-independent package checks.
46e53861 Reject unknown Python providedOutputs/requiredInputs payload names in opk-config-check, using the generated SDK registry without loading scripts.
f0504d4b Document rebinding each gaze model's input when changing RetinaFace detectors.
a0d7063b Align the Web README with explicit subject references and exact/same-major SDK compatibility.
8191fd06 Validate instance-mask pixels and compute foreground bounds in one complete scan; preserve standalone validation and malformed-pixel rejection.
d57c0ea1 Give unavailable embedding edges a finite cost above any all-valid assignment, preserving valid cosine -1 matches.

Runtime-CI model provisioning for NITEC, MobileGaze, mask models and YOLO tracker/OSD remains one separate deferred follow-up. Previously dropped assertions and accepted theoretical cases are not reopened by these fixes.

Compatibility and version decision

The optional field/table is wire-additive and affects only the Classifications schema closure. The version remains the repository's unreleased 1.0.0 development snapshot, with participants regenerated/rebuilt together.

The schema evaluator against origin/main reports minor impact and a version-bump error because both versions remain 1.0.0. Against published v0.1.3, the coordinated major transition to 1.0.0 passes. This follows the documented unreleased-snapshot policy; the evaluator was not changed or its error suppressed. The explicit empty-assessment behavior is documented for review.

Testing

Local validation of the combined branch at d57c0ea1 on 2026-11-18, based on main a58d18a0 (including #283):

  • Debug build and full native suite: 108 passed, 2 expected failures, 0 failures.
  • 55 Plumber tests, 147 WebUI tests, and 38 product release-tool tests passed.
  • SDK snapshot and browser bundle drift checks passed; bundle regenerated and matched the rebased source.
  • Configuration validation, scoped formatting/licence checks and documentation generation/validation passed. All seven follow-up commits have verified signatures and DCO trailers.
  • Follow-up changed-line clang-tidy completed with zero findings, using the checked-in analysis owner and host-computed Git ranges.

Regression demonstrations:

  • Packaging failed to include a v2 nested schema before the fix; tests now cover both schema versions, missing required roots, changed/extra files and unsafe trees.
  • Config preflight previously accepted unknown and non-root payload names; the CLI now rejects them with precise field paths before script loading.
  • Instance-mask resolution previously read 70 pixels for a 35-pixel bitmap; it now reads 35, while still rejecting an invalid final pixel outside the supplied box.
  • Embedding-only matching previously discarded both valid crossed-space assignments at cosine -1; square and rectangular regressions now pass, including missing embeddings.

Additional implementation validation before the rebase: 64 SDK release-tool tests, scoped formatting/licence checks, and zero changed-line clang-tidy findings across 193 translation units. Clang-tidy used the checked-in analysis owner with host-computed Git ranges after the container's Git lookup failed; no rule was disabled. The rebase did not alter the authored native/schema changes.

Deterministic model-shaped pipeline tests exercise real scheduling, preprocessing, keyword parsing and publication with overlapping/burst windows, alternate buffer partitioning, explicit empty assessments and 48 kHz input converted to 16 kHz processing. Scheduler/parser/consumer tests cover resets, inactivity, exact 64-bit values, absent/zero timestamps and malformed/conflicting contexts. Live microphone and real keyword-model accuracy were not qualified.

From the repository root inside the development container:

NINJAFLAGS=-j2 ./scripts/build.sh debug true
meson test -C development/build-active --print-errorlogs --num-processes 2
PYTHONPATH=tools/plumber:generated/open_perception_kit/python/src python3 -m unittest discover -s tools/plumber/tests
./scripts/perception-sdk.sh check
./scripts/opksink-web.sh test
./scripts/opksink-web.sh check
python3 tools/perception/tests/test_release.py
python3 scripts/release/TestReleaseTool.py
./tools/opk-config-check --root .

Documentation: ./scripts/gen-doc.sh from the host checkout. Full architecture release packages were not built for these follow-ups. Current-head CI is in the Checks tab.

Links

Task: EXPKITS-1576

Publish explicit source, session, segment and window timing for keyword
assessments, including successful empty results. Align SDKs, consumers,
comparison, inspection and regression coverage.

Signed-off-by: Zoltan Molnar <zoltan.molnar@arm.com>
@zmolnar
zmolnar requested review from a team as code owners November 17, 2026 14:54
Task: EXPKITS-1576

Align the field reference with the supported format-2 model schema.

Signed-off-by: Zoltan Molnar <zoltan.molnar@arm.com>
Task: EXPKITS-1576

Stage the complete schema tree and require the pipeline-v1 and model/opchain-v2
entry points, including when validating a package without its source checkout.

Signed-off-by: Zoltan Molnar <zoltan.molnar@arm.com>
Task: EXPKITS-1576

Reuse the generated payload registry for provided outputs and required
inputs before loading plugins or Python scripts.

Signed-off-by: Zoltan Molnar <zoltan.molnar@arm.com>
Task: EXPKITS-1576

Document each gaze model's explicit RetinaFace source and how to retain
the chosen connection across pipeline restarts.

Signed-off-by: Zoltan Molnar <zoltan.molnar@arm.com>
Task: EXPKITS-1576

Describe explicit subject IDs and exact or same-major SDK identity
acceptance instead of retired parent-based and exact-only rules.

Signed-off-by: Zoltan Molnar <zoltan.molnar@arm.com>
Task: EXPKITS-1576

Separate structural mask validation from the complete sample scan so the
instance resolver visits each pixel once while preserving binary checks.

Signed-off-by: Zoltan Molnar <zoltan.molnar@arm.com>
Task: EXPKITS-1576

Give unavailable embedding edges a finite cost above any all-valid
assignment. Preserve valid cosine -1 matches in square and rectangular cases.

Signed-off-by: Zoltan Molnar <zoltan.molnar@arm.com>

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant