[parmstro]

Summary

Fixes #2474

When all entries in a remote's includes field are sha256 digests or name=sha256:digest aliases, bypass /tags/list and fetch manifests directly. This avoids paginating 50,000+ tags when syncing a known set of digests, reducing sync time from minutes to seconds.

• Named aliases (name=sha256:digest) create a human-readable tag alongside the digest tag, enabling friendly image references in disconnected environments
• Cosign companion tags (.sig, .att, .sbom) are discovered via concurrent HEAD probes instead of filtering the full tag list; a new auto_discover_cosign field on ContainerRemote controls this behavior
• Safety guards: bypass is disabled for mirror mode, non-digest includes, and exclude patterns that could match included digests

Performance

Tested against quay.io/openshift-release-dev/ocp-v4.0-art-dev (50K+ tags, 189 digest includes):

Mode	Without optimization	With optimization
Metadata (on_demand)	3–8 minutes	~31 seconds
Full content (immediate)	8–12 minutes	~5 minutes

Production validation

This optimization has been production-validated on Red Hat Satellite 6.19 using the equivalent pulp_container 2.26 implementation, adapted here for the 2.29 API surface (includes/excludes fields, defaultdict-based _synced_digests, artifactless manifests). Customer requests for this optimization are tracked in SAT-49501.

Changes

• pulp_container/app/tasks/sync_stages.py — Core optimization: _parse_include_entry(), _can_bypass_taglist(), _tag_exists(), _discover_cosign_companions_without_taglist(), bypass logic in run(), alias tag creation in _process_manifests()
• pulp_container/app/models.py — Add auto_discover_cosign BooleanField to ContainerRemote
• pulp_container/app/serializers.py — Expose auto_discover_cosign in the REST API
• pulp_container/app/tasks/synchronize.py — Pass mirror flag to ContainerFirstStage
• pulp_container/app/migrations/0051_* — Migration for new field
• pulp_container/tests/unit/test_sync_stages.py — 21 unit tests covering parse, bypass logic, cosign discovery, and HEAD probes
• CHANGES/2474.feature — Changelog fragment

Test plan

• Unit tests pass (pytest pulp_container/tests/unit/test_sync_stages.py)
• Metadata-only sync with digest includes (on_demand mode)
• Full content sync with immediate mode
• Named alias tags created and pullable via podman pull
• Cosign companion tag discovery via HEAD probes
• Mixed digest + non-digest includes falls back to full tag list
• Mirror mode falls back to full tag list
• Local and remote podman pull validated
• CI functional tests

[gerrod3]

Note

Posted by an agent on behalf of @gerrod3.

Could we avoid adding auto_discover_cosign by making companion discovery a cached helper? Automatic discovery is already the existing behavior, and excludes can control which companions we probe and sync.

For example, initialize this state on the stage:

self._bypass_taglist = False
self._probed_cosign_digests = set()
self._cosign_probe_semaphore = asyncio.Semaphore(20)

Then replace _discover_cosign_companions_without_taglist() with a helper that populates the existing _cosign_tags collection:

async def _ensure_cosign_tags(self, digests):
    if not self._bypass_taglist:
        return  # The normal path already populated _cosign_tags.

    pending = sorted(set(digests) - self._probed_cosign_digests)
    candidates = []

    for digest in pending:
        if not digest.startswith("sha256:"):
            continue

        stem = digest.replace("sha256:", "sha256-", 1)
        candidates.extend(
            f"{stem}{suffix}"
            for suffix in (*COSIGN_TAG_SUFFIXES, "")
        )

    candidates = filter_resources(
        candidates, [], self.remote.excludes or []
    )

    async def probe(tag):
        async with self._cosign_probe_semaphore:
            return tag if await self._tag_exists(tag) else None

    results = await asyncio.gather(*(probe(tag) for tag in candidates))
    self._cosign_tags.extend(tag for tag in results if tag is not None)
    self._probed_cosign_digests.update(pending)

Filtering candidates before probing preserves exclusions and avoids unnecessary requests. For example, excludes=["sha256-*"] suppresses every companion probe. This also removes the need for the special "harmless exclusions" check in _can_bypass_taglist(). That check can instead require populated, explicit digest includes, using validation such as re.fullmatch(r"sha256:[0-9a-f]{64}", digest), while retaining the mirror fallback. Existing exclusions apply to tag names; the current code appends digest includes separately from tag filtering.

We need to call this helper in two places. First, before checking whether an image has a Cosign signature:

async def _has_cosign_signature(self, digest):
    await self._ensure_cosign_tags([digest])

    # Continue with the existing .sig / V3 manifest checks.
    ...

This fixes the ordering problem with signed_only: currently the bypass path checks signatures while _cosign_tags is empty, so it skips signed images before discovery happens when Cosign is the signature source.

Second, after processing the primary manifests in the bypass branch, set self._bypass_taglist from _can_bypass_taglist() before entering that branch and use:

await self._process_manifests(
    digest_list, signature_source, "Processing Manifests"
)

# Includes child manifests discovered inside an index.
await self._ensure_cosign_tags(self._synced_digests)

companion_tags = self._find_cosign_companion_tags()
if companion_tags:
    await self._process_manifests(
        companion_tags,
        signature_source,
        "Processing Cosign Companion Tags",
    )

The cache prevents repeating probes already performed during signature checks.

Finally, _tag_exists() should only interpret HTTP 404 as absence:

except aiohttp.ClientResponseError as exc:
    if exc.status == 404:
        return False
    raise

Authentication failures, rate limits, and upstream errors should propagate rather than silently omit companions.

With that structure, we can remove the model field, serializer field, and migration. I'd retain the mirror fallback initially and test digest-only signed sync, exclusions, child-manifest companions, caching, and probe errors.

I would also consider moving alias support into a separate change, since it introduces new includes syntax and creates local tags.