send_each_async loses all per-message results after numeric HTTP/2 transport error #995

[READ] Step 1: Are you in the right place?

Yes. This appears to originate from the FCM send_each_async() implementation in this repository and its async HTTP/2 transport.

[REQUIRED] Step 2: Describe your environment

• Operating System version: Debian GNU/Linux 12 (bookworm), python:3.14-slim-bookworm container
• Firebase SDK version: firebase-admin==7.7.0
• Firebase Product: Firebase Cloud Messaging (FCM)
• Python version: 3.14
• Pip version: N/A; dependencies are installed from uv.lock with uv
• Related dependencies: httpx==0.28.1, httpcore==1.0.9, h2==4.3.0, google-auth==2.55.1

[REQUIRED] Step 3: Describe the problem

Steps to reproduce:

We send approximately 8,000 token-targeted FCM notifications. The messages are split into sequential chunks of 500, and each chunk is sent with await messaging.send_each_async(messages, app=app). Calls are rate-limited to one chunk per second.

In a production run, approximately 6,000 of the 8,000 deliveries were reported as failed. Multiple 500-message chunks raised one batch-level exception instead of returning a BatchResponse with a result for every message:

firebase_admin.exceptions.UnknownError:
Unknown error while making remote service calls: 4213

The numeric value varies between occurrences; another observed value was 6641.

Expected behavior: individual transport or FCM failures should be represented by their corresponding SendResponse entries, while successful requests in the same call should remain available in the returned BatchResponse.

Actual behavior: send_each_async() raises one UnknownError for the entire group of up to 500 messages. The application cannot tell which requests succeeded. Retrying the whole group may duplicate notifications that were already accepted by FCM.

The SDK implementation starts one coroutine per message with asyncio.gather(). send_data() converts httpx.HTTPError into a SendResponse, but an exception outside that hierarchy escapes send_data(). The outer handler then wraps it as:

raise exceptions.UnknownError(
    message=f'Unknown error while making remote service calls: {error}',
    cause=error,
)

The numeric message is consistent with a raw KeyError(stream_id). The installed httpcore==1.0.9 still contains an unconditional del self._events[stream_id] in AsyncHTTP2Connection._response_closed(). A similar HTTP/2 error-masking problem was fixed in httpcore2 by pydantic/httpx2#1093. I also noticed #991, which proposes migrating this SDK to httpx2, is still open.

Could send_each_async() either:

1. avoid losing all per-message results when one coroutine raises an unexpected transport exception;
2. preserve and expose the underlying traceback/cause more clearly; and/or
3. move to a transport containing the HTTP/2 fix?

Relevant Code:

from firebase_admin import messaging


async def send_notifications(all_messages, app):
    results = []
    for offset in range(0, len(all_messages), 500):
        chunk = all_messages[offset : offset + 500]
        # A single unexpected transport exception raises UnknownError for
        # the complete chunk, so per-message results are unavailable.
        response = await messaging.send_each_async(chunk, app=app)
        results.extend(response.responses)
    return results

Each item is a standard token-targeted message:

messaging.Message(
    notification=messaging.Notification(title=title, body=body),
    data={"profile_key": profile_key, "push_id": push_id},
    token=device_token,
)

No concurrent send_each_async() calls are started by the application for the same Firebase app; the 500-message chunks are awaited sequentially.