Skip to content

[gremlin-python] [1/4] Add async HTTP streaming transport, AsyncResultSet, and GraphBinaryV4 stream reader - #3725

Open
masterhugo wants to merge 1 commit into
apache:masterfrom
masterhugo:feature/tp4-async-pr1-transport
Open

masterhugo wants to merge 1 commit into
apache:masterfrom
masterhugo:feature/tp4-async-pr1-transport

Conversation

@masterhugo

Copy link
Copy Markdown

Summary (Part 1 of 4: Native asyncio for gremlin-python 4.0)

This is PR 1 of 4 introducing native asyncio support to gremlin-python for TinkerPop 4.0 over HTTP streaming, without nest_asyncio, background ThreadPoolExecutor pools, or per-connection private event loops (asyncio.new_event_loop()).

Why native asyncio without nest_asyncio?

Currently, AiohttpHTTPTransport creates a private event loop per connection and drives aiohttp synchronously via run_until_complete(). When called from inside an already-running event loop (FastAPI/uvloop, Jupyter notebooks, async AWS Lambda handlers), it fails with RuntimeError: Cannot run the event loop while another loop is running, and nest_asyncio cannot patch Cython-based loops like uvloop.

What is included in PR 1 (Foundation layer)

  • gremlin_python/driver/aiohttp/async_transport.py:
    • AsyncBufferedStream: sliding-window bytearray buffer over aiohttp.ClientResponse.content with checkpoint(), restore(), compact(), fill_more(), and read_exactly(). Translates aiohttp.ServerTimeoutError into ReadTimeoutError.
    • AsyncAiohttpHTTPTransport: native aiohttp.ClientSession transport bound directly to asyncio.get_running_loop().
  • gremlin_python/driver/async_resultset.py:
    • AsyncResultSet: single-item streaming result set supporting await rs.one() (returning _EXHAUSTED on stream end so None result items are preserved), await rs.all(), and async for item in rs, using asyncio.wait(..., return_when=FIRST_COMPLETED) instead of polling loops.
  • gremlin_python/driver/serializer.py:
    • Added GraphBinarySerializersV4.deserialize_async_response_stream(stream) to incrementally decode GraphBinary v4 (0x84) objects and end-of-stream trailers across arbitrary HTTP chunk boundaries.
  • tests/unit/driver/test_async_http_streaming.py:
    • 27 unit tests covering AsyncResultSet, AsyncBufferedStream, deserialize_async_response_stream (including 2-byte chunk fragmentation across object/trailer boundaries), and AsyncAiohttpHTTPTransport.

Roadmap of the 4-PR Series

  • PR 1 (this PR): AsyncBufferedStream, AsyncAiohttpHTTPTransport, AsyncResultSet, deserialize_async_response_stream
  • PR 2: AsyncConnection & AsyncClient (asyncio.Queue connection pool)
  • PR 3: AsyncDriverRemoteConnection & AsyncGraphTraversal (async_traversal())
  • PR 4: AsyncTransaction (g.tx(), execute_in_tx), integration tests, examples, and documentation

@codecov-commenter

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 77.54%. Comparing base (a28cd1f) to head (60886f4).
⚠️ Report is 736 commits behind head on master.

Additional details and impacted files
@@             Coverage Diff              @@
##             master    #3725      +/-   ##
============================================
+ Coverage     76.35%   77.54%   +1.18%     
============================================
  Files          1012       32     -980     
  Lines         60341     7805   -52536     
  Branches       7075        0    -7075     
============================================
- Hits          46076     6052   -40024     
+ Misses        11548     1425   -10123     
+ Partials       2717      328    -2389     

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

@masterhugo
masterhugo force-pushed the feature/tp4-async-pr1-transport branch from 60886f4 to a3381f0 Compare October 20, 2026 00:36
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants