Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 4 additions & 0 deletions examples/ideas-board/.gitignore
Original file line number Diff line number Diff line change
@@ -0,0 +1,4 @@
.venv/
__pycache__/
*.pyc
.pytest_cache/
89 changes: 89 additions & 0 deletions examples/ideas-board/README.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,89 @@
# Concurrent Ideas Board (Python)

The Python twin of the Spring/.NET/Java/Go fundamentals demos. It proves the
library's hero feature: **automatic retry on serialization conflicts
(SQLSTATE 40001)** via the `@crdb_retry` decorator.

## What it shows

One idea row, many concurrent upvotes. Each upvote is a read-modify-write
(`SELECT votes` → `votes + 1` → `UPDATE`) under **SERIALIZABLE** isolation, so
colliding upvotes force CockroachDB to raise `40001`.

- `upvote_without_retry()` — no decorator. Conflicts propagate, votes are **lost**.
- `upvote()` — wrapped in `@crdb_retry(...)`. The decorator re-runs the whole
function in a fresh transaction on `40001`, so **every vote lands**.

```
PLAIN @crdb_retry OFF : 2/25 votes landed (23 lost to 40001)
RETRY @crdb_retry ON : 25/25 votes landed (0 failures)
```

## Files

| File | Role |
|------|------|
| `idea.py` | `Idea` dataclass (id, text, votes) |
| `repository.py` | psycopg read-modify-write helpers taking a cursor |
| `service.py` | `upvote_without_retry` (plain) and `upvote` (`@crdb_retry`) |
| `main.py` | seed one idea, fire N concurrent upvotes per path, print both |
| `test_unit.py` | fast unit test, no Docker (mocked psycopg) |
| `test_concurrency.py` | integration test, real CockroachDB via testcontainers |
| `crdb_container.py` | helper that boots an insecure CockroachDB container |

## Prerequisites

- Python 3.12+
- Docker (only for `main.py`'s throwaway cluster and the integration test)

## Install

The library ships on an anonymous PEP 503 feed, not public PyPI:

```bash
python -m venv .venv
source .venv/bin/activate
pip install -r requirements.txt
```

`requirements.txt` pins the **published** `cockroachdb==0.1.2` (plus
`cockroachdb-testing==0.1.1`) and carries the `--extra-index-url` line for the
feed. A single-package install looks like:

```bash
pip install 'cockroachdb==0.1.2' \
--extra-index-url https://cockroachlabs.github.io/cockroachdb-python-library/simple/
```

## Run

```bash
# Uses $CRDB_DSN if set, otherwise spins up a throwaway container (needs Docker).
python main.py

# Point at your own cluster instead:
CRDB_DSN=postgresql://root@localhost:26257/defaultdb python main.py
```

## Test

```bash
pytest # unit always runs; integration needs Docker
pytest test_unit.py # fast path only, no container
```

The unit test runs with **no container**. The integration test boots a real
CockroachDB container and asserts the retry path lands exactly 60 votes while
the plain path does not.

> Container note: the published `cockroachdb-testing` fixture and the
> `testcontainers` cockroachdb module boot CockroachDB in *secure* mode, which
> needs a client certificate this example does not ship (and psycopg3 cannot
> parse the module's `cockroachdb+psycopg2://` URL). So `crdb_container.py` runs
> the same image in `--insecure` mode and yields a plain `postgresql://` DSN —
> the exact connection style the library's own examples use.

> Source note: this example lives inside the `cockroachdb-python-library`
> checkout, so its tests were validated against the **local source** via
> `pip install -e packages/cockroachdb packages/cockroachdb-core`.
> Customers use the pinned published install above.
44 changes: 44 additions & 0 deletions examples/ideas-board/cdc_handler.py
Original file line number Diff line number Diff line change
@@ -0,0 +1,44 @@
"""CDC: a changefeed webhook handler for the ideas table.

The ``@changefeed_handler`` decorator tags a function with table/operations/endpoint
metadata; ``ChangefeedHandlerRegistry`` parses an incoming webhook envelope into
``ChangefeedEvent``s and dispatches each to the handler. Insert/update events are
projected onto an ``Idea``; deletes (after-image is ``None``) are counted.

This mirrors the Java ``IdeaChangefeedHandler``: it is pure event-processing logic,
so it is fully exercised without a live changefeed (see ``test_cdc.py``). A real
webhook-sink changefeed is created with ``ChangefeedManager`` against a live
cluster; that path needs an enterprise license + a reachable sink, so — like the
Java demo — there is no CDC integration test here.
"""

from __future__ import annotations

from typing import Any

from idea import Idea

# tag::cdc-handler[]
from cockroachdb import changefeed_handler


class IdeaChangefeedHandler:
"""Turns ``idea`` changefeed events into ``Idea``s and tracks deletes."""

TABLE = "ideas"

def __init__(self) -> None:
self.handled: list[Idea] = []
self.deletes = 0

@changefeed_handler(table=TABLE, operations=["insert", "update", "delete"])
def on_change(self, event: Any) -> None:
"""Project an insert/update ``after`` image onto an ``Idea``; count deletes."""
if event.after is None: # delete: no after-image
self.deletes += 1
return
after = event.after
self.handled.append(Idea(id=after["id"], text=after["text"], votes=after["votes"]))


# end::cdc-handler[]
6 changes: 6 additions & 0 deletions examples/ideas-board/conftest.py
Original file line number Diff line number Diff line change
@@ -0,0 +1,6 @@
"""Make the flat example modules importable when running pytest from here."""

import os
import sys

sys.path.insert(0, os.path.dirname(__file__))
59 changes: 59 additions & 0 deletions examples/ideas-board/crdb_container.py
Original file line number Diff line number Diff line change
@@ -0,0 +1,59 @@
"""Throwaway insecure CockroachDB via testcontainers.

The published `cockroachdb-testing` fixture and the `testcontainers` cockroachdb
module both boot CockroachDB in SECURE mode, which requires a client certificate
this example does not ship (and psycopg3 cannot parse the module's
`cockroachdb+psycopg2://` URL). So we run the same CockroachDB image in
`--insecure` mode via the generic testcontainers API and hand back a plain
`postgresql://` DSN — identical to how the library's own examples connect
(e.g. `postgresql://root@localhost:26257/defaultdb`).
"""

from __future__ import annotations

import contextlib
import time
from collections.abc import Iterator

import psycopg

IMAGE = "cockroachdb/cockroach:v24.3.0"


@contextlib.contextmanager
def insecure_crdb(image: str = IMAGE) -> Iterator[str]:
"""Start an insecure single-node CockroachDB; yield a psycopg DSN.

``image`` defaults to the pinned fundamentals image; the semantic-search IT
overrides it with a VECTOR-capable image (v25.2+).
"""
from testcontainers.core.container import DockerContainer
from testcontainers.core.waiting_utils import wait_for_logs

container = (
DockerContainer(image)
.with_command("start-single-node --insecure")
.with_exposed_ports(26257)
)
container.start()
try:
wait_for_logs(container, "CockroachDB node starting", timeout=90)
host = container.get_container_host_ip()
port = container.get_exposed_port(26257)
dsn = f"postgresql://root@{host}:{port}/defaultdb?sslmode=disable"
_wait_ready(dsn)
yield dsn
finally:
container.stop()


def _wait_ready(dsn: str, attempts: int = 30) -> None:
for _ in range(attempts):
try:
with psycopg.connect(dsn, connect_timeout=3) as conn:
conn.execute("SELECT 1")
return
except psycopg.OperationalError:
time.sleep(1)
msg = "CockroachDB did not become ready in time"
raise RuntimeError(msg)
56 changes: 56 additions & 0 deletions examples/ideas-board/follower_read_service.py
Original file line number Diff line number Diff line change
@@ -0,0 +1,56 @@
"""Follower-read leaderboard: read the top ideas AS OF SYSTEM TIME so the query
is served by the nearest replica instead of the leaseholder.

The library's ``build_staleness()`` helper and the ``@follower_read`` decorator
supply the AS OF SYSTEM TIME expression; ``FollowerReadContext`` carries it across
a call. NOTE: the plain-psycopg ``CockroachDBTemplate`` does NOT auto-inject
AS OF SYSTEM TIME (only the SQLAlchemy layer does), so this service applies the
resolved staleness to the SELECT itself — the cooperative pattern for a
psycopg app, exactly how the Java demo threads its staleness through.
"""

from __future__ import annotations

import psycopg
from idea import Idea

# tag::follower-read[]
from cockroachdb import build_staleness, follower_read
from cockroachdb.core import FollowerReadContext


@follower_read("follower_read_timestamp()")
def top_ideas(dsn: str, limit: int = 10) -> list[Idea]:
"""Top ideas by votes, read from a follower snapshot.

``@follower_read`` stamps the resolved staleness on this function
(``__crdb_follower_read__``); we read it back and apply it. The staleness is
library-controlled (a builtin like ``follower_read_timestamp()`` or an
interval like ``'-5s'``), never user input, so it is safe to interpolate.
"""
staleness = getattr(top_ideas, "__crdb_follower_read__", build_staleness())
sql = (
"SELECT id, text, votes FROM ideas "
f"AS OF SYSTEM TIME {staleness} "
"ORDER BY votes DESC, text ASC LIMIT %s"
)
with psycopg.connect(dsn, autocommit=True) as conn, conn.cursor() as cur:
cur.execute(sql, (limit,))
return [Idea(id=r[0], text=r[1], votes=r[2]) for r in cur.fetchall()]


def top_ideas_scoped(dsn: str, limit: int, *, seconds_ago: int) -> list[Idea]:
"""Same read, but the staleness comes from a ``FollowerReadContext`` block —
the signal a middleware or template layer would set for the whole request."""
with FollowerReadContext.scoped(seconds_ago=seconds_ago) as staleness:
sql = (
"SELECT id, text, votes FROM ideas "
f"AS OF SYSTEM TIME {staleness} "
"ORDER BY votes DESC, text ASC LIMIT %s"
)
with psycopg.connect(dsn, autocommit=True) as conn, conn.cursor() as cur:
cur.execute(sql, (limit,))
return [Idea(id=r[0], text=r[1], votes=r[2]) for r in cur.fetchall()]


# end::follower-read[]
24 changes: 24 additions & 0 deletions examples/ideas-board/idea.py
Original file line number Diff line number Diff line change
@@ -0,0 +1,24 @@
"""Domain entity for the Concurrent Ideas Board example."""

from __future__ import annotations

# tag::entity[]
from dataclasses import dataclass


@dataclass
class Idea:
"""A single idea on the board.

Attributes:
id: Primary key assigned by CockroachDB.
text: The idea's text.
votes: Running upvote count — the hotly-contended column.
"""

id: int
text: str
votes: int


# end::entity[]
100 changes: 100 additions & 0 deletions examples/ideas-board/main.py
Original file line number Diff line number Diff line change
@@ -0,0 +1,100 @@
"""Runnable demo: concurrent upvotes on one hot row, plain vs. @crdb_retry.

Seeds a single idea, then fires N concurrent upvotes through the plain path
(votes get lost to 40001) and the retry path (every vote lands).

Connection:
- Set CRDB_DSN to point at a running cluster, e.g.
CRDB_DSN=postgresql://root@localhost:26257/defaultdb
- Or leave it unset and the demo spins up a throwaway CockroachDB
container via cockroachdb-testing (requires Docker).

Usage:
python main.py
"""

from __future__ import annotations

import contextlib
import os
import threading
from collections.abc import Iterator

import repository
import service


@contextlib.contextmanager
def _dsn() -> Iterator[str]:
"""Yield a psycopg DSN — from CRDB_DSN, or a throwaway container."""
env = os.environ.get("CRDB_DSN")
if env:
yield env
return
from crdb_container import insecure_crdb

with insecure_crdb() as dsn:
yield dsn


# tag::runner[]
def _hammer(dsn: str, idea_id: int, fn, votes: int) -> int:
"""Fire *votes* concurrent upvotes through *fn*; return the failure count.

A barrier releases every thread at the same instant so the read-modify-write
windows overlap and CockroachDB is forced to serialize them.
"""
barrier = threading.Barrier(votes)
failures: list[Exception] = []
lock = threading.Lock()

def worker() -> None:
barrier.wait()
try:
fn(dsn, idea_id)
except Exception as exc:
with lock:
failures.append(exc)

threads = [threading.Thread(target=worker) for _ in range(votes)]
for t in threads:
t.start()
for t in threads:
t.join()
return len(failures)


def main() -> None:
votes = 25
with _dsn() as dsn, __import__("psycopg").connect(dsn, autocommit=True) as conn:
with conn.cursor() as cur:
repository.create_schema(cur)
idea_id = repository.seed(cur, "Add dark mode to the dashboard")

print(f"Seeded idea #{idea_id}; firing {votes} concurrent upvotes per path.\n")

# Plain path — no retry. Conflicts raise 40001 and those votes are lost.
failures = _hammer(dsn, idea_id, service.upvote_without_retry, votes)
with conn.cursor() as cur:
plain_total = repository.current_votes(cur, idea_id)
print(
f"PLAIN @crdb_retry OFF : {plain_total}/{votes} votes landed "
f"({failures} lost to 40001)"
)

# Reset, then the retry path — every vote eventually commits.
with conn.cursor() as cur:
repository.reset(cur, idea_id)
failures = _hammer(dsn, idea_id, service.upvote, votes)
with conn.cursor() as cur:
retry_total = repository.current_votes(cur, idea_id)
print(
f"RETRY @crdb_retry ON : {retry_total}/{votes} votes landed ({failures} failures)"
)


# end::runner[]


if __name__ == "__main__":
main()
Loading
Loading