Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
12 changes: 7 additions & 5 deletions docs/modules/ROOT/pages/how-to/bulk-loads.adoc
Original file line number Diff line number Diff line change
Expand Up @@ -65,13 +65,13 @@ with db.graph_batch(commit_every=5000) as batch: # also sets the create_vertice
...
----

== Known limitation: graph batches next to deletes
== Graph batches next to deletes (fixed in ArcadeDB 26.11.1)

WARNING: *A graph batch does not reuse the space of deleted edges.* The engine writes the edge records of each `graph_batch()` flush onto new pages and never looks at the free space that deletes left on older pages. The edge bucket gains one 64 KiB page per flush and never shrinks, while the live data stays flat. Both packages are affected. The engine issue is https://github.com/ArcadeData/arcadedb/issues/10746[ArcadeData/arcadedb#10746].
NOTE: *Fixed in ArcadeDB 26.11.1.* Before 26.11.1, a graph batch did not reuse the space of deleted edges (https://github.com/ArcadeData/arcadedb/issues/10746[ArcadeData/arcadedb#10746]). The engine wrote the edge records of each `graph_batch()` flush onto new pages and never looked at the free space that deletes left on older pages, so the edge bucket gained one 64 KiB page per flush while the live data stayed flat. From 26.11.1 (https://github.com/ArcadeData/arcadedb/pull/10836[ArcadeData/arcadedb#10836]) each edge goes where an ordinary insert would put it while an existing page has room, and only the rest of a flush goes onto new pages, so a large import stays as fast as before. The 26.11.1 packages ship this fix. The `26.11.1.devN` pre-releases embed engine 26.10.2 and still show the growth described below.

A one-off bulk load is not affected. The workloads that are affected call `graph_batch()` again and again on a type that also sees deletes, for example a sliding window that adds a small batch and deletes the oldest one.
On engine 26.10.2, a one-off bulk load is not affected. The workloads that are affected call `graph_batch()` again and again on a type that also sees deletes, for example a sliding window that adds a small batch and deletes the oldest one.

The measurement ran a sliding window of 60 rounds for 240 seconds on each package. Each round wrote 30 vertices and about 90 edges and deleted the oldest round with `DELETE FROM Node`, so the live edge count stayed flat. The delete order does not matter: deleting the edges before the vertices gave the same growth.
The measurement below ran on engine 26.10.2: a sliding window of 60 rounds for 240 seconds on each package. Each round wrote 30 vertices and about 90 edges and deleted the oldest round with `DELETE FROM Node`, so the live edge count stayed flat. The delete order does not matter: deleting the edges before the vertices gave the same growth.

[cols="3,2,2,2"]
|===
Expand Down Expand Up @@ -100,7 +100,9 @@ The measurement ran a sliding window of 60 rounds for 240 seconds on each packag

An engine-level Java run of the same shape confirms the split: after 2,000 rounds the batch had 2,000 edge pages (125.0 MiB), and `newEdge` in an ordinary transaction had 4 (0.3 MiB). The cost is disk space: in the same runs a scan of the edge type and a two-hop query did not slow down while the bucket grew.

If your workload deletes and inserts again, write the edges with SQL (`CREATE EDGE`) or with `Vertex.new_edge` inside `db.transaction()`. Both stayed flat in the measurement (0.25 MiB in the table). Keep `graph_batch()` for loads that add data and do not delete from the same type.
The engine-level Java run, repeated on the merge of the fix, kept the batch's edge bucket at 4 pages after 2,000 rounds, the same as `newEdge` in ordinary transactions, against 2,000 pages on its parent (https://github.com/ArcadeData/arcadedb/issues/10746#issuecomment-6462354150[verification on #10746]).

On a `26.11.1.devN` pre-release, if your workload deletes and inserts again, write the edges with SQL (`CREATE EDGE`) or with `Vertex.new_edge` inside `db.transaction()`. Both stayed flat in the measurement (0.25 MiB in the table). From 26.11.1 on, `graph_batch()` is fine for this workload too.

== If a graph batch is interrupted

Expand Down
14 changes: 7 additions & 7 deletions src/arcadedb/_native/database.py
Original file line number Diff line number Diff line change
Expand Up @@ -904,13 +904,13 @@ def graph_batch(self, **options):
wins. It is also the number of rows ``create_vertices`` sends, and commits, per call into the image.

Note:
``graph_batch()`` is recommended for bulk loads. When it is called again and
again on types that also see deletes (a sliding window, for example), each
flush writes its edge records onto new pages and never reuses the space
that deletes freed, so the edge bucket grows (ArcadeData/arcadedb#10746).
For that workload write the edges with SQL ``CREATE EDGE``, or
``Vertex.new_edge`` inside ``db.transaction()``. See "Known limitation:
graph batches next to deletes" in
``graph_batch()`` is recommended for bulk loads. Before ArcadeDB 26.11.1, calling
it again and again on types that also see deletes (a sliding window, for
example) grew the edge bucket by one page per flush, because each flush wrote
its edge records onto new pages (ArcadeData/arcadedb#10746, fixed in #10836).
The ``26.11.1.devN`` pre-releases embed engine 26.10.2 and still show it; on
those, write the edges with SQL ``CREATE EDGE``, or ``Vertex.new_edge`` inside
``db.transaction()``. See "Graph batches next to deletes" in
docs/modules/ROOT/pages/how-to/bulk-loads.adoc.
"""
from .extras import NativeGraphBatch
Expand Down
14 changes: 7 additions & 7 deletions src/arcadedb/core.py
Original file line number Diff line number Diff line change
Expand Up @@ -2080,13 +2080,13 @@ def graph_batch(
and 40,000 edges with and without ``parallel_flush``.

Note:
``graph_batch()`` is recommended for bulk loads. When it is called again and
again on types that also see deletes (a sliding window, for example), each
flush writes its edge records onto new pages and never reuses the space
that deletes freed, so the edge bucket grows (ArcadeData/arcadedb#10746).
For that workload write the edges with SQL ``CREATE EDGE``, or
``Vertex.new_edge`` inside ``db.transaction()``. See "Known limitation:
graph batches next to deletes" in
``graph_batch()`` is recommended for bulk loads. Before ArcadeDB 26.11.1, calling
it again and again on types that also see deletes (a sliding window, for
example) grew the edge bucket by one page per flush, because each flush wrote
its edge records onto new pages (ArcadeData/arcadedb#10746, fixed in #10836).
The ``26.11.1.devN`` pre-releases embed engine 26.10.2 and still show it; on
those, write the edges with SQL ``CREATE EDGE``, or ``Vertex.new_edge`` inside
``db.transaction()``. See "Graph batches next to deletes" in
docs/modules/ROOT/pages/how-to/bulk-loads.adoc.

Args:
Expand Down
Loading