Docs: what to do after a graph batch was interrupted before close()

docs/modules/ROOT/pages/how-to/bulk-loads.adoc

@@ -1,5 +1,5 @@
 = Bulk loads and commit size
-:description: The default commit size of insert_many, graph_batch and create_vertices on the arcadedb (native) and arcadedb-jvm packages, what it means for a failed load, and how to change it.
+:description: The default commit size of insert_many, graph_batch and create_vertices on the arcadedb (native) and arcadedb-jvm packages, what it means for a failed load, what to do after an interrupted graph batch, and how to change it.
 
 `insert_many`, `graph_batch` and `GraphBatch.create_vertices` load data in transactions. The commit size is the number of rows in one transaction. It decides how much work the engine holds in memory before it commits, and how many rows stay in the database when a load fails.
 
@@ -64,3 +64,63 @@ with db.graph_batch(commit_every=0) as batch:     # one commit for each flush, a
 with db.graph_batch(commit_every=5000) as batch:  # also sets the create_vertices chunk
     ...
 ----
+
+== If a graph batch is interrupted
+
+This section is about a `graph_batch()` load that stops before `close()` returns, for example when the process is killed (`kill -9`, the out-of-memory killer, power loss). An exception inside a `with db.graph_batch() as batch:` block still runs `close()`. The numbers below come from a kill probe on the `arcadedb` (native) package: `kill -9` in the middle of the edge phase, the default `commit_every=1000`, with and without `use_wal=True`. In the 40 kills of the probe, no committed vertex or edge chunk was lost, no record was torn, no edge pointed at a missing vertex, and every database reopened and accepted writes. The example below is one of these kills: 30,000 vertices and 70,000 edge records were in the database afterwards.
+
+=== What is in the database
+
+* The vertices and edge records that a returned call committed are there. A load that is killed loses the work after the last commit and nothing before it.
+* The edges are not traversable. The engine writes the edge records first and links them to their vertices (the head pointers and the incoming lists) in `close()`. Until `close()` has run, `outE()`, `inE()`, `out()` and `MATCH` find none of them, but `SELECT count(*) FROM E` counts them. In the example: 70,000 edge records, 0 reachable.
+* `CHECK DATABASE` reports each of these edges as an `ORPHAN RECORD` and counts them in `unreachableEdgeRecords`. It also uses the words "was not connected from", "is not referenced back by" and "missing from that vertex's OUT/IN list" for the same state. Read the statement in the xref:arcadedb::reference/sql/sql-database-admin.adoc#sql-check-database[`CHECK DATABASE` reference]; this page does not repeat it.
+
+=== Recovery that works
+
+[cols="3,2,2,3"]
+|===
+| Step | Edge records | Reachable by traversal | `CHECK DATABASE` afterwards
+
+| After the kill, nothing done
+| 70,000
+| 0
+| 70,000 `ORPHAN RECORD` warnings
+
+| `CHECK DATABASE` (report only)
+| 70,000
+| 0
+| the same 70,000
+
+| `CHECK DATABASE FIX`
+| 70,000
+| 0
+| the same 70,000. `FIX` relinks nothing and deletes nothing here.
+
+| `CHECK DATABASE FIX DELETE ORPHANS`
+| 0
+| 0
+| nothing left to report
+
+| Then load the edges again over the existing vertices and call `close()`
+| 90,000
+| 90,000
+| 0 warnings
+|===
+
+. Run `CHECK DATABASE`. The number of `ORPHAN RECORD` warnings is the number of committed edges that traversal cannot reach.
+. Do not expect `CHECK DATABASE FIX` to repair them. It reports the same warnings before and after, in the same process and after a restart.
+. Run `CHECK DATABASE FIX DELETE ORPHANS`. It deletes every edge record that no vertex lists. The vertices stay. Use it only when the interrupted batch is the only damage, because the engine cannot tell these records from other unlisted edge records.
+. Load the edges again. Do not call `create_vertices` again with the same keys: on a type with a unique index on the key it fails with `DuplicatedKeyException` and changes nothing, and without such an index it would create the vertices twice. Look up the RIDs of the vertices that exist, buffer the edges with `new_edges`, and let `close()` link them.
+
+[source,python]
+----
+db.command("sql", "CHECK DATABASE FIX DELETE ORPHANS")
+
+rid = {r["id"]: str(r["r"]) for r in db.query("sql", "SELECT @rid AS r, id FROM V").to_list()}
+with db.graph_batch() as batch:
+    batch.new_edges([rid[s] for s in sources], "E", [rid[d] for d in targets], properties)
+----
+
+If you load the edges again without step 3, the old records stay in the database next to the new ones. In the probe this gave 160,000 edge records for 90,000 distinct edges. Traversal is right, because the old records are unreachable, but the edge count of the type is wrong and `CHECK DATABASE` keeps reporting the 70,000 old records.
+
+Edges are linked only when `close()` returns, and each batch links its own edges. A long edge load split over several batches, each closed before the next opens, therefore leaves only the edges of the batch that was open at the interruption unlinked.