Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
3 changes: 2 additions & 1 deletion changelogs/unreleased.md
Original file line number Diff line number Diff line change
Expand Up @@ -2,7 +2,8 @@

## Changes after v3.4.0

No changes have been staged after 3.4.0. The Search index lifecycle API is assigned to the [3.4.0 versioned notes](./v3.4.0.md). Publication is established by the npm registry and immutable tag, not this file.
- Add GridFS buckets on `msq.gridfs()` and `model.gridfs()`, including upload, download, ranges, `ensureIndexes`, and phase-1 resumable upload. `writePathPolicy` gains a `gridfs` field and category. Policy violations stay `INVALID_OPERATION` with `reason: 'WRITE_PATH_POLICY'`. These notes stage the work after 3.4.0; the npm package version is unchanged and this file is not a 3.5.0 release. The Search index lifecycle API remains in the [3.4.0 versioned notes](./v3.4.0.md).
- Raise the CJS and ESM runtime package budgets for GridFS 3.5.0. `gridfs()` stays synchronous, so the single-file build keeps that code in the main bundle. The CJS ceiling is 873000 bytes after `contentType` moved onto file metadata.

## Archived preparation notes for the 3.3.2 candidate

Expand Down
5 changes: 3 additions & 2 deletions config/package-budget.json
Original file line number Diff line number Diff line change
@@ -1,7 +1,8 @@
{
"comment": "GridFS 3.5.0 stays in the main bundle: sync gridfs() cannot be split out of the single-file build.",
"artifacts": {
"dist/cjs/index.cjs": 840000,
"dist/esm/index.mjs": 840000,
"dist/cjs/index.cjs": 873000,
"dist/esm/index.mjs": 870000,
"dist/cjs/cli/data-task.cjs": 50000
}
}
3 changes: 3 additions & 0 deletions docs/en/api-index.md
Original file line number Diff line number Diff line change
Expand Up @@ -97,6 +97,7 @@ This index documents the current stable MongoDB adapter APIs and shared runtime
| [Aggregation pipeline](aggregate.md) | `aggregate()` | Aggregation pipeline query |
| [Vector search](vector-search.md) | Collection / Model `vectorSearch()` | MongoDB Vector Search query with Model soft-delete visibility when applicable |
| [Search indexes](search-indexes.md) | Collection / Model `createSearchIndex()` and related methods | Create, inspect, update, drop, and wait for Search / Vector Search indexes |
| [GridFS](gridfs.md) | `msq.gridfs()` / Model `gridfs()` | Upload, download, and resume GridFS files |

---

Expand Down Expand Up @@ -174,6 +175,7 @@ This index documents the current stable MongoDB adapter APIs and shared runtime
- [Aggregation pipeline](aggregate.md) - Aggregation pipelines
- [Vector search](vector-search.md) - MongoDB Vector Search queries
- [Search indexes](search-indexes.md) - Search / Vector Search index lifecycle
- [GridFS](gridfs.md) - GridFS upload, download, and resumable upload
- [Query plans](explain.md) - Query plans
- [Chain queries](chaining-api.md) - Chain queries

Expand Down Expand Up @@ -209,6 +211,7 @@ This index documents the current stable MongoDB adapter APIs and shared runtime
| [Create indexes in bulk](create-indexes.md) | Create indexes in bulk |
| [Drop an index](drop-index.md) | Drop an index |
| [List indexes](list-indexes.md) | List indexes |
| [GridFS](gridfs.md) | GridFS buckets, ranges, and resumable upload |
| [Find and count](find-and-count.md) | Find and count |
| [Expression functions](expression-functions.md) | Expression functions |
| [Quick upsert](quick-upsert.md) | Quick upsert |
Expand Down
7 changes: 4 additions & 3 deletions docs/en/capability-index.md
Original file line number Diff line number Diff line change
Expand Up @@ -14,14 +14,15 @@ Principles:
| Layer | Current status | Scope |
|-------|----------------|-------|
| Shared runtime | Stable where exposed by the current runtime | Database cache, pools, transactions, model constraints, sync, slow-query logging, and runtime diagnostics |
| MongoDB adapter | Stable | `collection()`, `db()`, `use()`, `pool()`, query/write helpers, aggregation, indexes, Search indexes, transactions, Change Streams |
| MongoDB adapter | Stable | `collection()`, `db()`, `use()`, `pool()`, query/write helpers, aggregation, indexes, Search indexes, GridFS, transactions, Change Streams |
| MySQL adapter | Planned / in development | Future database-native adapter, no current public runtime API |
| PostgreSQL adapter | Planned / in development | Future database-native adapter, no current public runtime API |

| Capability | Recommended entry | Notes |
|------------|----------------------|-------|
| Model | `msq.model()` / `MonSQLize.Model` | Supports model registration, relations, virtuals, populate, and common query entries |
| write-path-policy | `writePathPolicy` | Optional runtime guard for Model-only write namespaces |
| write-path-policy | `writePathPolicy` | Optional runtime guard for Model-only write namespaces, including the `gridfs` category |
| GridFS | `msq.gridfs()` / `model.gridfs()` | Upload, download, and phase-1 resumable GridFS upload. See [GridFS](./gridfs.md) |
| cache | Query `cache` options / `msq.cache` | Database query cache, multi-layer cache access, invalidation, and statistics |
| transaction | `startSession()` / `withTransaction()` | Supports MongoDB transaction wrappers and transaction statistics |
| pool | `pools` / `pool()` / `ConnectionPoolManager` | Declare `pools: PoolConfig[]` in the constructor, route with `pool()`, and use the low-level manager for advanced inspection or manual management |
Expand Down Expand Up @@ -50,7 +51,7 @@ Available entries:
- `writePathPolicy.namespaces`
- `WritePathPolicyOptions`

Use this when selected namespaces should be written through `msq.model()` instead of direct `collection()` writes. See [Write Path Policy](./write-path-policy.md).
Use this when selected namespaces should be written through `msq.model()` instead of direct `collection()` writes. See [Write Path Policy](./write-path-policy.md). GridFS writes use the `gridfs` field and are documented in [GridFS](./gridfs.md).

## 3. transaction

Expand Down
18 changes: 18 additions & 0 deletions docs/en/error-codes.md
Original file line number Diff line number Diff line change
Expand Up @@ -72,6 +72,12 @@ See [Data Tasks](./data-tasks.md) for the preview, approval, apply, backup, and
| `SEARCH_INDEX_UNSUPPORTED` | Deployment lacks Search index support | Atlas, MongoDB 8.2+ with mongot, or mongodb-atlas-local |
| `SEARCH_INDEX_BUILD_FAILED` | Search index build failed while waiting | `details[0].statusDetail`, index definition |
| `SEARCH_INDEX_WAIT_TIMEOUT` | waitForSearchIndex timed out | `timeoutMs`, `details[0].last` |
| `GRIDFS_FILE_NOT_FOUND` | GridFS file or resumable session is missing | id, filename, `ignoreMissing` |
| `GRIDFS_FILE_EXISTS` | GridFS upload conflicts with a visible file or an open session | `ifExists`, `sha256`, id |
| `GRIDFS_INTEGRITY_MISMATCH` | GridFS bytes do not match the declared sha256 or chunk size | `details[0].expected`, `details[0].actual` |
| `GRIDFS_UPLOAD_INCOMPLETE` | Resumable finalize found missing chunks | `details[0].missingChunks` |
| `GRIDFS_RESUME_MISMATCH` | Resumable append does not match the stored prefix | `fromChunk`, `expectedNextChunk` |
| `GRIDFS_RANGE_INVALID` | GridFS download range is not a legal half-open interval | `start`, `end`, `length` |
| `INVALID_RELATION_SELECT` | Relation default or populate selection is outside P0 inclusion syntax | Use non-empty top-level inclusion fields only |
| `RELATION_IN_USE` | Protected Model deletion found declared inbound references | Inspect `error.details[0].usages`; remove or retain references intentionally |
| `RELATION_USAGE_UNAVAILABLE` | Protected deletion could not complete its declared-relation scan | Inspect coverage/skipped details; do not retry as an unchecked delete |
Expand Down Expand Up @@ -210,6 +216,12 @@ See [Search index lifecycle](search-indexes.md) for when each code is raised; ma

---

### GRIDFS_*

See [GridFS](gridfs.md) for when each code is raised. `downloadStream` reports not-found and range failures as stream `'error'` events. Other driver failures are `MONGODB_ERROR` and keep the driver error as `cause`. `writePathPolicy` blocks are `INVALID_OPERATION`, not a GridFS code, with `details[0].reason` `WRITE_PATH_POLICY`.

---

### INVALID_VECTOR_SEARCH

**Explanation**: A local `vectorSearch()` option is invalid before MongoDB is called.
Expand Down Expand Up @@ -1041,6 +1053,12 @@ try {
| `SEARCH_INDEX_UNSUPPORTED` | Search indexes | Deployment lacks Search index support |
| `SEARCH_INDEX_BUILD_FAILED` | Search indexes | Search index build failed while waiting |
| `SEARCH_INDEX_WAIT_TIMEOUT` | Search indexes | waitForSearchIndex timed out |
| `GRIDFS_FILE_NOT_FOUND` | GridFS | GridFS file or resumable session is missing |
| `GRIDFS_FILE_EXISTS` | GridFS | GridFS upload conflicts with an existing file or session |
| `GRIDFS_INTEGRITY_MISMATCH` | GridFS | GridFS bytes do not match sha256 or chunk size |
| `GRIDFS_UPLOAD_INCOMPLETE` | GridFS | Resumable finalize found missing chunks |
| `GRIDFS_RESUME_MISMATCH` | GridFS | Resumable append does not match the stored prefix |
| `GRIDFS_RANGE_INVALID` | GridFS | GridFS download range is invalid |
| `INVALID_RELATION_SELECT` | Relations | Unsupported relation inclusion selection |
| `RELATION_IN_USE` | Relations | Protected delete found declared references |
| `RELATION_USAGE_UNAVAILABLE` | Relations | Protected delete scan coverage is incomplete or unavailable |
Expand Down
142 changes: 142 additions & 0 deletions docs/en/gridfs.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,142 @@
# GridFS

Instance and Model helpers store and read MongoDB GridFS files. The published npm package is still 3.4.0; this API is staged in the source after that release and is not a published 3.5.0 package.

GridFS is a client convention on ordinary collections (`<bucket>.files` and `<bucket>.chunks`). It does not require a Search or Vector Search deployment. Integration coverage uses `mongodb-memory-server` 7.0.37. The formal server matrix remains MongoDB 7.x and 8.x. Older servers, including 4.x, are unverified.

## Open a bucket

```ts
const bucket = msq.gridfs(bucketName?: string, opts?: {
db?: string;
pool?: string;
chunkSizeBytes?: number;
});

const modelBucket = model.gridfs(bucketName?: string, opts?: {
chunkSizeBytes?: number;
});
```

`bucketName` defaults to `fs`. An empty string is rejected. `chunkSizeBytes` defaults to 261120 (255 KiB) and must be a positive integer.

`msq.gridfs()` uses the runtime connection unless `pool` names a configured pool. A named pool uses that pool's client and `client.db(db)`. `db` defaults to the runtime database. `model.gridfs()` always uses the Model `poolName` and `dbName`; it does not accept `pool` or `db`.

Calling either method before `connect()` throws `NOT_CONNECTED`.

## Files API

| Method | Result |
|--------|--------|
| `upload(filename, data, opts?)` | Promise of file info. `data` is a `Buffer`, `Uint8Array`, or `Readable`. |
| `uploadStream(filename, opts?)` | Synchronous `Writable` with `id` and `done`. Await `done` for file info or the mapped error. |
| `download(idOrName)` | Promise of a `Buffer`. |
| `downloadStream(idOrName, range?)` | Synchronous `Readable`. Missing files and range errors arrive as `'error'` events. |
| `getFile(id)` | File info, or `null` when the id is absent. |
| `findFiles(filter?, opts?)` | File info documents. `limit` must be a positive integer. `skip` must be `>= 0`. Sort values are `1` or `-1`. |
| `delete(id, opts?)` | Removes the files document and its chunks. `ignoreMissing: true` resolves when the id is absent. |
| `rename(id, filename)` | Renames one visible file. |
| `drop()` | Drops `<bucket>.files`, `<bucket>.chunks`, and `<bucket>.uploads` when they exist. |
| `ensureIndexes()` | Creates the indexes below. Safe to call more than once. |

`idOrName` is an `ObjectId` or `{ filename, revision? }`. `revision` follows the driver: `0` is the oldest file with that name, and `-1` (the default) is the newest. `download({ filename })` therefore returns the latest file with that name.

### `ifExists` and sha256

`upload` defaults to `ifExists: 'error'`.

- When `sha256` is set, the conflict key is `metadata.sha256`. The digest must be 64 lowercase hex characters. `Buffer` and `Uint8Array` payloads are hashed before any GridFS write. A mismatch throws `GRIDFS_INTEGRITY_MISMATCH` and leaves no visible file.
- A `Readable` passed with `sha256` is buffered completely, then hashed, before any write. Large files should use the resumable API instead.
- `skip` with `sha256` returns the newest visible file with that digest. `skip` without `sha256` returns the newest visible file with the same filename. `duplicate` always creates another file.
- `uploadStream` stores `sha256` in metadata only. It does not check the bytes and it does not accept `ifExists`.

`metadata.sha256` written by these helpers overwrites a caller-supplied `metadata.sha256`. A provided `contentType` is stored as `metadata.contentType` and overwrites a caller-supplied value of that field. `FileInfo.contentType` returns it. An older file that still has a top-level `contentType` is returned the same way.

### Ranges

`downloadStream` ranges use an inclusive `start` and an exclusive `end`. Omitted `start` means `0`. Omitted `end` means the file length. A legal range satisfies `0 <= start < end <= length`. Anything else, including `start === end` and `start === length`, emits `GRIDFS_RANGE_INVALID`. When the failure depends on the file length, `details[0].length` is that length. Shape errors that can be rejected before a lookup omit `length`. `download()` reads the whole file and does not accept a range.

## Resumable upload

Phase 1 resumable upload keeps a pending session in `<bucket>.uploads` until finalize. The file stays invisible until `finalizeResumableUpload` succeeds. `uploadId` is the future `files._id`.

```ts
const session = await bucket.beginResumableUpload(filename, {
length,
sha256,
chunkSizeBytes?,
metadata?,
contentType?,
id?,
});
const progress = await bucket.appendChunks(session.uploadId, fromChunk, data);
const info = await bucket.finalizeResumableUpload(session.uploadId);
await bucket.abortResumableUpload(session.uploadId);
```

`beginResumableUpload` requires `length` and `sha256`. An id that already exists in `files` or `uploads` throws `GRIDFS_FILE_EXISTS`.

Chunk bytes must follow the session size:

- Every chunk except the last has length `chunkSizeBytes`.
- The last chunk has length `length % chunkSizeBytes`, or `chunkSizeBytes` when the length divides evenly.
- A payload that does not split into those sizes throws `INVALID_ARGUMENT`. The whole payload is checked before any chunk is inserted.

`appendChunks` is addressed by `fromChunk`:

- `fromChunk` past the contiguous prefix throws `GRIDFS_RESUME_MISMATCH` with `expectedNextChunk`.
- Replaying already stored chunks with the same bytes is idempotent and does not move `nextChunk` backward.
- Replaying an earlier chunk with different bytes throws `GRIDFS_RESUME_MISMATCH`.
- A payload that contains any new chunk must start at `nextChunk`. Starting earlier throws `GRIDFS_RESUME_MISMATCH` and writes nothing.
- A duplicate `{ files_id, n }` insert is accepted only when the stored bytes match.

`finalizeResumableUpload` requires a contiguous prefix, a total length equal to the session length, and a sha256 equal to the session digest. It then inserts the files document, sets `metadata.sha256`, and deletes the session. Missing chunk indexes throw `GRIDFS_UPLOAD_INCOMPLETE` with `details[0].missingChunks`. A bad digest or a chunk of the wrong size throws `GRIDFS_INTEGRITY_MISMATCH`. The files document is not inserted in those cases.

`abortResumableUpload` deletes the session and, when no files document is visible, that upload's chunks. A missing or already aborted session resolves. Aborting an id that already has a visible file does not delete that file's chunks.

`<bucket>.uploads` has a TTL index on `createdAt` with `expireAfterSeconds: 604800` (7 days). The TTL deletes upload session documents only. It does not delete chunks. Abort or finalize sessions that are no longer needed.

## Indexes

`upload`, `uploadStream` (before the driver stream is opened), `beginResumableUpload`, and `ensureIndexes` create:

| Collection | Key | Options |
|------------|-----|---------|
| `<bucket>.files` | `{ filename: 1, uploadDate: 1 }` | |
| `<bucket>.files` | `{ 'metadata.sha256': 1 }` | not unique |
| `<bucket>.chunks` | `{ files_id: 1, n: 1 }` | unique |
| `<bucket>.uploads` | `{ createdAt: 1 }` | `expireAfterSeconds: 604800` |

Reads do not create indexes. A write rejected by `writePathPolicy` does not create them either.

## writePathPolicy

GridFS writes are category `gridfs`. The namespace is `<bucket>.files`, matched with the usual order: instance id, `pool:db.collection`, `db.collection`, collection name, then `default`.

| `gridfs` | `mode` | `msq.gridfs()` writes | `model.gridfs()` writes |
|----------|--------|----------------------|-------------------------|
| `inherit` (default) | `allow-both` | allowed | allowed |
| `inherit` | `model-only` | blocked | blocked |
| `allow` | either | allowed | allowed |
| `block` | either | blocked | blocked |

`inherit` does not follow the management rule. Management `inherit` still allows Model management and blocks direct management. GridFS `inherit` under `model-only` blocks both sources. `model.gridfs().drop()` stays a management operation, so Model drop remains allowed under `model-only` while `msq.gridfs().drop()` is blocked.

A blocked call throws `INVALID_OPERATION` before any GridFS write. `details[0]` includes `category: 'gridfs'` (or `'management'` for `drop`) and `reason: 'WRITE_PATH_POLICY'`. `onViolation: 'warn'` logs that detail and allows the call. `getFile`, `findFiles`, `download`, and `downloadStream` are not governed.

## Errors

| Code | When |
|------|------|
| `GRIDFS_FILE_NOT_FOUND` | Download, delete, rename, or resumable session target is missing |
| `GRIDFS_FILE_EXISTS` | `ifExists: 'error'` finds a conflict, or the chosen id is already present |
| `GRIDFS_INTEGRITY_MISMATCH` | Bytes do not match `sha256`, or a finalized chunk has the wrong size |
| `GRIDFS_UPLOAD_INCOMPLETE` | Finalize found gaps. `details[0].missingChunks` lists them |
| `GRIDFS_RESUME_MISMATCH` | `fromChunk` is ahead of the contiguous prefix, or a replayed chunk differs |
| `GRIDFS_RANGE_INVALID` | `start` / `end` is not a legal half-open range |
| `INVALID_ARGUMENT` | Filename, bucket name, chunk size, digest, metadata key, or chunk payload is invalid |
| `INVALID_OPERATION` | `writePathPolicy` blocked the call |
| `NOT_CONNECTED` | The runtime is not connected, or the selected pool client is missing |
| `MONGODB_ERROR` | Other driver or server failures. `cause` is the original error |

`downloadStream` delivers `GRIDFS_FILE_NOT_FOUND` and `GRIDFS_RANGE_INVALID` on the stream. The method itself does not throw those errors.
2 changes: 2 additions & 0 deletions docs/en/support-matrix.md
Original file line number Diff line number Diff line change
Expand Up @@ -18,6 +18,8 @@
| MongoDB Driver | 6.x / 7.x | `npm run test:server-matrix` |
| MongoDB Server | 7.x / 8.x | strict `mongodb-memory-server` 7.0.37 / 8.0.26 single + replica set matrix |

GridFS is a client convention on `<bucket>.files` and `<bucket>.chunks`. Its integration tests run on `mongodb-memory-server` 7.0.37. That does not add MongoDB 4.x, or any server outside 7.x / 8.x, to this matrix.

## Default verification method

- **Default gate**: `npm test`
Expand Down
Loading
Loading