diff --git a/.changeset/mixed-storage-originals.md b/.changeset/mixed-storage-originals.md new file mode 100644 index 00000000..23983c13 --- /dev/null +++ b/.changeset/mixed-storage-originals.md @@ -0,0 +1,24 @@ +--- +'@transloadit/node': minor +'transloadit': minor +'@transloadit/mcp-server': patch +'@transloadit/utils': minor +--- + +Store mixed-media originals with `storeAsset()` and recover verified receipts without uploading. +The Storage CLI checkpoints both image and file receipts in one catalog, skips matching replays, +and stops with recoverable evidence if the server changes original bytes. Viewer remains image-only. +Add `getPublicStoredAssetUrl()` for version-pinned originals, downloads and bounded still previews. +Its new Built-ins require the corresponding API2 deployment before adoption; URL construction +does not publish assets. Existing `storeImage()` behavior is preserved. + +The generic CLI intentionally now requires unchanged original bytes, including for images: a +server-transformed upload exits nonzero after saving its actual receipt instead of silently +accepting a watermark. It uses the `file` upload field. Use the existing SDK `storeImage()` API +when deliberately accepting transformed image bytes. + +Plain and hashed replays both trust the local receipt's byte hash and do not verify live existence. +Refresh stale catalogs with `storage receipts sync`; incomplete legacy receipts require recovery +before replay. Sync cannot generate a missing checksum. Dimensionless files and unknown MIME types +remain in `files` outside Viewer's typed map; new uploads and sync require an explicit image MIME +type to classify an entry as a Viewer image. Existing catalogs remain readable. diff --git a/packages/node/README.md b/packages/node/README.md index 3a32ed30..e6cad6c3 100644 --- a/packages/node/README.md +++ b/packages/node/README.md @@ -1608,7 +1608,71 @@ string, so consumers inspecting Zod schema classes or validation issues must han validation. The exported `MODEL_CAPABILITIES` registry is read-only and frozen at the top level; do not mutate its entries. -### Store an image +### Mixed-media originals + +`storeAsset()` stores one original of any media type without a conversion step: + +```ts +const asset = await client.storeAsset('./beach-waves.mp4', { + path: 'website/blog-previews/beach-waves.mp4', +}) +await saveAsset(asset) +``` + +It verifies the returned size and MD5 against the local file; dimensions are optional. Existing +paths fail unless `overwrite: true` is deliberate. If stored bytes differ, the call rejects with +the completed receipt and Assembly ID in the error's `cause`; `onReceipt` also receives them. +That is a completed write, not a reason to retry. `getStoredAssetReceipt({ assemblyId, expected: +{ path, size, md5hash } })` recovers a receipt without uploading and verifies the same byte facts. +Neither method publishes files or edits a local catalog. + +The existing CLI accepts shell-expanded local files of different types: + +```console +transloadit storage store ./photo.jpg ./clip.mp4 ./guide.pdf website/blog-previews/ +``` + +Both use `transloadit.images.json` (or `--receipts`): dimensioned receipts with an `image/*` MIME +type go into `images`; other media and unknown MIME types go into `files`. Viewer uses only +`images`. Existing catalogs remain readable; sync reclassifies matching entries from current metadata. +Each successful upload is checkpointed; +rerunning skips paths whose saved receipt matches the local bytes. Changed bytes require a new +name or deliberate `--overwrite`; `--hashed` chooses content-addressed names instead. Sync +recovers mixed prefixes atomically and preserves unmatched entries; use this CLI version or +newer for mixed prefixes. A receipt mismatch stops the batch but preserves the actual write. + +Replay skips compare local bytes with the saved receipt, not current server existence or access. +Use `storage receipts sync` to refresh a stale catalog after server-side moves or replacements. +Legacy entries without byte hashes or a verified API origin need recovery before replay; the CLI +does not guess. Sync cannot generate a missing checksum: if none is available, choose a fresh name +or deliberately use `--overwrite` (never with `--hashed`). Images without dimensions remain +recoverable in `files`, but are not Viewer sources. +`--placeholder blur` explicitly requests server metadata extraction for every input in that command; +use a separate image-only batch when other files do not need it. Successful extraction adds metadata +usage of 20% of the input bytes. + +If the workspace transforms uploads (for example, adding a watermark), replacing or renaming a file +does not bypass that policy. Use an original-preserving plan or the SDK `storeImage()` API when you +deliberately accept transformed image bytes; its completed receipt describes the stored image. + +For a **deliberately published** directory, Utils builds original or preview links from the receipt: + +```ts +import { getPublicStoredAssetUrl } from '@transloadit/utils' + +const playbackUrl = getPublicStoredAssetUrl(asset) +const downloadUrl = getPublicStoredAssetUrl(asset, { download: true }) +const posterUrl = getPublicStoredAssetUrl(asset, { preview: { width: 960, preset: 'frame' } }) +``` + +This helper does not publish or check current access. Public downloads use the stored filename; +previews offer `auto`, `frame` or `icon`, not animated clips or arbitrary transforms. HTML/XML/SVG +originals download rather than execute inline; image/video/audio remain inline-capable. These +URLs require API2's `public-serve@0.0.3` / `public-preview@0.0.4` deployment before adoption. +Do not put private receipts or placeholder pixels in public catalogs. Unpublishing cannot recall +bytes already cached by a browser or CDN. Keep source originals and license/attribution records. + +### Image receipts `storeImage()` uploads one local image and returns a verified receipt ready for `Image`. Install `@transloadit/node` as a runtime dependency for server uploads; a dev dependency is enough @@ -1658,8 +1722,8 @@ for alpha handling and existing images without placeholders. Save the complete stored result from `results[producingStep][i]` after `ASSEMBLY_COMPLETED`, including its Workspace and returned path. Collision handling can rename the requested destination. -`storeImage()` returns this canonical record plus verified image metadata; other media use the -same `StoredAsset` contract without requiring image dimensions. +`storeImage()` returns this canonical record plus verified image metadata; `storeAsset()` supports +the same `StoredAsset` contract without requiring image dimensions. ```ts const page = await client.listStoredAssets({ prefix: 'uploads/', limit: 100 }) diff --git a/packages/node/src/Transloadit.ts b/packages/node/src/Transloadit.ts index f1d97749..e85ef333 100644 --- a/packages/node/src/Transloadit.ts +++ b/packages/node/src/Transloadit.ts @@ -54,6 +54,11 @@ import type { LintAssemblyInstructionsInput, LintAssemblyInstructionsResult, } from './lintAssemblyInstructions.ts' +import type { + GetStoredAssetReceiptOptions, + StoreAssetOptions, + StoredAssetReceipt, +} from './storageAsset.ts' import type { GetStoredImageReceiptOptions, StoredImageReceipt, @@ -108,6 +113,7 @@ import InconsistentResponseError from './InconsistentResponseError.ts' import { lintAssemblyInstructions as lintAssemblyInstructionsInternal } from './lintAssemblyInstructions.ts' import PaginationStream from './PaginationStream.ts' import PollingTimeoutError from './PollingTimeoutError.ts' +import { getStoredAssetReceipt, storeAsset } from './storageAsset.ts' import { getStoredImageReceipt, storeImage } from './storageImage.ts' import { normalizeStoragePublicPrefix, @@ -151,6 +157,12 @@ export type { RobotListResult, RobotParamHelp, } from './robots.ts' +export type { + GetStoredAssetReceiptOptions, + StoreAssetOptions, + StoredAssetExpectation, + StoredAssetReceipt, +} from './storageAsset.ts' export type { GetStoredImageReceiptOptions, StoredImageExpectation, @@ -573,6 +585,16 @@ export class Transloadit { this._defaultTimeout = timeout } + /** Stores one byte-identical local original of any media type and returns its verified receipt. */ + storeAsset(filePath: string, options: StoreAssetOptions): Promise { + return storeAsset(this, filePath, options) + } + + /** Recovers a generic original receipt without uploading or accepting changed bytes. */ + getStoredAssetReceipt(options: GetStoredAssetReceiptOptions): Promise { + return getStoredAssetReceipt(this, options) + } + /** Stores one local original at an explicit path and returns its verified image metadata. */ storeImage(filePath: string, options: StoreImageOptions): Promise { return storeImage(this, filePath, options) diff --git a/packages/node/src/cli/commands/storage.ts b/packages/node/src/cli/commands/storage.ts index 0b511592..61f47334 100644 --- a/packages/node/src/cli/commands/storage.ts +++ b/packages/node/src/cli/commands/storage.ts @@ -1,4 +1,4 @@ -import type { StoredImageReceipt } from '../../storageImage.ts' +import type { StoredAssetReceipt } from '../../storageAsset.ts' import { createHash } from 'node:crypto' import { createReadStream } from 'node:fs' @@ -25,6 +25,7 @@ import { assertStorageCatalogOrigin, assertStorageWorkspace, defaultStorageCatalog, + isStorageImageReceipt, readStorageCatalog, storageCatalogDelivery, storageTypesPath, @@ -40,7 +41,7 @@ abstract class StorageProjectCommand extends AuthenticatedCommand { .origin } receipts = Option.String('--receipts', defaultStorageCatalog, { - description: 'Committed project catalog with workspace, public prefixes and image receipts', + description: 'Project catalog with workspace, public prefixes and asset receipts', }) workspace = Option.String('--workspace', { description: @@ -203,12 +204,12 @@ export class StoragePublicationsCommand extends AuthenticatedCommand { } } -interface CliStoredImageReceipt extends StoredImageReceipt { +interface CliStoredAssetReceipt extends StoredAssetReceipt { source?: string apiOrigin?: string } -async function hashImageFile( +async function hashAssetFile( file: string, signal: AbortSignal, ): Promise<{ md5hash: string; size: number }> { @@ -219,7 +220,7 @@ async function hashImageFile( hash.update(chunk) size += chunk.length } - if (size === 0) throw new Error('Cannot store an empty image') + if (size === 0) throw new Error('Cannot store an empty file') signal.throwIfAborted() return { md5hash: hash.digest('hex'), size } } @@ -230,7 +231,7 @@ export class StorageStoreCommand extends StorageProjectCommand { static override usage = Command.Usage({ category: 'Storage', - description: 'Store original images and save verified metadata for Image', + description: 'Store original files and save verified, version-pinned asset receipts', details: ` Uses the CLI's Assembly credentials (environment, .env or ~/.transloadit/credentials). Storage writes must be enabled. Existing Storage paths conflict unless --overwrite is explicit. @@ -239,7 +240,7 @@ export class StorageStoreCommand extends StorageProjectCommand { --json includes uploaded for each receipt: false means this file reused a verified receipt. --placeholder blur requests server metadata on new uploads. Successful extraction adds metadata usage of 20% of the file bytes. Missing placeholders never retry a completed upload. - The project catalog binds workspace, published prefixes and image receipts. Each successful + The project catalog binds workspace, public prefixes, images and other files. Each successful upload is saved atomically before the next. Do not run two writers against the same catalog. The catalog defaults to transloadit.images.json; --receipts selects another file. `, @@ -247,6 +248,10 @@ export class StorageStoreCommand extends StorageProjectCommand { ['Store a hero image', 'transloadit storage store ./hero.jpg website/hero.jpg'], ['Store a content-addressed image', 'transloadit storage store ./hero.jpg website/ --hashed'], ['Store a directory of originals', 'transloadit storage store ./images/*.jpg website/'], + [ + 'Store mixed media', + 'transloadit storage store ./photo.jpg ./clip.mp4 ./guide.pdf website/media/', + ], ], }) @@ -267,7 +272,7 @@ export class StorageStoreCommand extends StorageProjectCommand { }) protected async run(): Promise { const file = resolve(this.receipts) - let stored: { receipt?: CliStoredImageReceipt } = {} + let stored: { receipt?: CliStoredAssetReceipt } = {} let destination = this.destination let workspace: string | undefined let saved = false @@ -282,9 +287,9 @@ export class StorageStoreCommand extends StorageProjectCommand { '--hashed cannot be combined with --overwrite; changed bytes get a new name', ) const apiOrigin = this.apiOrigin - const uploaded = new Map() + const uploaded = new Map() if (this.files.length > 1 && !this.destination.endsWith('/')) - throw new Error('Multiple images need a directory destination ending in /') + throw new Error('Multiple files need a directory destination ending in /') const inputs = this.files.map((input) => ({ file: input, path: this.destination.endsWith('/') @@ -293,14 +298,14 @@ export class StorageStoreCommand extends StorageProjectCommand { })) for (const input of inputs) { if (file === resolve(input.file)) - throw new Error('The receipts file cannot be the input image') + throw new Error('The receipts file cannot be the input file') if (resolve(storageTypesPath(file)) === resolve(input.file)) - throw new Error('The generated declarations file cannot be the input image') + throw new Error('The generated declarations file cannot be the input file') validateStoragePath(input.path) } if (!this.hashed && new Set(inputs.map((input) => input.path)).size !== inputs.length) throw new Error( - 'Input image basenames collide in the destination directory; rename them first', + 'Input file basenames collide in the destination directory; rename them first', ) const publicPrefix = this.publicDelivery ? normalizeStoragePublicPrefix( @@ -311,6 +316,7 @@ export class StorageStoreCommand extends StorageProjectCommand { for (const input of inputs) { let publicImage = false let unchanged = false + let uploadFailure: Error | undefined destination = input.path stored = {} saved = false @@ -327,24 +333,36 @@ export class StorageStoreCommand extends StorageProjectCommand { if (receipts?.workspace === workspace) assertStorageCatalogOrigin(receipts, apiOrigin, this.receipts) signal.throwIfAborted() - if (this.hashed) { - // This pass chooses the path and deduplicates. storeImage independently verifies its + if ( + !this.overwrite && + (this.hashed || + uploaded.has(destination) || + (receipts?.workspace === workspace && + (Object.hasOwn(receipts.images, destination) || + Object.hasOwn(receipts.files ?? {}, destination)))) + ) { + // Hash only to choose a hashed path or compare an existing receipt. storeAsset verifies its // upload input; sharing a caller-supplied checksum would need an immutable-file contract. - const { md5hash, size } = await hashImageFile(input.file, signal) - const extension = posix.extname(destination) - destination = `${destination.slice(0, destination.length - extension.length)}.${md5hash.slice(0, 8)}${extension}` + const { md5hash, size } = await hashAssetFile(input.file, signal) + if (this.hashed) { + const extension = posix.extname(destination) + destination = `${destination.slice(0, destination.length - extension.length)}.${md5hash.slice(0, 8)}${extension}` + } validateStoragePath(destination) // Explicit workspace overrides leave the catalog alone; still deduplicate this batch. const candidate = uploaded.get(destination) ?? (receipts?.workspace === workspace && Object.hasOwn(receipts.images, destination) ? receipts.images[destination] - : undefined) + : receipts?.workspace === workspace && + Object.hasOwn(receipts.files ?? {}, destination) + ? receipts.files?.[destination] + : undefined) if (candidate !== undefined) { const previous = hashedReceiptSchema.safeParse(candidate) if (!previous.success || previous.data.path !== destination) throw new Error( - `Catalog receipt for ${JSON.stringify(destination)} is incomplete or does not match this file. Run transloadit storage receipts sync ${quoteCliArgument(destination)} --receipts ${quoteCliArgument(this.receipts)}${this.endpoint === undefined ? '' : ` --endpoint ${quoteCliArgument(this.endpoint)}`} to recover version identity from the same API environment, then retry; nothing uploaded.`, + `Catalog receipt for ${JSON.stringify(destination)} is incomplete or does not match this file. Run transloadit storage receipts sync ${quoteCliArgument(destination)} --receipts ${quoteCliArgument(this.receipts)}${this.endpoint === undefined ? '' : ` --endpoint ${quoteCliArgument(this.endpoint)}`} to recover version identity and any available checksum from the same API environment. If the checksum remains unavailable, ${this.hashed ? 'choose another destination basename' : 'choose a fresh name or use --overwrite only to deliberately replace it'}; sync cannot generate a missing checksum. Nothing uploaded.`, ) if (previous.data.apiOrigin !== apiOrigin) throw new Error( @@ -352,39 +370,60 @@ export class StorageStoreCommand extends StorageProjectCommand { ) if (previous.data.md5hash !== md5hash || previous.data.size !== size) throw new Error( - `Stored bytes for ${JSON.stringify(destination)} differ from this file. An older deployment may have transformed the upload, or the short hashes collided. Restoring the same receipt will not help; choose another destination basename. Nothing uploaded or replaced.`, + this.hashed + ? `Stored bytes for ${JSON.stringify(destination)} differ from this file. An older deployment may have transformed the upload, or the short hashes collided. Restoring the same receipt will not help; choose another destination basename. Nothing uploaded or replaced.` + : `Stored bytes for ${JSON.stringify(destination)} differ from this file. Choose a fresh name or use --overwrite only to deliberately replace it. Nothing uploaded or replaced.`, ) stored.receipt = previous.data unchanged = true } } - stored.receipt ??= await this.client.storeImage(input.file, { - path: destination, - ...(placeholder === undefined ? {} : { placeholder }), - signal, - onReceipt: (receipt, expected, assemblyId) => { - const sizeMatches = receipt.size === expected.size - const md5Matches = receipt.md5hash === expected.md5hash - this.output.debug( - JSON.stringify({ - assemblyId, - result: receipt, - input: expected, - sizeMatches, - md5Matches, - }), - ) - if (!sizeMatches || !md5Matches) { - const difference = sizeMatches - ? 'same size, different MD5' - : `${expected.size.toLocaleString('en-US')} → ${receipt.size.toLocaleString('en-US')} bytes` - this.output.warn( - `Stored bytes differ from ${input.file} (${difference}); the workspace plan may have transformed the upload (for example, a Community-plan watermark on older deployments). The receipt describes the stored image.`, + stored.receipt ??= await this.client + .storeAsset(input.file, { + path: destination, + ...(placeholder === undefined ? {} : { placeholder }), + signal, + onReceipt: (receipt, expected, assemblyId) => { + // Receipt identity is validated before byte fidelity; retain an actual write even + // when the server transformed the original and storeAsset subsequently rejects. + stored.receipt = receipt + const sizeMatches = receipt.size === expected.size + const md5Matches = receipt.md5hash === expected.md5hash + this.output.debug( + JSON.stringify({ + assemblyId, + result: receipt, + input: expected, + sizeMatches, + md5Matches, + }), ) - } - }, - ...(this.overwrite ? { overwrite: true } : {}), - }) + if (!sizeMatches || !md5Matches) { + const difference = sizeMatches + ? 'same size, different MD5' + : `${expected.size.toLocaleString('en-US')} → ${receipt.size.toLocaleString('en-US')} bytes` + this.output.warn( + `Stored bytes differ from ${input.file} (${difference}); the workspace plan may have transformed the upload (for example, a Community-plan watermark on older deployments). The receipt describes the stored file.`, + ) + } + }, + ...(this.overwrite ? { overwrite: true } : {}), + }) + .catch((error: unknown) => { + // The error owns recovery evidence; the optional observer need not have run. + const completed = z + .object({ + receipt: storedAssetSchema.extend({ + md5hash: storedAssetSchema.shape.md5hash.unwrap(), + }), + }) + .safeParse(error instanceof InconsistentResponseError ? error.cause : undefined) + if (completed.success && completed.data.receipt.path === destination) + stored.receipt = completed.data.receipt + if (stored.receipt === undefined) throw error + uploadFailure = ensureError(error) + return stored.receipt + }) if (stored.receipt.workspace !== workspace) throw new Error( `The stored receipt belongs to Workspace ${stored.receipt.workspace}, not ${workspace}. The project catalog was not changed; inspect the Assembly before retrying.`, @@ -395,7 +434,7 @@ export class StorageStoreCommand extends StorageProjectCommand { ...(this.hashed ? { source: basename(input.file) } : {}), apiOrigin, } - if (this.hashed) uploaded.set(destination, stored.receipt) + uploaded.set(destination, stored.receipt) if (receipts !== undefined && receipts.workspace !== workspace) { this.output.notice( `Catalog ${this.receipts} was not changed; it belongs to ${receipts.workspace}. Use --receipts for a separate catalog.`, @@ -405,6 +444,16 @@ export class StorageStoreCommand extends StorageProjectCommand { publicImage = receipts?.public.some((prefix) => destination.startsWith(prefix)) ?? false // Replays must not reorder fields or discard application metadata from a saved receipt. if (unchanged) return receipts + let images = { ...receipts?.images } + let files = { ...receipts?.files } + // Updating an existing key preserves catalog order; remove only a former category. + if (isStorageImageReceipt(stored.receipt)) { + delete files[destination] + images = { ...images, [destination]: stored.receipt } + } else { + delete images[destination] + files = { ...files, [destination]: stored.receipt } + } return { ...receipts, workspace, @@ -413,7 +462,8 @@ export class StorageStoreCommand extends StorageProjectCommand { receipts?.delivery ?? storageCatalogDelivery(this.endpoint ?? this.cliConfig.endpoint), public: receipts?.public ?? [], - images: { ...receipts?.images, [stored.receipt.path]: stored.receipt }, + images, + ...(Object.keys(files).length > 0 || receipts?.files !== undefined ? { files } : {}), } }, () => { @@ -421,6 +471,7 @@ export class StorageStoreCommand extends StorageProjectCommand { }, ) if (stored.receipt === undefined) throw new Error('Storage did not return a receipt') + if (uploadFailure) throw uploadFailure if (publicPrefix !== undefined && published === undefined) { this.output.notice( `Publishing ${publicPrefix} recursively: all current and future objects under this prefix will be public.`, @@ -440,6 +491,13 @@ export class StorageStoreCommand extends StorageProjectCommand { }) } const receipt = stored.receipt + if (!isStorageImageReceipt(receipt)) { + this.output.print( + `${unchanged ? `Unchanged ${receipt.path}; no upload needed.\n` : ''}${saved ? `Saved ${receipt.path} in ${this.receipts}.` : `Stored ${receipt.path}; the different-workspace catalog was left unchanged.`}\nUse the asset/version receipt for original delivery or a Smart CDN preview.`, + { ...receipt, uploaded: !unchanged }, + ) + continue + } if (placeholder === 'blur' && receipt.thumbhash === undefined) { this.output.notice( unchanged @@ -613,8 +671,6 @@ const uploadEvidenceSchema = storedAssetSchema.extend({ const hashedReceiptSchema = uploadEvidenceSchema.extend({ md5hash: storedAssetSchema.shape.md5hash.unwrap(), - width: storedAssetSchema.shape.width.unwrap(), - height: storedAssetSchema.shape.height.unwrap(), }) /** Recovers pinned receipts from bounded native catalog pages without downloading originals. */ @@ -626,7 +682,9 @@ export class StorageReceiptsSyncCommand extends UnauthenticatedCommand { details: ` Uses native catalog pages and dam:write scope to also recover declared public prefixes. Adds or refreshes matched paths; never prunes unmatched entries. Local placeholder metadata - survives only for the same retained version. Every matched image must have catalog dimensions. + survives only for the same retained version. Known image MIME types with dimensions enter + images; other media and unknown MIME types enter files in the same catalog. + Mixed prefixes require this CLI version or newer. Any failure preserves the previous file. No Assembly, original download or remote write occurs. `, examples: [['Recover website images', 'transloadit storage receipts sync website/']], @@ -663,23 +721,13 @@ export class StorageReceiptsSyncCommand extends UnauthenticatedCommand { assertStorageCatalogOrigin(previous, endpoint, this.receipts) const assets = await listStorageAssets(client, workspace, this.prefix, signal) const entries = assets.map((asset) => { - try { - validateStoragePath(asset.path) - } catch (cause) { - throw new Error( - `Storage path ${JSON.stringify(asset.path)} is valid for listing but unsupported by the image renderer. Rename it before recovering an image catalog.`, - { cause }, - ) - } - if (asset.width === undefined || asset.height === undefined) { - throw new Error( - `Storage image ${JSON.stringify(asset.path)} needs positive catalog width and height. Select an image-only prefix and backfill missing dimensions before retrying.`, - ) - } const evidence = uploadEvidenceSchema.safeParse( previous?.workspace === workspace && Object.hasOwn(previous.images, asset.path) ? previous.images[asset.path] - : undefined, + : previous?.workspace === workspace && + Object.hasOwn(previous.files ?? {}, asset.path) + ? previous.files?.[asset.path] + : undefined, ) const retained = evidence.success && @@ -693,6 +741,18 @@ export class StorageReceiptsSyncCommand extends UnauthenticatedCommand { apiOrigin: endpoint, } if (asset.has_alpha !== undefined) delete merged.hasAlpha + // Listing already validates native paths. Only Viewer images use path-based delivery; + // generic files use asset/version IDs and may contain renderer delimiters such as |. + if (isStorageImageReceipt(merged)) { + try { + validateStoragePath(asset.path) + } catch (cause) { + throw new Error( + `Storage path ${JSON.stringify(asset.path)} is valid for listing but unsupported by the image renderer. Rename it before recovering an image catalog.`, + { cause }, + ) + } + } return [asset.path, merged] }) count = entries.length @@ -731,6 +791,17 @@ export class StorageReceiptsSyncCommand extends UnauthenticatedCommand { ) }) catalogUpdated = true + const images = new Map(Object.entries(previous?.images ?? {})) + const files = new Map(Object.entries(previous?.files ?? {})) + for (const [path, receipt] of Object.entries(synced)) { + if (isStorageImageReceipt(receipt)) { + files.delete(path) + images.set(path, receipt) + } else { + images.delete(path) + files.set(path, receipt) + } + } if (policy.public_prefixes.length === 0) this.output.notice( 'No public prefixes are declared on the server. For public delivery, deliberately publish a directory with storage publish; otherwise configure authorize for private images. Sync never publishes files.', @@ -743,7 +814,10 @@ export class StorageReceiptsSyncCommand extends UnauthenticatedCommand { previous?.delivery ?? storageCatalogDelivery(this.endpoint ?? config.credentialsEndpoint), public: policy.public_prefixes.map(({ prefix }) => prefix), - images: { ...previous?.images, ...synced }, + images: Object.fromEntries(images), + ...(files.size > 0 || previous?.files !== undefined + ? { files: Object.fromEntries(files) } + : {}), } }) this.output.print( diff --git a/packages/node/src/cli/storageReceipts.ts b/packages/node/src/cli/storageReceipts.ts index 70a5d3ac..d6c7df3c 100644 --- a/packages/node/src/cli/storageReceipts.ts +++ b/packages/node/src/cli/storageReceipts.ts @@ -15,11 +15,12 @@ const receiptsSchema = z.custom>( (value: unknown) => typeof value === 'object' && value !== null && !Array.isArray(value), ) const deliveryParameterSchema = z.union([z.string(), z.number(), z.boolean()]) +const workspaceSchema = z.string().regex(/^[a-zA-Z0-9][a-zA-Z0-9_-]{0,127}$/) /** Project identity and rendering metadata (including optional preview pixels), without credentials. */ export const storageCatalogSchema = z .object({ - workspace: z.string().regex(/^[a-zA-Z0-9][a-zA-Z0-9_-]{0,127}$/), + workspace: workspaceSchema, apiOrigin: z.string().url().optional(), public: z.array( z.string().refine((prefix) => { @@ -31,6 +32,7 @@ export const storageCatalogSchema = z }, 'Expected a normalized public directory'), ), images: receiptsSchema, + files: receiptsSchema.optional(), delivery: z .object({ baseUrl: z.string().url().optional(), @@ -42,6 +44,29 @@ export const storageCatalogSchema = z .optional(), }) .passthrough() + .superRefine((catalog, context) => { + for (const [path, value] of Object.entries(catalog.files ?? {})) { + const receipt = storedAssetSchema.safeParse(value) + if ( + !receipt.success || + receipt.data.path !== path || + receipt.data.workspace !== catalog.workspace + ) { + context.addIssue({ + code: 'custom', + path: ['files', path], + message: 'Expected a matching Storage asset receipt', + }) + } + if (Object.hasOwn(catalog.images, path)) { + context.addIssue({ + code: 'custom', + path: ['files', path], + message: 'A path cannot occur in both images and files', + }) + } + } + }) export type StorageProjectCatalog = z.infer /** Default project catalog; --receipts can select a separate project explicitly. */ @@ -60,6 +85,13 @@ const imageReceiptSchema = storedAssetSchema.extend({ height: z.number().int().positive().max(Number.MAX_SAFE_INTEGER), }) +/** Only image receipts with intrinsic dimensions belong in the Viewer map. */ +export function isStorageImageReceipt(value: unknown): value is z.infer { + const result = imageReceiptSchema.safeParse(value) + // Dimensions also describe videos; an unknown MIME type cannot establish a Viewer image. + return result.success && result.data.mime?.startsWith('image/') === true +} + const legacyTypesHeader = '// Generated by transloadit storage. Commit this file with the image catalog; do not edit.' const typesHeader = @@ -88,14 +120,20 @@ function catalogTypes(catalog: StorageProjectCatalog, file?: string): string { return [ typesHeader, ...(file === undefined ? [] : [`// Catalog: ${JSON.stringify(basename(file))}`]), - "import '@transloadit/viewer/next'", - '', - "declare module '@transloadit/viewer/next' {", - ' interface RegisteredStorageImages {', - ...properties, - ' }', - '}', - '', + // Headerless legacy ownership must still match the old empty interface exactly. Newly + // generated file-only catalogs have an owner line and need no Viewer dependency. + ...(properties.length === 0 && file !== undefined + ? ['export {}', ''] + : [ + "import '@transloadit/viewer/next'", + '', + "declare module '@transloadit/viewer/next' {", + ' interface RegisteredStorageImages {', + ...properties, + ' }', + '}', + '', + ]), ].join('\n') } @@ -128,7 +166,7 @@ export async function readStorageCatalog(file: string): Promise 0 || catalog.delivery !== undefined) ) throw new Error( `Catalog ${JSON.stringify(file)} has no verified API origin. Run storage receipts sync into a new --receipts file using the original API --endpoint, then review and replace this catalog. The existing file was preserved.`, ) - for (const [path, value] of Object.entries(catalog?.images ?? {})) { + for (const [path, value] of Object.entries({ ...catalog?.images, ...catalog?.files })) { const previous = receiptOriginSchema.safeParse(value) if ( !previous.success || diff --git a/packages/node/src/storageAsset.ts b/packages/node/src/storageAsset.ts new file mode 100644 index 00000000..b53dcbdf --- /dev/null +++ b/packages/node/src/storageAsset.ts @@ -0,0 +1,271 @@ +import type { AssemblyStatus } from './alphalib/types/assemblyStatus.ts' +import type { InterpolatableRobotUploadHandleInstructionsInput } from './alphalib/types/robots/upload-handle.ts' +import type { StoredAsset } from './alphalib/types/storageAsset.ts' +import type { CreateAssemblyOptions, Transloadit } from './Transloadit.ts' + +import { createHash } from 'node:crypto' +import { createReadStream } from 'node:fs' + +import debug from 'debug' +import { z } from 'zod' + +import { ApiError } from './ApiError.ts' +import { storedAssetSchema } from './alphalib/types/storageAsset.ts' +import InconsistentResponseError from './InconsistentResponseError.ts' + +/** + * Verified metadata for an original of any media type. Dimensions are optional. + */ +export interface StoredAssetReceipt extends Readonly { + /** Legacy catalog spelling; new server receipts use has_alpha. */ + readonly hasAlpha?: boolean + readonly md5hash: string +} + +/** Shared SDK/CLI opt-in for server extraction; rendering remains a separate choice. */ +export const storageImagePlaceholderSchema = z.enum(['blur', 'empty']) + +/** One explicit destination, with the existing Assembly upload and polling controls. */ +export interface StoreAssetOptions + extends Pick< + CreateAssemblyOptions, + 'chunkSize' | 'onAssemblyProgress' | 'onUploadProgress' | 'signal' | 'timeout' + > { + /** Complete relative Storage filename. Directories and interpolation expressions are rejected. */ + path: string + /** Explicit opt-in replacement of an existing path; defaults to false. */ + overwrite?: boolean + /** Request a server ThumbHash. Successful extraction adds metadata usage of 20% of the file bytes. */ + placeholder?: z.infer + /** Observe verified metadata. Not awaited; sync and async observer errors cannot undo a write. */ + onReceipt?: ( + receipt: StoredAssetReceipt, + input: StoredAssetExpectation, + assemblyId: string | undefined, + ) => void | Promise +} + +/** Trusted upload facts used to correlate a stored result with an application-owned upload. */ +export interface StoredAssetExpectation { + path: string + size: number + md5hash: string +} + +/** Recovers one original's receipt without re-uploading or trusting notification payloads. */ +export interface GetStoredAssetReceiptOptions { + assemblyId: string + expected: StoredAssetExpectation +} + +const positiveIntegerSchema = z.number().int().positive().max(Number.MAX_SAFE_INTEGER) +const receiptRequestSchema = z.object({ + assemblyId: z.string().regex(/^[A-Za-z0-9_-]+$/), + expected: z.object({ + path: z.string(), + size: positiveIntegerSchema, + md5hash: z.string().regex(/^[a-f0-9]{32}$/), + }), +}) +const completedAssetSchema = z.object({ + ok: z.literal('ASSEMBLY_COMPLETED'), + results: z.object({ + ':original': z.tuple([ + storedAssetSchema.extend({ + md5hash: storedAssetSchema.shape.md5hash.unwrap(), + size: positiveIntegerSchema, + }), + ]), + }), +}) +const completedImageSchema = completedAssetSchema.extend({ + results: z.object({ + ':original': z.tuple([ + completedAssetSchema.shape.results.shape[':original'].items[0].extend({ + width: positiveIntegerSchema, + height: positiveIntegerSchema, + }), + ]), + }), +}) + +function validateDestination(path: string): void { + if ( + typeof path !== 'string' || + path.trim() !== path || + Buffer.byteLength(path) > 1024 || + path.normalize('NFC') !== path || + /[\p{Cc}\p{Cs}\\|]/u.test(path) || + path.includes('${') || + path.split('/').some((segment) => segment.trim() === '' || segment === '.' || segment === '..') + ) { + throw new TypeError('Storage path must be a complete relative filename without interpolation') + } +} + +/** Stores one exact original; a mismatch is reported with the completed write's receipt. */ +export function storeAsset( + client: Transloadit, + filePath: string, + options: StoreAssetOptions, +): Promise { + return storeOriginal(client, filePath, options) +} + +/** Shared upload pipeline; the image entry point retains its existing transformed-byte policy. */ +export async function storeOriginal( + client: Transloadit, + filePath: string, + options: StoreAssetOptions, + image = false, +): Promise { + const { + path, + chunkSize, + onAssemblyProgress, + onUploadProgress, + onReceipt, + overwrite, + signal, + timeout, + } = options + validateDestination(path) + const placeholder = storageImagePlaceholderSchema.optional().parse(options.placeholder, { + path: ['placeholder'], + }) + if (overwrite !== undefined && typeof overwrite !== 'boolean') + throw new TypeError('overwrite must be a boolean') + signal?.throwIfAborted() + const checksum = createHash('md5') + let size = 0 + for await (const chunk of createReadStream(filePath, { signal })) { + checksum.update(chunk) + size += chunk.length + } + if (size === 0) + throw new Error(image ? 'Cannot store an empty image' : 'Cannot store an empty file') + const md5hash = checksum.digest('hex') + signal?.throwIfAborted() + const assembly = await client.createAssembly({ + chunkSize, + files: { [image ? 'image' : 'file']: filePath }, + onAssemblyProgress, + onUploadProgress, + params: { + steps: { + ...(placeholder === 'blur' + ? { + ':original': { + robot: '/upload/handle', + output_meta: { thumbhash: true }, + } satisfies InterpolatableRobotUploadHandleInstructionsInput, + } + : {}), + stored: { + robot: '/transloadit/store', + use: ':original', + path, + conflict_strategy: overwrite === true ? 'overwrite' : 'error', + }, + }, + }, + signal, + timeout, + waitForCompletion: true, + }) + const input = { path, size, md5hash } + // API2 can watermark uploads before Robots run; this exact write's result describes stored bytes. + const receipt = validateReceipt(assembly, input, true, image) + const observerFailed = (): void => { + debug('transloadit:warn')('Ignored onReceipt observer failure after a completed Storage write') + } + try { + // Neither a stalled observer nor its rejection may hide a successfully stored receipt. + void Promise.resolve(onReceipt?.({ ...receipt }, input, assembly.assembly_id)).catch( + observerFailed, + ) + } catch { + observerFailed() + } + if (!image && (receipt.size !== size || receipt.md5hash !== md5hash)) { + throw new InconsistentResponseError( + 'Stored bytes differ from the original file; do not upload again', + { + cause: { assemblyId: assembly.assembly_id, receipt, expected: input }, + }, + ) + } + return receipt +} + +/** Fetches authoritative Assembly status and verifies identity and original-byte fidelity. */ +export function getStoredAssetReceipt( + client: Transloadit, + options: GetStoredAssetReceiptOptions, +): Promise { + return getOriginalReceipt(client, options) +} + +/** Shared recovery keeps image-specific validation and Assembly diagnostics together. */ +export async function getOriginalReceipt( + client: Transloadit, + options: GetStoredAssetReceiptOptions, + image = false, +): Promise { + const { assemblyId, expected } = receiptRequestSchema.parse(options) + validateDestination(expected.path) + const assembly = await client.getAssembly(assemblyId) + if (assembly.assembly_id !== assemblyId) { + throw new InconsistentResponseError('The response did not match the requested Assembly', { + cause: { assemblyId }, + }) + } + return validateReceipt(assembly, expected, false, image) +} + +function validateReceipt( + assembly: AssemblyStatus, + expected: StoredAssetExpectation, + acceptTransformed = false, + image = false, +): StoredAssetReceipt { + const { path, size, md5hash } = expected + if (typeof assembly.error === 'string') throw new ApiError({ body: assembly }) + if (assembly.ok === 'ASSEMBLY_CANCELED') { + throw new InconsistentResponseError('The Storage Assembly ended with ASSEMBLY_CANCELED', { + cause: { assemblyId: assembly.assembly_id }, + }) + } + if ( + assembly.ok === 'ASSEMBLY_UPLOADING' || + assembly.ok === 'ASSEMBLY_EXECUTING' || + assembly.ok === 'ASSEMBLY_REPLAYING' + ) { + throw new InconsistentResponseError(`The Storage Assembly is not complete (${assembly.ok})`, { + cause: { assemblyId: assembly.assembly_id }, + }) + } + const parsed = (image ? completedImageSchema : completedAssetSchema).safeParse(assembly) + const result = parsed.success ? parsed.data.results[':original'][0] : undefined + const originals = assembly.results?.[':original'] + const receiptCheck = { + originalCount: Array.isArray(originals) ? originals.length : 0, + metadataValid: parsed.success, + pathMatches: result === undefined ? undefined : result.path === path, + sizeMatches: result === undefined ? undefined : result.size === size, + md5Matches: result === undefined ? undefined : result.md5hash === md5hash, + } + if ( + result === undefined || + result.path !== path || + (!acceptTransformed && (!receiptCheck.sizeMatches || !receiptCheck.md5Matches)) + ) { + throw new InconsistentResponseError( + `The Assembly did not return a matching Storage ${image ? 'image' : 'asset'} receipt`, + { + cause: { assemblyId: assembly.assembly_id, receiptCheck }, + }, + ) + } + return result +} diff --git a/packages/node/src/storageImage.ts b/packages/node/src/storageImage.ts index bb3f3130..9fea6668 100644 --- a/packages/node/src/storageImage.ts +++ b/packages/node/src/storageImage.ts @@ -1,46 +1,30 @@ -import type { AssemblyStatus } from './alphalib/types/assemblyStatus.ts' -import type { InterpolatableRobotUploadHandleInstructionsInput } from './alphalib/types/robots/upload-handle.ts' -import type { StoredAsset } from './alphalib/types/storageAsset.ts' -import type { CreateAssemblyOptions, Transloadit } from './Transloadit.ts' +import type { + GetStoredAssetReceiptOptions, + StoreAssetOptions, + StoredAssetExpectation, + StoredAssetReceipt, +} from './storageAsset.ts' +import type { Transloadit } from './Transloadit.ts' -import { createHash } from 'node:crypto' -import { createReadStream } from 'node:fs' - -import debug from 'debug' import { z } from 'zod' -import { ApiError } from './ApiError.ts' import { storedAssetSchema } from './alphalib/types/storageAsset.ts' import InconsistentResponseError from './InconsistentResponseError.ts' +import { getOriginalReceipt, storeOriginal } from './storageAsset.ts' + +export { storageImagePlaceholderSchema } from './storageAsset.ts' -/** - * Verified Storage metadata that can be saved and passed directly to an image renderer. - * Width and height reflect EXIF auto-orientation, matching Storage preview delivery. - */ -export interface StoredImageReceipt extends Readonly { - /** Legacy catalog spelling; new server receipts use has_alpha. */ - readonly hasAlpha?: boolean +/** Verified image metadata; dimensions reflect EXIF orientation used by preview delivery. */ +export interface StoredImageReceipt extends StoredAssetReceipt { readonly height: number - readonly md5hash: string readonly width: number } -/** Shared SDK/CLI opt-in for server extraction; rendering remains a separate choice. */ -export const storageImagePlaceholderSchema = z.enum(['blur', 'empty']) +export type StoredImageExpectation = StoredAssetExpectation +export type GetStoredImageReceiptOptions = GetStoredAssetReceiptOptions -/** One explicit destination, with the existing Assembly upload and polling controls. */ -export interface StoreImageOptions - extends Pick< - CreateAssemblyOptions, - 'chunkSize' | 'onAssemblyProgress' | 'onUploadProgress' | 'signal' | 'timeout' - > { - /** Complete relative Storage filename. Directories and interpolation expressions are rejected. */ - path: string - /** Explicit opt-in replacement of an existing path; defaults to false. */ - overwrite?: boolean - /** Request a server ThumbHash. Successful extraction adds metadata usage of 20% of the file bytes. */ - placeholder?: z.infer - /** Observe verified metadata. Not awaited; sync and async observer errors cannot undo a write. */ +/** Image-specific callback on the shared original upload pipeline. */ +export interface StoreImageOptions extends Omit { onReceipt?: ( receipt: StoredImageReceipt, input: StoredImageExpectation, @@ -48,190 +32,48 @@ export interface StoreImageOptions ) => void | Promise } -/** Trusted upload facts used to correlate a stored result with an application-owned upload. */ -export interface StoredImageExpectation { - path: string - size: number - md5hash: string -} - -/** Recovers one original's receipt without re-uploading or trusting notification payloads. */ -export interface GetStoredImageReceiptOptions { - assemblyId: string - expected: StoredImageExpectation -} - -const positiveIntegerSchema = z.number().int().positive().max(Number.MAX_SAFE_INTEGER) -const receiptRequestSchema = z.object({ - assemblyId: z.string().regex(/^[A-Za-z0-9_-]+$/), - expected: z.object({ - path: z.string(), - size: positiveIntegerSchema, - md5hash: z.string().regex(/^[a-f0-9]{32}$/), - }), -}) -const completedImageSchema = z.object({ - ok: z.literal('ASSEMBLY_COMPLETED'), - results: z.object({ - ':original': z.tuple([ - storedAssetSchema.extend({ - md5hash: storedAssetSchema.shape.md5hash.unwrap(), - height: positiveIntegerSchema, - width: positiveIntegerSchema, - size: positiveIntegerSchema, - }), - ]), - }), +const imageReceiptSchema = storedAssetSchema.extend({ + md5hash: storedAssetSchema.shape.md5hash.unwrap(), + height: z.number().int().positive().max(Number.MAX_SAFE_INTEGER), + width: z.number().int().positive().max(Number.MAX_SAFE_INTEGER), }) -function validateDestination(path: string): void { - if ( - typeof path !== 'string' || - path.trim() !== path || - Buffer.byteLength(path) > 1024 || - path.normalize('NFC') !== path || - /[\p{Cc}\p{Cs}\\|]/u.test(path) || - path.includes('${') || - path.split('/').some((segment) => segment.trim() === '' || segment === '.' || segment === '..') - ) { - throw new TypeError('Storage path must be a complete relative filename without interpolation') +function requireImage(receipt: StoredAssetReceipt): StoredImageReceipt { + const image = imageReceiptSchema.safeParse(receipt) + if (!image.success) { + throw new InconsistentResponseError( + 'The Assembly did not return a matching Storage image receipt', + { + cause: { receipt }, + }, + ) } + return image.data } -/** Stores one original without overwriting; receipt validation occurs after the Storage write. */ +/** Retains the image API's transformed-byte semantics; use storeAsset for exact originals. */ export async function storeImage( client: Transloadit, filePath: string, options: StoreImageOptions, ): Promise { - const { - path, - chunkSize, - onAssemblyProgress, - onUploadProgress, - onReceipt, - overwrite, - signal, - timeout, - } = options - validateDestination(path) - const placeholder = storageImagePlaceholderSchema.optional().parse(options.placeholder, { - path: ['placeholder'], - }) - if (overwrite !== undefined && typeof overwrite !== 'boolean') - throw new TypeError('overwrite must be a boolean') - signal?.throwIfAborted() - const checksum = createHash('md5') - let size = 0 - for await (const chunk of createReadStream(filePath, { signal })) { - checksum.update(chunk) - size += chunk.length - } - if (size === 0) throw new Error('Cannot store an empty image') - const md5hash = checksum.digest('hex') - signal?.throwIfAborted() - const assembly = await client.createAssembly({ - chunkSize, - files: { image: filePath }, - onAssemblyProgress, - onUploadProgress, - params: { - steps: { - ...(placeholder === 'blur' - ? { - ':original': { - robot: '/upload/handle', - output_meta: { thumbhash: true }, - } satisfies InterpolatableRobotUploadHandleInstructionsInput, - } - : {}), - stored: { - robot: '/transloadit/store', - use: ':original', - path, - conflict_strategy: overwrite === true ? 'overwrite' : 'error', - }, - }, + const receipt = await storeOriginal( + client, + filePath, + { + ...options, + onReceipt: (asset, input, assemblyId) => + options.onReceipt?.(requireImage(asset), input, assemblyId), }, - signal, - timeout, - waitForCompletion: true, - }) - const input = { path, size, md5hash } - // API2 can watermark uploads before Robots run; this exact write's result describes stored bytes. - const receipt = validateReceipt(assembly, input, true) - const observerFailed = (): void => { - debug('transloadit:warn')('Ignored onReceipt observer failure after a completed Storage write') - } - try { - // Neither a stalled observer nor its rejection may hide a successfully stored receipt. - void Promise.resolve(onReceipt?.({ ...receipt }, input, assembly.assembly_id)).catch( - observerFailed, - ) - } catch { - observerFailed() - } - return receipt + true, + ) + return requireImage(receipt) } -/** Fetches authoritative Assembly status and applies the same validation as a local image store. */ +/** Recovers one image receipt with the same strict byte checks as generic receipt recovery. */ export async function getStoredImageReceipt( client: Transloadit, options: GetStoredImageReceiptOptions, ): Promise { - const { assemblyId, expected } = receiptRequestSchema.parse(options) - validateDestination(expected.path) - const assembly = await client.getAssembly(assemblyId) - if (assembly.assembly_id !== assemblyId) { - throw new InconsistentResponseError('The response did not match the requested Assembly', { - cause: { assemblyId }, - }) - } - return validateReceipt(assembly, expected) -} - -function validateReceipt( - assembly: AssemblyStatus, - expected: StoredImageExpectation, - acceptTransformed = false, -): StoredImageReceipt { - const { path, size, md5hash } = expected - if (typeof assembly.error === 'string') throw new ApiError({ body: assembly }) - if (assembly.ok === 'ASSEMBLY_CANCELED') { - throw new InconsistentResponseError('The Storage Assembly ended with ASSEMBLY_CANCELED', { - cause: { assemblyId: assembly.assembly_id }, - }) - } - if ( - assembly.ok === 'ASSEMBLY_UPLOADING' || - assembly.ok === 'ASSEMBLY_EXECUTING' || - assembly.ok === 'ASSEMBLY_REPLAYING' - ) { - throw new InconsistentResponseError(`The Storage Assembly is not complete (${assembly.ok})`, { - cause: { assemblyId: assembly.assembly_id }, - }) - } - const parsed = completedImageSchema.safeParse(assembly) - const result = parsed.success ? parsed.data.results[':original'][0] : undefined - const originals = assembly.results?.[':original'] - const receiptCheck = { - originalCount: Array.isArray(originals) ? originals.length : 0, - metadataValid: parsed.success, - pathMatches: result === undefined ? undefined : result.path === path, - sizeMatches: result === undefined ? undefined : result.size === size, - md5Matches: result === undefined ? undefined : result.md5hash === md5hash, - } - if ( - result === undefined || - result.path !== path || - (!acceptTransformed && (!receiptCheck.sizeMatches || !receiptCheck.md5Matches)) - ) { - throw new InconsistentResponseError( - 'The Assembly did not return a matching Storage image receipt', - { - cause: { assemblyId: assembly.assembly_id, receiptCheck }, - }, - ) - } - return result + return requireImage(await getOriginalReceipt(client, options, true)) } diff --git a/packages/node/test/unit/cli/storage-receipts-sync.test.ts b/packages/node/test/unit/cli/storage-receipts-sync.test.ts index ecec8b79..7f82ef3a 100644 --- a/packages/node/test/unit/cli/storage-receipts-sync.test.ts +++ b/packages/node/test/unit/cli/storage-receipts-sync.test.ts @@ -585,12 +585,11 @@ test.each([ }) test.each([ - undefined, 0, -1, 1.5, Number.MAX_SAFE_INTEGER + 1, -])('fails atomically for missing/invalid image dimensions %j', async (width) => { +])('fails atomically for invalid supplied image dimensions %j', async (width) => { const previous = catalogJson({ unrelated: { keep: true } }) await writeFile('images.json', previous) const api = listed([{ ...asset, width }]) @@ -619,6 +618,70 @@ test.each([ expect(await readFile('images.json', 'utf8')).toBe(previous) }) +test.each([ + 'image/jpeg', + 'application/pdf', +])('keeps catalog order when refreshing %s', async (mime) => { + const current = { ...storedAsset({ mime }), apiOrigin: 'http://storage.invalid' } + const following = storedAsset({ path: 'website/later.jpg', mime }) + const entries = { [current.path]: current, [following.path]: following } + const category = mime === 'image/jpeg' ? 'images' : 'files' + await writeFile( + 'images.json', + JSON.stringify({ + apiOrigin: 'http://storage.invalid', + workspace: current.workspace, + public: [], + images: {}, + [category]: entries, + }), + ) + listed([storedAsset({ mime, version_id: 'C'.repeat(21) + 'A' })]) + await runSync() + expect(process.exitCode).toBeUndefined() + const catalog = JSON.parse(await readFile('images.json', 'utf8')) + expect(Object.keys(catalog[category])).toEqual([current.path, following.path]) + expect(catalog[category][current.path].version_id).toBe('C'.repeat(21) + 'A') +}) + +test.each([ + 'application/pdf', + null, +])('syncs native file paths without Viewer delimiters (%s)', async (mime) => { + const file = storedAsset({ path: 'website/report|final.pdf', mime }) + listed([file]) + await runSync() + expect(process.exitCode).toBeUndefined() + const catalog = JSON.parse(await readFile('images.json', 'utf8')) + expect(catalog.images).toEqual({}) + expect(catalog.files[file.path]).toMatchObject(file) + expect(await readFile('transloadit-images.d.ts', 'utf8')).not.toContain('@transloadit/viewer') +}) + +test('still refuses renderer delimiters for Viewer images without changing the catalog', async () => { + const previous = catalogJson({ keep: { ...recovered, path: 'keep' } }) + await writeFile('images.json', previous) + listed([storedAsset({ path: 'website/photo|final.jpg' })]) + await runSync() + expect(process.exitCode).toBe(1) + expect(OutputCtl.prototype.error).toHaveBeenCalledWith( + expect.stringContaining('unsupported by the image renderer'), + ) + expect(await readFile('images.json', 'utf8')).toBe(previous) +}) + +test('still refuses unsafe native paths for non-image files', async () => { + const previous = catalogJson({ keep: { ...recovered, path: 'keep' } }) + await writeFile('images.json', previous) + listed([storedAsset({ path: 'website/../report.pdf', mime: 'application/pdf' })]) + await runSync() + expect(process.exitCode).toBe(1) + expect(OutputCtl.prototype.error).toHaveBeenCalledWith( + expect.stringContaining('invalid asset path'), + ) + expect(await readFile('images.json', 'utf8')).toBe(previous) +}) + test.each([ 'other/a.jpg', 'website/../a.jpg', @@ -654,14 +717,36 @@ test.each([ expect(await readFile('images.json', 'utf8')).toBe(previous) }) -test('does not save an earlier image when a later image lacks height', async () => { +test('sync classifies dimensioned receipts with unknown MIME as files', async () => { + await writeFile('images.json', catalogJson({ [asset.path]: { ...recovered, mime: null } })) + listed([storedAsset({ mime: null })]) + await runSync() + expect(process.exitCode).toBeUndefined() + const catalog = JSON.parse(await readFile('images.json', 'utf8')) + expect(catalog.images).toEqual({}) + expect(catalog.files[asset.path]).toMatchObject({ + mime: null, + width: asset.width, + height: asset.height, + }) + expect(await readFile('transloadit-images.d.ts', 'utf8')).not.toContain('@transloadit/viewer') +}) + +test('recovers dimensionless files without losing the images or existing receipts', async () => { const previous = catalogJson({ keep: { ...recovered, path: 'keep' } }) await writeFile('images.json', previous) listed([asset, storedAsset({ path: 'website/b.jpg', height: undefined })]) await runSync() - expect(process.exitCode).toBe(1) - expect(OutputCtl.prototype.error).toHaveBeenCalledWith(expect.stringContaining('website/b.jpg')) - expect(await readFile('images.json', 'utf8')).toBe(previous) + expect(process.exitCode).toBeUndefined() + const catalog = JSON.parse(await readFile('images.json', 'utf8')) + expect(catalog.images[asset.path]).toEqual(recovered) + expect(catalog.images.keep).toEqual({ ...recovered, path: 'keep' }) + expect(catalog.files['website/b.jpg']).toMatchObject({ + path: 'website/b.jpg', + asset_id: asset.asset_id, + version_id: asset.version_id, + }) + expect(catalog.images['website/b.jpg']).toBeUndefined() }) test('rejects duplicate paths instead of choosing arbitrary version metadata', async () => { diff --git a/packages/node/test/unit/cli/storage-store.test.ts b/packages/node/test/unit/cli/storage-store.test.ts index d7c535b4..c59dbc5e 100644 --- a/packages/node/test/unit/cli/storage-store.test.ts +++ b/packages/node/test/unit/cli/storage-store.test.ts @@ -20,7 +20,7 @@ import { afterEach, beforeEach, describe, expect, test, vi } from 'vitest' import { ApiError } from '../../../src/ApiError.ts' import OutputCtl from '../../../src/cli/OutputCtl.ts' import { main } from '../../../src/cli.ts' -import { Transloadit } from '../../../src/Transloadit.ts' +import { InconsistentResponseError, Transloadit } from '../../../src/Transloadit.ts' import { storagePage, storedAsset } from './storage-fixtures.ts' vi.mock('node:fs/promises', async (importOriginal) => { @@ -94,6 +94,152 @@ function runStore(path = receipt.path): Promise { } describe('storage store', () => { + test.each([ + 'legacy.jpg', + 'clip.mp4', + ])('keeps dimensioned %s with unknown MIME out of Viewer', async (name) => { + const path = `website/${name}` + vi.spyOn(Transloadit.prototype, 'storeAsset').mockResolvedValue({ + ...receipt, + path, + mime: null, + }) + await runStore(path) + expect(process.exitCode).toBeUndefined() + const catalog = JSON.parse(await readFile('images.json', 'utf8')) + expect(catalog.images).toEqual({}) + expect(catalog.files[path]).toMatchObject({ mime: null, width: 800, height: 600 }) + const declarations = await readFile('transloadit-images.d.ts', 'utf8') + expect(declarations).toContain('export {}') + expect(declarations).not.toContain('@transloadit/viewer') + }) + + test('file-only catalogs emit standalone declarations without a Viewer dependency', async () => { + vi.spyOn(Transloadit.prototype, 'storeAsset').mockResolvedValue({ + ...receipt, + path: 'media/guide.pdf', + mime: 'application/pdf', + width: undefined, + height: undefined, + }) + await main(['storage', 'store', 'guide.pdf', 'media/guide.pdf']) + expect(process.exitCode).toBeUndefined() + const declarations = await readFile('transloadit-images.d.ts', 'utf8') + expect(declarations).toContain('export {}') + expect(declarations).not.toContain('@transloadit/viewer') + }) + + test('checkpoints a completed byte-mismatch receipt from the error without an observer callback', async () => { + const completed = { ...receipt, path: 'media/guide.pdf', mime: 'application/pdf' } + const store = vi.spyOn(Transloadit.prototype, 'storeAsset').mockRejectedValue( + new InconsistentResponseError( + 'Stored bytes differ from the original file; do not upload again', + { + cause: { assemblyId: 'completed', receipt: completed }, + }, + ), + ) + await main(['storage', 'store', 'guide.pdf', 'media/guide.pdf']) + expect(process.exitCode).toBe(1) + expect(store).toHaveBeenCalledOnce() + const catalog = JSON.parse(await readFile('transloadit.images.json', 'utf8')) + expect(catalog.files[completed.path]).toMatchObject(completed) + }) + + test('stores mixed originals in one catalog, replays unchanged bytes and keeps Viewer image-only', async () => { + const bytes = Buffer.from('one unchanged original') + const md5hash = createHash('md5').update(bytes).digest('hex') + await writeFile('photo.jpg', bytes) + await writeFile('clip.mp4', bytes) + await writeFile('guide.pdf', bytes) + const store = vi + .spyOn(Transloadit.prototype, 'storeAsset') + .mockImplementation(async (_file, options) => ({ + ...receipt, + path: options.path, + md5hash, + size: bytes.length, + mime: options.path.endsWith('.jpg') + ? 'image/jpeg' + : options.path.endsWith('.mp4') + ? 'video/mp4' + : 'application/pdf', + width: options.path.endsWith('.pdf') ? undefined : 800, + height: options.path.endsWith('.pdf') ? undefined : 600, + })) + const argv = ['storage', 'store', 'photo.jpg', 'clip.mp4', 'guide.pdf', 'website/media/'] + await main(argv) + expect(process.exitCode).toBeUndefined() + const first = await readFile('transloadit.images.json', 'utf8') + const catalog = JSON.parse(first) + expect(Object.keys(catalog.images)).toEqual(['website/media/photo.jpg']) + expect(Object.keys(catalog.files)).toEqual([ + 'website/media/clip.mp4', + 'website/media/guide.pdf', + ]) + expect(await readFile('transloadit-images.d.ts', 'utf8')).toContain('website/media/photo.jpg') + expect(await readFile('transloadit-images.d.ts', 'utf8')).not.toContain('clip.mp4') + await main(argv) + expect(process.exitCode).toBeUndefined() + expect(store).toHaveBeenCalledTimes(3) + expect(await readFile('transloadit.images.json', 'utf8')).toBe(first) + }) + + test('checkpoints a generic original before a later upload fails', async () => { + const store = vi + .spyOn(Transloadit.prototype, 'storeAsset') + .mockResolvedValueOnce({ + ...receipt, + path: 'media/a.mp4', + mime: 'video/mp4', + width: undefined, + height: undefined, + }) + .mockRejectedValueOnce(new Error('upload interrupted')) + await main(['storage', 'store', 'a.mp4', 'b.pdf', 'media/']) + expect(process.exitCode).toBe(1) + expect(store).toHaveBeenCalledTimes(2) + const catalog = JSON.parse(await readFile('transloadit.images.json', 'utf8')) + expect(catalog.images).toEqual({}) + expect(Object.keys(catalog.files)).toEqual(['media/a.mp4']) + }) + + test('stores prototype-looking file paths as ordinary JSON keys', async () => { + vi.spyOn(Transloadit.prototype, 'storeAsset').mockResolvedValue({ + ...receipt, + path: '__proto__', + mime: 'application/pdf', + }) + await main(['storage', 'store', 'a.pdf', '__proto__']) + expect(process.exitCode).toBeUndefined() + const catalog = JSON.parse(await readFile('transloadit.images.json', 'utf8')) + expect(Object.keys(catalog.files)).toEqual(['__proto__']) + expect(catalog.files.__proto__.path).toBe('__proto__') + }) + + test('refuses a files-only catalog from another API origin before any upload', async () => { + const before = JSON.stringify({ + workspace: 'my-app', + public: [], + images: {}, + apiOrigin: 'https://other.invalid', + files: { + 'media/a.pdf': { + ...receipt, + path: 'media/a.pdf', + mime: 'application/pdf', + apiOrigin: 'https://other.invalid', + }, + }, + }) + await writeFile('transloadit.images.json', before) + const store = vi.spyOn(Transloadit.prototype, 'storeAsset') + await main(['storage', 'store', 'new.pdf', 'media/new.pdf']) + expect(process.exitCode).toBe(1) + expect(store).not.toHaveBeenCalled() + expect(await readFile('transloadit.images.json', 'utf8')).toBe(before) + }) + test.each([ false, true, @@ -108,7 +254,7 @@ describe('storage store', () => { images: {}, }), ) - const store = vi.spyOn(Transloadit.prototype, 'storeImage').mockResolvedValue(enriched) + const store = vi.spyOn(Transloadit.prototype, 'storeAsset').mockResolvedValue(enriched) await main([ 'storage', 'store', @@ -133,7 +279,7 @@ describe('storage store', () => { }) test('a missing server hash is reported without hiding a completed store', async () => { - const store = vi.spyOn(Transloadit.prototype, 'storeImage').mockResolvedValue(receipt) + const store = vi.spyOn(Transloadit.prototype, 'storeAsset').mockResolvedValue(receipt) await main([ 'storage', 'store', @@ -164,7 +310,7 @@ describe('storage store', () => { await writeFile('hero.jpg', bytes) const catalog = `${JSON.stringify(JSON.parse(catalogJson({ [previous.path]: previous })), null, 2)}\n` await writeFile('images.json', catalog) - const store = vi.spyOn(Transloadit.prototype, 'storeImage') + const store = vi.spyOn(Transloadit.prototype, 'storeAsset') await main([ 'storage', 'store', @@ -185,14 +331,14 @@ describe('storage store', () => { }) test('rejects an unknown placeholder before uploading', async () => { - const store = vi.spyOn(Transloadit.prototype, 'storeImage') + const store = vi.spyOn(Transloadit.prototype, 'storeAsset') await main(['storage', 'store', './hero.jpg', receipt.path, '--placeholder', 'surprise']) expect(process.exitCode).toBe(1) expect(store).not.toHaveBeenCalled() }) test('plain uploads record API provenance so recovery cannot mix matching Workspace slugs', async () => { - vi.spyOn(Transloadit.prototype, 'storeImage').mockResolvedValue({ + vi.spyOn(Transloadit.prototype, 'storeAsset').mockResolvedValue({ ...storedAsset(), path: receipt.path, width: 800, @@ -209,7 +355,7 @@ describe('storage store', () => { await writeFile('hero.jpg', Buffer.from('image')) const previous = catalogJson({}) await writeFile('images.json', previous) - vi.spyOn(Transloadit.prototype, 'storeImage').mockResolvedValue({ + vi.spyOn(Transloadit.prototype, 'storeAsset').mockResolvedValue({ ...receipt, workspace: 'other-app', }) @@ -291,7 +437,7 @@ describe('storage store', () => { } await writeFile('hero.jpg', bytes) const store = vi - .spyOn(Transloadit.prototype, 'storeImage') + .spyOn(Transloadit.prototype, 'storeAsset') .mockResolvedValueOnce(first) .mockResolvedValueOnce(second) const args = ['storage', 'store', './hero.jpg', 'website/', '--hashed'] @@ -335,7 +481,7 @@ describe('storage store', () => { 'transloadit.images.json', catalogJson({ [path]: { path, md5hash, size: bytes.length, width: 800, height: 600 } }), ) - const store = vi.spyOn(Transloadit.prototype, 'storeImage') + const store = vi.spyOn(Transloadit.prototype, 'storeAsset') const args = ['storage', 'store', './hero.jpg', 'website/', '--hashed'] await main(args) expect(process.exitCode).toBe(1) @@ -361,6 +507,43 @@ describe('storage store', () => { ) }) + test.each([ + false, + true, + ])('explains recovery when a saved receipt has no checksum (hashed=%s)', async (hashed) => { + const bytes = Buffer.from('original bytes without a stored checksum') + const md5hash = createHash('md5').update(bytes).digest('hex') + const destination = 'website/hero.jpg' + const path = hashed ? `website/hero.${md5hash.slice(0, 8)}.jpg` : destination + const previous = catalogJson({ + [path]: { ...receipt, path, size: bytes.length, md5hash: undefined }, + }) + await writeFile('hero.jpg', bytes) + await writeFile('transloadit.images.json', previous) + const store = vi.spyOn(Transloadit.prototype, 'storeAsset') + await main(['storage', 'store', './hero.jpg', destination, ...(hashed ? ['--hashed'] : [])]) + expect(process.exitCode).toBe(1) + expect(store).not.toHaveBeenCalled() + expect(await readFile('transloadit.images.json', 'utf8')).toBe(previous) + expect(OutputCtl.prototype.error).toHaveBeenCalledWith( + expect.stringContaining('If the checksum remains unavailable'), + ) + expect(OutputCtl.prototype.error).toHaveBeenCalledWith( + expect.stringContaining( + hashed ? 'choose another destination basename' : 'choose a fresh name', + ), + ) + if (hashed) { + expect(OutputCtl.prototype.error).not.toHaveBeenCalledWith( + expect.stringContaining('--overwrite'), + ) + } else { + expect(OutputCtl.prototype.error).toHaveBeenCalledWith( + expect.stringContaining('--overwrite only to deliberately replace it'), + ) + } + }) + test.each([ 'checksum', 'size', @@ -384,7 +567,7 @@ describe('storage store', () => { await writeFile('hero.jpg', bytes) const previous = catalogJson({ [path]: old }) await writeFile('transloadit.images.json', previous) - const store = vi.spyOn(Transloadit.prototype, 'storeImage') + const store = vi.spyOn(Transloadit.prototype, 'storeAsset') await main(['storage', 'store', './hero.jpg', 'website/hero.jpg', '--hashed']) expect(process.exitCode).toBe(1) expect(store).not.toHaveBeenCalled() @@ -396,7 +579,7 @@ describe('storage store', () => { }) test('hashed and overwrite cannot be combined', async () => { - const store = vi.spyOn(Transloadit.prototype, 'storeImage') + const store = vi.spyOn(Transloadit.prototype, 'storeAsset') await main(['storage', 'store', './hero.jpg', receipt.path, '--hashed', '--overwrite']) expect(process.exitCode).toBe(1) expect(store).not.toHaveBeenCalled() @@ -417,7 +600,7 @@ describe('storage store', () => { }) await writeFile('hero.jpg', bytes) await writeFile('transloadit.images.json', previous) - const store = vi.spyOn(Transloadit.prototype, 'storeImage') + const store = vi.spyOn(Transloadit.prototype, 'storeAsset') await main(['storage', 'store', './hero.jpg', receipt.path, '--hashed']) expect(process.exitCode).toBe(1) expect(store).not.toHaveBeenCalled() @@ -440,7 +623,7 @@ describe('storage store', () => { await writeFile('a/hero.jpg', bytes) await writeFile('b/hero.jpg', bytes) const store = vi - .spyOn(Transloadit.prototype, 'storeImage') + .spyOn(Transloadit.prototype, 'storeAsset') .mockResolvedValueOnce({ ...receipt, path, md5hash, size: bytes.length }) .mockRejectedValueOnce(new ApiError({ body: { error: 'TRANSLOADIT_STORE_CONFLICT' } })) await main([ @@ -477,7 +660,7 @@ describe('storage store', () => { const previous = catalogJson({ [path]: stored }) await writeFile('hero.jpg', bytes) await writeFile('transloadit.images.json', previous) - const store = vi.spyOn(Transloadit.prototype, 'storeImage') + const store = vi.spyOn(Transloadit.prototype, 'storeAsset') await main(['storage', 'store', './hero.jpg', receipt.path, '--hashed']) expect(process.exitCode).toBe(1) expect(store).not.toHaveBeenCalled() @@ -496,7 +679,7 @@ describe('storage store', () => { test('hashed destinations with conflicting remote objects never suggest overwriting', async () => { await writeFile('hero.jpg', 'original') const hash = createHash('md5').update('original').digest('hex').slice(0, 8) - vi.spyOn(Transloadit.prototype, 'storeImage').mockRejectedValue( + vi.spyOn(Transloadit.prototype, 'storeAsset').mockRejectedValue( new ApiError({ body: { error: 'TRANSLOADIT_STORE_CONFLICT' } }), ) await main(['storage', 'store', './hero.jpg', receipt.path, '--hashed']) @@ -518,7 +701,7 @@ describe('storage store', () => { await writeFile('a/hero.jpg', 'a') await writeFile('b/hero.jpg', 'b') const store = vi - .spyOn(Transloadit.prototype, 'storeImage') + .spyOn(Transloadit.prototype, 'storeAsset') .mockImplementation(async (file, options) => ({ ...receipt, path: options.path, @@ -537,7 +720,7 @@ describe('storage store', () => { test('hash suffixes still respect the maximum Storage path length before uploading', async () => { await writeFile('hero.jpg', 'original') - const store = vi.spyOn(Transloadit.prototype, 'storeImage') + const store = vi.spyOn(Transloadit.prototype, 'storeAsset') await main(['storage', 'store', './hero.jpg', `${'x'.repeat(1020)}.jpg`, '--hashed']) expect(process.exitCode).toBe(1) expect(store).not.toHaveBeenCalled() @@ -546,7 +729,7 @@ describe('storage store', () => { test('saves optional ThumbHash metadata without advising blur for a private image', async () => { const blurred = { ...receipt, hasAlpha: true, thumbhash: '1QcSHQRnh493V4dIh4eXh1h4kJUI' } - vi.spyOn(Transloadit.prototype, 'storeImage').mockResolvedValue(blurred) + vi.spyOn(Transloadit.prototype, 'storeAsset').mockResolvedValue(blurred) await runStore() expect(process.exitCode).toBeUndefined() expect(JSON.parse(await readFile('images.json', 'utf8')).images[receipt.path]).toEqual(blurred) @@ -598,7 +781,7 @@ describe('storage store', () => { }), ) const stored = { ...receipt, thumbhash, ...(hasAlpha ? { hasAlpha: true } : {}) } - vi.spyOn(Transloadit.prototype, 'storeImage').mockResolvedValue(stored) + vi.spyOn(Transloadit.prototype, 'storeAsset').mockResolvedValue(stored) await runStore() expect(process.exitCode).toBeUndefined() const text = vi.mocked(OutputCtl.prototype.print).mock.calls[0]?.[0] @@ -608,7 +791,7 @@ describe('storage store', () => { test('private next steps name the files, application key and restart, using the custom catalog', async () => { await mkdir('src/app', { recursive: true }) - vi.spyOn(Transloadit.prototype, 'storeImage').mockResolvedValue(receipt) + vi.spyOn(Transloadit.prototype, 'storeAsset').mockResolvedValue(receipt) await runStore() expect(process.exitCode).toBeUndefined() const text = vi.mocked(OutputCtl.prototype.print).mock.calls[0]?.[0] @@ -630,7 +813,7 @@ describe('storage store', () => { const file = `next.config.${extension}` const source = 'throw new Error("the CLI must not execute this config")\n' await writeFile(file, source) - vi.spyOn(Transloadit.prototype, 'storeImage').mockResolvedValue(receipt) + vi.spyOn(Transloadit.prototype, 'storeAsset').mockResolvedValue(receipt) await runStore() expect(process.exitCode).toBeUndefined() const text = vi.mocked(OutputCtl.prototype.print).mock.calls[0]?.[0] @@ -644,7 +827,7 @@ describe('storage store', () => { test('omits wrapper advice when a config already uses the plugin', async () => { await writeFile('next.config.ts', 'export default withTransloaditImages(nextConfig)\n') - vi.spyOn(Transloadit.prototype, 'storeImage').mockResolvedValue(receipt) + vi.spyOn(Transloadit.prototype, 'storeAsset').mockResolvedValue(receipt) await runStore() expect(process.exitCode).toBeUndefined() const text = vi.mocked(OutputCtl.prototype.print).mock.calls[0]?.[0] @@ -655,7 +838,7 @@ describe('storage store', () => { const source = 'module.exports = { reactStrictMode: true }\n' await writeFile('next.config.js', source) await writeFile('package.json', '{"type":"commonjs"}\n') - vi.spyOn(Transloadit.prototype, 'storeImage').mockResolvedValue(receipt) + vi.spyOn(Transloadit.prototype, 'storeAsset').mockResolvedValue(receipt) await runStore() expect(process.exitCode).toBeUndefined() const text = vi.mocked(OutputCtl.prototype.print).mock.calls[0]?.[0] @@ -668,7 +851,7 @@ describe('storage store', () => { test('does not infer destination privacy from a different workspace catalog left unchanged', async () => { const previous = JSON.stringify({ workspace: 'other-app', public: ['website/'], images: {} }) await writeFile('images.json', previous) - vi.spyOn(Transloadit.prototype, 'storeImage').mockResolvedValue(receipt) + vi.spyOn(Transloadit.prototype, 'storeAsset').mockResolvedValue(receipt) await main([ 'storage', 'store', @@ -695,7 +878,7 @@ describe('storage store', () => { test('prints shared private setup and config advice once for a multi-file upload', async () => { await writeFile('next.config.ts', 'export default {}\n') - vi.spyOn(Transloadit.prototype, 'storeImage').mockImplementation(async (_file, options) => ({ + vi.spyOn(Transloadit.prototype, 'storeAsset').mockImplementation(async (_file, options) => ({ ...receipt, path: options.path, })) @@ -711,7 +894,7 @@ describe('storage store', () => { expect(output.split('is not wrapped yet')).toHaveLength(2) }) test('generated declarations cannot overwrite the catalog or input image', async () => { - const store = vi.spyOn(Transloadit.prototype, 'storeImage').mockResolvedValue(receipt) + const store = vi.spyOn(Transloadit.prototype, 'storeAsset').mockResolvedValue(receipt) await main([ 'storage', 'store', @@ -731,7 +914,7 @@ describe('storage store', () => { test('a handwritten declarations file is preserved before any upload', async () => { const previous = "declare module 'my-app' {}\n" await writeFile('transloadit-images.d.ts', previous) - const store = vi.spyOn(Transloadit.prototype, 'storeImage').mockResolvedValue(receipt) + const store = vi.spyOn(Transloadit.prototype, 'storeAsset').mockResolvedValue(receipt) await main(['storage', 'store', './hero.jpg', receipt.path]) expect(process.exitCode).toBe(1) expect(store).not.toHaveBeenCalled() @@ -739,7 +922,7 @@ describe('storage store', () => { }) test('upload output makes the declarations follow the catalog commit policy', async () => { - vi.spyOn(Transloadit.prototype, 'storeImage').mockResolvedValue(receipt) + vi.spyOn(Transloadit.prototype, 'storeAsset').mockResolvedValue(receipt) await runStore() expect(process.exitCode).toBeUndefined() expect(OutputCtl.prototype.print).toHaveBeenCalledWith( @@ -755,7 +938,7 @@ describe('storage store', () => { }) test('generated Viewer types follow the catalog commit policy and upgrade the old header', async () => { - vi.spyOn(Transloadit.prototype, 'storeImage').mockResolvedValue(receipt) + vi.spyOn(Transloadit.prototype, 'storeAsset').mockResolvedValue(receipt) await runStore() expect(process.exitCode).toBeUndefined() const types = await readFile('transloadit-images.d.ts', 'utf8') @@ -763,18 +946,58 @@ describe('storage store', () => { const legacyHeader = '// Generated by transloadit storage. Commit this file with the image catalog; do not edit.' await writeFile('transloadit-images.d.ts', types.replace(types.split('\n')[0], legacyHeader)) - await runStore() + await main([ + 'storage', + 'store', + './hero.jpg', + receipt.path, + '--receipts', + 'images.json', + '--overwrite', + ]) expect(process.exitCode).toBeUndefined() expect(await readFile('transloadit-images.d.ts', 'utf8')).toBe(types) }) + test('upgrades legacy empty Viewer declarations before a first upload', async () => { + await writeFile('images.json', JSON.stringify({ workspace: 'my-app', public: [], images: {} })) + await writeFile( + 'transloadit-images.d.ts', + [ + '// Generated by transloadit storage. Commit this file with the image catalog; do not edit.', + "import '@transloadit/viewer/next'", + '', + "declare module '@transloadit/viewer/next' {", + ' interface RegisteredStorageImages {', + ' }', + '}', + '', + ].join('\n'), + ) + const store = vi.spyOn(Transloadit.prototype, 'storeAsset').mockResolvedValue(receipt) + await runStore() + expect(process.exitCode).toBeUndefined() + expect(store).toHaveBeenCalledOnce() + const types = await readFile('transloadit-images.d.ts', 'utf8') + expect(types).toContain('// Catalog: "images.json"') + expect(types).toContain('"website/hero.jpg":') + }) + test('accepts generated declarations after a Windows checkout converts them to CRLF', async () => { - const store = vi.spyOn(Transloadit.prototype, 'storeImage').mockResolvedValue(receipt) + const store = vi.spyOn(Transloadit.prototype, 'storeAsset').mockResolvedValue(receipt) await runStore() expect(process.exitCode).toBeUndefined() const types = await readFile('transloadit-images.d.ts', 'utf8') await writeFile('transloadit-images.d.ts', types.replaceAll('\n', '\r\n')) - await runStore() + await main([ + 'storage', + 'store', + './hero.jpg', + receipt.path, + '--receipts', + 'images.json', + '--overwrite', + ]) expect(process.exitCode).toBeUndefined() expect(store).toHaveBeenCalledTimes(2) expect(await readFile('transloadit-images.d.ts', 'utf8')).toBe(types) @@ -784,7 +1007,7 @@ describe('storage store', () => { '\n', '\r\n', ])('upgrades headerless declarations with legacy alpha fields (%j)', async (eol) => { - const store = vi.spyOn(Transloadit.prototype, 'storeImage').mockResolvedValue(receipt) + const store = vi.spyOn(Transloadit.prototype, 'storeAsset').mockResolvedValue(receipt) await runStore() expect(process.exitCode).toBeUndefined() const types = await readFile('transloadit-images.d.ts', 'utf8') @@ -800,7 +1023,15 @@ describe('storage store', () => { expect(legacy).toContain('thumbhash?: string; hasAlpha?: boolean') await writeFile('transloadit-images.d.ts', legacy) - await runStore() + await main([ + 'storage', + 'store', + './hero.jpg', + receipt.path, + '--receipts', + 'images.json', + '--overwrite', + ]) expect(process.exitCode).toBeUndefined() expect(store).toHaveBeenCalledTimes(2) @@ -808,7 +1039,7 @@ describe('storage store', () => { }) test('the first public store creates a catalog and generated types without image init', async () => { - vi.spyOn(Transloadit.prototype, 'storeImage').mockResolvedValue(receipt) + vi.spyOn(Transloadit.prototype, 'storeAsset').mockResolvedValue(receipt) const publish = vi .spyOn(Transloadit.prototype, 'publishStoragePrefix') .mockImplementation(() => { @@ -845,7 +1076,7 @@ describe('storage store', () => { }) test('public store rejects a root object before any upload or policy change', async () => { - const store = vi.spyOn(Transloadit.prototype, 'storeImage') + const store = vi.spyOn(Transloadit.prototype, 'storeAsset') const publish = vi.spyOn(Transloadit.prototype, 'publishStoragePrefix') await main(['storage', 'store', './hero.jpg', 'hero.jpg', '--public']) expect(process.exitCode).toBe(1) @@ -868,14 +1099,14 @@ describe('storage store', () => { delivery, }), ) - vi.spyOn(Transloadit.prototype, 'storeImage').mockResolvedValue(receipt) + vi.spyOn(Transloadit.prototype, 'storeAsset').mockResolvedValue(receipt) await runStore() expect(process.exitCode).toBeUndefined() expect(JSON.parse(await readFile('images.json', 'utf8')).delivery).toEqual(delivery) }) test('public store checkpoints a receipt even when publication is denied', async () => { - vi.spyOn(Transloadit.prototype, 'storeImage').mockResolvedValue(receipt) + vi.spyOn(Transloadit.prototype, 'storeAsset').mockResolvedValue(receipt) vi.spyOn(Transloadit.prototype, 'publishStoragePrefix').mockRejectedValue(new Error('denied')) await main(['storage', 'store', './hero.jpg', receipt.path, '--public']) expect(process.exitCode).toBe(1) @@ -897,7 +1128,7 @@ describe('storage store', () => { 'credentials', 'TRANSLOADIT_KEY=saved-key\nTRANSLOADIT_SECRET=saved-secret\nTRANSLOADIT_WORKSPACE=my-app\nTRANSLOADIT_WORKSPACE_VERIFIED=true\nTRANSLOADIT_ENDPOINT=http://127.0.0.1:32189\n', ) - vi.spyOn(Transloadit.prototype, 'storeImage').mockResolvedValue(receipt) + vi.spyOn(Transloadit.prototype, 'storeAsset').mockResolvedValue(receipt) await main(['storage', 'store', './hero.jpg', receipt.path]) expect(process.exitCode).toBeUndefined() expect(JSON.parse(await readFile('transloadit.images.json', 'utf8')).delivery).toEqual({ @@ -937,7 +1168,7 @@ describe('storage store', () => { }, }) await main(['storage', 'store', './hero.jpg', receipt.path]) - expect(process.exitCode).toBeUndefined() + expect(process.exitCode).toBe(1) expect(OutputCtl.prototype.warn).toHaveBeenCalledWith( expect.stringContaining('same size, different MD5'), ) @@ -957,7 +1188,7 @@ describe('storage store', () => { }, }) await main(['storage', 'store', './hero.jpg', receipt.path, '--log-level', 'debug']) - expect(process.exitCode).toBeUndefined() + expect(process.exitCode).toBe(1) expect( JSON.parse(await readFile('transloadit.images.json', 'utf8')).images[receipt.path], ).toEqual(stored) @@ -975,7 +1206,9 @@ describe('storage store', () => { expect(OutputCtl.prototype.debug).toHaveBeenCalledWith( expect.stringContaining('"md5Matches":false'), ) - expect(OutputCtl.prototype.error).not.toHaveBeenCalled() + expect(OutputCtl.prototype.error).toHaveBeenCalledWith( + expect.stringContaining('Do not re-upload'), + ) expect(OutputCtl.prototype.warn).toHaveBeenCalledWith( expect.stringContaining('older deployments'), ) @@ -985,7 +1218,7 @@ describe('storage store', () => { const listeners = process.listeners('SIGINT') const previous = catalogJson({ 'website/earlier.jpg': receipt }) await writeFile('images.json', previous) - vi.spyOn(Transloadit.prototype, 'storeImage').mockImplementation((_file, options) => { + vi.spyOn(Transloadit.prototype, 'storeAsset').mockImplementation((_file, options) => { process.emit('SIGINT') expect(options.signal?.aborted).toBe(true) options.signal?.throwIfAborted() @@ -1007,7 +1240,7 @@ describe('storage store', () => { process.emit('SIGINT') await replace(from, to) }) - const store = vi.spyOn(Transloadit.prototype, 'storeImage').mockResolvedValue(receipt) + const store = vi.spyOn(Transloadit.prototype, 'storeAsset').mockResolvedValue(receipt) await main(['storage', 'store', './hero.jpg', './next.jpg', 'website/']) expect(process.exitCode).toBe(1) expect(store).toHaveBeenCalledTimes(1) @@ -1035,7 +1268,7 @@ describe('storage store', () => { 'transloadit.images.json', JSON.stringify({ workspace: 'project-app', public: [], images: {} }), ) - const store = vi.spyOn(Transloadit.prototype, 'storeImage').mockResolvedValue(receipt) + const store = vi.spyOn(Transloadit.prototype, 'storeAsset').mockResolvedValue(receipt) await main(['storage', 'store', './hero.jpg', receipt.path]) expect(process.exitCode).toBe(1) expect(store).not.toHaveBeenCalled() @@ -1046,7 +1279,7 @@ describe('storage store', () => { test('checkpoints earlier multi-file uploads if a later upload fails', async () => { const store = vi - .spyOn(Transloadit.prototype, 'storeImage') + .spyOn(Transloadit.prototype, 'storeAsset') .mockResolvedValueOnce({ ...receipt, path: 'website/a.jpg' }) .mockRejectedValueOnce(new Error('Second upload failed')) await main(['storage', 'store', './a.jpg', './b.jpg', 'website/']) @@ -1057,14 +1290,14 @@ describe('storage store', () => { }) test('refuses duplicate destination basenames before the first upload', async () => { - const store = vi.spyOn(Transloadit.prototype, 'storeImage').mockResolvedValue(receipt) + const store = vi.spyOn(Transloadit.prototype, 'storeAsset').mockResolvedValue(receipt) await main(['storage', 'store', './a/hero.jpg', './b/hero.jpg', 'website/']) expect(process.exitCode).toBe(1) expect(store).not.toHaveBeenCalled() }) test('stores multiple originals in a directory and commits both receipts', async () => { const store = vi - .spyOn(Transloadit.prototype, 'storeImage') + .spyOn(Transloadit.prototype, 'storeAsset') .mockImplementation(async (_file, options) => ({ ...receipt, path: options.path })) await main(['storage', 'store', './a.jpg', './b.jpg', 'website/']) expect(process.exitCode).toBeUndefined() @@ -1084,7 +1317,7 @@ describe('storage store', () => { }) test('refuses ambiguous multi-file destinations before uploading', async () => { - const store = vi.spyOn(Transloadit.prototype, 'storeImage').mockResolvedValue(receipt) + const store = vi.spyOn(Transloadit.prototype, 'storeAsset').mockResolvedValue(receipt) await main(['storage', 'store', './a.jpg', './b.jpg', 'website/hero.jpg']) expect(process.exitCode).toBe(1) expect(store).not.toHaveBeenCalled() @@ -1092,7 +1325,7 @@ describe('storage store', () => { }) test('the printed typed path has a readable filename alt', async () => { - vi.spyOn(Transloadit.prototype, 'storeImage').mockResolvedValue(receipt) + vi.spyOn(Transloadit.prototype, 'storeAsset').mockResolvedValue(receipt) await runStore() const text = vi.mocked(OutputCtl.prototype.print).mock.calls[0]?.[0] expect(text).toContain('alt="hero"') @@ -1120,7 +1353,7 @@ describe('storage store', () => { '.env', 'TRANSLOADIT_KEY=project-key\nTRANSLOADIT_SECRET=project-secret\nTRANSLOADIT_WORKSPACE=project-workspace\n', ) - vi.spyOn(Transloadit.prototype, 'storeImage').mockImplementation(() => { + vi.spyOn(Transloadit.prototype, 'storeAsset').mockImplementation(() => { if (setup === 'saved') expect(OutputCtl.prototype.notice).not.toHaveBeenCalled() else expect(OutputCtl.prototype.notice).toHaveBeenCalledWith( @@ -1140,7 +1373,7 @@ describe('storage store', () => { test.each([ '--private', ])('refuses the removed snippet-only flag %s before uploading', async (delivery) => { - const store = vi.spyOn(Transloadit.prototype, 'storeImage').mockResolvedValue(receipt) + const store = vi.spyOn(Transloadit.prototype, 'storeAsset').mockResolvedValue(receipt) await main(['storage', 'store', './hero.jpg', receipt.path, delivery]) expect(process.exitCode).toBe(1) expect(store).not.toHaveBeenCalled() @@ -1148,7 +1381,7 @@ describe('storage store', () => { }) test('stores a root object without implicitly generating workspace-wide authorization', async () => { const rootReceipt = { ...receipt, path: 'hero.jpg' } - vi.spyOn(Transloadit.prototype, 'storeImage').mockResolvedValue(rootReceipt) + vi.spyOn(Transloadit.prototype, 'storeAsset').mockResolvedValue(rootReceipt) await runStore(rootReceipt.path) expect(process.exitCode).toBeUndefined() expect(JSON.parse(await readFile('images.json', 'utf8')).images['hero.jpg']).toEqual( @@ -1167,7 +1400,7 @@ describe('storage store', () => { width, maxWidth, }) => { - vi.spyOn(Transloadit.prototype, 'storeImage').mockResolvedValue({ ...receipt, width }) + vi.spyOn(Transloadit.prototype, 'storeAsset').mockResolvedValue({ ...receipt, width }) await runStore() expect(process.exitCode).toBeUndefined() expect(OutputCtl.prototype.print).toHaveBeenCalledWith( @@ -1180,7 +1413,7 @@ describe('storage store', () => { test('printed JSX preserves special characters in Storage paths', async () => { const path = 'website/a&"b.jpg' - vi.spyOn(Transloadit.prototype, 'storeImage').mockResolvedValue({ ...receipt, path }) + vi.spyOn(Transloadit.prototype, 'storeAsset').mockResolvedValue({ ...receipt, path }) await runStore(path) expect(process.exitCode).toBeUndefined() expect(OutputCtl.prototype.print).toHaveBeenCalledWith( @@ -1190,7 +1423,7 @@ describe('storage store', () => { }) test('releases the writer lock even when temporary-file cleanup fails', async () => { - vi.spyOn(Transloadit.prototype, 'storeImage').mockResolvedValue(receipt) + vi.spyOn(Transloadit.prototype, 'storeAsset').mockResolvedValue(receipt) const remove = vi.mocked(rm).getMockImplementation() if (remove === undefined) throw new Error('Expected the real filesystem mock implementation') vi.mocked(rm).mockImplementation((path, options) => { @@ -1207,7 +1440,7 @@ describe('storage store', () => { test('retains the complete receipt and previous catalog when atomic replacement fails', async () => { const previous = catalogJson({ 'website/earlier.jpg': receipt }) await writeFile('images.json', previous) - vi.spyOn(Transloadit.prototype, 'storeImage').mockResolvedValue(receipt) + vi.spyOn(Transloadit.prototype, 'storeAsset').mockResolvedValue(receipt) vi.mocked(rename).mockRejectedValueOnce( Object.assign(new Error('EACCES: rename denied'), { code: 'EACCES' }), ) @@ -1228,7 +1461,7 @@ describe('storage store', () => { test('preserves an existing catalog mode across its atomic replacement', async () => { await writeFile('images.json', catalogJson({})) await chmod('images.json', 0o640) - vi.spyOn(Transloadit.prototype, 'storeImage').mockResolvedValue(receipt) + vi.spyOn(Transloadit.prototype, 'storeAsset').mockResolvedValue(receipt) await runStore() expect(process.exitCode).toBeUndefined() expect((await stat('images.json')).mode & 0o777).toBe(0o640) @@ -1236,7 +1469,7 @@ describe('storage store', () => { test('prints only the changed receipt for a src/app consumer', async () => { await mkdir('src/app', { recursive: true }) - vi.spyOn(Transloadit.prototype, 'storeImage').mockResolvedValue(receipt) + vi.spyOn(Transloadit.prototype, 'storeAsset').mockResolvedValue(receipt) await runStore() expect(process.exitCode).toBeUndefined() const snippet = vi.mocked(OutputCtl.prototype.print).mock.calls[0]?.[0] @@ -1249,7 +1482,7 @@ describe('storage store', () => { '.images.json', ])('stores a catalog named %s without printing an import', async (name) => { await mkdir('app') - vi.spyOn(Transloadit.prototype, 'storeImage').mockResolvedValue(receipt) + vi.spyOn(Transloadit.prototype, 'storeAsset').mockResolvedValue(receipt) await main(['storage', 'store', './hero.jpg', receipt.path, '--receipts', `app/${name}`]) expect(process.exitCode).toBeUndefined() expect(vi.mocked(OutputCtl.prototype.print).mock.calls[0]?.[0]).toContain( @@ -1257,8 +1490,44 @@ describe('storage store', () => { ) }) + test.each([ + 'image/jpeg', + 'application/pdf', + ])('keeps catalog order when replacing %s', async (mime) => { + const current = { ...receipt, mime } + const following = { ...current, path: 'website/later.jpg' } + const entries = { [current.path]: current, [following.path]: following } + const category = mime === 'image/jpeg' ? 'images' : 'files' + await writeFile( + 'images.json', + JSON.stringify({ + workspace: current.workspace, + public: [], + images: {}, + [category]: entries, + }), + ) + vi.spyOn(Transloadit.prototype, 'storeAsset').mockResolvedValue({ + ...current, + version_id: 'C'.repeat(21) + 'A', + }) + await main([ + 'storage', + 'store', + './hero.jpg', + current.path, + '--receipts', + 'images.json', + '--overwrite', + ]) + expect(process.exitCode).toBeUndefined() + const catalog = JSON.parse(await readFile('images.json', 'utf8')) + expect(Object.keys(catalog[category])).toEqual([current.path, following.path]) + expect(catalog[category][current.path].version_id).toBe('C'.repeat(21) + 'A') + }) + test('overwrites only when explicitly requested', async () => { - const store = vi.spyOn(Transloadit.prototype, 'storeImage').mockResolvedValue(receipt) + const store = vi.spyOn(Transloadit.prototype, 'storeAsset').mockResolvedValue(receipt) await main([ 'storage', 'store', @@ -1289,9 +1558,8 @@ describe('storage store', () => { ':original': [ { ...receipt, - height: undefined, + size: -1, md5hash: createHash('md5').update(bytes).digest('hex'), - size: bytes.length, }, ], }, @@ -1373,7 +1641,7 @@ describe('storage store', () => { test('keeps a receipts-file permission error and names the file before uploading', async () => { const previous = catalogJson({ [receipt.path]: receipt }) await writeFile('images.json', previous) - const store = vi.spyOn(Transloadit.prototype, 'storeImage').mockResolvedValue(receipt) + const store = vi.spyOn(Transloadit.prototype, 'storeAsset').mockResolvedValue(receipt) // Inject the OS error so this also exercises EACCES when the test process runs as root. vi.mocked(readFile).mockRejectedValueOnce( Object.assign(new Error('EACCES: permission denied'), { code: 'EACCES' }), @@ -1394,7 +1662,7 @@ describe('storage store', () => { ])('names a receipts %s without calling it invalid JSON', async (kind) => { if (kind === 'symlink') await symlink('credentials', 'images.json') else await mkdir('images.json') - const store = vi.spyOn(Transloadit.prototype, 'storeImage').mockResolvedValue(receipt) + const store = vi.spyOn(Transloadit.prototype, 'storeAsset').mockResolvedValue(receipt) await runStore() expect(process.exitCode).toBe(1) expect(store).not.toHaveBeenCalled() @@ -1408,7 +1676,7 @@ describe('storage store', () => { test('preserves every existing path, including ordinary JSON prototype-looking keys', async () => { const earlier = { ...receipt, path: '__proto__' } await writeFile('images.json', catalogJson({ [earlier.path]: earlier })) - vi.spyOn(Transloadit.prototype, 'storeImage').mockResolvedValue(receipt) + vi.spyOn(Transloadit.prototype, 'storeAsset').mockResolvedValue(receipt) await runStore() expect(JSON.parse(await readFile('images.json', 'utf8')).images).toEqual({ [earlier.path]: earlier, @@ -1416,10 +1684,10 @@ describe('storage store', () => { }) }) - test('uses storeImage and appends a keyed receipt with a ready-to-render snippet', async () => { + test('uses storeAsset and appends a keyed receipt with a ready-to-render snippet', async () => { const earlier = { ...receipt, path: 'website/earlier.jpg' } await writeFile('images.json', catalogJson({ [earlier.path]: earlier })) - const store = vi.spyOn(Transloadit.prototype, 'storeImage').mockResolvedValue(receipt) + const store = vi.spyOn(Transloadit.prototype, 'storeAsset').mockResolvedValue(receipt) await runStore() expect(process.exitCode).toBeUndefined() expect(store).toHaveBeenCalledExactlyOnceWith('./hero.jpg', { @@ -1447,8 +1715,9 @@ describe('storage store', () => { ]) }) - test('preserves the first receipt when a second store conflicts', async () => { - const store = vi.spyOn(Transloadit.prototype, 'storeImage').mockResolvedValueOnce(receipt) + test('preserves the first receipt when changed local bytes conflict, without uploading', async () => { + await writeFile('hero.jpg', 'changed local file') + const store = vi.spyOn(Transloadit.prototype, 'storeAsset').mockResolvedValueOnce(receipt) await runStore() const previous = await readFile('images.json', 'utf8') store.mockRejectedValueOnce(new ApiError({ body: { error: 'TRANSLOADIT_STORE_CONFLICT' } })) @@ -1457,17 +1726,18 @@ describe('storage store', () => { expect(await readFile('images.json', 'utf8')).toBe(previous) expect(await readdir(directory)).toEqual([ 'credentials', + 'hero.jpg', 'images.json', 'transloadit-images.d.ts', ]) expect(OutputCtl.prototype.error).toHaveBeenCalledWith( - 'Storage destination "website/hero.jpg" already exists. Choose a fresh name; use --overwrite only if you deliberately want to replace that object.', + 'Stored bytes for "website/hero.jpg" differ from this file. Choose a fresh name or use --overwrite only to deliberately replace it. Nothing uploaded or replaced.', ) }) test('leaves no receipts or temporary files after an upload failure', async () => { const store = vi - .spyOn(Transloadit.prototype, 'storeImage') + .spyOn(Transloadit.prototype, 'storeAsset') .mockRejectedValue(new Error('Offline')) await runStore() expect(store).toHaveBeenCalledOnce() @@ -1483,7 +1753,7 @@ describe('storage store', () => { '{', ])('rejects invalid existing receipts before uploading (%j)', async (previous) => { await writeFile('images.json', previous) - const store = vi.spyOn(Transloadit.prototype, 'storeImage').mockResolvedValue(receipt) + const store = vi.spyOn(Transloadit.prototype, 'storeAsset').mockResolvedValue(receipt) await runStore() expect(process.exitCode).toBe(1) expect(store).not.toHaveBeenCalled() @@ -1498,7 +1768,7 @@ describe('storage store', () => { test('refuses another writer before uploading instead of losing its receipts', async () => { await writeFile('images.json.lock', 'another-writer') - const store = vi.spyOn(Transloadit.prototype, 'storeImage').mockResolvedValue(receipt) + const store = vi.spyOn(Transloadit.prototype, 'storeAsset').mockResolvedValue(receipt) await runStore() expect(process.exitCode).toBe(1) expect(store).not.toHaveBeenCalled() @@ -1514,7 +1784,7 @@ describe('storage store', () => { vi.stubEnv('TRANSLOADIT_SMART_CDN_KEY', 'delivery-key') vi.stubEnv('TRANSLOADIT_SMART_CDN_SECRET', 'delivery-secret') await writeFile('credentials', 'TRANSLOADIT_KEY=stored-key\nTRANSLOADIT_SECRET=stored-secret\n') - const store = vi.spyOn(Transloadit.prototype, 'storeImage').mockResolvedValue(receipt) + const store = vi.spyOn(Transloadit.prototype, 'storeAsset').mockResolvedValue(receipt) await runStore() expect(process.exitCode).toBeUndefined() expect(store).toHaveBeenCalledOnce() @@ -1527,7 +1797,7 @@ describe('storage store', () => { vi.stubEnv('TRANSLOADIT_SMART_CDN_SECRET', 'delivery-secret') const previous = catalogJson({ [receipt.path]: receipt }) await writeFile('images.json', previous) - const store = vi.spyOn(Transloadit.prototype, 'storeImage').mockResolvedValue(receipt) + const store = vi.spyOn(Transloadit.prototype, 'storeAsset').mockResolvedValue(receipt) await runStore() expect(process.exitCode).toBe(1) expect(store).not.toHaveBeenCalled() diff --git a/packages/node/test/unit/storageAsset.test.ts b/packages/node/test/unit/storageAsset.test.ts new file mode 100644 index 00000000..f262439e --- /dev/null +++ b/packages/node/test/unit/storageAsset.test.ts @@ -0,0 +1,113 @@ +import { createHash } from 'node:crypto' +import { mkdtemp, rm, writeFile } from 'node:fs/promises' +import { tmpdir } from 'node:os' +import { join } from 'node:path' + +import { afterEach, expect, onTestFinished, test, vi } from 'vitest' + +import { InconsistentResponseError, Transloadit } from '../../src/Transloadit.ts' + +afterEach(() => vi.restoreAllMocks()) + +async function fixture(mime = 'application/pdf', extension = 'pdf') { + const directory = await mkdtemp(join(tmpdir(), 'store-asset-')) + onTestFinished(() => rm(directory, { recursive: true, force: true })) + const file = join(directory, `original.${extension}`) + const bytes = Buffer.from('original fixture bytes') + await writeFile(file, bytes) + const receipt = { + workspace: 'my-app', + asset_id: 'A'.repeat(22), + version_id: `${'B'.repeat(21)}A`, + path: `media/original.${extension}`, + size: bytes.length, + md5hash: createHash('md5').update(bytes).digest('hex'), + mime, + } + const response = { + assembly_id: 'completed', + ok: 'ASSEMBLY_COMPLETED', + results: { ':original': [receipt] }, + } + const client = new Transloadit({ + authKey: 'test', + authSecret: 'test', + endpoint: 'http://127.0.0.1:9', + }) + const create = vi.spyOn(client, 'createAssembly').mockResolvedValue(response) + const get = vi.spyOn(client, 'getAssembly').mockResolvedValue(response) + return { file, client, create, get, receipt, response } +} + +test.each([ + ['video/mp4', 'mp4'], + ['audio/mpeg', 'mp3'], + ['application/pdf', 'pdf'], + ['text/html', 'html'], + ['image/x-canon-cr2', 'cr2'], + ['application/zip', 'zip'], +])('stores and recovers %s without requiring dimensions or transforming bytes', async (mime, extension) => { + const { client, create, get, file, receipt } = await fixture(mime, extension) + await expect(client.storeAsset(file, { path: receipt.path })).resolves.toEqual(receipt) + expect(create).toHaveBeenCalledWith( + expect.objectContaining({ + files: { file }, + waitForCompletion: true, + params: { + steps: { + stored: { + robot: '/transloadit/store', + use: ':original', + path: receipt.path, + conflict_strategy: 'error', + }, + }, + }, + }), + ) + await expect( + client.getStoredAssetReceipt({ assemblyId: 'completed', expected: receipt }), + ).resolves.toEqual(receipt) + expect(create).toHaveBeenCalledOnce() + expect(get).toHaveBeenCalledWith('completed') +}) + +test('preserves the actual completed receipt when original-byte fidelity fails', async () => { + const { client, create, file, receipt, response } = await fixture() + const changed = { ...receipt, md5hash: 'a'.repeat(32) } + create.mockResolvedValue({ ...response, results: { ':original': [changed] } }) + const onReceipt = vi.fn() + await expect(client.storeAsset(file, { path: receipt.path, onReceipt })).rejects.toMatchObject({ + cause: { + assemblyId: 'completed', + receipt: changed, + expected: expect.objectContaining({ md5hash: receipt.md5hash }), + }, + }) + expect(onReceipt).toHaveBeenCalledWith( + changed, + expect.objectContaining({ md5hash: receipt.md5hash }), + 'completed', + ) + expect(create).toHaveBeenCalledOnce() +}) + +test('rejects mismatched recovery without another upload', async () => { + const { client, create, receipt } = await fixture() + await expect( + client.getStoredAssetReceipt({ assemblyId: 'completed', expected: { ...receipt, size: 999 } }), + ).rejects.toBeInstanceOf(InconsistentResponseError) + expect(create).not.toHaveBeenCalled() +}) + +test('requires deliberate overwrite and propagates cancellation without uploading', async () => { + const { client, create, file, receipt } = await fixture() + await client.storeAsset(file, { path: receipt.path, overwrite: true }) + expect(create.mock.calls[0]?.[0]?.params?.steps?.stored).toMatchObject({ + conflict_strategy: 'overwrite', + }) + create.mockClear() + const signal = AbortSignal.abort() + await expect(client.storeAsset(file, { path: receipt.path, signal })).rejects.toThrow() + expect(create).not.toHaveBeenCalled() +}) diff --git a/packages/node/test/unit/storageImage.test.ts b/packages/node/test/unit/storageImage.test.ts index 20ddfbbc..02f3a7f8 100644 --- a/packages/node/test/unit/storageImage.test.ts +++ b/packages/node/test/unit/storageImage.test.ts @@ -501,9 +501,10 @@ test.each([ ['non-finite height', { ...receipt, height: Number.POSITIVE_INFINITY }], ])('rejects a receipt with %s after writing', async (_name, invalid) => { const { client, create } = fixture({ ...completed, results: { ':original': [invalid] } }) - await expect(client.storeImage(filePath, { path: receipt.path })).rejects.toThrow( - InconsistentResponseError, - ) + await expect(client.storeImage(filePath, { path: receipt.path })).rejects.toMatchObject({ + name: 'InconsistentResponseError', + cause: { assemblyId: 'completed-assembly' }, + }) expect(create).toHaveBeenCalledOnce() vi.spyOn(client, 'getAssembly').mockResolvedValue({ ...completed, diff --git a/packages/transloadit/README.md b/packages/transloadit/README.md index 3a32ed30..e6cad6c3 100644 --- a/packages/transloadit/README.md +++ b/packages/transloadit/README.md @@ -1608,7 +1608,71 @@ string, so consumers inspecting Zod schema classes or validation issues must han validation. The exported `MODEL_CAPABILITIES` registry is read-only and frozen at the top level; do not mutate its entries. -### Store an image +### Mixed-media originals + +`storeAsset()` stores one original of any media type without a conversion step: + +```ts +const asset = await client.storeAsset('./beach-waves.mp4', { + path: 'website/blog-previews/beach-waves.mp4', +}) +await saveAsset(asset) +``` + +It verifies the returned size and MD5 against the local file; dimensions are optional. Existing +paths fail unless `overwrite: true` is deliberate. If stored bytes differ, the call rejects with +the completed receipt and Assembly ID in the error's `cause`; `onReceipt` also receives them. +That is a completed write, not a reason to retry. `getStoredAssetReceipt({ assemblyId, expected: +{ path, size, md5hash } })` recovers a receipt without uploading and verifies the same byte facts. +Neither method publishes files or edits a local catalog. + +The existing CLI accepts shell-expanded local files of different types: + +```console +transloadit storage store ./photo.jpg ./clip.mp4 ./guide.pdf website/blog-previews/ +``` + +Both use `transloadit.images.json` (or `--receipts`): dimensioned receipts with an `image/*` MIME +type go into `images`; other media and unknown MIME types go into `files`. Viewer uses only +`images`. Existing catalogs remain readable; sync reclassifies matching entries from current metadata. +Each successful upload is checkpointed; +rerunning skips paths whose saved receipt matches the local bytes. Changed bytes require a new +name or deliberate `--overwrite`; `--hashed` chooses content-addressed names instead. Sync +recovers mixed prefixes atomically and preserves unmatched entries; use this CLI version or +newer for mixed prefixes. A receipt mismatch stops the batch but preserves the actual write. + +Replay skips compare local bytes with the saved receipt, not current server existence or access. +Use `storage receipts sync` to refresh a stale catalog after server-side moves or replacements. +Legacy entries without byte hashes or a verified API origin need recovery before replay; the CLI +does not guess. Sync cannot generate a missing checksum: if none is available, choose a fresh name +or deliberately use `--overwrite` (never with `--hashed`). Images without dimensions remain +recoverable in `files`, but are not Viewer sources. +`--placeholder blur` explicitly requests server metadata extraction for every input in that command; +use a separate image-only batch when other files do not need it. Successful extraction adds metadata +usage of 20% of the input bytes. + +If the workspace transforms uploads (for example, adding a watermark), replacing or renaming a file +does not bypass that policy. Use an original-preserving plan or the SDK `storeImage()` API when you +deliberately accept transformed image bytes; its completed receipt describes the stored image. + +For a **deliberately published** directory, Utils builds original or preview links from the receipt: + +```ts +import { getPublicStoredAssetUrl } from '@transloadit/utils' + +const playbackUrl = getPublicStoredAssetUrl(asset) +const downloadUrl = getPublicStoredAssetUrl(asset, { download: true }) +const posterUrl = getPublicStoredAssetUrl(asset, { preview: { width: 960, preset: 'frame' } }) +``` + +This helper does not publish or check current access. Public downloads use the stored filename; +previews offer `auto`, `frame` or `icon`, not animated clips or arbitrary transforms. HTML/XML/SVG +originals download rather than execute inline; image/video/audio remain inline-capable. These +URLs require API2's `public-serve@0.0.3` / `public-preview@0.0.4` deployment before adoption. +Do not put private receipts or placeholder pixels in public catalogs. Unpublishing cannot recall +bytes already cached by a browser or CDN. Keep source originals and license/attribution records. + +### Image receipts `storeImage()` uploads one local image and returns a verified receipt ready for `Image`. Install `@transloadit/node` as a runtime dependency for server uploads; a dev dependency is enough @@ -1658,8 +1722,8 @@ for alpha handling and existing images without placeholders. Save the complete stored result from `results[producingStep][i]` after `ASSEMBLY_COMPLETED`, including its Workspace and returned path. Collision handling can rename the requested destination. -`storeImage()` returns this canonical record plus verified image metadata; other media use the -same `StoredAsset` contract without requiring image dimensions. +`storeImage()` returns this canonical record plus verified image metadata; `storeAsset()` supports +the same `StoredAsset` contract without requiring image dimensions. ```ts const page = await client.listStoredAssets({ prefix: 'uploads/', limit: 100 }) diff --git a/packages/utils/README.md b/packages/utils/README.md index d5c5bea2..f25c4899 100644 --- a/packages/utils/README.md +++ b/packages/utils/README.md @@ -40,6 +40,18 @@ const url = await getSignedSmartCdnUrl({ ### Smart CDN URL grammar +For a verified Storage receipt under an explicitly published prefix, use +`getPublicStoredAssetUrl(receipt, options?)`. No secret, network request or publication is involved. +Omit options for the original; use `{ download: true }` to download with its stored filename, or +`{ preview: { width: 960, preset: 'frame' } }` for a still preview. Presets are `auto`, `frame`, +and `icon`; width/height are bounded at 4096 and quality at 85. `format`, `resizeStrategy`, and +`background` expose the existing bounded public preview controls. Download and preview are +mutually exclusive; public custom filenames, clip controls and unknown options are rejected. +Identity comes only from `workspace`, `asset_id`, and `version_id`, never a mutable path. The +helper requires API2 `public-serve@0.0.3` / `public-preview@0.0.4`; use signed delivery for +private assets. A generated public URL is not proof of publication or authorization, and +unpublishing does not recall cached bytes. + The URL builders and parser share one grammar, so a URL built here parses back into the options that built it (and vice versa): diff --git a/packages/utils/src/index.ts b/packages/utils/src/index.ts index 10c721cf..82740328 100644 --- a/packages/utils/src/index.ts +++ b/packages/utils/src/index.ts @@ -24,6 +24,11 @@ export type { SmartCdnImageSource, SmartCdnImageSourceDimensions, } from './smartCdnImage.ts' +export type { + PublicStoredAssetPreview, + PublicStoredAssetReference, + PublicStoredAssetUrlOptions, +} from './storagePublicUrl.ts' export * from './assemblyInstructionsCompiler.ts' export { getSmartCdnUrl, parseSmartCdnUrl, stripSmartCdnAuth } from './smartCdn.ts' @@ -35,6 +40,7 @@ export { smartCdnImageMaxDimension, } from './smartCdnImage.ts' export { validateStoragePath, validateStoragePathPrefix } from './storagePath.ts' +export { getPublicStoredAssetUrl } from './storagePublicUrl.ts' const algorithmMap = { sha1: 'SHA-1', diff --git a/packages/utils/src/smartCdnImage.ts b/packages/utils/src/smartCdnImage.ts index 1c359b33..faff5a9a 100644 --- a/packages/utils/src/smartCdnImage.ts +++ b/packages/utils/src/smartCdnImage.ts @@ -14,7 +14,8 @@ export function getSmartCdnImageLimits(template: string): { // Do not assume customer Templates or future Built-in versions share this public contract. return template === 'builtin/public-preview@0.0.1' || template === 'builtin/public-preview@0.0.2' || - template === 'builtin/public-preview@0.0.3' + template === 'builtin/public-preview@0.0.3' || + template === 'builtin/public-preview@0.0.4' ? { maxDimension: 4096, maxQuality: 85 } : { maxDimension: smartCdnImageMaxDimension, maxQuality: 100 } } diff --git a/packages/utils/src/storagePublicUrl.ts b/packages/utils/src/storagePublicUrl.ts new file mode 100644 index 00000000..8a90b6c6 --- /dev/null +++ b/packages/utils/src/storagePublicUrl.ts @@ -0,0 +1,111 @@ +import type { SmartCdnUrlParams } from './smartCdn.ts' + +import { getSmartCdnUrl } from './smartCdn.ts' + +const previewFormats = ['avif', 'gif', 'jpeg', 'jpg', 'png', 'webp'] as const +const previewResizeStrategies = ['crop', 'fit', 'fillcrop', 'pad', 'stretch'] as const +const previewPresets = ['auto', 'frame', 'icon'] as const + +/** Immutable identity from a verified Storage receipt. A URL never publishes an asset. */ +export interface PublicStoredAssetReference { + workspace: string + asset_id: string + version_id: string +} + +/** Bounded still-preview controls; omitted values use the public Built-in's defaults. */ +export interface PublicStoredAssetPreview { + width?: number + height?: number + quality?: number + format?: (typeof previewFormats)[number] + resizeStrategy?: (typeof previewResizeStrategies)[number] + background?: string + preset?: (typeof previewPresets)[number] +} + +/** Choose original bytes (optionally downloaded) or one still preview, never both. */ +export type PublicStoredAssetUrlOptions = + | { download?: boolean; preview?: never } + | { preview: PublicStoredAssetPreview; download?: never } + +function boundedInteger(value: number | undefined, name: string, maximum: number): void { + if (value !== undefined && (!Number.isInteger(value) || value < 1 || value > maximum)) { + throw new RangeError(`${name} must be an integer between 1 and ${maximum}`) + } +} + +/** + * Build an unsigned, version-pinned URL for an asset under an explicitly published prefix. + * It does not verify current publication or make a network request. Use signed delivery for + * private assets. Existing cached bytes cannot be recalled by deleting or unpublishing an asset. + */ +export function getPublicStoredAssetUrl( + asset: PublicStoredAssetReference, + options: PublicStoredAssetUrlOptions = {}, +): string { + // Utils is dependency-free and cannot import the Node CLI's Zod catalog schema. + // Keep this small URL-boundary check local rather than making that schema a browser dependency. + if (!/^[a-zA-Z0-9][a-zA-Z0-9_-]{0,127}$/.test(asset.workspace)) { + throw new TypeError('workspace must be a valid Workspace slug') + } + const identity = /^[A-Za-z0-9_-]{21}[AQgw]$/u + if (!identity.test(asset.asset_id) || !identity.test(asset.version_id)) { + throw new TypeError('asset_id and version_id must be canonical Storage IDs') + } + if (Object.keys(options).some((key) => key !== 'download' && key !== 'preview')) { + throw new TypeError('Unknown public Storage URL option') + } + if (options.download !== undefined && typeof options.download !== 'boolean') { + throw new TypeError('download must be a boolean; public filenames come from Storage') + } + const urlParams: SmartCdnUrlParams = { v: asset.version_id } + let template = 'builtin/public-serve@0.0.3' + if (options.preview !== undefined) { + if (options.download !== undefined) + throw new TypeError('A preview cannot be an original download') + const preview = options.preview + const names = ['width', 'height', 'quality', 'format', 'resizeStrategy', 'background', 'preset'] + if ( + preview === null || + typeof preview !== 'object' || + Array.isArray(preview) || + Object.keys(preview).some((key) => !names.includes(key)) + ) { + throw new TypeError('Unknown public preview option') + } + boundedInteger(preview.width, 'width', 4096) + boundedInteger(preview.height, 'height', 4096) + boundedInteger(preview.quality, 'quality', 85) + if (preview.format !== undefined && !previewFormats.includes(preview.format)) + throw new TypeError('Unsupported preview format') + if ( + preview.resizeStrategy !== undefined && + !previewResizeStrategies.includes(preview.resizeStrategy) + ) + throw new TypeError('Unsupported preview resize strategy') + if (preview.preset !== undefined && !previewPresets.includes(preview.preset)) + throw new TypeError('Unsupported preview preset') + if ( + preview.background !== undefined && + !/^#[a-fA-F0-9]{6}([a-fA-F0-9]{2})?$/.test(preview.background) + ) + throw new TypeError('background must be #rrggbb or #rrggbbaa') + template = 'builtin/public-preview@0.0.4' + const fields = { + w: preview.width, + h: preview.height, + q: preview.quality, + f: preview.format, + r: preview.resizeStrategy, + bg: preview.background, + vs: preview.preset, + } + for (const [name, value] of Object.entries(fields)) { + if (value !== undefined) urlParams[name] = value + } + } else if (options.download) { + urlParams.download = '1' + } + return getSmartCdnUrl({ workspace: asset.workspace, input: asset.asset_id, template, urlParams }) +} diff --git a/packages/utils/test/storagePublicUrl.test.ts b/packages/utils/test/storagePublicUrl.test.ts new file mode 100644 index 00000000..6f259dc0 --- /dev/null +++ b/packages/utils/test/storagePublicUrl.test.ts @@ -0,0 +1,71 @@ +import { expect, test } from 'vitest' + +import { getPublicStoredAssetUrl, parseSmartCdnUrl } from '../src/index.ts' + +const asset = { workspace: 'my-app', asset_id: 'A'.repeat(22), version_id: `${'B'.repeat(21)}A` } + +test('builds stable original and download URLs without credentials or mutable filenames', () => { + expect(parseSmartCdnUrl(getPublicStoredAssetUrl(asset))).toMatchObject({ + workspace: 'my-app', + template: 'builtin/public-serve@0.0.3', + input: asset.asset_id, + urlParams: { v: asset.version_id }, + }) + const download = parseSmartCdnUrl(getPublicStoredAssetUrl(asset, { download: true })) + expect(download.urlParams).toEqual({ download: '1', v: asset.version_id }) + expect(download.auth).toBeUndefined() +}) + +test.each(['auto', 'frame', 'icon'] as const)('builds bounded %s preview URLs', (preset) => { + const url = getPublicStoredAssetUrl(asset, { + preview: { + width: 800, + height: 450, + quality: 75, + format: 'webp', + resizeStrategy: 'fit', + preset, + }, + }) + expect(parseSmartCdnUrl(url)).toMatchObject({ + template: 'builtin/public-preview@0.0.4', + input: asset.asset_id, + urlParams: { + w: '800', + h: '450', + q: '75', + f: 'webp', + r: 'fit', + vs: preset, + v: asset.version_id, + }, + }) +}) + +test.each([ + { width: 4097 }, + { height: 0 }, + { quality: 86 }, + { width: 1.5 }, + { background: 'red' }, +])('rejects unsupported preview values %j', (preview) => { + expect(() => getPublicStoredAssetUrl(asset, { preview })).toThrow() +}) + +test('rejects untyped attacker-controlled fields instead of generating unsigned work variants', () => { + const badInputs: unknown[] = [ + { preview: { preset: 'clip' } }, + { download: 'file.html' }, + { preview: { clip_duration: 30 } }, + { preview: {}, download: true }, + { preview: { resizeStrategy: 'min_fit' } }, + { urlParams: { v: 'latest' } }, + ] + for (const options of badInputs) { + // JavaScript callers do not have the TypeScript option union. + // @ts-expect-error Deliberately exercise the runtime boundary. + expect(() => getPublicStoredAssetUrl(asset, options)).toThrow() + } + expect(() => getPublicStoredAssetUrl({ ...asset, version_id: 'latest' })).toThrow() + expect(() => getPublicStoredAssetUrl({ ...asset, workspace: 'elsewhere/path' })).toThrow() +}) diff --git a/scripts/fixtures/img-next/seed.test.ts b/scripts/fixtures/img-next/seed.test.ts index 7155914e..1477f0b7 100644 --- a/scripts/fixtures/img-next/seed.test.ts +++ b/scripts/fixtures/img-next/seed.test.ts @@ -308,6 +308,11 @@ test('the packed CLI scaffolds an empty catalog and the actual constrained page output.push(typeof chunk === 'string' ? chunk : Buffer.from(chunk).toString()) return true }) + const errors: string[] = [] + t.mock.method(process.stderr, 'write', (chunk: string | Uint8Array) => { + errors.push(typeof chunk === 'string' ? chunk : Buffer.from(chunk).toString()) + return true + }) // Exercise the packed CLI and real receipt validation, including an older backend's // pre-Storage transformation. Only the remote Assembly response is simulated here. const stored = { @@ -346,7 +351,10 @@ test('the packed CLI scaffolds an empty catalog and the actual constrained page await cp(directory, join(originalCwd, 'app/cli-empty'), { recursive: true }) await writeFile('hero.jpg', bytes) await cli.main(['storage', 'store', './hero.jpg', 'website/hero.jpg']) - assert.equal(process.exitCode, undefined) + // The generic CLI requires original bytes, but the completed write is still recoverable. + // Rendering that actual receipt must not require a duplicate upload or accepting a false match. + assert.equal(process.exitCode, 1) + process.exitCode = undefined const catalog = JSON.parse(await readFile('transloadit.images.json', 'utf8')) assert.equal(catalog.images[stored.path].md5hash, stored.md5hash) assert.equal(catalog.images[stored.path].size, stored.size) @@ -375,9 +383,11 @@ test('the packed CLI scaffolds an empty catalog and the actual constrained page const page = await readFile('app/storage-image-example/page.tsx', 'utf8') assert.match(page, /width=\{960\}\s+preload/) assert( - printed.includes( - 'Render it with hero', - ), + errors + .join('') + .includes( + 'Receipt saved in transloadit.images.json. No further files were uploaded. Do not re-upload this object.', + ), ) assert(!printed.includes('export default function Page')) assert(page.includes("from '@transloadit/viewer/next'")) diff --git a/scripts/fixtures/upload-once-mock.ts b/scripts/fixtures/upload-once-mock.ts index 8b361911..d1bc5a3a 100644 --- a/scripts/fixtures/upload-once-mock.ts +++ b/scripts/fixtures/upload-once-mock.ts @@ -41,7 +41,7 @@ Transloadit.prototype.listPublicStoragePrefixes = async () => ({ ok: 'STORAGE_PUBLIC_PREFIXES_LISTED', public_prefixes: [], }) -Transloadit.prototype.storeImage = async (file: string, options: { path: string }) => { +Transloadit.prototype.storeAsset = async (file: string, options: { path: string }) => { if (process.env.TRANSLOADIT_INIT_FIXTURE_FAIL_UPLOAD === 'true') { // Exercise the real storage command's stderr, not only the demo's top-level catch. throw new Error('Invalid response at https://example.test/?signature=fixture-signed-secret')