Skip to content

fix(training): skip pretraining batches too small to form pairs - #530

Open
ChrisW09 wants to merge 2 commits into
mainfrom
fix/pretraining-single-row-batch
Open

ChrisW09 wants to merge 2 commits into
mainfrom
fix/pretraining-single-row-batch

Conversation

@ChrisW09

@ChrisW09 ChrisW09 commented Oct 21, 2026 •

Copy link
Copy Markdown
Collaborator

Fixes #529

Problem

The loaders keep a trailing partial batch, so pretraining sees tiny batches whenever the number of training or validation rows leaves a small remainder for batch_size. Such a batch can't form any pair:

  • one row, regression: get_knn() squeezed the (1, 1) distance matrix to shape (1,) and topk(...).indices[:, 1:] raised IndexError;
  • one or two rows, classification: no row has both a same-class and a different-class neighbor, so the "no usable sample in this batch" ValueError stopped pretraining.

Fix

ContrastivePretrainer._cannot_form_pairs(batch) is true for batches too small for any row to be an anchor: fewer than two rows for regression, fewer than three for classification. training_step, validation_step and test_step return None for such a batch, so Lightning skips it. Larger batches are handled exactly as before, including the documented ValueError when no row of a batch qualifies (e.g. three rows of three different classes).

#577 (PR #578) independently makes the training loader drop a one-row trailing training batch. This PR also covers one-row validation batches and two-row classification batches, which that change doesn't touch.

Tests

In tests/training/test_training_pretraining.py:

  • test_batch_too_small_to_form_pairs_is_skipped[regression-one-row|classification-one-row|classification-two-classes|classification-one-class]: the three steps return None.
  • test_three_row_batch_without_same_class_neighbors_still_raises: the documented error is kept for larger batches.
  • test_estimator_pretraining_completes_with_a_tiny_last_batch[regression-one-row-val-batch|classification-one-row-val-batch|classification-two-row-train-batch]: FTTransformer*.pretrain() with batch_size=16 and 17 validation rows or 34 training rows.

All new cases except the three-row guard fail on main; all pass with this change.

Docs

The pretraining note in docs/api/training/index.rst says that batches too small to form any pair are skipped rather than raising.

Checks

  • just test: 3244 passed, 46 skipped, 28 xfailed; coverage 91.30%
  • just docs: build succeeded (warnings treated as errors)
  • just check: all hooks passed (ruff, prettier, pyright)

A batch with one row has no neighbors to pair with: regression
pretraining crashed in get_knn() and classification raised the
"no usable sample" ValueError. Such a batch occurs whenever the number of
training or validation rows leaves a remainder of 1 for the batch size.
Skip it in the training, validation and test steps.

Fixes #529
A classification anchor needs both a same-class and a different-class
neighbor, so no row of a two-row batch can be one, and such a trailing
batch failed with the 'Use a larger batch size' error. The pretraining
note now says which batches are skipped.

Refs #529
@ChrisW09 ChrisW09 changed the title fix(training): skip single-row batches in contrastive pretraining fix(training): skip pretraining batches too small to form pairs Oct 21, 2026

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[BUG] Contrastive pretraining crashes on a batch too small to form pairs (e.g. a one-row last batch)

1 participant