Skip to content

Modernize the fides and NAG wrappers and add wrapper standards with a wrap-optimizer skill - #746

Draft
janosg wants to merge 7 commits into
mainfrom
worktree-modernize-nag-fides-wrappers
Draft

janosg wants to merge 7 commits into
mainfrom
worktree-modernize-nag-fides-wrappers

Conversation

@janosg

@janosg janosg commented Nov 18, 2026

Copy link
Copy Markdown
Member

What problem do you want to solve?

Many optimizer wrappers were added long ago and have not been touched since. Their options are only tested on trivial problems, some names are verbose or hide the structure of the wrapped library, some defaults are unexplained, and several options silently do nothing.

This PR sets a standard for wrappers and applies it to the first three:

  1. Standards and tooling. The guide on adding optimizers now contains the rules for names, option groups, defaults, deprecations, tests and tutorials. New tooling supports them: the mark.deprecated_options decorator, validated option groups, and wrapper tests that are selected with library-level markers.
  2. fides is updated to fides 0.8.2 and follows the standard. A new tutorial shows its advanced options.
  3. nag_dfols and nag_pybobyqa are updated to DFO-LS 1.6.5 and Py-BOBYQA 1.5.0 and follow the standard.
  4. A wrap-optimizer skill for coding agents runs the same workflow for the next wrappers. It stops for a human decision on the option table.

Follow-ups are tracked in #744 (least-squares version of fides) and #745 (scipy option aliases).

Changes for users

All old option names keep working until 0.7.0 and emit a FutureWarning that shows the new spelling.

fides

  • Requires fides>=0.8.2.
  • Options are renamed (table below). stepback_strategy="refine" is new.
  • For problems with bounds, the default of stepback_strategy will change from "truncate" to "reflect" (the fides default) in 0.7.0. Until then a warning is emitted if the option is not set.
  • Hessian approximations that need exact Hessians (HybridFixed, HybridFraction) are now rejected with a clear error instead of failing inside fides.
  • The result now also contains status (the exit flag of fides), n_fun_evals and n_jac_evals.

nag_dfols and nag_pybobyqa

  • Require DFO-LS>=1.6.5 and Py-BOBYQA>=1.5.0.
  • Options are bundled in groups that mirror the solvers, e.g. trustregion_options=NagTrustregionOptions(shrinking_factor=0.9). A group is passed as an instance of its class; dictionaries are rejected.
  • Restarts and noise each are one option that takes a boolean or a group: restarts=True or restarts=DfolsRestartOptions(max_unsuccessful_restarts=3), and noisy=True or noisy=NagNoiseOptions(additive_noise_level=0.1). Passing the group switches the feature on. With the default restarts=None the solvers restart if the function is noisy.
  • All solver options that optimagic can serve are available.
  • Bug fixes:
    • n_evals_per_point (formerly noise_n_evals_per_point) could not be used.
    • n_fun_evals counted points instead of evaluations for DFO-LS and was missing for Py-BOBYQA.
    • stopping_maxfun passed via algo_options was ignored by nag_pybobyqa.
    • The singular-value floor of the growing options was never passed, and the automatic choice of the growing method never switched.
  • n_iterations is now the number of iterations of all runs. The old wrappers reported one less, because they read a zero-based index. The wrappers switch off the most expensive part of the solvers' diagnostics (poisedness), which is not needed for counting iterations.
  • The result also contains jac: the solvers' estimate of the Jacobian of the residuals (DFO-LS) or of the gradient (Py-BOBYQA) at the solution. algorithm_output contains n_runs and n_evaluated_points.
  • Options the solvers reject now raise an InvalidAlgoOptionError instead of returning NaN parameters.

Option tables

fides

fides optimagic Default Replaces
hessian_update hessian_update "bfgs" hessian_update_strategy
maxiter stopping_maxiter 1_000_000 (harmonized)
maxtime stopping_maxtime inf stopping_max_seconds
fatol convergence_ftol_abs 0 (harmonized)
frtol convergence_ftol_rel 2e-9 (harmonized)
xtol convergence_xtol_abs 0
gatol convergence_gtol_abs 1e-5 (harmonized)
grtol convergence_gtol_rel 1e-8 (harmonized)
subspace_solver subspace_solver "full" (fides: "2D") trustregion_subspace_dimension
stepback_strategy stepback_strategy "truncate", "reflect" from 0.7.0 trustregion_stepback_strategy
theta_max max_stepback_fraction 0.95 trustregion_max_stepback_fraction
delta_init trustregion_initial_radius 1.0
mu trustregion_shrinking_threshold 0.25 trustregion_decrease_threshold
eta trustregion_expansion_threshold 0.75 trustregion_increase_threshold
gamma1 trustregion_shrinking_factor 0.25 trustregion_decrease_factor
gamma2 trustregion_expansion_factor 2.0 trustregion_increase_factor

Not exposed: history_file, verbose, funargs (handled by optimagic); resfun (#744); hess0, x_names, start_id.

nag_dfols (D) and nag_pybobyqa (P)

None means that the option is not passed and the solver uses its own default, which depends on the problem.

Solver option optimagic Default Solver Replaces (plain renames)
maxfun stopping_maxfun 1_000_000 (harmonized) D+P stopping_max_criterion_evaluations
rhoend convergence_min_trustregion_radius 1e-8 D+P convergence_minimal_trustregion_radius_tolerance
rhobeg trustregion_initial_radius None D+P
npt n_interpolation_points None D+P trustregion_n_interpolation_points
objfun_has_noise noisy (True or an instance of NagNoiseOptions) False D+P
nsamples n_evals_per_point None D+P noise_n_evals_per_point
model.abs_tol convergence_target_value D: 1e-12 / P: None D+P convergence_criterion_value
model.rel_tol convergence_ftol_scaled 0.0 D
seek_global_minimum seek_global_optimum False P
general.rounding_error_constant rounding_error_constant 0.1 D+P interpolation_rounding_error
general.safety_step_thresh safety_step_threshold 0.5 D+P threshold_for_safety_step
general.check_objfun_for_overflow check_fun_for_overflow True D clip_criterion_if_overflowing
init.random_initial_directions init_options.random_initial_directions False D+P
init.random_directions_make_orthogonal init_options.random_directions_make_orthogonal True D+P random_directions_orthogonal
tr_radius.eta1 trustregion_options.shrinking_threshold 0.1 D+P trustregion_threshold_successful
tr_radius.eta2 trustregion_options.expansion_threshold 0.7 D+P trustregion_threshold_very_successful
tr_radius.gamma_dec trustregion_options.shrinking_factor None D+P trustregion_shrinking_factor_not_successful
tr_radius.gamma_inc trustregion_options.expansion_factor 2.0 D+P trustregion_expansion_factor_successful
tr_radius.gamma_inc_overline trustregion_options.step_expansion_factor 4.0 D+P trustregion_expansion_factor_very_successful
tr_radius.alpha1 trustregion_options.min_radius_shrinking_factor None D+P trustregion_shrinking_factor_lower_radius
tr_radius.alpha2 trustregion_options.radius_factor_after_min_radius_shrinking None D+P trustregion_shrinking_factor_upper_radius
slow.history_for_slow slow_progress_options.history_for_slow 5 D+P
slow.thresh_for_slow slow_progress_options.threshold_for_slow 1e-8 (DFO-LS: 1e-4) D+P
slow.max_slow_iters slow_progress_options.max_slow_iterations None D+P
noise.quit_on_noise_level noisy.quit_on_noise_level True D+P
noise.scale_factor_for_quit noisy.scale_factor_for_quit 1.0 D+P
noise.multiplicative_noise_level noisy.multiplicative_noise_level None D+P
noise.additive_noise_level noisy.additive_noise_level None D+P
interpolation.precondition interpolation_options.precondition True D+P trustregion_precondition_interpolation
interpolation.throw_error_on_nans interpolation_options.throw_error_on_nans False D
interpolation.minimum_change_hessian interpolation_options.minimum_change_hessian True P trustregion_minimum_change_hession_for_underdetermined_interpolation
regression.num_extra_steps regression_options.n_extra_steps 0 D trustregion_n_extra_points_to_replace_successful
regression.increase_num_extra_steps_with_restart regression_options.increase_n_extra_steps_with_restart 0 D
regression.momentum_extra_steps regression_options.momentum_extra_steps False D
restarts.use_restarts restarts (True, False or an instance of the restart options) None D+P
restarts.max_unsuccessful_restarts restarts.max_unsuccessful_restarts 10 D+P
restarts.max_unsuccessful_restarts_total restarts.max_unsuccessful_restarts_total None P
restarts.rhobeg_scale_after_unsuccessful_restart restarts.initial_radius_scale_after_unsuccessful_restart None P
restarts.rhoend_scale restarts.convergence_min_radius_scale 1.0 D+P
restarts.use_soft_restarts restarts.use_soft_restarts True D+P
restarts.soft.num_geom_steps restarts.soft_n_geometry_steps 3 D+P
restarts.soft.move_xk restarts.soft_move_iterate True D+P
restarts.soft.max_fake_successful_steps restarts.soft_max_fake_successful_steps None D+P
restarts.hard.use_old_rk / use_old_fk restarts.hard_use_old_fun_value True D+P
restarts.auto_detect restarts.auto_detect True D+P
restarts.auto_detect.history restarts.auto_detect_history 30 D+P
restarts.auto_detect.min_chgJ_slope / min_chg_model_slope restarts.auto_detect_min_model_change_slope 0.015 D+P
restarts.auto_detect.min_correl restarts.auto_detect_min_correlation 0.1 D+P
restarts.increase_npt restarts.increase_n_interpolation_points False D
restarts.increase_npt_amt restarts.increase_n_interpolation_points_amount 1 D
restarts.hard.increase_ndirs_initial_amt restarts.hard_increase_n_initial_directions_amount 1 D
restarts.max_npt restarts.max_n_interpolation_points None D
restarts.throw_error_on_nans restarts.throw_error_on_nans False D
growing.ndirs_initial growing_options.n_initial_directions None D
growing.num_new_dirns_each_iter growing_options.n_new_directions_each_iteration 0 D
growing.delta_scale_new_dirns growing_options.radius_scale_new_directions None D
growing.do_geom_steps growing_options.do_geometry_steps False D
growing.reset_delta growing_options.reset_radius False D
growing.reset_rho growing_options.reset_min_radius False D
growing.gamma_dec growing_options.shrinking_factor None D
growing.safety.do_safety_step growing_options.do_safety_step True D
growing.safety.reduce_delta growing_options.safety_reduce_radius False D
growing.safety.full_geom_step growing_options.safety_full_geometry_step False D
growing.full_rank.use_full_rank_interp growing_options.use_full_rank_interpolation None D
growing.perturb_trust_region_step growing_options.perturb_trust_region_step None D
growing.full_rank.scale_factor growing_options.full_rank_scale_factor 0.01 D
growing.full_rank.svd_scale_factor growing_options.full_rank_svd_scale_factor 1.0 D
growing.full_rank.min_sing_val growing_options.full_rank_min_singular_value 1e-6 D
growing.full_rank.svd_max_jac_cond growing_options.full_rank_max_jacobian_condition_number 1e8 D

Deprecated options that are converted instead of renamed: the dictionaries trustregion_reset_options, trustregion_fast_start_options (D) and convergence_slow_progress, and initial_directions, noise_additive_level, noise_multiplicative_level, convergence_noise_corrected_criterion_tolerance, trustregion_method_to_replace_extra_points (D), clip_criterion_if_overflowing (P, no replacement).

Not exposed:

  • Logging and printing options and extra arguments of the function: handled by optimagic.
  • scaling_within_bounds: optimagic has its own scaling.
  • init.run_in_parallel: the solvers still evaluate the function one point at a time.
  • general.check_objfun_for_overflow (P): Py-BOBYQA only catches an OverflowError of the function, which optimagic handles before the solver sees it.
  • Regularizers (h, lh, prox_uh, func_tol.*, sfista.*), convex constraints given as projections (projections, dykstra.*, matrix_rank.*) and the evaluation database: not supported yet.

Evidence for defaults

All numbers are from the 54 Moré-Wild problems.

fides convergence criteria (BFGS). "Stopped early" means not within 1e-6 of the best value any setting reached.

Tolerances Ended by gradient Ended by ftol Other Stopped early Evaluations
optimagic defaults 35 18 1 0 5709
ftol switched off 50 0 4 0 7480
fides defaults 2 52 0 2 3814

The harmonized optimagic defaults stay.

fides subspace_solver. "full" and "2D" need about the same number of evaluations (median 16.5 vs. 16.0). On Rosenbrock and random quadratic problems with 100 to 200 parameters, an iteration with "full" takes less than half the time of one with "2D". optimagic therefore keeps "full".

fides stepback_strategy. With synthetic bounds, "truncate" and "reflect" solve the same problems with median 15.0 vs. 14.5 evaluations. There is no evidence for deviating from fides, so the default moves to "reflect".

DFO-LS slow-progress threshold. With optimagic's 1e-8 and with the DFO-LS default of 1e-4, 4 of 54 runs end by slow progress. With 1e-8 they stop at a relative error of 1e-10 to 1e-9, with 1e-4 at 5e-8 to 7e-6. The stricter value stays. For both NAG solvers the trust-region radius ends most runs.

Validation and remaining gaps

  • 679 wrapper tests (pixi run tests -m wrappers, about 10 seconds): contract tests against the installed libraries, behavior tests through om.minimize, forwarding tests and deprecation tests for every old name.
  • The default test suite, ty, the pre-commit hooks and the docs build pass locally.
  • The wrapper tests do not run in the CI of pull requests. The new wrapper-tests workflow runs them monthly against the newest releases of the wrapped libraries and can be started by hand; it can only run after the merge. The patch coverage reported by codecov is therefore lower than the real coverage of the wrappers.
  • The classes of the option groups are imported from optimagic.optimizers.nag_optimizers. There is no shorter public import path yet.
  • Two old NAG options only had an effect together with a noise level: the noise tolerance, and trustregion_reset_options without use_resets. They are translated correctly if everything is passed in one call. If they are set with with_option on an algorithm that is already noisy, they have no effect and the warning says so.
  • Two DFO-LS options (restarts.throw_error_on_nans, growing_options.use_full_rank_interpolation) only have forwarding tests.
  • In the DFO-LS benchmark, two runs that converge by the radius criterion end above the best known value. I did not check whether these are local optima.

AI assistance

The code, tests, documentation, benchmarks and this description were written with Claude Code. The design decisions (naming and grouping rules, deprecation policy, defaults, test strategy) were made by me in an interview at the start and at a checkpoint on the option tables. The results were checked with the tests and benchmarks described above. I have not yet reviewed the diff line by line; that is the remaining work before this leaves draft status.

Checklist

  • I have read the contributor guide.
  • I have described any additional validation I performed (e.g., manual checks, numerical comparisons, or benchmarks) and any remaining validation gaps.
  • I have added or updated tests for changed behavior.
  • I have updated documentation for user-facing changes.
  • I have linked any issues this PR resolves using Closes #XXXX.
  • I have described any substantial AI assistance and how I checked the result.
  • I have selected the appropriate base branch and described any remaining work.

🤖 Generated with Claude Code

janosg and others added 6 commits November 18, 2026 11:14
- mark.minimizer takes deprecated_options: old option names keep working
  in the constructor, with_option and algo_options and emit a
  FutureWarning with the new spelling.
- option_group validates dataclasses that bundle related options.
- resolve_changed_default announces a default that is about to change.
- Tests with a *_wrappers marker only run if selected with -m; a new
  workflow runs them on pull requests that touch wrappers and monthly
  against the newest releases of the wrapped libraries.
- Raise the minimal versions of fides, DFO-LS and Py-BOBYQA.

Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
- Update to fides 0.8.2 and add stepback_strategy="refine".
- Rename options to stay closer to fides and to the new trust-region
  convention; the old names are deprecated until 0.7.0.
- Announce the change of the stepback_strategy default to "reflect".
- Reject Hessian approximations that need exact Hessians or residuals
  based on their properties instead of a list of classes.
- Document every option by its effect.
- Add wrapper tests: contract with fides, behavior, forwarding and
  deprecations.
- Add a tutorial on the advanced options of fides.

Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
- Update to DFO-LS 1.6.5 and Py-BOBYQA 1.5.0.
- Bundle options in groups that mirror the solvers and expose all solver
  options optimagic can serve; the old names and dictionaries are
  deprecated until 0.7.0.
- Fix n_evals_per_point, n_fun_evals, stopping_maxfun of nag_pybobyqa and
  options that were never passed to the solvers.
- Stop switching on the diagnostics of the solvers, which slowed them
  down; n_iterations is no longer reported.
- Raise InvalidAlgoOptionError for options the solvers reject.
- Document every option by its effect.
- Add wrapper tests: contract with the solvers, behavior, forwarding and
  deprecations.

Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
- Rewrite the guide on adding optimizers as a markdown page with the
  standards for wrappers: option table, coverage, grouping, names,
  defaults, deprecations, tests and tutorials.
- Rewrite the guide on documenting optimizers around an effect-first
  template for option docstrings.
- Add the user-invoked wrap-optimizer skill and a pointer in AGENTS.md.
- Update the algorithm documentation and the changelog.

Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
- Move the handling of deprecated options from an argument of
  mark.minimizer to the separate decorator mark.deprecated_options.
- Option groups are only accepted as instances of their class.
  Dictionaries are rejected in the constructor, with_option and
  algo_options.
- nag_dfols and nag_pybobyqa report n_iterations again. The count comes
  from the diagnostic information of the solvers, with the expensive
  poisedness computation switched off.
- Add the rule that a wrapper keeps reporting what it reported before to
  the guide on adding optimizers and to the wrap-optimizer skill.

Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
- nag_dfols and nag_pybobyqa: `restarts` and `noisy` each take a boolean
  or an option group instead of a flag plus a separate group.
- Deprecated options can move into an option of the form
  `bool | <group>`, and converters see the other options of the call.
- fides reports status, n_fun_evals and n_jac_evals; the NAG wrappers
  report the solvers' estimate of the derivative as jac.
- Guide and skill: add the rule for features that can be switched on
  and off, and the rule that a wrapper fills as many fields of
  InternalOptimizeResult as it can.
- The wrapper-tests workflow only runs monthly and by hand.

Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
@review-notebook-app

Copy link
Copy Markdown

Review Notebooks

Notebooks changed:

Review visual diffs and comment on any notebook cell. All comments sync back to this PR.


Powered by ReviewNB

This is the case that #736 fixed for nag_pybobyqa.

Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
@codecov

codecov Bot commented Nov 18, 2026

Copy link
Copy Markdown

Codecov Report

❌ Patch coverage is 52.66352% with 702 lines in your changes missing coverage. Please review.

Files with missing lines Patch % Lines
tests/optimagic/optimizers/test_nag_wrappers.py 29.34% 484 Missing ⚠️
tests/optimagic/optimizers/test_fides_wrappers.py 31.27% 211 Missing ⚠️
src/optimagic/optimization/algo_option_tools.py 97.19% 3 Missing ⚠️
src/optimagic/optimizers/fides.py 96.38% 3 Missing ⚠️
...s/optimagic/optimization/test_algo_option_tools.py 99.47% 1 Missing ⚠️

❌ Your patch check has failed because the patch coverage (52.66%) is below the target coverage (80.00%). You can increase the patch coverage or adjust the target coverage.

Files with missing lines Coverage Δ
src/optimagic/mark.py 100.00% <100.00%> (ø)
src/optimagic/optimization/algorithm.py 94.85% <100.00%> (+0.19%) ⬆️
src/optimagic/optimizers/nag_optimizers.py 98.05% <ø> (+3.79%) ⬆️
tests/conftest.py 100.00% <100.00%> (ø)
tests/optimagic/optimizers/test_fides_options.py 92.50% <100.00%> (+1.59%) ⬆️
tests/optimagic/optimizers/test_nag_optimizers.py 100.00% <100.00%> (ø)
...s/optimagic/optimization/test_algo_option_tools.py 99.47% <99.47%> (ø)
src/optimagic/optimization/algo_option_tools.py 97.19% <97.19%> (ø)
src/optimagic/optimizers/fides.py 96.33% <96.38%> (-0.45%) ⬇️
tests/optimagic/optimizers/test_fides_wrappers.py 31.27% <31.27%> (ø)
... and 1 more

... and 5 files with indirect coverage changes

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant