Skip to content

Scala 3 migration: Lift Web to http4s, Lift Mapper to Doobie, Flyway to Liquibase - #2899

Open
hongwei1 wants to merge 333 commits into
OpenBankProject:developfrom
hongwei1:build/scala-3-migration
Open

hongwei1 wants to merge 333 commits into
OpenBankProject:developfrom
hongwei1:build/scala-3-migration

Conversation

@hongwei1

Copy link
Copy Markdown
Contributor

Migrates OBP-API to Scala 3, replacing the two frameworks that blocked it. 287 commits on top of
3df73fe11; 1027 files, +66009/-33154.

Opened for review of the whole line. It does not merge cleanly onto develop yet — the branch
is 26 commits behind and conflicts in 12 files, all of them where develop extended a Lift Mapper
entity this branch had already moved to Doobie. That merge is in progress separately and will be
pushed here; the conflicts are listed at the end so reviewers know what is coming rather than
discovering it from a red merge box.

What changed

Scala 3. Lift Mapper cannot compile under Scala 3 — the compiler crashes on the
object X extends class X shape every Mapper entity uses (reduced to a 5-line case). Both of its
consumers had to go first.

Lift Web → http4s, complete. net.liftweb.http no longer appears in any .scala source. There
is no Lift fallback in the request chain: an unmatched /obp/* path returns a JSON 404 from
notFoundCatchAll. API versions are unchanged by this — a framework migration happens in place
inside the existing version file, and a version bump still means a changed API signature.

Lift Mapper → Doobie, complete. ToSchemify.models is now Nil; Schemifier creates nothing.

Flyway → Liquibase. Flyway needed one hand-written script set per vendor: 118 for H2, 118 for
Postgres, and nothing for the three other drivers its vendorFolder would have booted against
silently, with no tables. One changelog now describes each change once and Liquibase emits the
dialect. The baseline is generated from a Postgres database the Flyway scripts built, not
hand-written, and is regenerated with scripts/GenerateChangelog.java plus a normaliser rather
than edited.

Defects found and fixed along the way

Migrating the data layer surfaced behaviour that lived in Mapper's field types rather than in the
entities, and which a column-by-column port drops silently:

  • a NULL MappedBoolean reads back as false whatever defaultValue declares, a NULL
    MappedLong as the declared default — read as a hardcoded -1, six call-limit columns turned
    "the configured limit" into "no limit", and that value is what the rate limiter enforces from
  • MappedEmail lowercases and trims on every set and validates on save; authuser lost all of it
  • two SQL injections in MappedMetrics, reachable from read-only roles, giving a boolean-blind
    oracle over the whole database

Security fixes on top: a locked account could still authenticate through OIDC and Keycloak (both
read v_oidc_users over JDBC and never call verify-credentials); consents.sca.enabled=false
accepted any SCA answer in production; the dynamic-code dependency validation inspected an empty
list because its scan was gated on an unrelated diagnostic prop; and compileScalaCode now refuses
to run when no SecurityManager can be installed (JEP 486) unless the operator says so explicitly.

Testing

3876 scenarios, 0 failures, on both H2 and Postgres. Postgres is not optional here: H2 tolerates
things Postgres does not, and the schema is generated from the changelog at boot on both.

Conflicts with current develop

develop added createdByUserId / updatedByUserId / a method-body hash to DynamicResourceDoc,
DynamicMessageDoc and ConnectorMethod, and one new Mapper entity (ChatEmailDigestState). This
branch had already moved the first three to Doobie, so the resolution ports the new fields into the
Doobie stores and the changelog rather than restoring the Mapper entities; ChatEmailDigestState
needs the same treatment, since with models = Nil its table would otherwise never be created.

The previous commit rewired the provider to Doobie and removed the
Mapper entity from Boot.scala's ToSchemify list, but missed deleting
the entity class itself - it lived in its own file, separate from the
provider file that got deleted. No remaining references; full suite
still green.
Replace the Lift Mapper tax-residence entity with a Doobie-backed
provider (forty-fifth table off Lift Mapper). mCustomerId is a
MappedLongForeignKey pointing at mappedcustomer.id (the customer's
internal BIGINT primary key, not their UUID customerId) - the Doobie
provider resolves customerId back to the UUID via a MappedCustomer
lookup by id, falling back to the raw long id as a string if the
customer row is missing, matching the Mapper entity's own getter
exactly.

Two indexes carried over: a plain index on mcustomerid (from the
foreign-key field) and a UNIQUE INDEX on
(mcustomerid, mdomain, mtaxnumber), confirmed against a booted
instance's information_schema.indexes - caught a first attempt at the
migration script that used ALTER TABLE ... ADD CONSTRAINT ... UNIQUE,
which H2 backs with an auto-suffixed index name rather than the literal
constraint name; switched to CREATE UNIQUE INDEX to match Schemifier's
actual output, per MigratedTablesExistTest.

deletion.DeleteCustomerCascade's cascade delete moves to raw SQL
against the new table.
Replace the Lift Mapper customer-link entity with a Doobie-backed
provider (forty-sixth table off Lift Mapper). Unique index on
customerlinkid; plain indexes on customerid and othercustomerid,
confirmed against a booted instance's information_schema.indexes.

No test exercised this table's provider or the connector methods
wired to it (bank-to-bank customer relationships, e.g. spouse/parent
at another bank), so a characterization test
(CustomerLinkProviderTest) covering full CRUD plus bulkDelete was
added and confirmed green against the pristine Mapper entity before
the migration, then again against the Doobie provider.
Replace the Lift Mapper counterparty-limit entity with a Doobie-backed
provider (forty-seventh table off Lift Mapper). Two unique indexes:
one on counterpartylimitid, one on the composite
(bankid, accountid, viewid, counterpartyid) - at most one limit per
tuple - confirmed against a booted instance's information_schema.indexes.
Amount fields are NUMERIC(16,10); count fields default to -1 and
amount fields to 0 at the application layer on create, matching the
Mapper fields' own defaultValue overrides.

toJValue moves onto the new CounterpartyLimitRow case class verbatim,
since CounterpartyLimitTrait extends JsonAble.

MigrationOfCounterpartyLimitFieldType, a historical migration, switches
to the tableExistsByName overload used by the other historical
migrations in this series.
Replace the Lift Mapper customer-account-link entity with a
Doobie-backed provider (forty-eighth table off Lift Mapper). Two
unique indexes: one on customeraccountlinkid, one on the composite
(accountid, customerid) - a customer has at most one link per account
- confirmed against a booted instance's information_schema.indexes.

createAgentAccountLink builds its own AgentAccountLinkTraitCommons
from the CustomerAccountLinkTrait result fields (documented in the
entity as "customer and agent share the same model"), so the row type
only needs to implement CustomerAccountLinkTrait.

The endpoint test covers create/read/update/delete, but not
getOrCreateCustomerAccountLink, the unfiltered getCustomerAccountLinks,
or bulkDeleteCustomerAccountLinks, so a characterization test
(CustomerAccountLinkProviderTest) covering those three was added and
confirmed green against the pristine Mapper entity before the
migration, then again against the Doobie provider.

Two cascade-delete call sites move to the new provider:
deletion.DeleteBankCascade's account-scoped lookup (filters by
accountId only, no bankId) and deletion.DeleteCustomerCascade's
cascade delete. A third direct reference in
LocalMappedConnector.getBankAccountsForUser also moves over.
Replace the Lift Mapper user-customer-link entity with a Doobie-backed
provider (forty-ninth table off Lift Mapper). Two unique indexes: one
on musercustomerlinkid, one on the composite (muserid, mcustomerid) -
a user has at most one link per customer - confirmed against a booted
instance's information_schema.indexes. mdateinserted is a separate
column from the createdat/updatedat pair the CreatedUpdated mixin
also adds; the trait's dateInserted getter reads the former.

getOCreateUserCustomerLink preserves the Mapper version's
find-then-insert-with-retry-on-conflict shape exactly, including the
scala.util.Try wrapping around the insert: ConcurrentDuplicateCreationTest
scenario L races 8 concurrent calls for the same (userId, customerId)
and asserts no exception and exactly one row, relying on the unique
index to reject the losing insert so it can be caught and retried as a
re-fetch. Confirmed green against the pristine Mapper entity first,
then again after the migration.

Two direct callers by name (not through the DI seam) move to the new
object: deletion.DeleteCustomerCascade's cascade delete and
MappedCustomerProvider.getCustomersByUserId.
Replace the Lift Mapper CRM-event entity with a Doobie-backed provider
(fiftieth table off Lift Mapper). Three indexes: UNIQUE on
mcrmeventid, plain on mbankid, plus a plain index on muserid that the
entity's own dbIndexes never declared - Lift auto-indexes
MappedLongForeignKey columns, confirmed against a booted instance's
information_schema.indexes. mUserId stores ResourceUser's internal
BIGINT primary key; the row's user accessor resolves it back to a
live ResourceUser the same way the Mapper entity's getter did,
including throwing if the id doesn't resolve.

The sandbox importer (LocalMappedConnectorDataImport) built a
not-yet-saved MappedCrmEvent via Mapper's create/.validate/Saveable
pattern; it now builds a CrmEventCreateParams (a transient
CrmEventTrait implementation whose user accessor throws, matching
what the Mapper version would have thrown for the same reason - the
importer never sets mUserId) and saves it through
DoobieCrmEventProvider.createEvent via a new SaveableCrmEvent,
mirroring the SaveableAtm pattern already used for the Atm table.

MappedCrmEventProviderTest, which called the Mapper entity directly,
was rewritten to go through DoobieCrmEventProvider's create/get
methods; confirmed green against the pristine Mapper entity before
the migration under its original form. SandboxDataLoadingTest's "should
create CRM Events ok" scenario exercises the sandbox-import path
end-to-end.
Table 51/140 in the Lift Mapper to Doobie strangler migration.

Replaces MappedUserRefreshes with DoobieUserRefreshesProvider, backed
by a Flyway migration matching the probed schema (muserid varchar(44),
createdat/updatedat timestamps, unique index on muserid). This table
drives the refresh_user.interval login flow that periodically forces
a fresh credential check.

AuthUserTest exercises this table's row-count assertions extensively
across its login scenarios; rewrote its direct MappedUserRefreshes
calls (findAll().length, bulkDelete_!!()) to the new provider's
count()/bulkDelete() helpers, added specifically to support those
assertions. All 5 AuthUserTest scenarios pass unchanged against the
new backing store, along with the full suite (3638 tests, 0
failures).
Table 52/140 in the Lift Mapper to Doobie strangler migration.

MappedThing was never wired into Boot.scala's ToSchemify list, so its
table was never created by Schemifier, and Thing.thingProvider had no
caller anywhere in the codebase - a leftover example/tutorial entity
with zero live callers and zero test coverage. There is no data to
migrate and no table to create a Flyway script for, so the correct
move is deleting the scaffold rather than building a Doobie provider
nothing would ever invoke.

Removes the code.examplething package entirely and its exemption
entry in MappedClassNameTest. Full suite passes unchanged (3638
tests, 0 failures).
Table 53/140 in the Lift Mapper to Doobie strangler migration.

Replaces MappedPayeeLookupProvider with DoobiePayeeLookupProvider,
backed by a Flyway migration matching the probed schema (unique index
on lookupid, plain index on expiresat). This is the short-lived
lookup cache behind the mobile-wallet payee-lookup flow in
Http4s700 - createPayeeLookup writes a row with a ttl, later reads
filter out expired rows in application code without deleting them,
preserved exactly as before.

Covered by the existing Http4s700RoutesTest suite; no new
characterization test needed. Full suite passes unchanged (3638
tests, 0 failures).
Table 54/140 in the Lift Mapper to Doobie strangler migration.

MetricsArchiveRun's concrete Mapper type leaked into two cross-file
signatures - RunCompleted(run: MetricsArchiveRun) in
MetricsArchiveScheduler and metricsArchiveRunToJson(r:
MetricsArchiveRun) in JSONFactory7.0.0 - so this introduces
MetricsArchiveRunTrait with plain getters and widens both call sites
to it, backed by MetricsArchiveRunRow. The plain-object API
(recordRun, lastRun, lastSuccessfulRun, pruneToMostRecent,
maxRowsToKeep, count, bulkDelete_!!) keeps its original names so only
field-level access (PascalCase MappedField.get -> lowercase trait
getter) needed to change at call sites, in both production code and
MetricsArchiveSchedulerTest.

Flyway migration matches the probed schema: unique index on runid,
plain index on startedat. Full suite passes unchanged (3638 tests, 0
failures).
Table 55/140 in the Lift Mapper to Doobie strangler migration.

Introduces OpenCorridorFeeAccrualTrait so the plain-object API
(accrue, unswept, find, markSwept) can keep returning domain values
instead of the Mapper entity. The one behavioural translation is the
sweep's per-row mutation - Lift called
`accrual.FeeSettlementId(id).saveMe()` on each row in place; the
Doobie replacement is `OpenCorridorFeeAccrual.markSwept(trId, id)`,
an explicit UPDATE keyed on the unique transaction_request_id, called
once per accrual from OpenCorridorFees.sweep exactly as before.

Flyway migration matches the probed schema: unique index on
transaction_request_id (idempotent accrual per covered promise),
plain indexes on debtor_bank_id and fee_settlement_id. Covered by the
existing Http4s700RoutesTest fee-sweep scenario; no new
characterization test needed. Full suite passes unchanged (3638
tests, 0 failures).
Table 56/140 in the Lift Mapper to Doobie strangler migration.

Replaces MappedUtilityPaymentCallbackProvider with
DoobieUtilityPaymentCallbackProvider behind the existing
UtilityPaymentCallbackProvider trait, so UtilityCallbackDispatcher and
the injector wiring in UtilityPaymentCallbacks are unaffected. This is
the one-shot callback registry for UTILITY transaction-requests that
supply a callback_url - distinct from the standing account-event
webhook system.

Flyway migration matches the probed schema: unique index on
callbackid, plain index on transactionrequestid. Covered by the
existing Http4s700RoutesTest scenarios for createUtilityVendResult
and createTransactionRequestUtility.

Full suite passes (3638 tests, 0 failures) on a clean, uncontaminated
run; an earlier sharded run's single failure in
ConcurrentBackoffCounterSelfHealTest (an unrelated in-memory timing
test with no dependency on this table) turned out to be a flake
introduced by running an isolated retest concurrently against the
same target/ directory - confirmed by both an isolated rerun and this
clean full-suite rerun passing.
Table 57/140 in the Lift Mapper to Doobie strangler migration.

MappedWebUiPropsProvider is referenced directly by name (no DI
injector indirection) from roughly fifteen call sites across
AuthUser, I18NUtil, Glossary, APIUtil, and several APIMethods/Http4s
version files, all going through the provider-interface methods
(getAll/getByName/createOrUpdate/delete/getWebUiPropsValue) rather
than raw Mapper fields - so the object keeps its name and only its
backing implementation moves to Doobie, avoiding a large-radius
rename.

createOrUpdate preserves the original's find-by-name-then-upsert
shape, including the quirk that a newly created row's webUiPropsId is
always a freshly generated UUID - the caller's own webUiPropsId
field, if any, is ignored on create, matching Lift's MappedUUID
default-value behaviour where .create() never had that field set
explicitly.

Flyway migration matches the probed schema: unique indexes on both
webuipropsid and name. Covered by three dedicated WebUiPropsTest
suites (v3.1.0/v5.1.0/v6.0.0, 37 scenarios). Full suite passes (3638
tests, 0 failures).
Table 58/140 in the Lift Mapper to Doobie strangler migration.

MappedUserScope had zero callers anywhere in the codebase - no
endpoint, no test, nothing reaching UserScope.userScope.vend. Its
sibling table Scope/MappedScope (plural, backing the actual v4.0.0
scopes endpoints) is unrelated and stays untouched. There is no data
to migrate for a table nothing ever writes to or reads from, so the
scaffold is deleted rather than given a Doobie provider nothing would
invoke.

Removes the code.scope.UserScope and
code.scope.MappedUserScopeProvider files entirely, their Boot.scala
ToSchemify entry, and the MappedClassNameTest exemption. Full suite
passes unchanged (3638 tests, 0 failures).
Table 59/140 in the Lift Mapper to Doobie strangler migration.

Backs the v6.0.0 management/groups CRUD endpoints (a named bundle of
roles that can be granted to a user's entitlement in one shot, scoped
to a bank or system-wide). Had zero existing test coverage - no
endpoint test ever exercised group creation, retrieval, update, or
deletion - so this adds GroupTest.scala (13 scenarios covering
create/get/list/update/delete, role-gating for bank-scoped vs
system-level groups, and 404s) as a characterization suite, confirmed
green against the pristine Mapper implementation before migrating,
then confirmed green again against DoobieGroupProvider.

Flyway migration matches the probed schema: two plain (non-unique)
indexes on groupid and bankid, matching the entity's own
Index(...)/Index(...) declaration. Full suite passes (3651 tests, 0
failures).
Table 60/140 in the Lift Mapper to Doobie strangler migration.

Replaces MappedOrganisationProvider with DoobieOrganisationProvider
behind the existing OrganisationProvider trait, so the injector wiring
in Organisations and every call site through
Organisations.organisation.vend is unaffected.

updateOrganisation preserves the original's partial-update semantics:
each Option field overwrites only when present, and lastupdate is
stamped on every successful update. Note that website and logoUrl use
orElse against the existing value rather than getOrElse, matching
Lift's behaviour where a None in the request left the stored column
untouched rather than blanking it.

Flyway migration matches the probed schema: one unique index on
organisationid. Covered by the existing Http4s700RoutesTest
organisation CRUD scenarios. Full suite passes (3651 tests, 0
failures).
The three internal id-mapping tables (account, customer, transaction)
each declared their unique indexes as:

    UniqueIndex(<id>) :: UniqueIndex(<id>, <plainTextReference>)

Neither constrains the reference column. The id column is a MappedUUID
- a fresh random value on every insert, so it never collides - and the
composite is strictly implied by the single-column unique index on that
same id, so it can never reject a row the first index would have
allowed. It is dead weight: no constraint value, and no lookup value
either, since its leading column is already covered.

The consequence is that getOrCreate*Id, a SELECT-then-INSERT with
nothing underneath it, lets two concurrent calls for the SAME reference
both miss the SELECT, both INSERT, and both succeed - minting two
different OBP ids for one underlying bank reference. A later read
(LIMIT 1, no ORDER BY) then returns an arbitrary one of them, so data
written under one id is invisible under the other. This sits on the hot
path: Helper.convertToId runs it for every inbound message on the
RabbitMQ, gRPC, REST and stored-procedure connectors.

The providers already carry a "unique-index violation from a concurrent
insert - re-fetch the committed row" retry branch, written for exactly
the constraint that was never created. This makes that branch live
rather than dead code; no provider changes are needed.

V057 collapses any duplicates the missing constraint already permitted
(keeping the lowest id per reference - the earliest row, the one most
likely to have downstream data keyed to it), drops the redundant
composite indexes, and creates the single-column unique indexes. Rows
with a NULL reference are left alone: unique indexes permit multiple
NULLs, so they cannot violate the new constraint.

Only db/migration/h2 exists today and flyway.enabled defaults to false,
so this changes the test schema now and becomes the template for the
other vendor folders later. V057 carries a caveat for that port: on
PostgreSQL a constraint violation aborts the surrounding transaction,
so the retry branch only works if the INSERT is not sharing one with
later statements.

ConcurrentIdMappingRaceTest pins the invariant. It asserts the
constraint directly rather than racing: a first draft fired N
concurrent getOrCreateAccountId calls and passed with the fix reverted,
because the first insert always won before the others reached their
SELECT. A test that passes without the fix guards nothing.
The "completes immediately (no regression)" scenario failed twice in
full-suite runs under load with "openFuturesCount(ok)=1: 1 was not
equal to 0", while passing every time in isolation.

It is a genuine race in the test, not an environment artefact.
futureWithLimits returns the ORIGINAL future rather than one derived
from its own onComplete callback, so Await.result can return while that
callback - the one that runs decrementOnce - is still queued on the
ExecutionContext. Asserting the counter on the first read after the
await therefore samples a value that is correct but not yet settled.

Polls with `eventually` instead. The sibling scenario deliberately
keeps its Thread.sleep: there the counter is already 0 and what is
being guarded against is the completion callback wrongly taking it to
-1, so `eventually` would pass on its first poll, potentially before
that callback has run at all, and would not exercise the hazard.
Verified with three consecutive green runs.

Also documents the target/classes trap that surfaced while verifying a
Flyway migration in this area: Flyway loads from the classpath, and
maven never deletes a script you removed from src, so stashing the .sql
and re-running gives a false green off the stale copy.
Table 61/140 in the Lift Mapper to Doobie strangler migration.

The entity type leaked out of its package: AttributeDefinition appears
in Connector.scala's method signatures, LocalMappedConnector's
overrides, NewStyle's AttributeDocumentation wrapper and
JSONFactory4.0.0. The Doobie row case class therefore keeps the name
rather than being called DoobieAttributeDefinition, so none of those
signatures change and this stays a swap of the storage layer instead of
a rename rippling through the connector API. The four generated
connectors (RabbitMQ, gRPC, REST, stored-procedure) do not override
these three methods, so the trait defaults in Connector.scala are the
only definition sites.

Four call sites outside the provider queried the Mapper companion
directly - DeleteProductCascade and the already-migrated Doobie
account/transaction/transaction-request attribute providers, which were
still reaching into AttributeDefinition.findAll(By(...)). They now use
findAllByBankIdAndCategory / findAllByBankIdsAndCategory /
deleteByAttributeDefinitionId on the new companion, which reproduce the
same By / ByList shapes in SQL.

Preserves a Mapper quirk deliberately: canBeSeenOnViews is stored as a
";"-joined string and read back with a bare split, so an empty column
yields List("") rather than Nil. Callers filter that list by
membership, where the empty-string element is inert, but normalising it
would be a behaviour change smuggled in with a storage swap.

Flyway migration matches the probed schema: one unique index on
(bankid, name, category), the natural key the provider's
find-then-update keys off. Covered by the eight existing attribute
suites. Full suite passes (3653 tests, 0 failures).
Table 62/140 in the Lift Mapper to Doobie strangler migration.

jobscheduler is a lock table rather than a job-history log: a row exists
only while a job holds the lock and is deleted when the job finishes, so
in healthy operation it is empty and any row present is either a running
job or a stale lock left by a JVM that died before its finally block.

The concrete type leaks into JSONFactory7.0.0's createSchedulerJobsJsonV700
signature, so the Doobie row case class keeps the name JobScheduler and
that signature is untouched; only the field reads inside it change from
Mapper accessors (r.JobId.get) to trait ones (r.jobId). createdAt is
carried on the row because both schedulers and the v7 diagnostics
endpoint use it to distinguish a running job from an abandoned lock.

The Mapper query syntax at the call sites is replaced with named
finders - findAllByApiInstanceId / findAllByName /
findAllCreatedOnOrBefore / findByName / findByJobId - and create/delete
with createJob / delete.

Preserves a pre-existing bug in DataBaseCleanerScheduler verbatim: its
boot-time cleanup matches Name against apiInstanceId, which never hits,
because lock rows store Name=jobName and ApiInstanceId=apiInstanceId.
MetricsArchiveScheduler carries a comment describing this exact mismatch
and was fixed to key on ApiInstanceId; this scheduler was not. Left
as-is with a comment - correcting it would change self-heal-on-redeploy
behaviour under cover of a storage swap.

Flyway migration matches the probed schema: one unique index on jobid.
Covered by MetricsArchiveSchedulerTest and Http4s700RoutesTest's
scheduler-job-lock scenarios. Full suite passes (3653 tests, 0 failures).
Table 63/140 in the Lift Mapper to Doobie strangler migration.

The concrete type is what BankAccountBalanceProviderTrait is written
in, so the Doobie row case class keeps the name and the provider
signatures are untouched. The Connector trait is already stated in
terms of the obp-commons BankAccountBalanceTrait, so nothing leaks that
far and no connector changes.

The Flyway script reproduces a surprising fact rather than tidying it:
this table has no primary key and no indexes at all. The entity is a
KeyedMapper[String, _] declaring BalanceId_ as its primaryKeyField, but
Schemifier emitted a bare CREATE TABLE with none of that -
information_schema.indexes returns nothing for it on a booted instance.
Adding the declared primary key would be a schema change beyond a
storage swap, and on an existing database it could fail outright if
duplicate balanceid_ values were already let in by its absence.

balanceamount is stored in the smallest currency unit, so reading it
back needs the account's currency, which lives on mappedbankaccount.
Mapper resolved that with a per-row lookup in a val evaluated at row
construction - an N+1 - falling back to "EUR" when the account was
missing. The same first-match-or-EUR resolution is now a correlated
subquery inside the one SELECT: same answer, without the extra round
trips.

createOrUpdateBankAccountBalance keeps both of its Empty paths: an
unknown account is Empty (the account is what supplies the currency),
and on the update branch an unknown balanceId is Empty rather than an
insert - it is create-or-update on a supplied id, not upsert-by-id.

Covered by BankAccountBalanceTest plus the v2.1.0/v4.0.0
TransactionRequests suites. Full suite passes (3653 tests, 0 failures).
Table 64/140 in the Lift Mapper to Doobie strangler migration.

The concrete type is used directly by LocalMappedConnector's twelve
endpoint-tag methods, so the Doobie row case class keeps the name and
those signatures are untouched; only the Mapper query syntax inside
them becomes named finders. Both the provider trait and the Connector
trait are already stated in terms of the obp-commons EndpointTagT, so
nothing leaks further.

A tag is system-level when bankId is absent and bank-level otherwise.
Mapper wrote the system-level case as a literal null into the column and
read it back as None for null-or-empty; that null/empty equivalence is
preserved on the read side, and inserts now bind None.

Preserves a pre-existing bug in getBankLevelEndpointTag verbatim: it
repeats By(TagName, tagName) where By(BankId, bankId) was clearly
meant, so a bank-level lookup has always resolved exactly like a
system-level one, ignoring the bank entirely. Left as-is with a comment
- correcting it would change which tag callers get back, under cover of
a storage swap.

Flyway migration matches the probed schema: one unique index on
endpointtagid, and deliberately nothing on (operationid, tagname), so
duplicate tags for one endpoint remain possible exactly as before.
Covered by EndpointTagTest. Full suite passes (3653 tests, 0 failures).
Table 65/140 in the Lift Mapper to Doobie strangler migration.

Every external caller already went through MappedApiProductsProvider
and ApiProductTrait, so nothing outside the package sees the entity
type; the Doobie row case class keeps the name anyway to leave the
trait and provider signatures untouched.

Had no direct test coverage - the only suite naming ApiProduct
exercised the separate api-product-ATTRIBUTE table - so this adds
ApiProductsProviderTest, written against the Lift implementation and
confirmed green there before the rewrite, then green again after. It
pins the parts a storage swap can silently break: createOrUpdate is
keyed on (bankId, apiProductCode) and updates in place, preserving the
generated apiProductId the caller never supplies; tags are normalised
on the way in (trim, lower-case, strip embedded pipes, de-duplicate,
drop empties); and the tag filter matches a whole tag rather than a
substring, so filtering by "beta" must not return a product tagged only
"beta-2".

That last one is the reason for the storage format: tags live
pipe-delimited WITH leading and trailing pipes ("|featured|beta|") so
LIKE '%|beta|%' cannot match inside a longer tag. Kept verbatim.

Flyway migration matches the probed schema: unique index on
(bankid, apiproductcode) plus a plain index on bankid. Full suite
passes (3660 tests, 0 failures).
Table 66/140 in the Lift Mapper to Doobie strangler migration.

Per-bank AMQP broker coordinates: where OBP-API publishes messages
destined for a bank's own infrastructure. Every caller already went
through the three companion methods (findByBankId / upsert /
deleteByBankId), so replacing the storage behind them touches nothing
in Http4s700, OpenCorridorPublisher, OpenCorridorSettlement or
OpenCorridorFees.

Two details kept deliberately. password stays on the row - it is needed
to connect - but remains write-only by contract: AmqpBankBrokerJsonV700
has no password field, so no endpoint echoes it, and the migration
comment records that. And upsert stays keyed on bank_id, which the
unique index enforces as one broker per bank; that index is what makes
the find-then-insert-or-update well-defined.

The Mapper class also overrode save to stamp UpdatedAt on every write;
the Doobie upsert writes updated_at explicitly on both branches, so
that behaviour survives without the override.

Flyway migration matches the probed schema: one unique index on
bank_id. Covered by the Http4s700RoutesTest amqp-broker scenarios.
Full suite passes (3660 tests, 0 failures).
Table 67/140 in the Lift Mapper to Doobie strangler migration.

The name is kept from the Lift entity because DeleteProductCascade and
three historical MigrationOf* scripts refer to it directly, and the row
type is what ProductFeeTrait callers already see through the provider.

The `type` field is stored in a column called TYPE_C, not TYPE:
Schemifier appends "_c" to column names that collide with SQL reserved
words. The Flyway script reproduces that name exactly - calling it TYPE
would create a column the code never reads.

Both declared indexes are plain Index, not UniqueIndex, so productfeeid
is NOT unique at the database level even though the provider treats it
as an identifier. Reproduced as-is rather than tightened; adding
uniqueness would be a schema decision beyond a storage swap, and could
fail outright on data its absence has already allowed.

createOrUpdateProductFee keeps its asymmetry: a supplied productFeeId
means update-that-row-or-Empty, no id means insert with a generated
one - and the update branch rewrites bankId and productCode too, so a
fee can be moved between products by id.

The three MigrationOf* scripts move to the String-based
DbFunction.tableExistsByName overload now the Mapper object is gone.

Flyway migration matches the probed schema, including DECIMAL(34,2) for
amount. Covered by ProductFeeTest. Full suite passes (3660 tests, 0
failures).
Table 68/140 in the Lift Mapper to Doobie strangler migration.

The transactional outbox: business events commit their outbound
messages as rows in the same DB transaction, and the relay publishes
them afterwards with at-least-once redelivery. This is the table that
closes the atomicity gap between a DB commit and a broker publish, so
its mutation semantics matter more than most.

Unlike the read-mostly tables migrated so far, this one is written from
several places as the row moves PENDING -> DELIVERED or STICKY. The
Mapper form was field-mutation-then-saveMe scattered across the relay;
that becomes named intent-revealing updates - recordAttempt,
markDelivered, markSticky, resetForRetry - which keeps each call site's
meaning visible and, crucially, makes the updated_at stamp
unmissable.

updated_at is load-bearing rather than bookkeeping: the relay's
exponential backoff is computed from it (row.updatedAt + backoff <= now),
so a write that forgot to stamp it would make that row retry
immediately, forever. Mapper guaranteed the stamp with a `save`
override; every update helper here writes it explicitly instead, and
the migration comment records why.

Flyway migration matches the probed schema: three plain indexes
(status, subject_id, outbox_type), none unique. Covered by the
Http4s700RoutesTest operator-endpoint scenarios. Full suite passes
(3660 tests, 0 failures).
Table 69/140 in the Lift Mapper to Doobie strangler migration.

The OIDC token set obtained for an AuthUser at login, keyed to that
user by authuserprimarykey - AuthUser's internal row id, not a user_id
UUID. Every caller already went through the provider, so replacing the
storage behind it touches nothing in AuthUser.

Two facts reproduced rather than tidied. The table has NO index on
authuserprimarykey even though every read filters on it: the entity
overrode dbIndexes with nothing but super.dbIndexes, so Schemifier
created only the primary key. And rows accumulate rather than being
replaced - createToken always inserts and the read takes the newest by
createdat - so a user's token history is retained. Both left as-is;
adding an index or switching to upsert would be decisions beyond a
storage swap.

MigrationOfOpnIDConnectToken moves to the String-based
DbFunction.tableExistsByName overload now the Mapper object is gone,
and the MappedClassNameTest exemption for the entity is removed.

Exercised through the AuthUser login path. Full suite passes (3660
tests, 0 failures).
Table 70/140 in the Lift Mapper to Doobie strangler migration.

Had no direct coverage, so this adds UserAgreementProviderTest, written
against the Lift implementation first. That baseline run earned its
keep: it failed, and the failure was the test being wrong rather than
the code.

getLastUserAgreement does NOT return the newest row when a user
re-accepts the same agreement type on the same day. The date column is
MappedDate - day precision, no time - so both rows tie, and Lift broke
the tie with a stable sort over rows in insertion order, which means
the OLDEST of the tied rows wins despite the method's name. So
re-accepting terms today keeps reporting yesterday's... in fact
today's earlier... text.

The Doobie query reproduces that exactly with ORDER BY date_c DESC,
id ASC. Without the explicit id tie-break SQL would be free to return
either tied row, which would have been a real behaviour drift hidden
behind a green suite. The test pins the quirk with a comment saying
plainly that it is a defect being preserved, not endorsed; fixing it
needs its own change and a decision about existing rows.

Two more details carried over: agreementhash is DERIVED (a SHA-256 of
the text that Mapper recomputed in a beforeSave hook, so a caller could
never store a hash disagreeing with the text) and is computed on insert
here for the same reason; and the `date` field lives in a column called
DATE_C, since Schemifier appends "_c" to names colliding with SQL
reserved words - the same treatment ProductFee's `type` gets.

LiftUsers' batched multi-user lookup moves from findAll(ByList(...)) to
findAllByUserIds. Full suite passes (3663 tests, 0 failures).
Table 71/140 in the Lift Mapper to Doobie strangler migration.

Every caller already went through the provider, so replacing the
storage behind it is confined to this package plus one accessor change
at the claim endpoint.

Three things carried over deliberately rather than tidied.

secretKey is the sole credential on an invitation link:
getUserInvitationBySecretLink resolves an invitation from it alone,
with no bank scoping. So the Doobie insert generates it with the same
CSPRNG rather than accepting one from a caller. The column has neither
an index nor a uniqueness constraint despite every claim filtering on
it - reproduced as-is, since changing that is a schema decision, but
worth knowing it is the shape of the thing.

createdAt is not bookkeeping: the claim endpoint expires an invitation
24 hours after it, so the row carries it. Dropping it as a
CreatedUpdated artefact was my first instinct and the compiler caught
it - the field is load-bearing and is now commented as such.

scramble, the personal-data erasure path, replaces each field with a
random string OF THE ORIGINAL FIELD'S LENGTH. That is what the Mapper
version did and it is preserved: a fixed-width scramble would change
how much a retained row still reveals.

Flyway migration matches the probed schema: one unique index on
userinvitationid. Covered by UserInvitationApiTest, AgentDelegationTest
and Http4s700RoutesTest. Full suite passes (3663 tests, 0 failures).
getTopUsers and getTopConsumers called createQueriesByHttpParamsFuture
directly on the raw request params instead of going through
APIMetrics.applyMetricsFromDateDefault the way every other metrics-reading
endpoint does. With no from_date, APIUtil.getFromDate substitutes the
epoch, which makes MappedMetrics.determineMetricsCacheTTL classify the
query as "only stable data" and pick the 24-hour TTL - so the default,
no-parameter call an operator dashboard would make froze for a day while
traffic kept arriving, and the first miss of that day scanned the whole
metric table since 1970. Corrected the two ResourceDoc descriptions to
match (they claimed "defaults to one year ago" / "the current date",
which was never the actual range).
createAccountJSON's Links.Self does list.head.AccountId unconditionally.
getAccount builds that list by filtering the caller's own private accounts
down to the requested accountId, which is legitimately empty for an id
that does not exist (or belongs to someone else) - the same shape a real
TPP integration hits on a typo or a stale id. That empty list reached
list.head and threw NoSuchElementException, answering 500 instead of the
404 UK Open Banking's spec calls for.

Found by extending the endpoint auth/crash sweep to cover Berlin Group and
UK Open Banking (previously OBP-standard only) - FailureSweepTest calls
every endpoint with a nonexistent id and asserts none of them 5xx.
…n Banking

EndpointCatalog.all was Http4s700.allResourceDocs - the OBP-standard
aggregation only. AuthSweepTest, SuccessSweepTest and FailureSweepTest all
read their coverage from it, so every Berlin Group and UK Open Banking
endpoint was silently outside the anonymous-401/crash sweep: a doc in
those standards missing AuthenticatedUserIsRequired with empty roles would
let anonymous callers reach account data and nothing would catch it.

Switching to ResourceDocRegistry.allStaticResourceDocs (the same
cross-standard union APIUtil.getAllResourceDocs already exposes) needed
three follow-on fixes, all specific to a catalog that now spans multiple
independent route trees rather than one:

- EndpointCatalog.concretePath hard-coded "/obp/" + apiShortVersion.
  Berlin Group and UK Open Banking routes match on Root / urlPrefix /
  apiShortVersion with no "/obp" segment at all (see e.g.
  Http4sBGv13AIS.bgV13Prefix) - urlPrefix is "obp" for the OBP standard by
  construction (ApiVersion.setUrlPrefix patches it to the configured
  apiPathZero at boot), so using implementedInApiVersion.urlPrefix
  uniformly reproduces the old OBP behaviour while giving BG/UK their own
  real prefix instead of a path that 404s before reaching any route.

- AuthSweepTest.messageOf only read the top-level "message" field. Berlin
  Group requests get a PSD2-mandated {"tppMessages": [{"text": ...}]}
  envelope instead (ErrorResponseConverter.toBgErrorBody) - the endpoint
  was correctly answering 401, the sweep just could not see the message
  text to compare it against. Fall back to tppMessages[0].text, which
  carries the identical string the OBP envelope would have.

- SweepCoverageTest's "deduplicated by (url, verb)" check assumed one
  route shape maps to one operation, true for OBP but not for Berlin
  Group: several SCA sub-steps (e.g. updatePsuAuthentication /
  selectPsuAuthenticationMethod / transactionAuthorisation) legitimately
  share one URL and verb, disambiguated by request body rather than path.
  Replaced with a check on operationId uniqueness, which is the union's
  actual by-construction guarantee and still catches a genuine duplicate
  (e.g. two ResourceDoc objects registered under the same operation id).

Two categories of endpoint answer non-2xx to SuccessSweepTest's
fully-entitled-but-consentless caller and are documented in
expectedNon2xx rather than treated as failures: Berlin Group AIS and UK
Open Banking account-read endpoints both require an established,
standard-tagged consent regardless of role, which the sweep's generic
fixture (grants every role, creates no consent) does not provide. The
403 in both cases is the endpoint correctly refusing, not a defect - the
anonymous case is what AuthSweepTest already covers independently.
…ie stores

develop's 73 commits here are four features and one large teardown.

The features: maker/checker for runtime-supplied code (a Dynamic Change Request holds a
proposed create/update/delete of a dynamic resource doc, connector method, message doc or ABAC
rule until a second person approves its payload hash, and the runtime executes only rows whose
body hash equals the approved one); API Product Subscriptions (one Consumer holds one product
for a period, with a status machine and the Scope and RateLimiting rows it created); Dynamic
Glossary Items, which an operator adds at runtime and which may shadow a static item; and the
on-behalf-of attribution framework, which renames accountableUserId to onBehalfOfUserId, pulls
the resolver into Users.onBehalfOfUserIdOf, and states in UserReference.scala, per column,
whether a write by a consent user is recorded against the consent user or the human it acts for.

The teardown: the 12 OBPAPIx_y_z aggregator objects and the VersionedOBPApis trait are gone,
along with the APIMethodsXYZ.scala shims. Version enumeration is now explicit in
Http4sResourceDocAggregation.allVersions, and the Lift ResourceDoc text those files carried as
comments was exported first to scripts/resource_doc_baseline/*.json, which is what the parity
audit now reads.

All four features were written against Lift Mapper entities and created their tables through
Schemifier. This branch has neither: ToSchemify.models is empty, so a Mapper column here would
compile and then not exist. The resolution carries the behaviour across instead of restoring the
entities - DynamicChangeRequest, DynamicGlossaryItem, ApiProductSubscription, its Scope join
table and its Attribute table are case classes over Doobie stores, MakerChecker's 33 Mapper call
sites read through the Doobie providers, and the five tables plus the approvedhash/isactive
columns on the four target types and dynamicentity.authmode come from a new
db.changelog-develop-merge-2.yaml.

Three of develop's entries in the new attribution policy name entities this branch does not
have, and one names a column it does not have:

  - 16 of the 65 classes UserReference.scala names are Lift Mapper entities this branch already
    moved to Doobie. They are only ever read to build the log line in Users.attributionOf, but
    AgentDelegationTest resolves each one, so they are repointed at the class that actually owns
    the write here, with the real column names beside them.
  - PemUsageLastUser and UserScopeUser are dropped: 6bfcb4b deleted the PemUsage scaffold (its
    table was never created in any environment) and ac72b13 deleted MappedUserScope (no
    callers). A policy entry for a table that does not exist is not a policy.
  - abacrule already carries isactive from the baseline, unlike the three dynamic-code tables, so
    its maker/checker changeset adds only approvedhash. Liquibase aborted the boot on the
    duplicate column, which reads as a test-infrastructure failure rather than a schema one.

Two defects were introduced while resolving conflicts and are fixed here rather than left for CI:

The attribution policy was not actually applied to dynamic data. MapppedDynamicDataProvider
defines ownerOf and never calls it - save, update, get, getAll and existsData all passed the raw
caller id - and MapppedDynamicEntityProvider.createOrUpdate stored dynamicEntity.userId
unresolved. Taking our side of those conflicts kept the Doobie method bodies and dropped the
wrapping develop had put around them, so a consent user's rows and definitions were owned by the
agent rather than by the human it acts for. That is the whole point of the feature, and it is
silent: nothing fails, the row is simply attributed to the wrong user.

Four resource-doc examples published a dangling $ref to Object. A field of an erased type takes
its published type from its example value, so ProvenanceJsonV700.is_active and three of
RateLimiterLimitJsonV700's windows, left as None by develop's examples, had no type to recover.
DynamicChangeRequestJsonV700.current_payload is a separate case: it is a bare JValue and is
genuinely JNothing for a CREATE, which json4s omits from the body entirely - that now publishes
as {"type":"object"}, which is what the null branch already answered for the same field, while
every other declared type with an empty example still fails. buildSwaggerSchema's own failures
named neither the class nor the field, so translateEntity now adds that; it is how the offending
field was found at all.

Alongside: the four table-count assertions move from 147 to 152 for the five new tables, and
run_tests_parallel.sh's per-shard hard kill becomes SHARD_TIMEOUT_SECONDS, defaulting to 2400s
on Postgres. That kill exists to stop a JVM whose Pekko threads hang after the tests finish, not
to bound how long tests may take, and at 1200s three of the four Postgres shards were killed
mid-run - each still emitting test output at the moment it died. A killed shard reports zero
failures while dropping ~800 tests from the audit line, so the run reads as a pass unless the
per-shard durations are checked.

Suite: 4233 tests, 0 failures on H2 and the same 4233 on Postgres, where the three shards now
finish in 20:43, 21:58 and 22:39 - which is what the old 20-minute kill was cutting off.
… their roles

ResourceDocMatcher picked the first doc whose template matched, which is registration order.
Two docs can share (verb, version, segment count) and both match a request when one spells a
segment as an all-caps wildcard: v4's catch-all
/banks/BANK_ID/accounts/ACCOUNT_ID/GRANT_VIEW_ID/transaction-request-types/TRANSACTION_REQUEST_TYPE/transaction-requests
also matches .../FREE_FORM/..., and it is registered in the object body while the per-type
alias docs are added later by initBatch9AliasResourceDocs(). The catch-all declares no roles,
so ResourceDocMiddleware - which reads roles from the matched doc only - ran no role check at
all, and the FREE_FORM doc's Some(List(canCreateAnyTransactionRequest)) was never enforced. A
user holding only the owner view could create a FREE_FORM transaction request and get 201;
Lift returned 403, and Lift had no wildcard doc for this URL at all - all nine were literal
per-type entries, so this was introduced by the migration.

Two changes, both at the mechanism rather than at the call site:

findResourceDoc now picks the match with the most literal segments instead of the first one.
sortBy is stable, so docs of equal specificity keep their registration order and nothing else
moves.

literalAllCapsSegments gains the seven transaction-request types that were missing from it -
BULK, HOLD, MOBILE_WALLET, UTILITY, CARDANO, ETH_SEND_TRANSACTION, ETH_SEND_RAW_TRANSACTION.
Specificity ordering cannot separate two wildcards, and MOBILE_WALLET was shadowing UTILITY and
OPEN_CORRIDOR_PROMISE in v7 (HOLD and CARDANO shadowing the two ETH_* docs in v6) the same way,
which attached the wrong operationId to those requests - so api_disabled_endpoints, the JSON
schema and auth-type allow-lists, and the metric rows all keyed on the wallet endpoint. Its own
doc comment already said to add a value here when a path uses a new all-caps literal.

TRANSACTION_REQUEST_TYPE stays a wildcard: it is the only one of the eighteen that is a
placeholder rather than a type.
… not the human

createTransactionRequestImpl210 is the only writer of on_behalf_of_user_id. develop resolved it
with Users.attributionOf(userId, UserReference.TransactionRequest); porting the method to the
Doobie store kept the Doobie body and dropped the attribution block, leaving

  onBehalfOfUserId = callContext.flatMap(cc => cc.onBehalfOfUser.or(cc.consenter).map(_.userId))

CallContext.onBehalfOfUser is consentCreator.or(consenter).or(user) - a plain field fallback. It
does not consult the consent chain; the resolver that does lives in the String sibling
(ApiSession.onBehalfOfUserId -> Users.onBehalfOfUserIdOf), whose own doc describes three steps.
The expression above did step 1 and then jumped to step 3, so for a consent user whose
CallContext carries only `user` - the ordinary case, and the one AgentDelegationTest asserts is
resolved through ResourceUser.createdByConsentId - the row was written with userId ==
onBehalfOfUserId == the agent's id. The request succeeds and the attribution WARN is never
logged, so nothing surfaces: the payment is simply owned by the wrong user.

Restores the attributionOf call, keeping develop's precedence rule that a consentCreator or
consenter populated by a middleware still wins over the database chain. (.or(cc.consenter) in
the old expression was dead in any case - consenter is already inside onBehalfOfUser's chain.)

The other five attributionOf call sites survived the port; this was the one that did not.
UserReference.TransactionRequest still declared policy UseOnBehalfOfUserId with no callers left.
…same id

deleteAtm issued DELETE FROM mappedatm WHERE matmid = ?, keyed on the ATM id alone. The unique
index mappedatm_mbankid_matmid is on the pair, so two banks may legitimately hold the same
atm_id, and every other statement in this store - getAtmFromProvider, the createOrUpdateAtm
re-read, updateColumn - filters on both columns. deleteAtm was the only one that did not, though
atm.bankId.value was in hand.

The Lift code this replaced was also bank-unscoped, but it deleted through
find(By(mAtmId, ...)), which returns at most one row - so the blast radius was one wrong row
rather than all of them. Widening that to a set-based DELETE is what the migration introduced.

Note deleteAtmAttributesByAtmId, called immediately after this in Http4s510, is unscoped for the
same reason and atmattribute does carry a bankid column - but its signature comes from the
Connector trait, so fixing it changes a connector method signature and is left out of this
commit deliberately.
…mn, a dead tryo

MetricsArchiveRun.pruneToMostRecent built DELETE ... WHERE id < (SELECT MIN(id) FROM (...)) with
no alias on the derived table. PostgreSQL required one until version 16, so the statement is a
syntax error on any PG 15 or older deployment. Nothing catches it: CI pins postgres:16-alpine
and H2 never required an alias. The consequence is worse than a failed prune, because
MetricsArchiveScheduler calls recordRun again from inside its own catch - the second throw is
unguarded and escapes runOnce, so the tick ends in an exception and the table this class
documents as self-capping grows without bound.

buildTopConsumersQuery reused buildFilterConditions, whose consumerId fragment is the
unqualified `consumerid`. That query selects FROM metric, consumer and both tables carry the
column, so ?consumer_id=X on /management/metrics/top-consumers fails with an ambiguous column
reference. getTopConsumersByConsumerId avoided the reuse for exactly this reason and says so in
a comment. buildFilterConditions already took resolvedUserIdExpr so a caller could qualify
userid; consumerIdExpr does the same for this one, and buildTopApisQuery - which selects FROM
metric alone - keeps the bare default.

getAllAggregateMetricsBox and getTopConsumersFuture wrapped an already-evaluated val in tryo.
The query ran on the line above, so tryo had nothing to catch and a failure threw straight past
it, bypassing the caller's unboxFullOrFail and its OBP error code. The query now runs inside the
tryo, which is how the other four memoized methods in this file were already written.
Every other version reaches the chain through gate(), which resolves to HttpRoutes.empty when
APIUtil.versionIsAllowed is false. The three Berlin Group trees were spliced in raw. Neither BG
wrapper re-checks the prop, and ResourceDocMiddleware deliberately does not re-check
version-level enablement per request - it says so at isEndpointEnabled, because the check is
meant to happen once at startup in exactly this place. So Berlin Group had no version gate at
any layer.

The asymmetry is what makes it a problem rather than an omission: with
api_disabled_versions=BGv1.3 set, GET /obp/v7.0.0/api/versions reports BG as inactive while
/berlin-group/v1.3/accounts/ACCOUNT_ID/transactions keeps returning consent-scoped account data,
and UKv3.1 disabled by the same prop does go dark.
…eware skipped them

ResourceDocMatcher indexes on (verb, apiVersion, segment count). elasticSearchWarehouse's
template was "/search/warehouse" - two segments - while its route is
`prefixPath / "search" / "warehouse" / queryString`, three. A real call therefore looked up a
key the doc was not filed under and found nothing, and there is no prefix or longest-prefix
fallback. elasticSearchMetrics had the same mismatch.

With no doc, ResourceDocMiddleware takes its unmatched branch, which calls
resolveCallerWithoutRateLimiting instead of anonymousAccess and never runs validateOnly. Four
things silently did not happen for these endpoints: the Consumer rate limiter never counted the
call, so this Elasticsearch proxy could be driven unthrottled and any per-endpoint limit row for
it was inert; api_disabled_endpoints could not disable them; auth-type and JSON-schema
validation never ran; and the metric row carried an empty operationId. Authentication and the
canSearchWarehouse / canSearchMetrics checks are inline in the handlers, so this was a control
bypass rather than data exposure.

The templates gain a third segment. SEARCH_QUERY is outside literalAllCapsSegments so it stays a
wildcard. That diverges from the Lift baseline, which is the documented placeholder-rename case:
both are recorded in parity_allowlist.json with the digests allowlist_helper.py computed.
resetDatabaseForTestClass states that every table is listed explicitly, since no entity is a
Lift Mapper any more and there is no model loop to clear them. The five tables
db.changelog-develop-merge-2.yaml creates were not in the list, so that statement was false for
them and their rows survived into later test classes - the suite runs forkMode=once against one
h2:mem database with DB_CLOSE_DELAY=10, so the leak spans the whole JVM.

Nothing failed because of it: the suites that write these tables use UUID-suffixed ids, and the
one fixed glossary title is deleted by every scenario that creates it. That is luck rather than
isolation, and check_test_isolation.py would not catch the next table added without it - it
checks setPropsValues placement, not table coverage.
…em were bare

check_nullable_column_reads.py exists to prove that no store reads a nullable column into a
non-nullable Scala type, because doobie's Get throws NonNullableColumnRead on a SQL NULL and
fails the whole query rather than the one row - one legacy row turns a listing endpoint into a
500. It printed "OK: every nullable column is read through Option" while structurally unable to
see most of the code it was pointed at. Two independent blind spots:

The scan matched only `type XxxRow = (...)` alias declarations. About a third of the stores are
written that way; the rest write the row type inline at the `.query[(...)]` call site or use a
case class, and neither was ever examined. Adding those two shapes - plus comment stripping,
without which this script's own scaladoc example (a `case class TopApi` beside a
`SELECT ... FROM metric`) matched itself and reported three violations against a util class that
runs no queries - takes the count from 0 to 597 across 47 stores.

read_ddl broke out of a createTable block the moment it saw `tableName:`. That works only
because Liquibase's writer emits keys alphabetically and so puts tableName after columns; a
hand-written changelog puts it first, and the scan then registered the table with no columns at
all. Empty is worse than absent here: find_violations skips any column not in the map, so every
column of those tables was silently exempt. Six tables from the develop merge were in that
state, addColumn changesets were never read, and flow-style `- column: {name: x}` entries were
not parsed. The block is now read by indentation, in either key order and either style.

The 597 reads are fixed the way the script's own docstring prescribes - bound as Option and
collapsed to what Mapper's reader returned: String to null, Boolean to false, numerics to the
field's default, Timestamp to a null Date. Three shapes appear across the stores and each keeps
its callers unchanged: a tuple consumed as `r._N`, a `case (...)` destructuring (binders renamed
and given a collapsing val), and a case-class row (fields become `xRaw: Option[T]` with a
defaulted `def x: T` accessor). Where a row class is constructed by name outside its store -
DoobieInvestigationQueries.CustomerRow is built by Http4s600 - the class keeps its shape and the
collapse happens at the query instead.

Spot-checked rather than taken on faith: customerlink declares every column except its primary
key nullable, and the store read eight of them bare at six query sites.
The attribute providers bind the free-text columns that come straight from the request body -
name and value - as bare String. Doobie's Put for a non-nullable type calls unsafeSetNonNullable,
which throws on a null, and the create branch is not inside tryo, so the exception escapes the
Future and the caller gets a 500 with no OBP frame in the trace.

A null does reach the bind. The JSON case classes declare value as a plain String, both of this
repo's custom String deserializers explicitly exclude JNull, and json4s 4.1.1's default
ExtractionNullStrategy is Keep - so extraction yields null rather than raising, and no upstream
validation rejects it: the endpoint validates `type` and passes `value` through untouched. Lift's
MappedString did not override setFilter, so it stored SQL NULL and returned 201.

Binding as Option restores that. The same wrapping lands on the name filter in the attribute
WHERE clauses, which is the behaviour Lift's By(field, null) had: it renders `= NULL` and matches
nothing, rather than throwing at bind time.

Ten providers, 48 binds.
AuthSweepTest keeps a signed-off list of endpoints whose observed authentication differs from
what their ResourceDoc declares, and a scenario that fails when an entry stops being needed -
an exemption nobody removes is read by the next person as still true.

createTransactionRequestFreeForm was on that list for answering 400 rather than the 403 its doc
declares. The written reason - that the endpoint does no upfront role check and leaves the
decision to the connector - described the symptom rather than the cause: the doc the middleware
matched was v4's wildcard catch-all, which declares no roles at all, so no role check ran for
any transaction-request type. With the matcher now preferring the most specific template, the
per-type doc wins and its canCreateAnyTransactionRequest is enforced.

The sweep caught this on its own, which is the whole point of that scenario.
@sonarqubecloud

Copy link
Copy Markdown

develop's 6b1b49f put the AGPL header at the top of 1119 files. This branch has changed the
imports directly under the package line in most of them, so merging that commit as it stands
turns a comment-only change into a conflict in every one of those files.

Applying the same patch here first makes the two sides agree on the top of each file: 916 files
took develop's diff unchanged, and 118 whose leading context had moved got the identical 27
lines prepended. The 85 the commit touches that this branch has deleted are left deleted.

No header is added to files that exist only on this branch - develop itself carries several
hundred Scala files without one, so it is a one-off change there rather than an enforced rule.
develop adds 496 commits, written against Lift Mapper and Schemifier, which this
branch no longer has. Their features are carried across as Doobie stores, with the
schema they need declared in Liquibase:

- db.changelog-develop-merge-3.yaml: the 21 new tables (files, assets, retention,
  tenants, domain APIs, platform apps, consumer account access, Dynamic Entity
  webhooks, account creation, group memberships, self-service banks), the new
  columns (Dynamic Entity record metadata, metric forwarded_for/domain_api_url,
  signing basket consumer, user auth context update bank, counterparty
  on-behalf-of creator, dynamic resource doc lang) and indexes, taken from the
  schema upstream's Schemifier built at the merge base and at db74dd4.
- db.changelog-dynamic-entity-space.yaml: system level Dynamic Entity rows move
  from a NULL bank id to SYS, and the indexes the space-scoped ones replace are
  dropped.
- The Dynamic Entity, record, access and resource doc stores address a row by
  (bank or SYS, entity, id); records keep who created and last updated them.
- Data migrations upstream wrote with Mapper (role renames, Organisation to
  Tenant, self-service banks, currency upper-casing) read and write the rows
  through JDBC; MakerChecker re-hashes dynamic code with its language once.
- Organisation is replaced by Tenant, as upstream did.
- The dynamic code dependency scan stays ungated: getDependentMethods and
  getDynamicCodeDependentMethods accept upstream's `force` argument but always
  scan, so the security validation never sees an empty list.
Takes develop's v7.0.0 customers-at-one-bank attribute filter. The new test imports the json4s extraction implicit that this branch's Scala 3 build needs.
…nario call

With the brace on the next line the body is a separate statement of the Feature, so the suite aborted while loading.
…NULL reply or error can be settled, and one failing row no longer stops a relay pass
…uthor's code, and the dry-run output is deleted at once
…defines, so restricted calls inside closures and local objects are rejected
… classes of Java snippets, and does not reject the wrapper's own calls
… per-line tag, and lines of the template report as 0
@sonarqubecloud

Copy link
Copy Markdown

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants