diff --git a/compiler-architecture.md b/compiler-architecture.md index d5af560a..60cef75a 100644 --- a/compiler-architecture.md +++ b/compiler-architecture.md @@ -456,9 +456,6 @@ check_enough_funds($pulled_sn, $share_n) ### Optimisations -> [!NOTE] -> Peephole optimisations aren't implemented yet. There is a POC in another branch, to measure how much perf could be impacted, but it's too soon to consider - You may have noticed that the previous compilation examples emit _a lot_ of garbage. That's done by design: the compiler must be simple and declarative. We don't want dozens of special cases in the compilation logic, which must express a general, albeit redundant, template which focuses on correctness. @@ -468,29 +465,57 @@ That change also *enables* a peephole that was previously out of reach. `assert_ However, the `irInstr` layer allows us to easily rewrite the instructions so that we remove garbage instructions, precomputing more aggressively, rewrite them into more efficient code (this is called [peephole optimisation](https://en.wikipedia.org/wiki/Peephole_optimization)) -Each peephole is independent and is expressed as a `func(instr []irInstr) []irInstr`, which returns the new instructions set, or nil if it didn't change. -Each peephole is independently testable and reviewable. +Each peephole is independent and is expressed as an `ir.Peephole`, a `func([]ir.Instr) []ir.Instr` which returns the new instruction stream, or nil if it didn't change anything. A pass never mutates its input. +Each peephole is independently testable and reviewable: one file per pass under `internal/ir/peephole_*.go`, with its tests written against the textual IR. + +We apply each peephole optimisation sequentially, and repeat until a full sweep changes nothing (the program `p` such that `f(p) == p` for every peephole `f`). + +Note that proving that a peephole function _does_ have a fixed point is usually simple, whereas proving that the function composition of all the peepholes isn't. Pratically speaking, we avoid a non-terminating optimisation by imposing a max amount of sweeps (`maxOptimizeSweeps`). A clever order of peepholes makes convergence fast anyway: `ir.DefaultPeepholes()` is ordered so that one sweep normally does it all and the second one confirms the fixed point. + +Optimisation is **opt-in**. `compiler.Compile` assembles the plain stream; `compiler.CompileWithOptions(program, flags, compiler.Options{Optimize: true})` — `numscript.CompileOptions{Optimize: true}` on the public API — runs `ir.Optimize` first, and typechecks the result again before assembling it. The host decides, per compilation. + +The contract of every pass is that the optimized program is **behaviorally identical** to the plain one: same postings in the same order, same metadata, same errors — including which error surfaces when more than one could. Three things enforce it: + +- the spec corpus (`TestCompilerScripts`) runs the plain and the optimized program on every case and requires identical results and identical error structs, before checking the plain one against the spec; +- `TestCompiledCorpusPassesVerify` runs both through `vm.Verify` and the `Encode`/`Decode` round trip; +- the differential sweep (`internal/difftest`) has a `vm vs optimized vm` leg compared with `CompareIdentical`: no aggregation, and one tolerance, a capacity bound of the plain program the optimized one does not hit. -We apply each peephole optimisation sequentially, and repeat until we reach a fixed point for each peephole (the program `p` such that `f(p) == p`, where `f` is the peephole function). +#### Implemented peepholes -Note that proving that a peephole function _does_ have a fixed point is usually simple, whereas proving that the function composition of all the peepholes isn't. Pratically speaking, we can avoid non-terminating optimisation passes by imposing a max amount of optimisation passes. A clever order of peepholes should make convergence quite fast anyway. +Three of them share one analysis, `walkConsts`: a forward walk that knows, before each instruction, which registers hold the same constant on every path reaching it. Jumps only go forward, so every edge into a label is known when the walk reaches it and one pass is exact. Unlike a "written once" rule, it does not depend on the typechecker, which checks reads against writes in program order, not per path. + +1. **`constProp`** — a pure op whose result is the same int, str or bool on every path reaching it becomes a load of that value. Portions are tracked but not materialized, as there is no portion load. Each op is folded exactly as the VM computes it; a `mk_portion` with a zero denominator is left to fail at run time. `check_enough_funds` fails unless `got == needed`, so after it `got` takes `needed`'s value: with a literal amount, the whole allotment split is computed at compile time. +2. **`dropProvenAsserts`** — a check that passes on every path reaching it goes: the `assert_*` family and `check_enough_funds`. A check that may fail stays where it is, so the first error a run hits is unchanged. Dropping `assert_leftover` is what frees the portion arithmetic for `deadCode`. +3. **`pruneDeadBranches`** — a conditional jump on a constant bool becomes an unconditional `jmp` or nothing; the instructions between a `jmp` and the next label a reachable jump targets go; a jump to the next instruction goes; a label no jump targets goes. +4. **`deadCode`** — a pure instruction whose dests no path reads goes. Store reads, pulls and every check stay; a `mk_portion` goes only where its denominator is known not to be zero. + +The `@world` diamond (see [Plain account source/dest](#plain-account-sourcedest-bounded)) is what the folding and the pruning are for: `str_eq` over two `load_str` folds, the branch on it goes, and a literal `@world` source is back to a single unbounded `pull_account` — any other literal account to a single bounded one. The amount clamp (`max($amount, 0)`) collapses the same way when the amount is a literal. + +`send [USD/2 42] (source = @world destination = @dest)` optimizes to: + +``` + $r0 = "USD/2" + $r1 = 42 + set_current_asset($r0) + $r2 = 42 + $r3 = "world" + $r4 = pull_account(account: $r3, cap: $r2) + check_enough_funds($r4, $r1) + $r5 = "dest" + send_to_account(account: $r5) +``` -> TODO list some peepholes +and `send [USD/2 100] (source = @src destination = {1/3 to @a 1/3 to @b remaining to @c})` to three sends of the constants 34, 33 and 33: no portion op is left. -The `@world` diamond (see [Plain account source/dest](#plain-account-sourcedest-bounded)) is -the clearest case to date, and it needs two passes that compose: +The passes can trade one capacity bound of the assembler for another (a shorter stream with longer-lived registers), so when the optimized stream hits `ErrRegisterBankOverflow` or `ErrProgramTooLarge` the compiler assembles the plain one instead. Optimizing can lift a bound (`deadCode` drops unused `load_var`s, so a script declaring more vars than a bank holds may compile), never impose one. -1. **Const-fold `str_eq`** — when both operands trace back to `load_str` constants, the - comparison is statically decidable: replace it with `true` or `false`. -2. **Dead-branch elimination** — a conditional jump on a register holding a known bool - is either a no-op or an unconditional `jmp`; then everything between a `jmp` and the - next reachable label is unreachable. `is_zero` over a `load_int` folds the same way, - which is what makes the quantity branches reachable for this pass too. +#### Planned peepholes -Together they collapse a literal `@world` source back to a single unbounded -`pull_account`, and any other literal account back to a single bounded one — i.e. to -exactly the code the compiler emitted when the VM still special-cased the name. The -prologue's `load_str "world"` also becomes dead once no branch refers to it. +- **Post from unbounded** — a send statement with a single unbounded source (`@world`, or `allowing unbounded overdraft`) and a single destination becomes a direct posting, skipping the funds queue. +- **Dead credit** — the destination credit of a direct posting whose destination's balance is never read. +- **Funds bypass** — the Pull-then-Send → take + post rewrite for a bounded single source, a fan-out (one source, allotment destinations) and a fan-in (statically distinct sources, one destination). Never N×M, `oneof` or scaling. +- **Integer allotment split** — `portion_to_int(mul_portion(p, int_to_portion(x)))` as one integer op, so an allotment over a var amount allocates no `big.Rat`. Needs a new opcode. +- **Register allocation** — only if the benchmarks justify it; see below. ### Registers allocator diff --git a/internal/compiler/bench_test.go b/internal/compiler/bench_test.go index 5c7db2b9..18f424fb 100644 --- a/internal/compiler/bench_test.go +++ b/internal/compiler/bench_test.go @@ -260,12 +260,23 @@ func BenchmarkCompiledVMCapped(b *testing.B) { // reuse one Vm, measure only the run. func benchCompiledVM(b *testing.B, src string, store benchStore) { b.Helper() + benchCompiledVMWith(b, src, store, compiler.Options{}) +} + +// benchCompiledVMOpt is benchCompiledVM on the optimized program. +func benchCompiledVMOpt(b *testing.B, src string, store benchStore) { + b.Helper() + benchCompiledVMWith(b, src, store, compiler.Options{Optimize: true}) +} + +func benchCompiledVMWith(b *testing.B, src string, store benchStore, opts compiler.Options) { + b.Helper() parsed := parser.Parse(src) if len(parsed.Errors) != 0 { b.Fatalf("parse errors: %v", parsed.Errors) } - _, program, err := compiler.Compile(parsed.Value, nil) + _, program, err := compiler.CompileWithOptions(parsed.Value, nil, opts) if err != nil { b.Fatalf("compile: %v", err) } @@ -315,6 +326,38 @@ func BenchmarkCompiledVMAllotmentThirds(b *testing.B) { }}) } +// --- Optimized programs ---------------------------------------------------- +// The same scripts through compiler.Options{Optimize: true}, so each peephole's +// effect is a before/after on the same benchmark. + +func simpleStore() benchStore { + return benchStore{balances: map[funds.PairKey]*big.Int{ + {Account: "src", Asset: "USD/2", Color: ""}: big.NewInt(100), + }} +} + +func BenchmarkCompiledVMOpt(b *testing.B) { + benchCompiledVMOpt(b, benchSrc, simpleStore()) +} + +func BenchmarkCompiledVMCappedOpt(b *testing.B) { + benchCompiledVMOpt(b, benchSrcCapped, cappedStore()) +} + +// The most common numscript shape: funding an account from @world. +const benchSrcWorld = `send [USD/2 42] ( + source = @world + destination = @dest +)` + +func BenchmarkCompiledVMWorld(b *testing.B) { + benchCompiledVM(b, benchSrcWorld, benchStore{}) +} + +func BenchmarkCompiledVMWorldOpt(b *testing.B) { + benchCompiledVMOpt(b, benchSrcWorld, benchStore{}) +} + // Fan-in: {1/3 from @a; 1/3 from @b; 1/3 from @c} -> @dest. Allotment on the // source side, no early-exit jump. const benchSrcFanIn = `send [USD/2 30] ( @@ -333,3 +376,72 @@ func BenchmarkCompiledVMFanIn(b *testing.B) { {Account: "c", Asset: "USD/2", Color: ""}: big.NewInt(100), }}) } + +func BenchmarkCompiledVMAllotmentOpt(b *testing.B) { + benchCompiledVMOpt(b, benchSrcAllotment, benchStore{balances: map[funds.PairKey]*big.Int{ + {Account: "src", Asset: "USD/2", Color: ""}: big.NewInt(1000), + }}) +} + +func BenchmarkCompiledVMAllotmentThirdsOpt(b *testing.B) { + benchCompiledVMOpt(b, benchSrcAllotmentThirds, benchStore{balances: map[funds.PairKey]*big.Int{ + {Account: "src", Asset: "USD/2", Color: ""}: big.NewInt(1000), + }}) +} + +func BenchmarkCompiledVMFanInOpt(b *testing.B) { + benchCompiledVMOpt(b, benchSrcFanIn, benchStore{balances: map[funds.PairKey]*big.Int{ + {Account: "a", Asset: "USD/2", Color: ""}: big.NewInt(100), + {Account: "b", Asset: "USD/2", Color: ""}: big.NewInt(100), + {Account: "c", Asset: "USD/2", Color: ""}: big.NewInt(100), + }}) +} + +// The same split with the amount from a var: no constant reaches the portion +// arithmetic, so it runs on every execution. +const benchSrcAllotmentVar = `vars { monetary $m } +send $m ( + source = @src + destination = { + 10% to @fees + remaining to @b + } +)` + +func benchVarsWith(b *testing.B, opts compiler.Options) { + parsed := parser.Parse(benchSrcAllotmentVar) + enc, program, err := compiler.CompileWithOptions(parsed.Value, nil, opts) + if err != nil { + b.Fatal(err) + } + vars, err := enc.Encode(map[string]string{"m": "USD/2 100"}) + if err != nil { + b.Fatal(err) + } + store := benchStore{balances: map[funds.PairKey]*big.Int{{Account: "src", Asset: "USD/2"}: big.NewInt(1000)}} + machine := vm.NewVm(program) + b.ReportAllocs() + b.ResetTimer() + for i := 0; i < b.N; i++ { + if _, err := vm.Exec(context.Background(), machine, &vars, store); err != nil { + b.Fatal(err) + } + } +} + +func BenchmarkCompiledVMAllotmentVar(b *testing.B) { benchVarsWith(b, compiler.Options{}) } +func BenchmarkCompiledVMAllotmentVarOpt(b *testing.B) { + benchVarsWith(b, compiler.Options{Optimize: true}) +} + +const benchSrcAllotmentPct = `send [USD/2 100] ( + source = @src + destination = { + 10% to @fees + remaining to @b + } +)` + +func BenchmarkCompiledVMAllotmentPctOpt(b *testing.B) { + benchCompiledVMOpt(b, benchSrcAllotmentPct, benchStore{balances: map[funds.PairKey]*big.Int{{Account: "src", Asset: "USD/2"}: big.NewInt(1000)}}) +} diff --git a/internal/compiler/compiler.go b/internal/compiler/compiler.go index 24b6ec02..296ba03e 100644 --- a/internal/compiler/compiler.go +++ b/internal/compiler/compiler.go @@ -1,6 +1,8 @@ package compiler import ( + "errors" + "fmt" "maps" "math/big" "slices" @@ -21,6 +23,18 @@ import ( // interpreter.RunProgram, the script's own #![feature(..)] declarations are // unioned in. func Compile(program parser.Program, featureFlags map[string]struct{}) (VarsEncoder, vm.Program, error) { + return CompileWithOptions(program, featureFlags, Options{}) +} + +// Options configures CompileWithOptions. The zero value is what Compile uses. +type Options struct { + // Optimize runs ir.Optimize over the instruction stream before assembling. + // The optimized program produces the same postings, metadata and errors as + // the plain one. + Optimize bool +} + +func CompileWithOptions(program parser.Program, featureFlags map[string]struct{}, opts Options) (VarsEncoder, vm.Program, error) { compiled, cErr := compileProgramToIR(program, featureFlags) if cErr != nil { return VarsEncoder{}, vm.Program{}, cErr @@ -30,7 +44,21 @@ func Compile(program parser.Program, featureFlags map[string]struct{}) (VarsEnco return VarsEncoder{}, vm.Program{}, err } - prog, err := ir.Assemble(compiled.instructions) + instrs := compiled.instructions + if opts.Optimize { + instrs = ir.Optimize(instrs) + // a pass that breaks register typing is a bug in the pass + if err := ir.Typecheck(instrs); err != nil { + return VarsEncoder{}, vm.Program{}, fmt.Errorf("optimized program: %w", err) + } + } + + prog, err := ir.Assemble(instrs) + // the passes can trade one capacity bound for another (a shorter stream + // with longer-lived registers), so the plain stream decides those errors + if opts.Optimize && (errors.Is(err, ir.ErrRegisterBankOverflow) || errors.Is(err, ir.ErrProgramTooLarge)) { + prog, err = ir.Assemble(compiled.instructions) + } if err != nil { return VarsEncoder{}, vm.Program{}, err } diff --git a/internal/compiler/optimize_test.go b/internal/compiler/optimize_test.go new file mode 100644 index 00000000..35510ff5 --- /dev/null +++ b/internal/compiler/optimize_test.go @@ -0,0 +1,125 @@ +package compiler + +import ( + "strings" + "testing" + + "github.com/formancehq/numscript/internal/ir" + "github.com/formancehq/numscript/internal/parser" + "github.com/gkampitakis/go-snaps/snaps" + "github.com/stretchr/testify/require" +) + +// getOptimizedOutput is getCompiledOutput after ir.Optimize. The dump is +// canonicalized through a parse: a pass that deletes an instruction leaves a +// gap in the register numbering, and the parser renumbers by first appearance. +func getOptimizedOutput(t *testing.T, source string) string { + t.Helper() + program := parser.Parse(source) + require.Empty(t, program.Errors) + compiled, err := compileProgramToIR(program.Value, nil) + require.Nil(t, err) + require.NoError(t, ir.Typecheck(compiled.instructions)) + + optimized := ir.Optimize(compiled.instructions) + require.NoError(t, ir.Typecheck(optimized)) + + instrs, errs := ir.Parse(ir.Dump(optimized)) + require.Empty(t, errs, "the optimized dump does not parse back") + return "\n" + ir.Dump(instrs) +} + +// A literal @world keeps only the unbounded arm of its branch. +func TestOptimizedWorldSend(t *testing.T) { + out := getOptimizedOutput(t, ` + send [USD/2 42] ( + source = @world + destination = @dest + ) + `) + + snaps.MatchInlineSnapshot(t, out, snaps.Inline(` + $r0 = "USD/2" + $r1 = 42 + set_current_asset($r0) + $r2 = 42 + $r3 = "world" + $r4 = pull_account(account: $r3, cap: $r2) + check_enough_funds($r4, $r1) + $r5 = "dest" + send_to_account(account: $r5) +`)) +} + +// Any other literal account keeps only the bounded one. +func TestOptimizedBoundedSend(t *testing.T) { + out := getOptimizedOutput(t, ` + send [USD/2 10] ( + source = @src + destination = @dest + ) + `) + + snaps.MatchInlineSnapshot(t, out, snaps.Inline(` + $r0 = "USD/2" + $r1 = 10 + set_current_asset($r0) + $r2 = 10 + $r3 = "src" + $r4 = 0 + $r5 = pull_account(account: $r3, cap: $r2, overdraft: $r4) + check_enough_funds($r5, $r1) + $r6 = "dest" + send_to_account(account: $r6) +`)) +} + +// An account from a var is decided at run time: both arms stay. +func TestOptimizedSendFromVarKeepsBothArms(t *testing.T) { + out := getOptimizedOutput(t, ` + vars { account $src } + send [USD/2 10] ( + source = $src + destination = @dest + ) + `) + + snaps.MatchInlineSnapshot(t, out, snaps.Inline(` + $r0 = "world" + $r1 = load_var(0) + $r2 = "USD/2" + $r3 = 10 + set_current_asset($r2) + $r4 = 10 + $r5 = 0 + $r6 = str_eq($r1, $r0) + jmp_if_false($r6, #not_world_1) + $r7 = pull_account(account: $r1, cap: $r4) + jmp(#pull_end_2) +#not_world_1 + $r7 = pull_account(account: $r1, cap: $r4, overdraft: $r5) +#pull_end_2 + check_enough_funds($r7, $r3) + $r8 = "dest" + send_to_account(account: $r8) +`)) +} + +// The unbounded arm the compiler emits for a literal @world is reached by a +// branch on str_eq; folding and pruning it must leave a single pull. Checked on +// the plain dump too, so a change in how the compiler spells the branch shows +// up here rather than as a silently unoptimized program. +func TestOptimizeCollapsesLiteralWorldBranch(t *testing.T) { + src := `send [USD/2 42] (source = @world destination = @dest)` + + plain := getCompiledOutput(t, src) + require.Equal(t, 2, strings.Count(plain, "pull_account("), "the plain program has both arms") + require.Contains(t, plain, "str_eq(") + + out := getOptimizedOutput(t, src) + require.Equal(t, 1, strings.Count(out, "pull_account(")) + require.NotContains(t, out, "overdraft:", "the bounded arm is the dead one") + require.NotContains(t, out, "str_eq(") + require.NotContains(t, out, "jmp") + require.NotContains(t, out, "#") +} diff --git a/internal/compiler/scripts_test.go b/internal/compiler/scripts_test.go index 7a4f4d6e..63796b5c 100644 --- a/internal/compiler/scripts_test.go +++ b/internal/compiler/scripts_test.go @@ -63,6 +63,9 @@ func TestCompilerScripts(t *testing.T) { } } +// runScriptSpec checks the plain program against the spec, and the optimized +// program against the plain one: on every case the two must return the same +// result and the same error, struct for struct. func runScriptSpec(t *testing.T, specs specs_format.Specs, src string) { parsed := parser.Parse(src) require.Empty(t, parsed.Errors) @@ -74,6 +77,8 @@ func runScriptSpec(t *testing.T, specs specs_format.Specs, src string) { enc, program, cErr := compiler.Compile(parsed.Value, featureFlags) require.Nil(t, cErr) + _, optimized, cErr := compiler.CompileWithOptions(parsed.Value, featureFlags, compiler.Options{Optimize: true}) + require.Nil(t, cErr) hasFocused := slices.ContainsFunc(specs.TestCases, func(tc specs_format.TestCase) bool { return tc.Focus @@ -91,9 +96,12 @@ func runScriptSpec(t *testing.T, specs specs_format.Specs, src string) { balances := specs_format.MergeBalances(specs.Balances, tc.Balances) - machine := vm.NewVm(program) store := scriptStore(balances, specs.Meta, tc.Meta) - res, execErr := vm.Exec(context.Background(), machine, &vars, store) + res, execErr := vm.Exec(context.Background(), vm.NewVm(program), &vars, store) + + optimizedRes, optimizedErr := vm.Exec(context.Background(), vm.NewVm(optimized), &vars, store) + require.Equal(t, execErr, optimizedErr, "case %q: the optimized program must fail as the plain one does", tc.It) + require.Equal(t, res, optimizedRes, "case %q: the optimized program must produce what the plain one does", tc.It) if tc.ExpectMissingFunds { require.IsType(t, vm.MissingFundsError{}, execErr, "case %q", tc.It) diff --git a/internal/compiler/verify_corpus_test.go b/internal/compiler/verify_corpus_test.go index 3544ff6a..c7b7bed2 100644 --- a/internal/compiler/verify_corpus_test.go +++ b/internal/compiler/verify_corpus_test.go @@ -1,6 +1,7 @@ package compiler_test import ( + "bytes" "encoding/json" "path/filepath" "slices" @@ -17,7 +18,8 @@ import ( // Compile does not call vm.Verify — the VM assumes its own compiler's output is // well formed, and paying for a full static pass on every compile would make // that assumption cost something. This test is what earns the assumption: every -// script in the corpus must produce bytecode the verifier accepts. +// script in the corpus must produce bytecode the verifier accepts, plain and +// optimized alike, and that bytecode must survive its encoding. func TestCompiledCorpusPassesVerify(t *testing.T) { rawSpecs, err := specs_format.ReadSpecsFiles([]string{scriptsFolder}) require.NoError(t, err) @@ -42,10 +44,19 @@ func TestCompiledCorpusPassesVerify(t *testing.T) { parsed := parser.Parse(rawSpec.NumscriptContent) require.Empty(t, parsed.Errors) - _, program, cErr := compiler.Compile(parsed.Value, featureFlags) - require.Nil(t, cErr) + for name, opts := range map[string]compiler.Options{"plain": {}, "optimized": {Optimize: true}} { + t.Run(name, func(t *testing.T) { + _, program, cErr := compiler.CompileWithOptions(parsed.Value, featureFlags, opts) + require.Nil(t, cErr) - require.NoError(t, vm.Verify(program)) + require.NoError(t, vm.Verify(program)) + + encoded := program.Encode() + decoded, err := vm.DecodeProgram(encoded) + require.NoError(t, err) + require.True(t, bytes.Equal(encoded, decoded.Encode()), "Encode/Decode/Encode is not byte-stable") + }) + } }) } } diff --git a/internal/difftest/compare.go b/internal/difftest/compare.go index c5722d71..4851c294 100644 --- a/internal/difftest/compare.go +++ b/internal/difftest/compare.go @@ -2,6 +2,7 @@ package difftest import ( "fmt" + "maps" "math/big" ) @@ -41,6 +42,61 @@ func CompareEngines(vmRes, newRes SideResult) Verdict { return compare(vmRes, newRes, "vm", "new interpreter", false) } +// CompareIdentical is the comparison with no normalization: the optimized +// program against the plain one. Same compiler, same engine, so every stage's +// error text, the posting list in its order and the metadata must match to the +// letter, unless the plain program hit a capacity bound the optimized one does +// not. +func CompareIdentical(aRes, bRes SideResult, aLabel, bLabel string) Verdict { + if aRes.InternalErr != "" { + return mismatch("%s reported an internal error: %s", aLabel, aRes.InternalErr) + } + if bRes.InternalErr != "" { + return mismatch("%s reported an internal error: %s", bLabel, bRes.InternalErr) + } + // the passes may lift a capacity bound of the b-side (fewer live registers, a + // shorter stream), never impose one: the compiler falls back to the plain + // stream when the optimized one hits a bound + if (bRes.RegisterOverflow || bRes.ProgramTooLarge) && aRes.CompileErr == "" { + return tolerated("capacity bound lifted by optimization") + } + for _, stage := range []struct { + name string + a, b string + }{ + {"compileErr", aRes.CompileErr, bRes.CompileErr}, + {"resolveErr", aRes.ResolveErr, bRes.ResolveErr}, + {"runErr", aRes.RunErr, bRes.RunErr}, + } { + if stage.a != stage.b { + return mismatch("%s differs: %s=%q, %s=%q", stage.name, aLabel, stage.a, bLabel, stage.b) + } + } + if aRes.Failed() { + return ok() + } + + if len(aRes.Postings) != len(bRes.Postings) { + return mismatch("posting count differs: %s has %d, %s has %d\n%s: %+v\n%s: %+v", + aLabel, len(aRes.Postings), bLabel, len(bRes.Postings), aLabel, aRes.Postings, bLabel, bRes.Postings) + } + for i := range aRes.Postings { + a, b := aRes.Postings[i], bRes.Postings[i] + if a.Source != b.Source || a.Destination != b.Destination || a.Asset != b.Asset || a.Color != b.Color || a.Amount.Cmp(b.Amount) != 0 { + return mismatch("posting %d differs: %s=%+v, %s=%+v\n%s: %+v\n%s: %+v", + i, aLabel, a, bLabel, b, aLabel, aRes.Postings, bLabel, bRes.Postings) + } + } + + if !maps.Equal(aRes.TxMeta, bRes.TxMeta) { + return mismatch("tx metadata differs\n%s: %v\n%s: %v", aLabel, aRes.TxMeta, bLabel, bRes.TxMeta) + } + if !maps.Equal(aRes.AccountMeta, bRes.AccountMeta) { + return mismatch("account metadata differs\n%s: %v\n%s: %v", aLabel, aRes.AccountMeta, bLabel, bRes.AccountMeta) + } + return ok() +} + func compare(aRes, bRes SideResult, aLabel, bLabel string, oracle bool) Verdict { // Checked before anything else, and symmetrically: an engine breaking its // own contract is never an expected outcome, and every tolerance below is diff --git a/internal/difftest/compare_identical_test.go b/internal/difftest/compare_identical_test.go new file mode 100644 index 00000000..0ddb3d19 --- /dev/null +++ b/internal/difftest/compare_identical_test.go @@ -0,0 +1,24 @@ +package difftest + +import "testing" + +func TestCompareIdenticalToleratesOnlyLiftedCapacityBounds(t *testing.T) { + overflow := SideResult{CompileErr: "register bank overflow: ...", RegisterOverflow: true} + tooLarge := SideResult{CompileErr: "program too large: ...", ProgramTooLarge: true} + compiled := SideResult{} + + for _, plain := range []SideResult{overflow, tooLarge} { + if v := CompareIdentical(compiled, plain, "optimized vm", "vm"); v != tolerated("capacity bound lifted by optimization") { + t.Errorf("lifted bound %q: got %+v", plain.CompileErr, v) + } + } + + // imposing a bound, or trading one for another, is a mismatch: the compiler + // falls back to the plain stream when the optimized one hits a bound + if v := CompareIdentical(overflow, compiled, "optimized vm", "vm"); !v.Mismatch { + t.Errorf("imposed bound: got %+v", v) + } + if v := CompareIdentical(overflow, tooLarge, "optimized vm", "vm"); !v.Mismatch { + t.Errorf("traded bound: got %+v", v) + } +} diff --git a/internal/difftest/difftest.go b/internal/difftest/difftest.go index 189bd5cc..4eec0e38 100644 --- a/internal/difftest/difftest.go +++ b/internal/difftest/difftest.go @@ -21,6 +21,9 @@ type Case struct { New SideResult Oracle SideResult VM SideResult + // VMOpt is the vm on the program the compiler's peephole passes produce + // from the same script. + VMOpt SideResult // OracleVsNew compares this repo's interpreter against the vendored // legacy machine, which is the only independent ground truth available: @@ -38,6 +41,12 @@ type Case struct { OracleVsNew Verdict OracleVsVM Verdict NewVsVM Verdict + + // VMVsVMOpt compares the optimized program against the plain one with + // CompareIdentical: the passes promise the same postings, in the same order, + // and the same errors. The only tolerance is a capacity bound of the plain + // program that the optimized one does not hit. + VMVsVMOpt Verdict } // RunOne generates one program from rng, runs it against all three engines, @@ -53,6 +62,7 @@ func RunOne(ctx context.Context, rng *rand.Rand) Case { newRes := runNew(ctx, g.Script, g.Vars, g.Balances, g.Metadata, g.Flags) vmRes := runVM(ctx, g.Script, g.Vars, g.Balances, g.Metadata, g.Flags) + vmOptRes := runVMOptimized(ctx, g.Script, g.Vars, g.Balances, g.Metadata, g.Flags) c := Case{ Script: g.Script, @@ -60,8 +70,10 @@ func RunOne(ctx context.Context, rng *rand.Rand) Case { Shape: g.Shape, New: newRes, VM: vmRes, + VMOpt: vmOptRes, - NewVsVM: CompareEngines(vmRes, newRes), + NewVsVM: CompareEngines(vmRes, newRes), + VMVsVMOpt: CompareIdentical(vmOptRes, vmRes, "optimized vm", "vm"), } if g.OracleCompatible { @@ -76,8 +88,8 @@ func RunOne(ctx context.Context, rng *rand.Rand) Case { return c } -// Legs returns the three pairwise verdicts with stable names, for callers -// that report per leg. +// Legs returns the pairwise verdicts with stable names, for callers that +// report per leg. func (c Case) Legs() []struct { Name string Verdict Verdict @@ -89,11 +101,12 @@ func (c Case) Legs() []struct { {"oracle vs new", c.OracleVsNew}, {"oracle vs vm", c.OracleVsVM}, {"new vs vm", c.NewVsVM}, + {"vm vs optimized vm", c.VMVsVMOpt}, } } -// AnyMismatch reports whether any of the three pairwise verdicts found a +// AnyMismatch reports whether any of the pairwise verdicts found a // divergence. func (c Case) AnyMismatch() bool { - return c.OracleVsNew.Mismatch || c.OracleVsVM.Mismatch || c.NewVsVM.Mismatch + return c.OracleVsNew.Mismatch || c.OracleVsVM.Mismatch || c.NewVsVM.Mismatch || c.VMVsVMOpt.Mismatch } diff --git a/internal/difftest/numscript_only_shapes_test.go b/internal/difftest/numscript_only_shapes_test.go index d86c89dc..94e2c6bf 100644 --- a/internal/difftest/numscript_only_shapes_test.go +++ b/internal/difftest/numscript_only_shapes_test.go @@ -316,10 +316,14 @@ send $m ( ctx := context.Background() newRes := runNew(ctx, tc.script, tc.vars, tc.balances, tc.metadata, tc.flags) vmRes := runVM(ctx, tc.script, tc.vars, tc.balances, tc.metadata, tc.flags) + vmOptRes := runVMOptimized(ctx, tc.script, tc.vars, tc.balances, tc.metadata, tc.flags) if v := CompareEngines(vmRes, newRes); v.Mismatch { t.Fatalf("vm mismatch: %s\nvm: %+v\nnew: %+v", v.Reason, vmRes, newRes) } + if v := CompareIdentical(vmOptRes, vmRes, "optimized vm", "vm"); v.Mismatch { + t.Fatalf("optimized vm mismatch: %s\noptimized: %+v\nvm: %+v", v.Reason, vmOptRes, vmRes) + } if tc.bothReject { if !newRes.Failed() || !vmRes.Failed() { t.Fatalf("expected both engines to reject\nnew: %+v\nvm: %+v", newRes, vmRes) diff --git a/internal/difftest/run_vm.go b/internal/difftest/run_vm.go index 07116619..ede937e8 100644 --- a/internal/difftest/run_vm.go +++ b/internal/difftest/run_vm.go @@ -42,11 +42,22 @@ func (s vmStore) GetMetadata(_ context.Context, account, scope, key string) (str } func runVM(ctx context.Context, script string, vars map[string]string, balances map[gen.BalanceKey]*big.Int, metadata map[gen.MetaKey]string, featureFlags []string) SideResult { + return runVMWith(ctx, script, vars, balances, metadata, featureFlags, false) +} + +// runVMOptimized is runVM on the program the compiler's peephole passes +// produce. Same compiler and same engine as runVM, so its result must match +// runVM's to the letter: see CompareIdentical. +func runVMOptimized(ctx context.Context, script string, vars map[string]string, balances map[gen.BalanceKey]*big.Int, metadata map[gen.MetaKey]string, featureFlags []string) SideResult { + return runVMWith(ctx, script, vars, balances, metadata, featureFlags, true) +} + +func runVMWith(ctx context.Context, script string, vars map[string]string, balances map[gen.BalanceKey]*big.Int, metadata map[gen.MetaKey]string, featureFlags []string, optimize bool) SideResult { flagSet := make(map[string]struct{}, len(featureFlags)) for _, f := range featureFlags { flagSet[f] = struct{}{} } - varsEncoder, program, err := numscript.CompileWithFeatureFlags(script, flagSet) + varsEncoder, program, err := numscript.CompileWithOptions(script, numscript.CompileOptions{FeatureFlags: flagSet, Optimize: optimize}) if err != nil { return SideResult{ CompileErr: err.Error(), diff --git a/internal/ir/peephole.go b/internal/ir/peephole.go new file mode 100644 index 00000000..7533ec97 --- /dev/null +++ b/internal/ir/peephole.go @@ -0,0 +1,50 @@ +package ir + +// A Peephole rewrites an instruction stream and returns the result, or nil when +// it found nothing to change. A pass never mutates its input. +// +// Every pass must preserve the program's observable behavior exactly: the same +// postings, metadata and errors, including which error surfaces when several +// could. +type Peephole func(instrs []Instr) []Instr + +// maxOptimizeSweeps bounds the fixed-point iteration. Each pass reaches a fixed +// point on its own; their composition is not proven to. +const maxOptimizeSweeps = 16 + +// DefaultPeepholes is the ordered set Optimize applies. The order matters for +// how fast a sweep converges, not for the result: the folding feeds the +// pruning, and a dropped check frees the values only it read for deadCode. +func DefaultPeepholes() []Peephole { + return []Peephole{ + constProp, + dropProvenAsserts, + pruneDeadBranches, + deadCode, + } +} + +// Optimize applies DefaultPeepholes in order, sweeping again while a sweep +// changed something, up to maxOptimizeSweeps. +// +// Optimization is opt-in: the compiler assembles the plain stream unless asked +// otherwise, and the optimized stream is behaviorally identical to it. +func Optimize(instrs []Instr) []Instr { + return optimize(instrs, DefaultPeepholes()) +} + +func optimize(instrs []Instr, passes []Peephole) []Instr { + for range maxOptimizeSweeps { + changed := false + for _, pass := range passes { + if next := pass(instrs); next != nil { + instrs = next + changed = true + } + } + if !changed { + break + } + } + return instrs +} diff --git a/internal/ir/peephole_const_prop.go b/internal/ir/peephole_const_prop.go new file mode 100644 index 00000000..c766cf05 --- /dev/null +++ b/internal/ir/peephole_const_prop.go @@ -0,0 +1,299 @@ +package ir + +import ( + "maps" + "math/big" + "slices" +) + +// constProp replaces a pure op whose result is the same int, str or bool on +// every path reaching it with a load of that value. Portion results are not +// materialized, there is no portion load, but they are still tracked for the +// ops that read them. +func constProp(instrs []Instr) []Instr { + var out []Instr + walkConsts(instrs, func(i int, env constEnv) { + if env == nil { + return + } + in := instrs[i] + switch in.(type) { + case BinaryOp, UnaryOp: + default: + return + } + v, ok := evalPure(in, env) + if !ok { + return + } + load, ok := loadOf(in.dests()[0], v) + if !ok { + return + } + if out == nil { + out = slices.Clone(instrs) + } + out[i] = load + }) + return out +} + +// A constEnv maps a register to its value: a *big.Int, string, bool or +// *big.Rat, never mutated once stored. +type constEnv map[Reg]any + +// walkConsts calls visit with the registers known before each instruction, or +// nil when no path reaches it. env is only valid during the call. +// +// Jumps only go forward, so every edge into a label is known when the walk +// reaches it and one pass is exact. +func walkConsts(instrs []Instr, visit func(i int, env constEnv)) { + incoming := map[Label]constEnv{} + addEdge := func(l Label, env constEnv) { + if prev, ok := incoming[l]; ok { + incoming[l] = meetEnv(prev, env) + } else { + incoming[l] = env + } + } + + env := constEnv{} + for i, in := range instrs { + if l, ok := in.(LabelMarker); ok { + if inc, ok := incoming[l.Label]; ok { + if env == nil { + env = inc + } else { + env = meetEnv(env, inc) + } + } + continue + } + visit(i, env) + if env == nil { + continue + } + + switch in := in.(type) { + case Jmp: + addEdge(in.Target, env) + env = nil + + case JmpIfFalse: + env = branch(env, in.Cond, false, in.Target, addEdge) + + case JmpIfTrue: + env = branch(env, in.Cond, true, in.Target, addEdge) + + case CheckEnoughFunds: + // it fails unless got == needed + if v, ok := env[in.Needed]; ok { + env[in.Got] = v + } else if v, ok := env[in.Got]; ok { + env[in.Needed] = v + } + + default: + dests := in.dests() + if len(dests) == 0 { + continue + } + v, known := evalPure(in, env) + for _, d := range dests { + delete(env, d) + } + if known { + env[dests[0]] = v + } + } + } +} + +// branch records the taken edge of a jump on cond == jumpsOn and returns the +// fallthrough env, nil when cond makes either edge impossible. +func branch(env constEnv, cond Reg, jumpsOn bool, target Label, addEdge func(Label, constEnv)) constEnv { + v, known := env[cond].(bool) + if !known || v == jumpsOn { + taken := maps.Clone(env) + taken[cond] = jumpsOn + addEdge(target, taken) + } + if known && v == jumpsOn { + return nil + } + env[cond] = !jumpsOn + return env +} + +func meetEnv(a, b constEnv) constEnv { + out := constEnv{} + for r, v := range a { + if w, ok := b[r]; ok && sameConst(v, w) { + out[r] = v + } + } + return out +} + +func sameConst(a, b any) bool { + switch a := a.(type) { + case *big.Int: + b, ok := b.(*big.Int) + return ok && a.Cmp(b) == 0 + case *big.Rat: + b, ok := b.(*big.Rat) + return ok && a.Cmp(b) == 0 + case string: + b, ok := b.(string) + return ok && a == b + case bool: + b, ok := b.(bool) + return ok && a == b + } + return false +} + +func loadOf(dest Reg, v any) (Instr, bool) { + switch v := v.(type) { + case *big.Int: + l := LoadInt{Dest: dest} + l.Value.Set(v) + return l, true + case string: + return LoadStr{Dest: dest, Value: v}, true + case bool: + return ConstBool{Dest: dest, Value: v}, true + } + return nil, false +} + +// evalPure is the value a load or pure op writes under env. Each case must +// compute exactly what the VM's opcode does. +func evalPure(in Instr, env constEnv) (any, bool) { + switch in := in.(type) { + case LoadInt: + return &in.Value, true + case LoadStr: + return in.Value, true + case ConstBool: + return in.Value, true + case BinaryOp: + l, lok := env[in.Left] + r, rok := env[in.Right] + if lok && rok { + return evalBinary(in.Op, l, r) + } + case UnaryOp: + if a, ok := env[in.Arg]; ok { + return evalUnary(in.Op, a) + } + } + return nil, false +} + +func evalBinary(op BinKind, l, r any) (any, bool) { + li, lInt := l.(*big.Int) + ri, rInt := r.(*big.Int) + lp, lPort := l.(*big.Rat) + rp, rPort := r.(*big.Rat) + ls, lStr := l.(string) + rs, rStr := r.(string) + ints, portions, strs := lInt && rInt, lPort && rPort, lStr && rStr + + switch op.(type) { + case OpAddInt: + if ints { + return new(big.Int).Add(li, ri), true + } + case OpSubInt: + if ints { + return new(big.Int).Sub(li, ri), true + } + case OpLtInt: + if ints { + return li.Cmp(ri) < 0, true + } + case OpEqInt: + if ints { + return li.Cmp(ri) == 0, true + } + case OpAddString: + if strs { + return ls + rs, true + } + case OpStrEq: + if strs { + return ls == rs, true + } + case OpLtPortion: + if portions { + return lp.Cmp(rp) < 0, true + } + case OpEqPortion: + if portions { + return lp.Cmp(rp) == 0, true + } + case OpAddPortion: + if portions { + return new(big.Rat).Add(lp, rp), true + } + case OpSubPortion: + if portions { + return new(big.Rat).Sub(lp, rp), true + } + case OpMulPortion: + if portions { + return new(big.Rat).Mul(lp, rp), true + } + case OpMakePortion: + // a zero denominator is a runtime error, left to the VM + if ints && ri.Sign() != 0 { + return new(big.Rat).SetFrac(li, ri), true + } + case OpMonetaryToString: + if lStr && rInt { + return ls + " " + ri.String(), true + } + } + return nil, false +} + +func evalUnary(op UnKind, a any) (any, bool) { + ai, aInt := a.(*big.Int) + ap, aPort := a.(*big.Rat) + ab, aBool := a.(bool) + + switch op.(type) { + case OpIntCopy, OpPortionCopy, OpStrCopy, OpBoolCopy: + return a, true + case OpNegInt: + if aInt { + return new(big.Int).Neg(ai), true + } + case OpIntToString: + if aInt { + return ai.String(), true + } + case OpIsZero: + if aInt { + return ai.Sign() == 0, true + } + case OpNot: + if aBool { + return !ab, true + } + case OpPortionToString: + if aPort { + return ap.String(), true + } + case OpIntToPortion: + if aInt { + return new(big.Rat).SetInt(ai), true + } + case OpPortionToInt: + if aPort { + return new(big.Int).Div(ap.Num(), ap.Denom()), true + } + } + return nil, false +} diff --git a/internal/ir/peephole_dead_code.go b/internal/ir/peephole_dead_code.go new file mode 100644 index 00000000..0c960199 --- /dev/null +++ b/internal/ir/peephole_dead_code.go @@ -0,0 +1,92 @@ +package ir + +import "maps" + +// deadCode removes a pure instruction whose dests no path reads. Store reads, +// pulls and every check stay, as they can fail or have effects. +func deadCode(instrs []Instr) []Instr { + // a mk_portion fails on a zero denominator, so it is pure only where the + // denominator is known not to be zero + safePortions := map[int]bool{} + walkConsts(instrs, func(i int, env constEnv) { + b, ok := instrs[i].(BinaryOp) + if !ok || env == nil { + return + } + if _, mk := b.Op.(OpMakePortion); mk { + if _, ok := evalPure(b, env); ok { + safePortions[i] = true + } + } + }) + + pure := func(i int) bool { + switch in := instrs[i].(type) { + case LoadInt, LoadStr, ConstBool, LoadVar, UnaryOp: + return true + case BinaryOp: + if _, mk := in.Op.(OpMakePortion); mk { + return safePortions[i] + } + return true + } + return false + } + + // backwards liveness: jumps only go forward, so a label's live set is + // known before any jump to it is reached + labelLive := map[Label]map[Reg]bool{} + live := map[Reg]bool{} + dead := map[int]bool{} + for i := len(instrs) - 1; i >= 0; i-- { + in := instrs[i] + switch j := in.(type) { + case LabelMarker: + labelLive[j.Label] = maps.Clone(live) + continue + case Jmp: + live = maps.Clone(labelLive[j.Target]) + continue + case JmpIfFalse: + maps.Copy(live, labelLive[j.Target]) + live[j.Cond] = true + continue + case JmpIfTrue: + maps.Copy(live, labelLive[j.Target]) + live[j.Cond] = true + continue + } + + dests := in.dests() + if len(dests) != 0 && pure(i) && !anyLive(live, dests) { + dead[i] = true + continue + } + for _, d := range dests { + delete(live, d) + } + for _, s := range in.sources() { + live[s] = true + } + } + + if len(dead) == 0 { + return nil + } + out := make([]Instr, 0, len(instrs)-len(dead)) + for i, in := range instrs { + if !dead[i] { + out = append(out, in) + } + } + return out +} + +func anyLive(live map[Reg]bool, regs []Reg) bool { + for _, r := range regs { + if live[r] { + return true + } + } + return false +} diff --git a/internal/ir/peephole_drop_proven_asserts.go b/internal/ir/peephole_drop_proven_asserts.go new file mode 100644 index 00000000..1e6409c6 --- /dev/null +++ b/internal/ir/peephole_drop_proven_asserts.go @@ -0,0 +1,71 @@ +package ir + +import ( + "math/big" + + "github.com/formancehq/numscript/internal/funds" +) + +// dropProvenAsserts removes a check that passes on every path reaching it. A +// check that may fail stays where it is, so the first error a run hits is +// unchanged. +func dropProvenAsserts(instrs []Instr) []Instr { + drop := map[int]bool{} + walkConsts(instrs, func(i int, env constEnv) { + if env != nil && checkPasses(instrs[i], env) { + drop[i] = true + } + }) + if len(drop) == 0 { + return nil + } + out := make([]Instr, 0, len(instrs)-len(drop)) + for i, in := range instrs { + if !drop[i] { + out = append(out, in) + } + } + return out +} + +func checkPasses(in Instr, env constEnv) bool { + intOf := func(r Reg) (*big.Int, bool) { v, ok := env[r].(*big.Int); return v, ok } + strOf := func(r Reg) (string, bool) { v, ok := env[r].(string); return v, ok } + portionOf := func(r Reg) (*big.Rat, bool) { v, ok := env[r].(*big.Rat); return v, ok } + + switch in := in.(type) { + case AssertNonNegativeAmount: + v, ok := intOf(in.Amount) + return ok && v.Sign() >= 0 + case AssertNonNegativeBalance: + v, ok := intOf(in.Balance) + return ok && v.Sign() >= 0 + case AssertNonNegativePortion: + v, ok := portionOf(in.Portion) + return ok && v.Sign() >= 0 + case AssertLeftover: + v, ok := portionOf(in.Portion) + return ok && v.Sign() >= 0 && (!in.Exact || v.Sign() == 0) + case AssertSameAsset: + l, lok := strOf(in.Left) + r, rok := strOf(in.Right) + return lok && rok && l == r + case AssertValidAccount: + v, ok := strOf(in.Account) + return ok && funds.ValidateAccount(v) + case AssertValidColor: + v, ok := strOf(in.Color) + return ok && funds.ValidateColor(v) + case AssertValidScope: + v, ok := strOf(in.Scope) + return ok && funds.ValidateScope(v) + case AssertUnscoped: + v, ok := strOf(in.Scope) + return ok && v == "" + case CheckEnoughFunds: + g, gok := intOf(in.Got) + n, nok := intOf(in.Needed) + return gok && nok && g.Cmp(n) == 0 + } + return false +} diff --git a/internal/ir/peephole_prune_dead_branches.go b/internal/ir/peephole_prune_dead_branches.go new file mode 100644 index 00000000..5617a825 --- /dev/null +++ b/internal/ir/peephole_prune_dead_branches.go @@ -0,0 +1,155 @@ +package ir + +// pruneDeadBranches removes control flow that can be decided statically: +// +// - a conditional jump on a constant bool becomes an unconditional jmp, or +// nothing; +// - the instructions between an unconditional jmp and the next label some +// reachable jump targets can never run, and go; +// - a jump whose target is the next instruction is a no-op, and goes; +// - a label no jump targets any more goes. +// +// Jumps only go forward, so one pass in program order knows at every label +// whether a reachable jump targets it: every jump to a label sits before it. +func pruneDeadBranches(instrs []Instr) []Instr { + conds := constConds(instrs) + changed := false + + // jumps on a constant condition + folded := make([]Instr, 0, len(instrs)) + for i, in := range instrs { + switch j := in.(type) { + case JmpIfFalse: + if v, ok := conds[i]; ok { + changed = true + if !v { + folded = append(folded, Jmp{Target: j.Target}) + } + continue + } + case JmpIfTrue: + if v, ok := conds[i]; ok { + changed = true + if v { + folded = append(folded, Jmp{Target: j.Target}) + } + continue + } + } + folded = append(folded, in) + } + + // unreachable instructions. Labels are all kept here: the ones left with no + // jump are dropped at the end. + live := map[Label]bool{} + reachable := true + kept := make([]Instr, 0, len(folded)) + for _, in := range folded { + if l, ok := in.(LabelMarker); ok { + if live[l.Label] { + reachable = true + } + kept = append(kept, in) + continue + } + if !reachable { + changed = true + continue + } + kept = append(kept, in) + switch j := in.(type) { + case Jmp: + live[j.Target] = true + reachable = false + case JmpIfFalse: + live[j.Target] = true + case JmpIfTrue: + live[j.Target] = true + } + } + + // jumps to the next instruction + out := make([]Instr, 0, len(kept)) + for i, in := range kept { + if target, ok := jumpTarget(in); ok && jumpsToNext(kept, i, target) { + changed = true + continue + } + out = append(out, in) + } + + // labels no jump targets + referenced := map[Label]bool{} + for _, in := range out { + if target, ok := jumpTarget(in); ok { + referenced[target] = true + } + } + n := 0 + for _, in := range out { + if l, ok := in.(LabelMarker); ok && !referenced[l.Label] { + changed = true + continue + } + out[n] = in + n++ + } + out = out[:n] + + if !changed { + return nil + } + return out +} + +func jumpTarget(in Instr) (Label, bool) { + switch j := in.(type) { + case Jmp: + return j.Target, true + case JmpIfFalse: + return j.Target, true + case JmpIfTrue: + return j.Target, true + default: + return "", false + } +} + +// jumpsToNext reports whether the jump at instrs[i] targets a label that sits +// before the next instruction, i.e. with only labels in between. +func jumpsToNext(instrs []Instr, i int, target Label) bool { + for _, in := range instrs[i+1:] { + l, ok := in.(LabelMarker) + if !ok { + return false + } + if l.Label == target { + return true + } + } + return false +} + +// constConds maps each conditional jump to the value its condition has on +// every path reaching it, when it has one. +func constConds(instrs []Instr) map[int]bool { + conds := map[int]bool{} + walkConsts(instrs, func(i int, env constEnv) { + if env == nil { + return + } + var cond Reg + switch j := instrs[i].(type) { + case JmpIfFalse: + cond = j.Cond + case JmpIfTrue: + cond = j.Cond + default: + return + } + if v, ok := env[cond].(bool); ok { + conds[i] = v + } + }) + return conds +} diff --git a/internal/ir/peephole_test.go b/internal/ir/peephole_test.go new file mode 100644 index 00000000..929be9d2 --- /dev/null +++ b/internal/ir/peephole_test.go @@ -0,0 +1,590 @@ +package ir + +import ( + "testing" + + "github.com/stretchr/testify/require" +) + +// runPass parses src, applies one pass and returns its dump, or src's own +// canonical dump and false when the pass changed nothing. Registers keep the +// numbers the parse gave them, so an instruction a pass deletes leaves a gap. +func runPass(t *testing.T, pass Peephole, src string) (string, bool) { + t.Helper() + instrs, errs := Parse(src) + require.Empty(t, errs, "IR errors: %v", errs) + require.NoError(t, Typecheck(instrs)) + + out := pass(instrs) + if out == nil { + return Dump(instrs), false + } + require.NoError(t, Typecheck(out), "the pass broke register typing") + return Dump(out), true +} + +func requirePass(t *testing.T, pass Peephole, src, want string) { + t.Helper() + got, changed := runPass(t, pass, src) + require.True(t, changed, "the pass changed nothing:\n%s", got) + require.Equal(t, want, got) +} + +func requireUnchanged(t *testing.T, pass Peephole, src string) { + t.Helper() + got, changed := runPass(t, pass, src) + require.False(t, changed, "the pass changed the program:\n%s", got) +} + +// --- constProp --- + +func TestConstProp(t *testing.T) { + testCases := []struct { + name string + src string + want string + }{ + { + name: "str_eq of equal literals", + src: ` + $r0 = "world" + $r1 = "world" + $r2 = str_eq($r0, $r1) +`, + want: ` $r0 = "world" + $r1 = "world" + $r2 = true +`, + }, + { + name: "str_eq of different literals", + src: ` + $r0 = "src" + $r1 = "world" + $r2 = str_eq($r0, $r1) +`, + want: ` $r0 = "src" + $r1 = "world" + $r2 = false +`, + }, + { + name: "eq_int and lt_int", + src: ` + $r0 = 0 + $r1 = 42 + $r2 = eq_int($r0, $r1) + $r3 = lt_int($r0, $r1) + $r4 = lt_int($r1, $r0) + $r5 = eq_int($r1, $r1) +`, + want: ` $r0 = 0 + $r1 = 42 + $r2 = false + $r3 = true + $r4 = false + $r5 = true +`, + }, + { + name: "is_zero", + src: ` + $r0 = 0 + $r1 = -3 + $r2 = is_zero($r0) + $r3 = is_zero($r1) +`, + want: ` $r0 = 0 + $r1 = -3 + $r2 = true + $r3 = false +`, + }, + } + for _, tc := range testCases { + t.Run(tc.name, func(t *testing.T) { + requirePass(t, constProp, tc.src, tc.want) + }) + } +} + +func TestConstPropLeavesNonConstants(t *testing.T) { + testCases := []struct { + name string + src string + }{ + { + name: "operand from a var", + src: ` + $r0 = load_var(0) + $r1 = "world" + $r2 = str_eq($r0, $r1) +`, + }, + { + // written twice, so its value depends on the path taken + name: "operand written on both arms of a branch", + src: ` + $r0 = load_var(0) + $r1 = 0 + $r3 = "world" + $r4 = str_eq($r0, $r3) + jmp_if_true($r4, #end) + $r1 = 1 +#end + $r2 = is_zero($r1) +`, + }, + } + for _, tc := range testCases { + t.Run(tc.name, func(t *testing.T) { + requireUnchanged(t, constProp, tc.src) + }) + } +} + +func TestConstPropFoldsThroughOps(t *testing.T) { + testCases := []struct { + name string + src string + want string + }{ + { + name: "arithmetic and copies", + src: ` + $r0 = 1 + $r1 = 2 + $r2 = $r0 + $r1 + $r3 = int_copy($r2) + $r4 = is_zero($r3) +`, + want: ` $r0 = 1 + $r1 = 2 + $r2 = 3 + $r3 = 3 + $r4 = false +`, + }, + { + // the check fails unless got == needed, so got is needed after it + name: "check_enough_funds fixes got", + src: ` + $r0 = "USD/2" + set_current_asset($r0) + $r1 = 100 + $r2 = "src" + $r3 = 0 + $r4 = pull_account(account: $r2, cap: $r1, overdraft: $r3) + check_enough_funds($r4, $r1) + $r5 = 1 + $r6 = 2 + $r7 = mk_portion($r5, $r6) + $r8 = int_to_portion($r4) + $r9 = mul_portion($r7, $r8) + $r10 = portion_to_int($r9) +`, + want: ` $r0 = "USD/2" + set_current_asset($r0) + $r1 = 100 + $r2 = "src" + $r3 = 0 + $r4 = pull_account(account: $r2, cap: $r1, overdraft: $r3) + check_enough_funds($r4, $r1) + $r5 = 1 + $r6 = 2 + $r7 = mk_portion($r5, $r6) + $r8 = int_to_portion($r4) + $r9 = mul_portion($r7, $r8) + $r10 = 50 +`, + }, + { + name: "both arms agree", + src: ` + $r0 = load_var(0) + $r1 = 7 + $r2 = "world" + $r3 = str_eq($r0, $r2) + jmp_if_true($r3, #end) + $r1 = 7 +#end + $r4 = is_zero($r1) +`, + want: ` $r0 = load_var(0) + $r1 = 7 + $r2 = "world" + $r3 = str_eq($r0, $r2) + jmp_if_true($r3, #end) + $r1 = 7 +#end + $r4 = false +`, + }, + } + for _, tc := range testCases { + t.Run(tc.name, func(t *testing.T) { + requirePass(t, constProp, tc.src, tc.want) + }) + } +} + +func TestConstPropLeavesDivisionByZero(t *testing.T) { + requireUnchanged(t, constProp, ` + $r0 = 1 + $r1 = 0 + $r2 = mk_portion($r0, $r1) + $r3 = portion_to_int($r2) +`) +} + +// --- dropProvenAsserts --- + +func TestDropProvenAsserts(t *testing.T) { + requirePass(t, dropProvenAsserts, ` + $r0 = 42 + assert_non_negative_amount($r0) + $r1 = "USD/2" + $r2 = "USD/2" + assert_same_asset($r1, $r2) + $r3 = 1 + $r4 = 2 + $r5 = mk_portion($r3, $r4) + assert_leftover($r5) +`, ` $r0 = 42 + $r1 = "USD/2" + $r2 = "USD/2" + $r3 = 1 + $r4 = 2 + $r5 = mk_portion($r3, $r4) +`) +} + +func TestDropProvenAssertsKeepsFailingAndUnknown(t *testing.T) { + testCases := []struct { + name string + src string + }{ + { + name: "fails", + src: ` + $r0 = -1 + assert_non_negative_amount($r0) +`, + }, + { + name: "unknown", + src: ` + $r0 = load_var(0) + assert_non_negative_amount($r0) +`, + }, + { + name: "mismatched assets", + src: ` + $r0 = "USD/2" + $r1 = "EUR/2" + assert_same_asset($r0, $r1) +`, + }, + } + for _, tc := range testCases { + t.Run(tc.name, func(t *testing.T) { + requireUnchanged(t, dropProvenAsserts, tc.src) + }) + } +} + +// --- deadCode --- + +func TestDeadCode(t *testing.T) { + requirePass(t, deadCode, ` + $r0 = "USD/2" + set_current_asset($r0) + $r1 = "unused" + $r2 = 1 + $r3 = 2 + $r4 = mk_portion($r2, $r3) + $r5 = load_var(0) + $r6 = 10 + $r7 = str_eq($r5, $r0) + jmp_if_true($r7, #end) + $r6 = 20 +#end + $r8 = int_copy($r6) + $r9 = 0 + $r10 = pull_account(account: $r5, cap: $r6, overdraft: $r9) +`, ` $r0 = "USD/2" + set_current_asset($r0) + $r5 = load_var(0) + $r6 = 10 + $r7 = str_eq($r5, $r0) + jmp_if_true($r7, #end) + $r6 = 20 +#end + $r9 = 0 + $r10 = pull_account(account: $r5, cap: $r6, overdraft: $r9) +`) +} + +func TestDeadCodeKeepsEffects(t *testing.T) { + requireUnchanged(t, deadCode, ` + $r0 = "USD/2" + set_current_asset($r0) + $r1 = load_var(0) + $r2 = load_var(1) + $r3 = mk_portion($r1, $r2) + $r4 = "src" + $r5 = balance($r4, $r0) + $r6 = 0 + $r7 = pull_account(account: $r4, cap: $r1, overdraft: $r6) +`) +} + +// --- pruneDeadBranches --- + +func TestPruneDeadBranches(t *testing.T) { + testCases := []struct { + name string + src string + want string + }{ + { + name: "jmp_if_false on true falls through, the label goes", + src: ` + $r0 = true + jmp_if_false($r0, #skip) + $r1 = 1 +#skip + $r2 = 2 +`, + want: ` $r0 = true + $r1 = 1 + $r2 = 2 +`, + }, + { + name: "jmp_if_false on false skips the arm", + src: ` + $r0 = false + jmp_if_false($r0, #skip) + $r1 = 1 +#skip + $r2 = 2 +`, + want: ` $r0 = false + $r2 = 2 +`, + }, + { + name: "jmp_if_true on true skips the arm", + src: ` + $r0 = true + jmp_if_true($r0, #skip) + $r1 = 1 +#skip + $r2 = 2 +`, + want: ` $r0 = true + $r2 = 2 +`, + }, + { + name: "jmp_if_true on false falls through", + src: ` + $r0 = false + jmp_if_true($r0, #skip) + $r1 = 1 +#skip + $r2 = 2 +`, + want: ` $r0 = false + $r1 = 1 + $r2 = 2 +`, + }, + { + name: "the world diamond collapses to its unbounded arm", + src: ` + $r0 = "USD/2" + set_current_asset($r0) + $r1 = "world" + $r2 = 10 + $r3 = 0 + $r4 = true + jmp_if_false($r4, #not_world) + $r5 = pull_account(account: $r1, cap: $r2) + jmp(#pull_end) +#not_world + $r5 = pull_account(account: $r1, cap: $r2, overdraft: $r3) +#pull_end + check_enough_funds($r5, $r2) +`, + want: ` $r0 = "USD/2" + set_current_asset($r0) + $r1 = "world" + $r2 = 10 + $r3 = 0 + $r4 = true + $r5 = pull_account(account: $r1, cap: $r2) + check_enough_funds($r5, $r2) +`, + }, + { + // the jmp(#b) is itself unreachable, so #b stays unreferenced and goes + // with it, while #a's code is reachable through the jump to it + name: "code after an unconditional jmp goes, up to a live label", + src: ` + $r0 = true + jmp_if_true($r0, #a) + $r1 = 1 + jmp(#b) +#a + $r2 = 2 +#b + $r3 = 3 +`, + want: ` $r0 = true + $r2 = 2 + $r3 = 3 +`, + }, + { + name: "a jump to the next instruction goes", + src: ` + $r0 = load_var(0) + $r1 = is_zero($r0) + jmp_if_true($r1, #next) +#next + $r2 = 1 +`, + want: ` $r0 = load_var(0) + $r1 = is_zero($r0) + $r2 = 1 +`, + }, + { + name: "a label with no jump goes", + src: ` + $r0 = 1 +#orphan + $r1 = 2 +`, + want: ` $r0 = 1 + $r1 = 2 +`, + }, + } + for _, tc := range testCases { + t.Run(tc.name, func(t *testing.T) { + requirePass(t, pruneDeadBranches, tc.src, tc.want) + }) + } +} + +func TestPruneDeadBranchesLeavesLiveControlFlow(t *testing.T) { + testCases := []struct { + name string + src string + }{ + { + name: "jump on a run-time condition", + src: ` + $r0 = load_var(0) + $r1 = is_zero($r0) + jmp_if_true($r1, #end) + $r2 = 1 +#end + $r3 = 2 +`, + }, + { + name: "both arms reachable", + src: ` + $r0 = load_var(0) + $r1 = is_zero($r0) + jmp_if_false($r1, #else) + $r2 = 1 + jmp(#end) +#else + $r2 = 2 +#end + $r3 = int_copy($r2) +`, + }, + } + for _, tc := range testCases { + t.Run(tc.name, func(t *testing.T) { + requireUnchanged(t, pruneDeadBranches, tc.src) + }) + } +} + +// --- Optimize --- + +// The compiler's output for `send [USD/2 42] (source = @world destination = +// @dest)`, minus the amount clamp: the passes have to compose for the +// literal @world to end up as a single unbounded pull. +func TestOptimizeCollapsesWorldDiamond(t *testing.T) { + src := ` + $r0 = "world" + $r1 = "USD/2" + $r2 = 42 + assert_non_negative_amount($r2) + set_current_asset($r1) + $r3 = "world" + $r4 = 0 + $r5 = str_eq($r3, $r0) + jmp_if_false($r5, #not_world) + $r6 = pull_account(account: $r3, cap: $r2) + jmp(#pull_end) +#not_world + $r6 = pull_account(account: $r3, cap: $r2, overdraft: $r4) +#pull_end + check_enough_funds($r6, $r2) + $r7 = "dest" + send_to_account(account: $r7) +` + instrs, errs := Parse(src) + require.Empty(t, errs) + require.NoError(t, Typecheck(instrs)) + + out := Optimize(instrs) + require.NoError(t, Typecheck(out)) + require.Equal(t, ` $r1 = "USD/2" + $r2 = 42 + set_current_asset($r1) + $r3 = "world" + $r6 = pull_account(account: $r3, cap: $r2) + check_enough_funds($r6, $r2) + $r7 = "dest" + send_to_account(account: $r7) +`, Dump(out)) +} + +func TestOptimizeReturnsInputWhenNothingApplies(t *testing.T) { + src := ` + $r0 = "USD/2" + $r1 = load_var(0) + set_current_asset($r0) + $r2 = load_var(0) + $r3 = 0 + $r4 = pull_account(account: $r2, cap: $r1, overdraft: $r3) + check_enough_funds($r4, $r1) + $r5 = "dest" + send_to_account(account: $r5) +` + instrs, errs := Parse(src) + require.Empty(t, errs) + require.Equal(t, Dump(instrs), Dump(Optimize(instrs))) +} + +// A pass whose output is never a fixed point of the next must still terminate. +func TestOptimizeBoundsTheSweeps(t *testing.T) { + calls := 0 + flip := func(instrs []Instr) []Instr { + calls++ + return append([]Instr{}, instrs...) + } + optimize(nil, []Peephole{flip}) + require.Equal(t, maxOptimizeSweeps, calls) +} diff --git a/numscript.go b/numscript.go index db3b5c64..6c908060 100644 --- a/numscript.go +++ b/numscript.go @@ -255,15 +255,34 @@ func (p ParseResult) Compile() (VarsEncoder, CompiledProgram, error) { // CompileWithFeatureFlags compiles the program, rejecting any construct gated // behind an experimental feature flag that isn't in featureFlags. func (p ParseResult) CompileWithFeatureFlags(featureFlags map[string]struct{}) (VarsEncoder, CompiledProgram, error) { + return p.CompileWithOptions(CompileOptions{FeatureFlags: featureFlags}) +} + +// CompileOptions configures CompileWithOptions. The zero value is what Compile +// uses. +type CompileOptions struct { + // FeatureFlags is the set of experimental features the host allows, as in + // CompileWithFeatureFlags. + FeatureFlags map[string]struct{} + + // Optimize runs the compiler's peephole passes over the program before it + // is assembled. The optimized program produces the same postings, metadata + // and errors as the plain one; it executes fewer instructions and allocates + // less. Off by default. + Optimize bool +} + +func (p ParseResult) CompileWithOptions(opts CompileOptions) (VarsEncoder, CompiledProgram, error) { if len(p.parseResult.Errors) != 0 { return VarsEncoder{}, CompiledProgram{}, p.parseResult.Errors[0] } + featureFlags := opts.FeatureFlags if featureFlags == nil { featureFlags = make(map[string]struct{}) } - return compiler.Compile(p.parseResult.Value, featureFlags) + return compiler.CompileWithOptions(p.parseResult.Value, featureFlags, compiler.Options{Optimize: opts.Optimize}) } func Compile(source string) (VarsEncoder, CompiledProgram, error) { @@ -274,6 +293,10 @@ func CompileWithFeatureFlags(source string, featureFlags map[string]struct{}) (V return Parse(source).CompileWithFeatureFlags(featureFlags) } +func CompileWithOptions(source string, opts CompileOptions) (VarsEncoder, CompiledProgram, error) { + return Parse(source).CompileWithOptions(opts) +} + var DecodeCompiledProgram = vm.DecodeProgram // BytecodeVersion is the version of the bytecode wire format a compiled