Testing drift and cost
MEMORYFIELD
Project memory for AI tools
I built deterministic evaluation gates and hash-checked context snapshot checks that act like a bouncer checking IDs before an LLM judge ever runs, then modeled token savings from explicit workload assumptions.
確かめる
試した範囲と、その先の予測は、ちゃんと分けて見せたい。
The Bouncer Before the Judge.
Deterministic invariant gates → hash-snapshot watchdog → 1 / 0 binary verdict
Pin the claim
Bind every rule and claim to its input fixture.
Check before LLM
Block broken structure in milliseconds.
1 Pass / 0 Halt
Every pass or refusal seals an inspectable receipt.
STEP 1 / 3 · Pin SnapshotELI5: Seal the RuleML / Systems: Each carried rule and benchmark claim is bound to its target file path, token budget, and SHA-256 content digest.Plain English (ELI5): Stamp every rule with an ID badge before the session starts so the watchdog knows exactly what clean data looks like.
Verification schematic. Select a step to inspect the mechanism. Recorded test results and receipts follow below.
Separate recorded envelopes: threat matrix 91fe6f47 and controlled drift run fa289286.Stress-testing AST compaction across 18 simulated domains and 20 real repositories
A compiler that only works on one codebase is a brittle demo. I first generated an 18-domain corpus matrix spanning 198 nodes and 2,628 dependency edges, and then ran mf benchmark --fleet across 20 local repositories, including several checkouts of this portfolio (3,383 Markdown files and 5,558 resolved links, receipt 6f40789f).
On my two largest gamedev repositories, archiving older session notes out of the active prompt (mf compact --rollover --apply, receipt 47304b7c) trimmed Tripletap-Game from 145 active notes to 6 (93.1% reduction, saving 69,827 active tokens) and Pineapple MTN from 92 active notes to 6 (93.5% reduction, saving 34,262 active tokens) for 104,089 active-prompt tokens saved with 1-hop hash links back to the archive.
Across all 18 simulated domains, the compiler shrank 502,110 estimated raw tokens to 280,185 (a 44.20% reduction, receipt 91fe6f47). Across the 20-repository fleet, it shrank 6,285,760 estimated tokens to 2,293,995 (a 63.50% reduction, receipt 6f40789f). Across a 25-task (100 verbatim ground-truth facts) recall evaluation (receipt 728dc1df), AST compaction preserved 100.0% of hard facts (100 of 100) versus 19.0% (19 of 100) under naive tail truncation.
- SIMULATED DOMAINS
- 18
Constructed corpus matrix; 198 nodes and 2,628 edges (plus 20 real fleet repos, receipt 6f40789f).
- ESTIMATED TOKEN REDUCTION
- 44.20%
502,110 to 280,185 estimated tokens in this matrix (63.50% across the 20-repo fleet, receipt 6f40789f).
The Bouncer Before the Judge: deterministic gates precede LLM evals
Asking an LLM to grade whether a file link is broken or a context snapshot was tampered with is slow, expensive, and probabilistic. I put deterministic invariant gates ahead of any model call: like a bouncer checking IDs at the door, they block broken graph links, token overflows, and case collisions in milliseconds with an inspectable receipt.
I refused to use an LLM-as-a-judge for structural invariants: a broken reference, token overflow or case collision should fail deterministically in milliseconds, not pass by majority vote.
File-hash watchdog: curated attacks in one run, random drift in another
Each carried rule travels inside a hash-checked context snapshot watched by a live watchdog. In the constructed threat-matrix run, deterministic invariant gates intercepted 25 of 25 curated prompt and reference attacks; coverage on that suite, not on every future prompt injection.
In a separate 100-draw Sobol quasi-random perturbation run, the watchdog caught all 30 injected mutations while triggering 0 false flags across 70 clean workspace draws.
SHA-256 hashed receipts let any session verify whether a stored record or benchmark fixture was altered on disk. Their scope is file and snapshot integrity; hardware attestation and third-party lab audits require separate verification.
- CURATED ATTACKS CAUGHT
- 25 / 25
Constructed threat-matrix test envelope.
- FALSE FLAGS ON CLEAN DRAWS
- 0 / 70
One controlled Sobol run; 30 adversarial draws were also caught.
Modeling token ROI from explicit 22-day assumptions
I keep measured token compaction strictly separate from projected dollar savings. The default model assumes 5 developers, 100 agent turns per day, 22 working days per month, and a $3 per million input-token rate with provider caching off.
Developer hours recovered, hourly engineering rates, and platform fees live as adjustable sliders in the calculator below rather than hard claims from the write-loss benchmark.
Under that 22-day baseline using the 101-file workspace compaction ratio, monthly raw input cost drops from $4,789 to $1,602 after compilation (excluding output tokens, tool calls, and hosting).
Open the interactive simulator below to stress-test how team size, turn volume, or provider prompt caching shifts the monthly and annual projections.
- MODELED MONTHLY INPUT SAVINGS
- $3,187
22-day baseline, caching off; uses golden workspace token estimates, not the 18-domain matrix.
- MODELED ANNUAL INPUT SAVINGS
- $38,248
The monthly projection multiplied by 12.
Model input-token costs and operating assumptions
Model the cost of carrying context
The token counts come from one recorded compiler benchmark. Rates, cache behavior, recovered time and platform cost are editable assumptions. This model does not measure customer savings.
$3,187 / mo modeled input savings (66.56% reduction) · 101-file workspace (145,119 → 48,534 tokens) across 5 developers × 100 turns/day × 22 working days at $3/M input tokens ($4,789 raw → $1,602 compiled). Receipt 58b0499d
The recorded baseline uses $3. Edit this assumption to compare another rate; this is not a current price feed.
Off by default. Enabling it applies the same hit and rate fractions to both raw and compiled inputs; the benchmark does not establish a cache hit rate.
Monthly input-token savings
$3,187$4,789 raw → $1,602 compiled
66.56% modeled reduction · $38,248 / year
Assumed recovered time
Enter hoursEnter recovered hours and an hourly value to model time savings. Write-loss tests do not measure recovered engineering hours.
Modeled net return
Enter costIncludes token savings and assumed time value, less $0 / month platform cost.
Modeled payback
UnavailableBased on 22 working days per month. This is a scenario, not an observed deployment result.
ABOUT
I’m Matt Stenquist, a Product Design Engineer and former professional snowboarder. I started coding at nine, shipped modded hardware and games by 16, and turned pro by 22. A few broken bones and a gold tooth later, I design and code consumer apps, Figma component systems, and AI developer tools from scratch.
LEARN MORE