Add Feature-activation Matrix runner (gated) to validate FeatureSet changes

[TanmayDhobale]

Add a gated “Feature Matrix” to Mollusk to run the same fixtures under a baseline and one or more feature-enabled variants, compare results for parity, and report compute unit deltas. It’s off by default (feature flag), includes a simple CLI command (mollusk matrix) with --variant name:b58,b58 syntax, and can emit Markdown/JSON reports with thresholds to flag regressions. This helps validate feature activations quickly without a validator and keeps default users unaffected.

[TanmayDhobale]

@joncinque hey should i add a gated ‘Feature Matrix’ to Mollusk (runs fixtures under baseline vs feature variants, compares parity, reports CU deltas via CLI), or would you prefer a different approach?

[buffalojoec]

Interesting! Yeah, if this is useful to you, you're welcome to make a PR!

Can you maybe show a more fleshed-out example of the CLI command in a code snippet? I think I understand what you're going after, but I might be able to help guide you in the implementation a bit.

[TanmayDhobale]

example showing the full workflow:

Feature Matrix CLI Example

# Example: Testing a new precompile feature against current mainnet behavior
# This validates that the new feature doesn't break existing program behavior

mollusk matrix \
  ./programs/token/src/elf/token.so \
  ./harness/tests \
  11111111111111111111111111111112 \
  --baseline-slot 250000000 \
  --variant "precompile-on:5G9...,9sV..." \
  --variant "both-features:5G9...,9sV...,11111111111111111111111111111112" \
  --max-cu-delta-abs 1000 \
  --max-cu-delta-percent 5.0 \
  --out-dir ./feature-validation-reports \
  --markdown \
  --json

What This Does

1. Baseline: Runs all fixtures against FeatureSet from slot 250M (current mainnet)
2. Variants: Runs same fixtures with:
   • precompile-on: New precompile feature enabled
   • both-features: New feature + existing test features
3. Validation: Checks that:
   • Program results are identical (no behavior changes)
   • Return data matches (no output changes)
   • Account states are equivalent (no side effects)
   • CU usage is within thresholds (performance impact bounded)

Output Example

# Feature Matrix Report

Baseline: slot 250000000

## Result Parity
| Variant | Pass | ProgramResult | ReturnData | Accounts |
|---------|------|---------------|------------|----------|
| precompile-on | PASS | OK | OK | OK |
| both-features | PASS | OK | OK | OK |

## Compute Units
Baseline CU: 1500

| Variant | CU | Delta | % | Pass |
|---------|----|-------|---|------|
| precompile-on | 1520 | +20 | +1.3% | PASS |
| both-features | 1550 | +50 | +3.3% | PASS |

[TanmayDhobale]

@buffalojoec what do u think

[buffalojoec]

I think this looks pretty good! I just want to bikeshed the arguments a bit.

For --baseline-slot, I don't see why we wouldn't just call it --slot, but the value for Slot is already part of the provided SlotContext in the fixture. I think that value should come from the fixture, not from some CLI override, right? So we shouldn't include this option.

If you really want this thing to be a true matrix, then you should instead just provide a list of feature IDs and let the runner just flip them on and off in a Cartesian product, similar to test_matrix. How about something like this?

mollusk matrix \
  ./programs/token/src/elf/token.so \
  ./harness/tests \
  --feature 2yDdYzg56LgRBzX1UeMNcrsXphJ4yTZe4cCvEoGzbXDc \
  --feature 5GDFSoKFJWWEkttfwTwWXfehKH7DGiL9v8bNChjJ5Q5g \
  --feature JC3tpDPKRUqdc3iDzqxeYWRoo1qWCp1RRpaej4sdKbkH \
  --feature 7BfGJBKBfZV4Q9GGzdbhBzPLA4brmLHNVGywttr4EDBR \
  --max-cu-delta-abs 1000 \
  --max-cu-delta-percent 5.0 \
  --out-dir ./feature-validation-reports \
  --markdown \
  --json

A comma or space-separate list/string is also reasonable if the multi-flag approach doesn't look all that great. What do you think?

[TanmayDhobale]

I think this looks pretty good! I just want to bikeshed the arguments a bit.

For --baseline-slot, I don't see why we wouldn't just call it --slot, but the value for Slot is already part of the provided SlotContext in the fixture. I think that value should come from the fixture, not from some CLI override, right? So we shouldn't include this option.

If you really want this thing to be a true matrix, then you should instead just provide a list of feature IDs and let the runner just flip them on and off in a Cartesian product, similar to test_matrix. How about something like this?

mollusk matrix \
  ./programs/token/src/elf/token.so \
  ./harness/tests \
  --feature 2yDdYzg56LgRBzX1UeMNcrsXphJ4yTZe4cCvEoGzbXDc \
  --feature 5GDFSoKFJWWEkttfwTwWXfehKH7DGiL9v8bNChjJ5Q5g \
  --feature JC3tpDPKRUqdc3iDzqxeYWRoo1qWCp1RRpaej4sdKbkH \
  --feature 7BfGJBKBfZV4Q9GGzdbhBzPLA4brmLHNVGywttr4EDBR \
  --max-cu-delta-abs 1000 \
  --max-cu-delta-percent 5.0 \
  --out-dir ./feature-validation-reports \
  --markdown \
  --json

A comma or space-separate list/string is also reasonable if the multi-flag approach doesn't look all that great. What do you think?

Totally agree on both points