Skip to content

Implies endpoint in booster #3846

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 22 commits into from
May 8, 2024
Merged

Implies endpoint in booster #3846

merged 22 commits into from
May 8, 2024

Conversation

goodlyrottenapple
Copy link
Contributor

@goodlyrottenapple goodlyrottenapple commented May 2, 2024

Closes #3778 #3601

This PR introduces a simple check implication implementation, which uses the matching algorithm to try to find a substitution or fail to unify the given configurations.

If matching the RHS configuration to the LHS succeeds and produces a substitution σ, we filter out any predicates in the consequent which appear in the antecedent and then check if all predicates in σ(filtered_preds) evaluate to true/#top. (Question: do we want to apply σ to the RHS preds before filtering?). With this simple algorithm, we can successfully discharge almost 85% of implication checks in KEVM and 88% in Kontrol.

Copy link
Member

@jberthold jberthold left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I realise this is not quite ready, so I am just leaving two comments on the rpc-client changes - feel free to ignore.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think we should re-visit emitting right-assoc for collections. I tried to have a look at the diff here, but it is really difficult when it gets so obscured by large sets being re-ordered between kore and booster

@goodlyrottenapple
Copy link
Contributor Author

KEVM

Test sam-implication time master-88ef647ec time (sam-implication/master-88ef647ec) time
kontrol/test-expectcalltest-testexpectstaticcall-0-spec.k 46.49 56.01 0.8300303517229067
kontrol/test-arithmetictest-test_wmul_weakly_increasing_positive-uint256-uint256-0-spec.k 64.28 76.51 0.8401516141680826
kontrol/test-arithmetictest-test_wdiv_rounding-uint256-uint256-0-spec.k 51.68 60.58 0.8530868273357544
mcd/flipper-tau-pass-spec.k 54.46 63.2 0.8617088607594936
bihu/forwardToHotWallet-success-1-spec.k 51.18 59.3 0.8630691399662732
mcd/cat-exhaustiveness-spec.k 97.68 111.24 0.8781014023732472
mcd/vat-mului-pass-spec.k 101.08 114.42 0.8834119909106799
erc20/ds/transfer-failure-1-a-spec.k 70.22 78.9 0.8899873257287705
mcd/vat-subui-pass-rough-spec.k 118.04 130.16 0.9068838352796559
mcd/vat-addui-pass-spec.k 114.11 125.23 0.9112033857701828
mcd/functional-spec.k 523.63 574.17 0.9119772889562325
kontrol/test-expectreverttest-test_expectrevert_bytes4-0-spec.k 81.37 88.67 0.9176722679598511
kontrol/test-countertest-testincrement-0-spec.k 81.45 88.59 0.9194039959363359
mcd/vat-subui-pass-spec.k 114.11 124.09 0.9195745023773068
mcd/vat-muluu-pass-spec.k 95.25 102.57 0.9286341035390465
bihu/forwardToHotWallet-failure-4-spec.k 86.35 92.53 0.9332108505349616
kontrol/test-allowchangestest-testallow-0-spec.k 55.01 58.93 0.9334804004751399
kontrol/test-storetest-teststoreload-0-spec.k 47.38 50.43 0.9395201269085862
kontrol/test-owneruponlytest-testincrementasowner-0-spec.k 66.8 70.82 0.9432363739056764
kontrol/test-expectreverttest-testfail_expectrevert_failandsuccess-0-spec.k 54.81 57.98 0.9453259744739566
erc20/ds/transferFrom-failure-1-c-spec.k 97.62 102.97 0.9480431193551521
mcd/flipper-bids-pass-rough-spec.k 75.82 79.94 0.948461346009507
erc20/ds/transferFrom-failure-2-a-spec.k 72.54 76.37 0.9498494173104622
erc20/ds/transferFrom-failure-2-b-spec.k 87.65 92.15 0.9511665762344005
mcd/dsvalue-peek-pass-rough-spec.k 51.52 54.1 0.9523105360443623
erc20/ds/approve-failure-spec.k 45.92 48.15 0.9536863966770509
kontrol/test-expectreverttest-testfail_expectrevert_false-0-spec.k 72.37 75.74 0.9555056773171376
erc20/ds/transferFrom-failure-1-b-spec.k 134.89 140.92 0.9572097644053363
erc20/ds/transfer-failure-1-b-spec.k 89.23 93.17 0.9577117097778255
kontrol/test-expectreverttest-test_expectrevert_message-0-spec.k 55.33 57.75 0.9580952380952381
kontrol/test-expectcalltest-testexpectregularcall-0-spec.k 46.21 48.14 0.959908599916909
kontrol/test-expectreverttest-testfail_expectrevert_bytes4-0-spec.k 62.34 64.92 0.9602587800369686
benchmarks/keccak00-spec.k 45.43 47.29 0.9606682173821104
mcd/vat-subui-fail-rough-spec.k 69.73 72.45 0.9624568668046929
mcd/dstoken-approve-fail-rough-spec.k 82.23 85.34 0.963557534567612
mcd/vat-addui-fail-rough-spec.k 71.67 74.36 0.9638246369015601
mcd/end-subuu-pass-spec.k 62.55 64.87 0.9642361646369662
benchmarks/storagevar02-overflow-spec.k 40.14 41.58 0.9653679653679654
benchmarks/storagevar01-spec.k 40.82 42.26 0.9659252247988642
erc20/ds/transfer-success-1-spec.k 134.9 126.52 1.0662345874170092
kontrol/test-allowchangestest-testfailallowchangestostorage-0-spec.k 74.95 69.88 1.072552947910704
mcd/dsvalue-read-pass-summarize-spec.k 57.73 53.07 1.087808554739024
kontrol/test-allowchangestest-testallow_fail-0-spec.k 56.61 51.3 1.1035087719298247
mcd/vat-slip-pass-rough-spec.k 112.63 100.81 1.1172502727903977
kontrol/test-expectreverttest-test_expectrevert_returnvalue-0-spec.k 65.6 57.93 1.1324011738304849
TOTAL 3781.81 4006.31 0.943963397740065

@goodlyrottenapple goodlyrottenapple marked this pull request as ready for review May 7, 2024 14:50
@goodlyrottenapple goodlyrottenapple requested a review from geo2a May 7, 2024 14:50
@goodlyrottenapple
Copy link
Contributor Author

Kontrol

Test sam-implication time master-538d07959 time (sam-implication/master-538d07959) time
src/tests/integration/test_foundry_prove.py::test_foundry_xml_report 22.53 29.57 0.7619208657423064
src/tests/integration/test_foundry_prove.py::test_foundry_refute_node 14.51 17.03 0.8520258367586612
PlainPrankTest.test_startPrankWithOrigin_true() 162.47 188.48 0.862001273344652
HevmTests.prove_revert 18.37 21.27 0.8636577338975083
src/tests/integration/test_foundry_prove.py::test_foundry_auto_abstraction 44.58 51.36 0.8679906542056075
BytesTypeTest.test_bytes4(bytes4) 31.03 35.35 0.8777934936350777
AddrTest.test_addr_true()-trace_options1 29.94 33.58 0.8916021441334129
BMCLoopsTest.test_countdown_concrete() 18.27 20.45 0.893398533007335
src/tests/integration/test_foundry_prove.py::test_foundry_prove_skips_setup 161.44 180.18 0.895992895992896
Setup2Test.test_setup() 29.96 33.29 0.8999699609492341
AccountParamsTest.testDealConcrete()-trace_options0 28.85 32.02 0.900999375390381
FreshCheatcodes.test_address() 59.1 65.59 0.9010519896325659
src/tests/integration/test_foundry_prove.py::test_foundry_remove_node 45.38 50.34 0.9014700039729837
HevmTests.proveFail_all_branches 56.63 62.36 0.9081141757536884
ConstructorTest.run_constructor() 69.05 75.98 0.9087917873124506
AddrTest.test_notBuiltinAddress_concrete() 27.07 29.72 0.9108344549125169
StructTypeTest.test_vars((uint8,uint32,bytes32)) 40.45 44.39 0.9112412705564317
ConstructorTest.testFail_constructor() 55.54 60.85 0.9127362366474938
HevmTests.proveFail_require_assert 48.0 52.41 0.9158557527189468
HevmTests.prove_require_assert_true 48.7 53.15 0.9162746942615241
CoinBaseTest.test_coinbase_setup() 62.41 67.91 0.9190104550139891
AddrTest.test_builtInAddresses() 27.45 29.83 0.9202145491116326
BlockParamsTest.testChainId(uint256) 44.3 48.11 0.9208064851382248
StoreTest.testGasStoreColdVM() 37.37 40.55 0.9215782983970406
AssertTest.test_assert_false() 73.92 80.19 0.9218106995884774
AssertTest.testFail_assert_true() 47.98 52.04 0.9219830899308225
BytesTypeTest.test_bytes32(bytes32) 27.83 30.11 0.9242776486217203
SymbolicStorageTest.testFail_SymbolicStorage(uint256) 125.28 135.5 0.9245756457564576
ConstructorTest.test_constructor() 115.06 124.32 0.9255148005148006
AssertTest.checkFail_assert_false() 32.82 35.46 0.9255499153976311
SetUpTest.testSetupData() 74.05 79.98 0.925856464116029
BlockParamsTest.testWarp(uint256)-trace_options2 33.29 35.88 0.9278149386845038
AddConst.applyOp(uint256) 66.09 71.19 0.9283607248209019
GasTest.testSetGas() 35.84 38.57 0.9292196007259529
RollTest.test_roll_setup() 61.43 66.07 0.9297714545179356
SetUpDeployTest.test_extcodesize() 75.8 81.43 0.9308608620901386
InitCodeTest.test_init() 116.3 124.81 0.9318163608685202
ArithmeticTest.test_max2(uint256,uint256) 69.38 74.41 0.9324015589302513
StoreTest.testStoreLoadNonExistent() 50.7 54.36 0.9326710816777043
BlockParamsTest.testCoinBase() 35.66 38.22 0.9330193615907901
MethodDisambiguateTest.test_method_call() 27.88 29.88 0.9330655957161981
AssumeTest.testFail_assume_false(uint256,uint256) 61.62 65.99 0.9337778451280497
ArithmeticContract.add(uint256,uint256) 19.5 20.88 0.9339080459770115
Setup2Test.testFail_setup() 69.81 74.64 0.9352893890675241
AssertTest.testFail_expect_revert() 98.25 105.0 0.9357142857142857
ArithmeticCallTest.test_double_add_sub_external(uint256,uint256,uint256) 402.21 429.77 0.9358726760825558
StoreTest.testLoadNonExistent() 40.44 43.21 0.935894468872946
AssertTest.test_assert_true() 57.45 61.34 0.9365829801108575
StartPrankTestOrigin.test_startprank_origin_setup() 115.46 123.17 0.9374035885361695
ArithmeticCallTest.test_double_add(uint256,uint256) 285.52 303.98 0.9392723205474043
ArithmeticTest.test_max1(uint256,uint256) 80.53 85.33 0.943747802648541
ArithmeticCallTest.test_double_add_double_sub(uint256,uint256) 379.96 402.37 0.9443049929169669
SymbolicStorageTest.testFail_SymbolicStorage1(uint256) 142.07 150.43 0.9444259788605995
AssertTest.test_assert_true_branch(uint256) 74.54 78.9 0.9447401774397972
BlockParamsTest.testFee(uint256) 45.68 48.26 0.946539577289681
HevmTests.prove_require_assert_false 57.39 60.62 0.9467172550313429
PlainPrankTest.test_stopPrank_notExistent() 43.55 45.99 0.9469449880408783
FreshCheatcodes.test_int128() 44.42 46.9 0.9471215351812368
FreshCheatcodes.test_bool() 44.33 46.71 0.9490473132091629
StoreTest.testGasLoadWarmUp() 75.29 79.26 0.9499116830683826
AssumeTest.test_assume_false(uint256,uint256) 90.04 94.77 0.9500896908304317
AccountParamsTest.test_GetNonce_true() 44.34 46.61 0.9512980047200172
EmitContractTest.testExpectEmit() 87.82 92.27 0.9517719735558686
CSETest.test_identity(uint256,uint256) 234.61 246.36 0.9523055690858906
InitCodeTest.testFail_init() 48.19 50.55 0.9533135509396637
PrankTestMsgSender.test_msgsender_setup() 117.88 123.64 0.9534131349077968
AssertTest.test_failing_branch(uint256) 98.33 103.11 0.9536417418291145
AssertTest.test_revert_branch(uint256,uint256) 94.12 98.69 0.9536933833215119
AddrTest.test_notBuiltinAddress_symbolic(address) 67.64 70.85 0.9546930134086098
SymbolicStorageTest.testEmptyInitialStorage(uint256) 45.77 47.94 0.9547350855235712
CSETest.test_add_const(uint256,uint256) 218.83 229.15 0.9549639973816277
]) 65.01 68.04 0.9554673721340388
LoopsTest.sum_N(uint256) 250.5 262.13 0.9556327013314004
AccountParamsTest.test_Nonce_NonExistentAddress() 54.32 56.81 0.9561696884351346
AssumeTest.test_assume_true(uint256,uint256) 43.05 44.95 0.9577308120133481
PrankTestOrigin.test_origin_setup() 114.56 119.61 0.9577794498787727
BlockParamsTest.testBlockNumber() 27.49 28.64 0.9598463687150838
FreshBytesTest.test_symbolic_bytes_length 109.86 114.41 0.9602307490603968
PlainPrankTest.testFail_startPrank_internalCall() 53.48 55.68 0.9604885057471264
MockCallTest.testSelectorMockCall() 59.63 62.06 0.9608443441830486
src/tests/integration/test_foundry_prove.py::test_foundry_merge_nodes 49.64 51.64 0.9612703330751355
FreshCheatcodes.test_freshSymbolicWord() 44.36 46.13 0.9616301755907218
FreshBytesTest.test_symbolic_bytes_3 153.42 159.52 0.9617602808425274
PlainPrankTest.test_startPrank_zeroAddress_true() 163.21 169.5 0.9628908554572272
StartPrankTestMsgSender.test_startprank_msgsender_setup() 115.41 119.72 0.9639993317741397
AccountParamsTest.testDealSymbolic(uint256) 53.08 54.95 0.9659690627843494
LabelTest.testLabel() 36.22 34.97 1.0357449242207606
PlainPrankTest.test_prank_zeroAddress_true() 225.47 199.49 1.130232091834177
PlainPrankTest.test_prank_expectRevert() 115.43 101.91 1.1326660779118831
TOTAL 7272.4400000000005 7703.040000000001 0.9440999916915919

Comment on lines +531 to +532
MatchFailed{} ->
doesNotImply (sortOfPattern substPatL) req.antecedent.term req.consequent.term
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The legacy backend would first simplify both patterns and return the simplified implication.

Not sure whether we would want to keep that behaviour (did it matter in the tests?)

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

i guess this would matter if e.g. the antecedent goes to #bottom? i guess here i am assuming things are defined and the constraints don't contradict each other

then implies (sortOfPattern substPatL) req.antecedent.term req.consequent.term subst
else -- pure . Left . RpcError.backendError . RpcError.ImplicationCheckError . RpcError.ErrorOnly $ "implies"

ApplyEquations.evaluateConstraints doTracing def mLlvmLibrary solver mempty filteredConsequentPreds >>= \case
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I would have expected that the substPatL.constraints be used as preconditions when checking that substPatR.constraints (minus the identical ones, i.e., filteredConsequentPreds) hold here. evaluateConstraints should probably take an additional argument for things that are assumed true when checking.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

yeah i was surprised even this most naive version seems to work reasonably well especially in KEVM. don't know if we should try more complex versions in this PR or follow up later. What do you think?

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

So we are essentially throwing the known constraints away. We should definitely have a follow-up PR to see if considering the substPatL.constraints allows us to cover the rest of the cases that Kore is still needed for.

Comment on lines +610 to +615
evaluateConstraints' constraints = do
pushConstraints constraints
-- evaluate all existing constraints, once
traverse_ simplifyAssumedPredicate . predicates =<< getState
-- this may yield additional new constraints, left unevaluated
predicates <$> getState
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Wrt. the comment above in JsonRpc about the constraints of the antecedent:
This function would need to push the constraints from the antecedent, but then only check the ones from the consequent, and we want them all to be TrueBool (independent of each other). This is different from the simplifyAssumedPredicate idea which was to simplify (not remove) constraints.
Probably we should put the antecedent constraints into the state, and then call simplifyConstraint' on each (substituted) predicate of the consequent to check that it yields true.

match1 _ t1@ConsApplication{} t2@DomainValue{} = failWith $ DifferentSymbols t1 t2
match1 _ t1@ConsApplication{} t2@Injection{} = failWith $ DifferentSymbols t1 t2
match1 _ t1@ConsApplication{} t2@KMap{} = failWith $ DifferentSymbols t1 t2
match1 _ t1@ConsApplication{} t2@KList{} = failWith $ DifferentSymbols t1 t2
match1 _ t1@ConsApplication{} t2@KSet{} = failWith $ DifferentSymbols t1 t2
match1 matchTy (ConsApplication symbol1 sorts1 args1) (ConsApplication symbol2 sorts2 args2) = matchSymbolAplications matchTy symbol1 sorts1 args1 symbol2 sorts2 args2
match1 Rewrite t1@ConsApplication{} t2@FunctionApplication{} = addIndeterminate t1 t2
match1 Eval (ConsApplication symbol1 sorts1 args1) (FunctionApplication symbol2 sorts2 args2) = matchSymbolAplications Eval symbol1 sorts1 args1 symbol2 sorts2 args2
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

(off-topic) I was stumbling over this line first... why would we try to match those obviously-different things in Eval mode?
After reading matchSymbolApplication it turns out to be somewhat trivial... since a symbol cannot be both constructor and function, it will always end up in addIndeterminate.

. Left
. RpcError.backendError
. RpcError.ImplicationCheckError
. RpcError.ErrorWithContext "does-not-imply2"
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is the "2" in "does-not-imply2" spurious?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

ah, that's now a pointless check. thanks for pointing out. After the matching code re-factor, we already throw this error earlier via SubjectVariableMatch and return not implies.

then implies (sortOfPattern substPatL) req.antecedent.term req.consequent.term subst
else -- pure . Left . RpcError.backendError . RpcError.ImplicationCheckError . RpcError.ErrorOnly $ "implies"

ApplyEquations.evaluateConstraints doTracing def mLlvmLibrary solver mempty filteredConsequentPreds >>= \case
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

So we are essentially throwing the known constraints away. We should definitely have a follow-up PR to see if considering the substPatL.constraints allows us to cover the rest of the cases that Kore is still needed for.

@rv-jenkins rv-jenkins merged commit a0ca443 into master May 8, 2024
7 checks passed
@rv-jenkins rv-jenkins deleted the sam/implication branch May 8, 2024 15:24
geo2a added a commit that referenced this pull request May 14, 2024
geo2a pushed a commit that referenced this pull request May 15, 2024
Partially reverts #3846 by disabling the implication endpoint in booster
until we figure out and address #3857
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Implement check-implication in booster
4 participants