Add an experimental batch module #1134

siv2r · 2022-08-21T08:55:46Z

Overview

This PR adds support for batch verifying Schnorr signatures and tweaked x-only public key checks. It is based on the work of @jonasnick in #760.

Batch Verification

This implementation does not strictly follow the BIP340 batch verification spec. The API design is loosely based on this suggestion: #760 (comment). Prior development discussion of this PR can be found in siv2r#2.

Speed Up

batch verifying Schnorr signatures is 20% faster - graph here
batch verifying tweak pubkey checks is 50% faster - graph here

Fixes #1087

siv2r · 2022-08-21T09:02:36Z

This implementation does not strictly follow the BIP340 batch verification spec.

For example,

the random numbers (or randomizers) aren’t generated by passing a seed (hash of all inputs) to CSPRNG
allows mixing of schnorrsig and tweak checks
uses the tag “BIP0340/batch” for initializing the sha256 obj (not in the BIP340 specs)

Alternative Design Options

batch module design
- the current code has: (new batch module)
  - create, destroy, and verify APIs in the batch module
  - batch_add_* APIs in schnorrsig and extrakeys modules (with #ifdef ENABLE_MODULE_BATCH)
- alternate option 1: (new batch module)
  - place the batch_add_* APIs in the batch module (with schnorrsig and extrakeys header guards)
- alternate option 2: (no module)
  - place create, destroy, and verify in src/secp256k1.c
  - place batch_add_* APIs in schnorrsig and extrakeys modules
- relevant discussions:
  - Schnorr Batch Verification Interface siv2r/secp256k1#2 (comment)
  - Schnorr Batch Verification Interface siv2r/secp256k1#2 (comment)
Delayed randomizer generation
- the current code generates a randomizer just after a user enters input (in _batch_add_*)
- we could instead generate them (in batch_verify) after the user enters all their input
- Pros: follows bip340 specs. Cons: consumes more memory.
- more info: General batch verification API context #1087 (comment) and here
Use BLAKE256 instead of SHA256 for generating randomizers (to improve speed).
In _batch_create, we could:
- Get memory size instead of max_terms.
- Provide pre-determined sizes (small, medium, and large) instead of max_terms.
  - Better test coverage?

siv2r · 2022-08-21T09:03:50Z

Questions

Streaming batch API (this PR) vs Single call batch API?
Is transparent verification required?
- the current code implements transparent verification (inside batch_add_* APIs)
- without transparent verification, the user needs to check for space in the batch before calling _batch_add APIs:
```
if (batch_enough_space_for_schnorrsig) {
  batch_add_schnorrsig()
}
```
On an empty batch, Should _batch_verify return 0 or 1?
- the current code returns 1
  - simple implementation
- if we want to return 0
  - needs an extra param in batch to avoid ANDing the result with its initial value
  - provides better security?
xonly_pubkey_tweak_add_check recommends ctx to be initialized for verification, but it can work even if ctx is initialized as none.
- Since batch_add_tweak_xonlypub_check is based on tweak_add_check, should it also recommend that ctx be initialized for verification?
- Can it recommend that ctx be initialized for none instead?
A better name for secp256k1_batch_usable?
- the current name is confusing
- here, “usable” means if the batch can be used by the batch_add_* functions

siv2r · 2022-08-21T10:35:46Z

In extrakeys/bench_impl.h, cast pointers to (void *) before freeing to avoid MSVC warning.

sipa · 2022-08-23T17:11:47Z

doc/speedup-batch/bench_output.txt

+
+schnorrsig_sign                    ,    50.4       ,    50.5       ,    50.7    
+schnorrsig_verify                  ,    89.1       ,    89.2       ,    89.3    
+schnorrsig_batch_verify_1          ,   104.0       ,   104.0       ,   104.0    


batch_verify_1 shouldn't be slower than non-batch verify. Is it possible to revert to using non-batch validation logic for this case?

Not possible with the current design.

The non-batch validation (secp256k1_schnorrsig_verify) logic looks something like this:

calc rj using secp256k1_ecmult: Rj = sG - eP

convert rj (gej) to r (ge)

check if the r.x = sig[0:32] and r.y = even

one schnorrsig occupies two points in the batch, and one tweak check occupies one point in the batch. If a batch contains two points, there is no guarantee that they are from a schnorrsig (R, P). It could be from two tweak checks. So, we can't use the r.y = even check.

Hence, I tried implementing a slightly modified schnorrsig_verify logic (not implement in this PR):

calc neg_rj using secp256k1_ecmult: neg_Rj = -s*G + batch.scalars[1]*batch.points[1]

check if neg_rj + batch.points[0] == inf using _gej_add_var

batch.scalars[0] = 1 always. So, we don't need to use ecmult again

This gives somewhat better benchmarks than before:

Benchmark , Min(us) , Avg(us) , Max(us) schnorrsig_sign , 49.1 , 50.1 , 53.4 schnorrsig_verify , 86.6 , 87.2 , 88.4 schnorrsig_batch_verify_1 , 94.7 , 95.0 , 95.2

But schnorrsig_batch_verify_1 is still slower than schnorrsig_verify.

jonasnick

This implementation does not strictly follow the BIP340 batch verification spec. [...] the random numbers (or randomizers) aren’t generated by passing a seed (hash of all inputs) to CSPRNG

The security argument for batch verification should relatively easily translate to the approach in this PR.

Is transparent verification required?

I initially wasn't a fan of transparent verification because developers generally want to know how long a certain function call takes and don't want to be surprised by some batch_add calls taking much longer than others. But I changed my mind on this.

The batch verification API right now is used as follows:

for (i = 0; i < N_SIGS; i++) {
    if(!batch_usable(batch) || !batch_add_schnorrsig(batch, sig[i], msg[i], sizeof(msg[i]), &pk)) {
        return 0;
    }
}
if(!batch_verify(ctx, batch)) {
    return 0;
}

Without TV, then one option is to have the user create a batch that is large enough. Of course that's not alway possible because the batch can get larger than the available memory. Moreover, this would require adding an API that, given a number of schnorrsigs and tweaks (and more in the future) returns the size of the required batch. That seems way more complicated than TV.

If there's no TV and the batch is not guaranteed to be large enough, then users need to essentially reimplement something like transparent verification:

for (i = 0; i < N_SIGS; i++) {
    if(!schnorrsig_batch_has_space(batch)) {
        if(!batch_verify(batch)) {
            return 0;
        }
    }
    if (!batch_add_schnorrsig(batch, sig[i], msg[i], sizeof(msg[i]), &pk)) {
        return 0;
    }
}
if(!batch_verify(ctx, batch)) {
    return 0;
}

This is more code compared to having TV built into batch_add.
*_batch_has_space is specific to whatever you're trying to add to the batch, i.e., we would also have to add tweak_has_space, for example.
The batch_usable function in the current implementation (with TV) only allows for earlier aborts and is not essential (unless you want to determine if a batch_add failed because its input is obviously malformed or because the previous batch_add triggered a batch_verify that failed).
If users don't want to use TV (for whatever reason), then they don't have to - even in the current implementation. This requires counting the terms that have been added to the batch and verifying before it's full (perhaps we can make this simpler).

xonly_pubkey_tweak_add_check recommends ctx to be initialized for verification, but it can work even if ctx is initialized as none. [...] Since batch_add_tweak_xonlypub_check is based on tweak_add_check, should it also recommend that ctx be initialized for verification?

I don't think so. #1126 removes "initialized for verification" from the API docs of xonly_pubkey_tweak_add_check. For consistency it would be better if you'd remove "(can be initialized for none)" from the API doc.

real-or-random · 2022-08-24T18:54:59Z

I haven't had a closer look yet but I agree with @jonasnick's comments about TV.

fjahr · 2022-12-25T19:54:27Z

@siv2r Are you still working on this topic? Do you plan to address @jonasnick 's comments?

siv2r · 2022-12-26T06:34:40Z

@fjahr, this PR needs review from other contributors regarding the (batch API) design decision made here. The #1134 (review) comment showed its support for the "Transparent Verification" feature, which was implemented.

IIRC, two documentation changes are required (at the time of writing):

remove "initialized for verification" from the batch_add_tweak_xonlypub_check doc.
document that batch_verify_1 is slower than schnorrsig_verify (see #1134 (comment)).

I avoided making these changes immediately to keep the commit history clean for reviewers. I would be happy to work on any required changes after it gets enough review.

fjahr

I have opened a draft PR for using this in bitcoin core: bitcoin/bitcoin#29491

Here is a rebased branch of the code that I am using: https://github.com/fjahr/secp256k1/tree/pr1134-rebase-2024

I hope this can create some new interest and motivate people to review here as well.

fjahr · 2024-02-27T23:57:54Z

src/modules/batch/main_impl.h

+
+        /* `_strauss_batch_internal` should not fail due to insufficient memory.
+         * `batch_create` will allocate memeory needed by `_strauss_batch_internal`. */
+        VERIFY_CHECK(strauss_ret != 0);


Getting an unused variable warning about strauss_ret here when building this as part of bitcoin core, probably because the check is removed with optimizations?

The macro VERIFY_CHECK(code) is defined to be the empty string in production builds, that's why you need to suppress the warning using (void)strauss_ret;.

include/secp256k1_batch.h

fjahr · 2024-02-28T15:21:38Z

src/modules/batch/tests_impl.h

+    secp256k1_batch *batch_vrfy;
+    secp256k1_batch *batch_both;
+    secp256k1_batch *batch_sttc;
+    unsigned char aux_rand16[32];


Getting another warning here when compiling together with Bitcoin Core:

src/modules/batch/tests_impl.h:68:19: warning: mixing declarations and code is a C99 extension [-Wdeclaration-after-statement] unsigned char aux_rand16[32]; ^

In C89 all the variable declarations must precede statements within any block.

Thanks @sipa, yeah, I should have added that I have this fixed now in my rebase branch: fjahr@7c6b9df#diff-fa3fe2044a5385d080fb813c89c9d0707b50ec7083cef9cd9c114b1f2e5483c3R41

fjahr · 2024-12-03T22:07:19Z

Here is a rebased branch of the code that I am using: https://github.com/fjahr/secp256k1/tree/pr1134-rebase-2024

FWIW, I have updated the rebased code recently and also added the changes necessary to build the module with cmake.

7edaf8b Benchmark Chainstate::ConnectBlock duration (Eunovo) Pull request description: Introduce benchmarks to evaluate ConnectBlock performance for: - Blocks containing only Schnorr signatures - Blocks containing both Schnorr and ECDSA signatures - Blocks containing only ECDSA signatures The benchmarks in this PR, focus on signature validation. Additional benchmarks may be added in the future to assess other aspects of ConnectBlock. This is the first step toward implementing Batch Verification of Schnorr Signatures in Core. It provides a way to test and measure the performance improvements of batch verification on Core. For more details on batch validation, refer to the [batch-verify module on secp](bitcoin-core/secp256k1#1134) and [batch-verify on core](#29491). ACKs for top commit: josibake: reACK 7edaf8b fjahr: utACK 7edaf8b l0rinc: ACK 7edaf8b Tree-SHA512: 883c8a5e4e4de401ffb9ac9b6789b7fe0737afefbdaf02c6d7e1645392efc4f0d2d28b423ba7e34366a33608e0835793f5e7a1312b5c8063de14446319529cc7

This commit adds the foundational configuration, build scripts, and an initial structure for experimental batch module.

This commit adds the batch_create and batch_destroy APIs. Relevant Links: 1. batch_scratch_size allocation formula is taken from bench ecmult: https://github.com/bitcoin-core/secp256k1/blob/694ce8fb2d1fd8a3d641d7c33705691d41a2a860/src/bench_ecmult.c#L312. 2. aux_rand16 param in batch_create enables synthetic randomness for randomizer generation: sipa/bips#204.

This commit refactors ecmult_strauss_batch and adds _batch_verify API. The current ecmult_strauss_batch only works on empty scratch space. To make batch_verify compatible, we need ecmult_strauss_batch to support a scratch space pre-filled with scalars and points. So, it was refactored to do exactly that. The batch_verify API always uses the Strauss algorithm. It doesn't switch to Pippenger (unlike ecmult_multi_var). ecmult_pippenger_batch represents points as secp256k1_ge whereas ecmult_strauss_batch represents points as secp256k1_gej. This makes supporting both Pippenger and Strauss difficult (at least with the current batch object design). Hence, batch_verify only supports Strauss for simplicity.

This commit adds the batch APIs: 1. batch_add_schnorrsig Adds a Schnorr signature to the batch. 2. batch_add_xonlypub_tweak_check Adds a tweaked x-only pubkey check to the batch. 3. batch_usable Checks if a batch can be used by _batch_add_* APIs. **Side Note:** Exposing batch_add_schnorrsig in the secp256k1_schnorrsig.h header file (with batch module header guards) will force the user to define ENABLE_MODULE_BATCH during their code compilation. Hence, it is in a standalone secp256k1_schnorrsig_batch.h header file. A similar argument could be made for batch_add_xonlypub_tweak_check.

This commit adds an example C program using the batch API. GNU Autotools and CMake will compile this example only if both batch and schnorrsig modules are enabled.

This commit adds the following tests: 1. GitHub workflow 2. Batch API tests (ordered) 3. Tagged SHA256 test 4. BIP340 test vectors: https://github.com/bitcoin/bips/blob/master/bip-0340/test-vectors.csv 5. Large random test for `strauss_batch` refactor

This commit adds the following tests: 1. Random bitflip test for randomizer generating function 2. Random bitflip in Schnorr Signature (batch_add_schnorrsig test) 3. NULL arg tests (for both batch_add APIs)

This commit adds benchmarks for: 1. Batch verifying Schnorr signatures 2. Batch verifying tweaked pubkey checks 3. Normal tweaked pubkey check in extrakeys module For batch verify benchmark, the number of sigs (or checks) in the batch varies from 1 to SECP256K1_BENCH_ITERS with a 20% increment.

This commit generates two semi-log graphs that visualize the batch verification speed up over single verification (y-axis) wrt the number of signatures (or tweak checks) in the batch (x-axis). The input data points are taken from the batch verify benchmark. GNU plot was used to generate these graphs (plot.gp file). The instructions to reproduce these graphs (on your local machine) are given in doc/speedup-batch.md file. The value of `STRAUSS_MAX_TERMS_PER_BATCH` was calculated (approx) from the generated graphs. Relevant discussion: #2 (comment)

siv2r · 2025-07-04T07:41:50Z

Just updated the PR on top of the current master and added a CMake build option. Planning to work on Pippenger next. Would appreciate any reviews in the meantime. I’ll fix the failed CI checks soon.

fjahr

Thanks for the update @siv2r! Leaving a few comments from a superficial read-through. I will integrate the latest code into the Bitcoin Core PR and take a deeper look soon.

fjahr · 2025-07-06T18:12:21Z

README.md

@@ -22,6 +22,7 @@ Features:
 * Optional module for Schnorr signatures according to [BIP-340](https://github.com/bitcoin/bips/blob/master/bip-0340.mediawiki).
 * Optional module for ElligatorSwift key exchange according to [BIP-324](https://github.com/bitcoin/bips/blob/master/bip-0324.mediawiki).
 * Optional module for MuSig2 Schnorr multi-signatures according to [BIP-327](https://github.com/bitcoin/bips/blob/master/bip-0327.mediawiki).
+* Optional module for Batch Verification (experimental).


Could also mention "according to BIP340" here since batch verification is also specified there.

fjahr · 2025-07-06T18:17:59Z

ci/ci.sh

@@ -80,6 +80,7 @@ esac
    --enable-module-extrakeys="$EXTRAKEYS" \
    --enable-module-schnorrsig="$SCHNORRSIG" \
    --enable-module-musig="$MUSIG" \
+    --enable-module-batch="$BATCH"


Missing \ here?

fjahr · 2025-07-06T18:29:58Z

src/tests.c

+    *nonzero_inp_len = num_nonzero;
+    /* ptr to g_scalar*/
+    g_scalar = g_scalar_ptr;
+    /* is mulciplicand of g nonzero? */


nit: typo

Suggested change

/* is mulciplicand of g nonzero? */

/* is multiplicand of g nonzero? */

fjahr · 2025-07-06T18:36:02Z

src/modules/schnorrsig/tests_impl.h

@@ -242,6 +263,9 @@ static void test_schnorrsig_bip_vectors(void) {
        };
        test_schnorrsig_bip_vectors_check_signing(sk, pk, aux_rand, msg, sizeof(msg), sig);
        test_schnorrsig_bip_vectors_check_verify(pk, msg, sizeof(msg), sig, 1);
+        #ifdef ENABLE_MODULE_BATCH


nit: Not sure if that is something that would be popular but I think you could limit these ifdefs to just the test function itself by just having the function body be enclosed in an batch ifdef. At least that would save a few lines of code. Just an idea though, this doesn't bother me much.

fjahr · 2025-07-06T19:14:55Z

src/modules/batch/main_impl.h

+ *     sha256: contains hash of all the inputs (schnorrsig/tweaks) present in
+ *             the batch object, expect the first input. Used for generating a random secp256k1_scalar
+ *             for each term added by secp256k1_batch_add_*.
+ *     sha256: contains hash of all inputs (except the first one) present in the batch.


sha256 arg is listed twice here

fjahr · 2025-07-06T19:15:27Z

src/modules/batch/main_impl.h

+        if (batch->data == NULL) {
+            return NULL;
+        }
+        /* allocate memeory for `max_terms` number of scalars and points on scratch space */


nit

Suggested change

/* allocate memeory for `max_terms` number of scalars and points on scratch space */

/* allocate memory for `max_terms` number of scalars and points on scratch space */

fjahr · 2025-07-06T19:17:38Z

include/secp256k1_batch.h

+/** Verify the set of schnorr signatures or tweaked pubkeys present in the secp256k1_batch.
+ *
+ *  Returns: 1: every schnorrsig/tweak (in batch) is valid
+ *           0: atleaset one of the schnorrsig/tweak (in batch) is invalid


nit

Suggested change

* 0: atleaset one of the schnorrsig/tweak (in batch) is invalid

* 0: at least one of the schnorrsig/tweak (in batch) is invalid

fjahr · 2025-07-06T19:29:00Z

include/secp256k1_batch.h

+ *  work correctly. It simply makes it hard to understand the reason behind
+ *  `secp256k1_batch_add_*` failure (if occurs).
+ */
+SECP256K1_API int secp256k1_batch_usable(


I understand the rationale given here but I am still not sure we really need this function. As far as I understand this only returns what result is set to and only the verify function could set result to false. So this means for this to be useful, the user would first need to have called verify and ignored the result. I am not sure in which context this would really be needed, maybe in some concurrency scenarios? But even then it feels like users should be handling that differently. Happy to be convinced otherwise but currently I would vote to remove it if there isn't some use case I am missing.

siv2r force-pushed the batch-verify-interface branch from f48072a to eb49e93 Compare August 21, 2022 10:33

sipa reviewed Aug 23, 2022

View reviewed changes

jonasnick reviewed Aug 24, 2022

View reviewed changes

real-or-random mentioned this pull request May 10, 2023

Rework or get rid of scratch space #1302

Open

fjahr mentioned this pull request Feb 27, 2024

[EXPERIMENTAL] Schnorr batch verification for blocks bitcoin/bitcoin#29491

Draft

5 tasks

fjahr reviewed Feb 28, 2024

View reviewed changes

real-or-random added feature performance labels Feb 28, 2024

fjahr reviewed Feb 28, 2024

View reviewed changes

fjahr mentioned this pull request Jun 28, 2024

WIP: Add schnorrsig batch verification #760

Closed

3 tasks

Eunovo mentioned this pull request Jan 20, 2025

Benchmark Chainstate::ConnectBlock duration bitcoin/bitcoin#31689

Merged

fjahr mentioned this pull request Mar 12, 2025

Script Validation Performance Improvement Tracking Issue bitcoin/bitcoin#32042

Open

siv2r added 9 commits June 30, 2025 18:00

batch: Initialize an experimental batch module

800233a

This commit adds the foundational configuration, build scripts, and an initial structure for experimental batch module.

batch: Add example

e5bbca6

This commit adds an example C program using the batch API. GNU Autotools and CMake will compile this example only if both batch and schnorrsig modules are enabled.

batch: Add tests for batch_add_* APIs

642901f

This commit adds the following tests: 1. Random bitflip test for randomizer generating function 2. Random bitflip in Schnorr Signature (batch_add_schnorrsig test) 3. NULL arg tests (for both batch_add APIs)

siv2r force-pushed the batch-verify-interface branch from eb49e93 to 410abb2 Compare July 4, 2025 07:35

siv2r mentioned this pull request Jul 4, 2025

Schnorr Batch Verification Interface siv2r/secp256k1#2

Closed

6 tasks

fjahr reviewed Jul 6, 2025

View reviewed changes

	/* is mulciplicand of g nonzero? */
	/* is multiplicand of g nonzero? */

	/* allocate memeory for `max_terms` number of scalars and points on scratch space */
	/* allocate memory for `max_terms` number of scalars and points on scratch space */

	* 0: atleaset one of the schnorrsig/tweak (in batch) is invalid
	* 0: at least one of the schnorrsig/tweak (in batch) is invalid

Add an experimental batch module #1134

Are you sure you want to change the base?

Add an experimental batch module #1134

Uh oh!

Conversation

siv2r commented Aug 21, 2022 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Overview

Batch Verification

Speed Up

Uh oh!

siv2r commented Aug 21, 2022 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Alternative Design Options

Uh oh!

siv2r commented Aug 21, 2022

Questions

Uh oh!

siv2r commented Aug 21, 2022

Uh oh!

sipa Aug 23, 2022 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

siv2r Aug 23, 2022 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

jonasnick left a comment

Choose a reason for hiding this comment

Uh oh!

real-or-random commented Aug 24, 2022

Uh oh!

fjahr commented Dec 25, 2022

Uh oh!

siv2r commented Dec 26, 2022

Uh oh!

fjahr left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

fjahr commented Dec 3, 2024

Uh oh!

siv2r commented Jul 4, 2025

Uh oh!

fjahr left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

siv2r commented Aug 21, 2022 •

edited

Loading

siv2r commented Aug 21, 2022 •

edited

Loading

sipa Aug 23, 2022 •

edited

Loading

siv2r Aug 23, 2022 •

edited

Loading