Skip to content

Commit 6c06b13

Browse files
dignifiedquiredignifiedquire
authored
feat: implement election post
1 parent a02147e commit 6c06b13

32 files changed

+1522
-506
lines changed

‎.circleci/config.yml

Lines changed: 4 additions & 26 deletions
Original file line numberDiff line numberDiff line change
@@ -79,25 +79,6 @@ jobs:
7979
command: |
8080
cargo +stable test --verbose --release --all
8181
RUSTFLAGS="-D warnings" cargo +stable build --examples --release --all
82-
test_mem_trees:
83-
docker:
84-
- image: filecoin/rust:latest
85-
working_directory: /mnt/crate
86-
resource_class: 2xlarge
87-
steps:
88-
- checkout
89-
- attach_workspace:
90-
at: "."
91-
- restore_cache:
92-
keys:
93-
- cargo-v13-{{ checksum "rust-toolchain" }}-{{ checksum "Cargo.toml" }}-{{ checksum "Cargo.lock" }}-{{ arch }}
94-
- parameter-cache-{{ .Revision }}
95-
- run:
96-
name: Test (nightly) mem-trees feature
97-
command: |
98-
cargo +$(cat rust-toolchain) test --verbose --release -p storage-proofs --features mem-trees -Z package-features
99-
environment:
100-
FIL_PROOFS_REPLICATED_TREES_DIR: replicated-disk-trees
10182
10283
test_ignored_release:
10384
docker:
@@ -222,15 +203,15 @@ jobs:
222203
cat stacked-benchmarks.json
223204
no_output_timeout: 60m
224205
- run:
225-
name: Run Rational PoST benchmarks using a 1GiB sector
206+
name: Run Election PoST benchmarks using a 1GiB sector
226207
command: |
227-
./fil-proofs-tooling/scripts/benchy-remote.sh "${CIRCLE_BRANCH}" "${BENCHMARK_SERVER_SSH_USERNAME}@${BENCHMARK_SERVER_IP_ADDR}" rational-post --size=$((1024*1024)) > rational-post-benchmarks.json
228-
cat rational-post-benchmarks.json
208+
./fil-proofs-tooling/scripts/benchy-remote.sh "${CIRCLE_BRANCH}" "${BENCHMARK_SERVER_SSH_USERNAME}@${BENCHMARK_SERVER_IP_ADDR}" election-post --size=$((1024*1024)) > election-post-benchmarks.json
209+
cat election-post-benchmarks.json
229210
no_output_timeout: 60m
230211
- run:
231212
name: Aggregate benchmarks into single JSON document
232213
command: |
233-
./fil-proofs-tooling/scripts/aggregate-benchmarks.sh stacked-benchmarks.json micro-benchmarks.json hash-constraints.json rational-post-benchmarks.json > aggregated-benchmarks.json
214+
./fil-proofs-tooling/scripts/aggregate-benchmarks.sh stacked-benchmarks.json micro-benchmarks.json hash-constraints.json election-post-benchmarks.json > aggregated-benchmarks.json
234215
cat aggregated-benchmarks.json
235216
- store_artifacts:
236217
path: stacked-benchmarks.json
@@ -359,9 +340,6 @@ workflows:
359340
- test_release:
360341
requires:
361342
- cargo_fetch
362-
- test_mem_trees:
363-
requires:
364-
- cargo_fetch
365343
- test_ignored_release:
366344
name: test_ignored_release_storage_proofs
367345
crate: "storage-proofs"

‎README.md

Lines changed: 0 additions & 15 deletions
Original file line numberDiff line numberDiff line change
@@ -161,21 +161,6 @@ To check that it's working you can inspect the replication log to find `using pa
161161

162162
(You can also verify if the cache is working by inspecting the time each layer takes to encode, `encoding, layer:` in the log, where the first two layers, forward and reverse, will take more time than the rest to populate the cache while the remaining 8 should see a considerable time drop.)
163163

164-
In the most extreme case, to reduce time at the cost of *a lot* of memory consumption you can turn on the feature that stores MTs on memory (`mem-trees`) instead of on disk (the default) to generate them all on RAM and avoid disk I/O (if the HW doesn't have enough RAM to handle the MTs, roughly 20x the sector size, this won't have the desired effect as the OS will start backing them on disk anyway). For example, to run the `stacked` example with this feature turned on you'd need to indicate so to `cargo`,
165-
166-
```
167-
# NEEDS TO BE RUN INSIDE `storage-proofs\` directory
168-
# for the `--features` to take effect.
169-
cargo run \
170-
-p filecoin-proofs \
171-
--release \
172-
--example stacked \
173-
--features \
174-
mem-trees \
175-
-- \
176-
--size 1048576
177-
```
178-
179164
**Speed Optimized Pedersen Hashing** - we use Pedersen hashing to generate Merkle Trees and verify Merkle proofs. Batched Pedersen hashing has the property that we can pre-compute known intermediary values intrinsic to the Pedersen hashing process that will be reused across hashes in the batch. By pre-computing and cacheing these intermediary values, we decrease the runtime per Pedersen hash at the cost of increasing memory usage. We optimize for this speed-memory trade-off by varying the cache size via a Pedersen Hash parameter known as the "window-size". This window-size parameter is configured via the [`pedersen_hash_exp_window_size` setting in `storage-proofs`](https://github.com/filecoin-project/rust-fil-proofs/blob/master/storage-proofs/src/settings.rs). By default, Bellman has a cache size of 256 values (a window-size of 8 bits), we increase the cache size to 65,536 values (a window-size of 16 bits) which results in a roughly 40% decrease in Pedersen Hash runtime at the cost of a 9% increase in memory usage. See the [Pedersen cache issue](https://github.com/filecoin-project/rust-fil-proofs/issues/697) for more benchmarks and expected performance effects.
180165

181166
### Memory

‎fil-proofs-tooling/scripts/aggregate-benchmarks.sh

Lines changed: 3 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -5,13 +5,13 @@ set -e
55
stacked_path=$1
66
micro_path=$2
77
hash_constraints_path=$3
8-
rational_post_path=$4
8+
election_post_path=$4
99

1010
jq --sort-keys -s '{ benchmarks: { "stacked-benchmarks": { outputs: { "max-resident-set-size-kb": .[0] } } } } * .[1]' \
1111
<(jq '.["max-resident-set-size-kb"]' $stacked_path) \
12-
<(jq -s '.[0] * { benchmarks: { "hash-constraints": .[1], "stacked-benchmarks": .[2], "micro-benchmarks": .[3], "rational-post-benchmarks": .[4] } }' \
12+
<(jq -s '.[0] * { benchmarks: { "hash-constraints": .[1], "stacked-benchmarks": .[2], "micro-benchmarks": .[3], "election-post-benchmarks": .[4] } }' \
1313
<(jq 'del (.benchmarks)' $micro_path) \
1414
<(jq '.benchmarks' $hash_constraints_path) \
1515
<(jq '.benchmarks' $stacked_path) \
1616
<(jq '.benchmarks' $micro_path) \
17-
<(jq '.benchmarks' $rational_post_path))
17+
<(jq '.benchmarks' $election_post_path))

‎fil-proofs-tooling/src/bin/benchy/rational_post.rs renamed to ‎fil-proofs-tooling/src/bin/benchy/election_post.rs

Lines changed: 41 additions & 9 deletions
Original file line numberDiff line numberDiff line change
@@ -7,8 +7,8 @@ use filecoin_proofs::types::{
77
UnpaddedBytesAmount,
88
};
99
use filecoin_proofs::{
10-
add_piece, generate_piece_commitment, generate_post, seal_commit, seal_pre_commit, verify_post,
11-
PrivateReplicaInfo, PublicReplicaInfo,
10+
add_piece, generate_candidates, generate_piece_commitment, generate_post, seal_commit,
11+
seal_pre_commit, verify_post, PrivateReplicaInfo, PublicReplicaInfo,
1212
};
1313
use log::info;
1414
use storage_proofs::sector::SectorId;
@@ -30,6 +30,7 @@ struct Inputs {
3030
#[derive(Serialize)]
3131
#[serde(rename_all = "kebab-case")]
3232
struct Outputs {
33+
candidates_cpu_time_ms: u64,
3334
proving_cpu_time_ms: u64,
3435
proving_wall_time_ms: u64,
3536
verifying_wall_time_ms: u64,
@@ -52,7 +53,7 @@ impl Report {
5253
}
5354

5455
pub fn run(sector_size: usize) -> Result<(), failure::Error> {
55-
info!("Benchy Rational PoSt: sector-size={}", sector_size);
56+
info!("Benchy Election PoSt: sector-size={}", sector_size);
5657

5758
let sector_size_unpadded_bytes_ammount =
5859
UnpaddedBytesAmount::from(PaddedBytesAmount(sector_size as u64));
@@ -139,21 +140,52 @@ pub fn run(sector_size: usize) -> Result<(), failure::Error> {
139140
// Measure PoSt generation and verification.
140141
let post_config = PoStConfig(SectorSize(sector_size as u64));
141142

142-
let gen_post_measurement =
143-
measure(|| generate_post(post_config, &CHALLENGE_SEED, &priv_replica_info))
144-
.expect("failed to generate PoSt");
143+
let gen_candidates_measurement = measure(|| {
144+
generate_candidates(post_config, &CHALLENGE_SEED, &priv_replica_info, PROVER_ID)
145+
})
146+
.expect("failed to generate post candidates");
147+
148+
let candidates = &gen_candidates_measurement.return_value;
149+
150+
let gen_post_measurement = measure(|| {
151+
generate_post(
152+
post_config,
153+
&CHALLENGE_SEED,
154+
&priv_replica_info,
155+
candidates
156+
.iter()
157+
.cloned()
158+
.map(Into::into)
159+
.collect::<Vec<_>>(),
160+
PROVER_ID,
161+
)
162+
})
163+
.expect("failed to generate PoSt");
145164

146165
let proof = &gen_post_measurement.return_value;
147166

148-
let verify_post_measurement =
149-
measure(|| verify_post(post_config, &CHALLENGE_SEED, proof, &pub_replica_info))
150-
.expect("failed to verify PoSt");
167+
let verify_post_measurement = measure(|| {
168+
verify_post(
169+
post_config,
170+
&CHALLENGE_SEED,
171+
proof,
172+
&pub_replica_info,
173+
&candidates
174+
.iter()
175+
.cloned()
176+
.map(Into::into)
177+
.collect::<Vec<_>>(),
178+
PROVER_ID,
179+
)
180+
})
181+
.expect("failed to verify PoSt");
151182

152183
// Create a JSON serializable report that we print to stdout (that will later be parsed using
153184
// the CLI JSON parser `jq`).
154185
let report = Report {
155186
inputs: Inputs { sector_size },
156187
outputs: Outputs {
188+
candidates_cpu_time_ms: gen_candidates_measurement.cpu_time.as_millis() as u64,
157189
proving_cpu_time_ms: gen_post_measurement.cpu_time.as_millis() as u64,
158190
proving_wall_time_ms: gen_post_measurement.wall_time.as_millis() as u64,
159191
verifying_cpu_time_ms: verify_post_measurement.cpu_time.as_millis() as u64,

‎fil-proofs-tooling/src/bin/benchy/main.rs

Lines changed: 6 additions & 6 deletions
Original file line numberDiff line numberDiff line change
@@ -3,8 +3,8 @@ extern crate serde;
33

44
use clap::{value_t, App, Arg, SubCommand};
55

6+
mod election_post;
67
mod hash_fns;
7-
mod rational_post;
88
mod stacked;
99

1010
fn main() {
@@ -89,8 +89,8 @@ fn main() {
8989
.help("Extract data after proving and verifying.")
9090
);
9191

92-
let rational_post_cmd = SubCommand::with_name("rational-post")
93-
.about("Benchmark Rational PoST")
92+
let election_post_cmd = SubCommand::with_name("election-post")
93+
.about("Benchmark Election PoST")
9494
.arg(
9595
Arg::with_name("size")
9696
.long("size")
@@ -105,7 +105,7 @@ fn main() {
105105
let matches = App::new("benchy")
106106
.version("0.1")
107107
.subcommand(stacked_cmd)
108-
.subcommand(rational_post_cmd)
108+
.subcommand(election_post_cmd)
109109
.subcommand(hash_cmd)
110110
.get_matches();
111111

@@ -132,11 +132,11 @@ fn main() {
132132
})
133133
.expect("stacked failed");
134134
}
135-
("rational-post", Some(m)) => {
135+
("election-post", Some(m)) => {
136136
let sector_size_kibs = value_t!(m, "size", usize)
137137
.expect("could not convert `size` CLI argument to `usize`");
138138
let sector_size = sector_size_kibs * 1024;
139-
rational_post::run(sector_size).expect("rational-post failed");
139+
election_post::run(sector_size).expect("election-post failed");
140140
}
141141
("hash-constraints", Some(_m)) => {
142142
hash_fns::run().expect("hash-constraints failed");

‎fil-proofs-tooling/src/bin/benchy/stacked.rs

Lines changed: 2 additions & 7 deletions
Original file line numberDiff line numberDiff line change
@@ -15,7 +15,6 @@ use fil_proofs_tooling::{measure, FuncMeasurement, Metadata};
1515
use storage_proofs::circuit::metric::MetricCS;
1616
use storage_proofs::circuit::stacked::StackedCompound;
1717
use storage_proofs::compound_proof::{self, CompoundProof};
18-
use storage_proofs::crypto::pedersen::JJ_PARAMS;
1918
use storage_proofs::drgraph::*;
2019
use storage_proofs::hasher::{Blake2sHasher, Domain, Hasher, PedersenHasher, Sha256Hasher};
2120
use storage_proofs::porep::PoRep;
@@ -316,16 +315,13 @@ fn do_circuit_work<H: 'static + Hasher>(
316315

317316
let compound_public_params = compound_proof::PublicParams {
318317
vanilla_params: pp.clone(),
319-
engine_params: &*JJ_PARAMS,
320318
partitions: Some(*partitions),
321319
};
322320

323321
if *bench || *circuit {
324322
let mut cs = MetricCS::<Bls12>::new();
325-
<StackedCompound as CompoundProof<_, StackedDrg<H, Sha256Hasher>, _>>::blank_circuit(
326-
&pp, &JJ_PARAMS,
327-
)
328-
.synthesize(&mut cs)?;
323+
<StackedCompound as CompoundProof<_, StackedDrg<H, Sha256Hasher>, _>>::blank_circuit(&pp)
324+
.synthesize(&mut cs)?;
329325

330326
report.outputs.circuit_num_inputs = Some(cs.num_inputs() as u64);
331327
report.outputs.circuit_num_constraints = Some(cs.num_constraints() as u64);
@@ -344,7 +340,6 @@ fn do_circuit_work<H: 'static + Hasher>(
344340
let gparams =
345341
<StackedCompound as CompoundProof<_, StackedDrg<H, Sha256Hasher>, _>>::groth_params(
346342
&compound_public_params.vanilla_params,
347-
&JJ_PARAMS,
348343
)?;
349344

350345
let multi_proof = {

‎filecoin-proofs/examples/stacked.rs

Lines changed: 2 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -26,7 +26,6 @@ use bellperson::Circuit;
2626
use storage_proofs::circuit::metric::*;
2727
use storage_proofs::circuit::stacked::StackedCompound;
2828
use storage_proofs::compound_proof::{self, CompoundProof};
29-
use storage_proofs::crypto::pedersen::JJ_PARAMS;
3029
use storage_proofs::drgraph::*;
3130
use storage_proofs::example_helper::prettyb;
3231
use storage_proofs::fr32::fr_into_bytes;
@@ -287,15 +286,14 @@ fn do_the_work<H: 'static>(
287286
if circuit || groth || bench {
288287
let compound_public_params = compound_proof::PublicParams {
289288
vanilla_params: pp.clone(),
290-
engine_params: &*JJ_PARAMS,
291289
partitions: Some(partitions),
292290
};
293291
if circuit || bench {
294292
info!("Performing circuit bench.");
295293
let mut cs = MetricCS::<Bls12>::new();
296294

297295
<StackedCompound as CompoundProof<_, StackedDrg<H, Blake2sHasher>, _>>::blank_circuit(
298-
&pp, &JJ_PARAMS,
296+
&pp,
299297
)
300298
.synthesize(&mut cs)
301299
.expect("failed to synthesize circuit");
@@ -320,7 +318,7 @@ fn do_the_work<H: 'static>(
320318
// and skip replication/vanilla-proving entirely.
321319
info!("Performing circuit groth.");
322320
let gparams =
323-
<StackedCompound as CompoundProof<_, StackedDrg<H, Blake2sHasher>, _>>::groth_params(&compound_public_params.vanilla_params, &JJ_PARAMS)
321+
<StackedCompound as CompoundProof<_, StackedDrg<H, Blake2sHasher>, _>>::groth_params(&compound_public_params.vanilla_params)
324322
.unwrap();
325323

326324
let multi_proof = {

‎filecoin-proofs/parameters.json

Lines changed: 32 additions & 32 deletions
Original file line numberDiff line numberDiff line change
@@ -1,44 +1,44 @@
11
{
2-
"v17-proof-of-spacetime-rational-535d1050e3adca2a0dfe6c3c0c4fa12097c9a7835fb969042f82a507b13310e0.params": {
3-
"cid": "QmeWrgK6EnuX4GBF1v7FtYSKYJQ9ykWLAmvyfGLSwapwdJ",
4-
"digest": "6350d9da17c4cad3f56b8bde1bc687f2",
5-
"sector_size": 16777216
6-
},
7-
"v17-proof-of-spacetime-rational-535d1050e3adca2a0dfe6c3c0c4fa12097c9a7835fb969042f82a507b13310e0.vk": {
8-
"cid": "QmYdxLfqg4MEjSyKFstizuKKPW1GtQSPLvZYf8x8WhdkGP",
9-
"digest": "eb14fa3adae47c0b717b718672fd522b",
10-
"sector_size": 16777216
11-
},
12-
"v17-proof-of-spacetime-rational-b99f15d0bdaaf4ffb68b2ca72b69ea8d915f66a2a56f667430ad69d87aa5febd.params": {
13-
"cid": "QmZNP7e6iJkAJotmCNnHGTtRPkjv6swjZ6w1fAcvqMJT77",
14-
"digest": "c0fcbd6bb7c7c94deee0e7040c3ac2c6",
15-
"sector_size": 1073741824
16-
},
17-
"v17-proof-of-spacetime-rational-b99f15d0bdaaf4ffb68b2ca72b69ea8d915f66a2a56f667430ad69d87aa5febd.vk": {
18-
"cid": "QmNkGKYiCuCnHcFi7R6y9ZRVYouErFyM8BnpAJvDbC8BdL",
19-
"digest": "401b9034863cdd4fb570b6986da7f815",
20-
"sector_size": 1073741824
21-
},
22-
"v17-proof-of-spacetime-rational-ba14a058a9dea194f68596f8ecf6537074f038a15c8d1a8550e10e31d4728912.params": {
23-
"cid": "Qmebne2YhvJ6ZRqxWxmV6v8D617yzqs6feNy4KmQ9AUFjo",
24-
"digest": "3f17b79c853de3b860ec625117b72b24",
2+
"v17-proof-of-spacetime-election-54266a3979de672c5f59a521e8e4a3fc2108da1eb019af74089ff6d05bfe83aa.params": {
3+
"cid": "QmYZLjoyx8J5aZoWbDD3vmTporR5b16bForUVfX2sVeeXH",
4+
"digest": "20b9c96462158f8bcb325d848234563a",
255
"sector_size": 1024
266
},
27-
"v17-proof-of-spacetime-rational-ba14a058a9dea194f68596f8ecf6537074f038a15c8d1a8550e10e31d4728912.vk": {
28-
"cid": "QmSA5zPNdFgMsSKEaGCzsbKENWP5zCee5cGxA9yYJWqu8E",
29-
"digest": "5a3e73274ae556367a01dd1584dda287",
7+
"v17-proof-of-spacetime-election-54266a3979de672c5f59a521e8e4a3fc2108da1eb019af74089ff6d05bfe83aa.vk": {
8+
"cid": "QmZFpWh8Cjw7p9TAbky4Tcky3xgUiKemzu1hXXG4vbGLqd",
9+
"digest": "a571d112489e6a6e22794b0d5c29f650",
3010
"sector_size": 1024
3111
},
32-
"v17-proof-of-spacetime-rational-c2ae2b440e693ee69fd6da9e85c4294c5c70c1a46d5785ca5f2a676d6cd4c8de.params": {
33-
"cid": "QmUCPsKSySgs1Fa5ypf5SeBr7V7V3erVPHKZo87A14JP19",
34-
"digest": "d0771726ef23c7c891ad9a8f2a830b4e",
12+
"v17-proof-of-spacetime-election-5dafbb8a1d6802f3b68f0dd2d3eff36d203263b4a867df986b1d434f1852726c.params": {
13+
"cid": "QmdDHHvQjgvhPmQZsBFKUK24NrMNRpUF5KSkFYFskxbWfJ",
14+
"digest": "14b90c09bc79a516024fffdf87c7df52",
3515
"sector_size": 268435456
3616
},
37-
"v17-proof-of-spacetime-rational-c2ae2b440e693ee69fd6da9e85c4294c5c70c1a46d5785ca5f2a676d6cd4c8de.vk": {
38-
"cid": "QmbKk6AwLokHETnpyvgRd1ZHQaCfkHsyNw57wZo9B8CehB",
39-
"digest": "8d6e73d14fcf63a1506028fe8fd58ff8",
17+
"v17-proof-of-spacetime-election-5dafbb8a1d6802f3b68f0dd2d3eff36d203263b4a867df986b1d434f1852726c.vk": {
18+
"cid": "QmXidpxzFXcgipsCSuaS7VfZ67rFefEzAJqD8oJW9Bs2m5",
19+
"digest": "a4ec1080131da0d21c8a115a0c06611c",
4020
"sector_size": 268435456
4121
},
22+
"v17-proof-of-spacetime-election-8923799c7eba68497833a94802c1f687aca037964aacbec838188c6d3912b718.params": {
23+
"cid": "QmVGafFXTC66w14YKp2xEncPiEPjWAxyfsd33e5QjN5hQE",
24+
"digest": "6833890a37442f94878a166b5b249448",
25+
"sector_size": 16777216
26+
},
27+
"v17-proof-of-spacetime-election-8923799c7eba68497833a94802c1f687aca037964aacbec838188c6d3912b718.vk": {
28+
"cid": "Qme5js3sto2rQCcx1MJ9bdzABtZ486uD55jkKq2R6qWCQ1",
29+
"digest": "46530f7bade51c44b1686ee92dd04204",
30+
"sector_size": 16777216
31+
},
32+
"v17-proof-of-spacetime-election-8ac1992c7b71f15757f62e11af51abc3d3b9db8ed8ed6804f4f06842d6d23bc5.params": {
33+
"cid": "QmXq4pUVkTioDUssjQC5ptWAWueWZjL7UMxWBh1qGyp2aM",
34+
"digest": "9d4ee0259e12b631204d65f694472806",
35+
"sector_size": 1073741824
36+
},
37+
"v17-proof-of-spacetime-election-8ac1992c7b71f15757f62e11af51abc3d3b9db8ed8ed6804f4f06842d6d23bc5.vk": {
38+
"cid": "QmdeNG4HVF82g5K9fNEzCZ9cspC8AmuYdCiRcHnuwiaPGG",
39+
"digest": "e005916854080fe243f86b9635e4aced",
40+
"sector_size": 1073741824
41+
},
4242
"v17-stacked-proof-of-replication-0c0b444c6f31d11c8e98003cc99a3b938db26b77a296d4253cda8945c234266d.params": {
4343
"cid": "QmU5eYfCwyjsJ4perRgVW6KTq1bDLfQ7XcBhg9Z1mH42kQ",
4444
"digest": "3d6d4ba0b2eae2695dfd73c916a66c95",

0 commit comments

Comments
 (0)