clio and rippled config

[signalise]

Hey Guys,

Does rippled need to have the ledger history set to full for Clio to download the full history?

I followed the install instructions step by step and have a working Clio server. Although I want more history and by default, I only get the most recent rippled ledgers.

Also, a big thank you to the Clio project; its exactly what I needed ;)

Andy

[cjcobb23]

Hey there. clio will start extracting at the most recently ledger validated by default. You can specify start_sequence in the config, at the top level, which will cause clio to extract starting at that specific ledger. If rippled doesn't have the ledger in question, clio will wait until the ledger is present. Hope that helps!

[signalise]

Hey Mate, if I understand currently, I should align Clio and rippled ledgers by disabling rippled's #online_delete and setting Clio's [start_sequence] within rippled's [ledger_history] range.

For example, to start backfilling data from approximately 3 years ago I will attempt the following:

1. Update the rippled server's config to backfill approximately 3 years of ledgers /etc/opt/ripple/rippled.cfg

# At the rate of 1000 ledgers per hour for 3 years, 1000*8760*3
[ledger_history]
26280000

[node_db]
type=NuDB
path=/var/lib/rippled/db/nudb
#online_delete=2000
#advisory_delete=0

# Oversite on my part, rippled needs ips_fixed to get data.
[ips_fixed]
65.109.58.182 51235
18.117.241.219 51235
3.219.105.145 51235
13.49.231.59 51235
116.202.148.26 51235
94.130.172.27 51235
3.112.125.152 51235

3. Edit the Clio server's config file to start syncing within the rippled range /opt/clio/etc/config.json.

# Offset start sequence from current ledger - 26280000 (3 years of ledgers)
"start_sequence":  48718549

"etl_sources":
[
    {
        "ip":"127.0.0.1",
        "ws_port":"6006",
        "grpc_port":"50051"
    }
]

I'll give it a crack and let you know how it goes.

Thanks again,

Andy

[signalise]

OK, I have it working using the method above. Unfortunately, I calculate it will take 139 days to download the 3 years of ledgers I'm interested in. That said, I'm in contact with someone who might be able to provide me with a full db dump.

And for bonus points :) can you query the Cassandra db table clio.transactions and decode the transaction hash without using the API for analytics?

SELECT DECODE(transaction) FROM clio.transactions;

That would be cool.