Not ready for prime time...

[philsie1]

There's some potential here but there are some serious performance and data management issues that needed to be fixed if this is ever to see the light of day again. I have successfully gotten it up to date with the new JSON fields, fixed problems that arise when you don't have the entire block chain, added features to arbitrarily download portions of block chain, and simplified a lot of things that cause it to choke down. I haven't uploaded my changes yet but hope to soon.

However, my recommendation is to learn what you can from this and design your own; and ditch Gephi and use later Neo4J... GLTA

[willmoss]

You can perform this same exercise just using Neo4J without any need for external software

[philsie1]

Yes, I’m trying that; using my limited Neo4J skills.

Have you been working with this as well? I had this dropped in my lap a few months ago and asked if I could make it work. It all works (after a lot of fixes) but there are still a lot of issues that often require manual fixing; and it’s glacially slow.

Thanks for the feedback…

                Phil

Philip Brown
Data Engineering & Biometrics
The MITRE Corporation

"I drank what?" -- Socrates

From: willmoss [mailto:notifications@github.com]
Sent: Tuesday, November 18, 2014 4:29 PM
To: thallium205/BitcoinVisualizer
Cc: Brown, Phil
Subject: Re: [BitcoinVisualizer] Not ready for prime time... (#17)

You can perform this same exercise just using Neo4J without any need for external software

—
Reply to this email directly or view it on GitHubhttps://github.com//issues/17#issuecomment-63548350.

[willmoss]

Hi, Yes I have completed this exercise already. Infact I'm working on a web front-end for the Neo4J database. You can view it at http://blocksnfo.bxmediaus.com ... although it's only a small section of the blockchain I indexed. Kind of a work in progress.

There are other examples of blockchain indexing around. Best idea is to check the bitcointalk.org forums. However afaik this is the only one that uses Neo4J.

[nii236]

Willmoss are you still working on this? I'm about to start redo-ing this project and would be nice to work to see where you're at.

[philsie1]

FYI, after some significant changes I was able to get BCV up and running and performing all functions except for scraping.

However, due to serious performance and design issues in BCV I decided on a new approach that gets me from a year’s worth of JSON files to a fully populated graph DB in hours instead of weeks. If interested I can share some of my insights and changes.

                            p

Philip Brown
Data Engineering & Biometrics
The MITRE Corporation

"I drank what?" -- Socrates

From: John Nguyen [mailto:notifications@github.com]
Sent: Thursday, May 07, 2015 9:57 AM
To: thallium205/BitcoinVisualizer
Cc: Brown, Phil
Subject: Re: [BitcoinVisualizer] Not ready for prime time... (#17)

Willmoss are you still working on this? I'm about to start redo-ing this project and would be nice to work to see where you're at.

—
Reply to this email directly or view it on GitHubhttps://github.com//issues/17#issuecomment-99877065.

[nii236]

I just found out about Cayley DB and think it suits our purposes. I'll hack away at it for awhile first then let you know how I go!

[nii236]

@philsie1, I'm preparing to load the entire blockchain into a graph db. The methodology for working out the owner level graph is quiet complicated though. Are you following the same system that was outlined in this repo? Or have you made improvements?

[philsie1]

I made significant changes in those regards. With respect to loading the DB I highly recommend using the Bulk-load feature in the later versions of Neo4J. It works a little backwards from traditional where you’d create a node then add attributes. You create attributes and then create the node and provide the attr list. Also, commits don’t occur until you CLOSE the db and if it fails then you have to start over. But, you can break the job down to subsets of the data to ensure failure doesn’t cost you everything.

In regards to owner-level graph it doesn’t need to be so complicated. Don’t rely entirely on the DB to build this. It’s 1000 times more efficient (at the cost of RAM) to create a Map outside of the DB to build up related addresses. If you build the owner graph by iterating through the entire DB for every address in the DB then it may finish sometime shortly after Bitcoin is no longer in use. ☺ In general, anything you can do in memory to get the DB built will be infinitely faster than traversing the graph.

I also created an in-memory Map of the nodes as I was creating them for fast lookup. When bulk-loading you need only the node number to create relationships.

Also, using the Jackson JSON parser was about 100 times faster than the one used in the current release. I highly recommend getting at least 16GB of RAM and using a solid state drive to get the DB built.

To give you a sense of scale, with my stripped down version I was populating the DB with transactions at a starting rate of 10,000/sec and 1000-2000/sec as the DB got larger. There are 86,400 seconds in a day and many millions of transactions; plan wisely… ☺

                            Phil

From: John Nguyen [mailto:notifications@github.com]
Sent: Tuesday, May 12, 2015 4:07 AM
To: thallium205/BitcoinVisualizer
Cc: Brown, Phil
Subject: Re: [BitcoinVisualizer] Not ready for prime time... (#17)

@philsie1https://github.com/philsie1, I'm preparing to load the entire blockchain into a graph db. The methodology for working out the owner level graph is quiet complicated though. Are you following the same system that was outlined in this repo? Or have you made improvements?

—
Reply to this email directly or view it on GitHubhttps://github.com//issues/17#issuecomment-101182748.

[Rory-Powell]

@philsie1 I've recently taken into restoring this as well myself with the newest version of neo4j, I'd love to have a look at the progress you made before going much further if you still have your changes?

[philsie1]

There were some good ideas but I ended up rethinking the whole process. One huge factor was getting a better JSON reader; the one originally used would take 3-20 seconds per file. Use this instead:

import org.codehaus.jackson.JsonParseException;
import org.codehaus.jackson.map.JsonMappingException;
import org.codehaus.jackson.map.ObjectMapper;

It was 100 times faster. Next, I used the bulk-loading functionality now available. This is also a huge improvement over traditional loading; the tradeoff is that it does NOT log commits so there’s no recovery from failure. So it’s great to create the initial DB but for ongoing updates it’s best to use the other way.

My code base is entirely different from BCV so it wont be helpful to push my versions; plus mine contains MITRE copyright notices. However, I can give snippets easily enough.

Oh, when it comes to managing the wallets I found that keeping a hash-table during the load was a huge perf gain rather than querying the graph for existing nodes. Also, for my own purposes I preparsed all the JSON files into a BCV format with files for blocks and transactions inputs and outputs. Loading from that into a fresh graph was achieving 10-20 thousand transactions per second. I loaded a year of BTC Txs in 35 minutes. The BCV, once working, took 35 days to load as much.

Here’s my function for storing a transaction in the graph (I used relationship properties for the input/output values rather than create actual nodes for each input/output).

public void storeTransaction2(CSVTransaction tran)
{
long txNode;
long walletNode;
Map<String, Object> txProps = new HashMap<String, Object>();

          txProps.put("Index", tran.getTx_index());
          txProps.put("Hash", tran.getHash());
          txProps.put("Timestamp", tran.getTime());
          txProps.put("BlockIndex", tran.getBlockIndex());
          txNode = graphIns.createNode(txProps, txLabel);

          if (tran.getInputs().size() > 1) {
                 walletNode = graphIns.createNode(null, walletLabel);
          } else {
                 walletNode = 0;
          }

          // create node for each input operation and link it to the originating address
          for (CSVInput in : tran.getInputs())
          {
                 /**
                 * create input node with props, link to tx
                 */
                 Map<String, Object> inProps = new HashMap<String, Object>();

// inProps.put("InAddr", in.getAddrHash());
inProps.put("Timestamp", in.getTimestamp());
SimpleDateFormat sdf = new SimpleDateFormat("yyyy-MM-dd HH:mm:ss");
inProps.put("Date", sdf.format(in.getTimestamp() * 1000)); // takes millis
inProps.put("Amount", (double) ( (double) in.getAmount() / SATOSHI));
inProps.put("FromTxIndex", in.getPriorOutputTx());
inProps.put("FromTxN", in.getPriorOutputIndex());
// long inNode = graphIns.createNode(inProps, inLabel);
// long rel = graphIns.createRelationship(txNode, inNode, MBCRelationships.inputFrom, null);

                 /**
                 * lookup existing or create new address node
                 */
                 Long addrNode = addrMap2.get(in.getAddrHash());
                 if (addrNode == null) {
                       Map<String, Object> addrProps = new HashMap<String, Object>();
                       addrProps.put("Addr", in.getAddrHash());
                       addrNode = graphIns.createNode(addrProps, addrLabel);

                       addrMap2.put(in.getAddrHash(), addrNode);
                 }

                 // input withdraws from the address

// graphIns.createRelationship(inNode, addrNode, MBCRelationships.withdrawl, null);
graphIns.createRelationship(addrNode, txNode, MBCRelationships.withdraw, inProps);

                 if (walletNode != 0) {
                       graphIns.createRelationship(walletNode, addrNode, MBCRelationships.sameOwner, null);
                 }

          }

          // create output node for each operation and link to the destination address
          for (CSVOutput out : tran.getOutputs())
          {
                 Map<String, Object> outProps = new HashMap<String, Object>();

// outProps.put("OutAddr", out.getAddrHash());
SimpleDateFormat sdf = new SimpleDateFormat("yyyy-MM-dd HH:mm:ss");
outProps.put("Date", sdf.format(out.getTimestamp() * 1000)); // takes millis
outProps.put("Timestamp", out.getTimestamp());
outProps.put("Amount", (double) ((double) out.getAmount() / SATOSHI));
outProps.put("OutputNum", out.getN());
outProps.put("TxIndex", tran.getTx_index());
// long outNode = graphIns.createNode(outProps, outLabel);
// graphIns.createRelationship(txNode, outNode, MBCRelationships.outputTo, null);

                 Long addrNode = addrMap2.get(out.getAddrHash());
                 if (addrNode == null) {
                       Map<String, Object> addrProps = new HashMap<String, Object>();
                       addrProps.put("Addr", out.getAddrHash());
                       addrNode = graphIns.createNode(addrProps, addrLabel);

                       addrMap2.put(out.getAddrHash(), addrNode);
                 }

                 // output deposits into the address
                 graphIns.createRelationship(txNode, addrNode, MBCRelationships.deposit, outProps);

          }

   }

In the main function the creation of the graph inserter is:

Map<String, String> config = new HashMap<String, String>() {{
put("dump_configuration","false");
put("cache_type","none");
put("use_memory_mapped_buffers","false");
put("neostore.nodestore.db.mapped_memory","1G");
put("neostore.relationshipstore.db.mapped_memory","2G");
put("neostore.propertystore.db.mapped_memory","2G");
put("neostore.propertystore.db.strings.mapped_memory","1G");
}};

   BatchInserter graphIns = BatchInserters.inserter(pathToGraphDB, config);

NOTE: When you use a bulk inserter, most of the work is done when you CLOSE the graph DB at the end of the program. So fully expect that when you CLOSE the graph your program will grind for a long time while it creates the actual nodes and relationships. If the close fails then your load just failed.

System.out.print("Initiating graph shutdown.... this may take quite a while.\n");
graphIns.shutdown();
System.out.print("Graph is now shutdown and ready to use\n");

Cheers.

From: Rory Powell [mailto:notifications@github.com]
Sent: Saturday, March 26, 2016 3:51 PM
To: thallium205/BitcoinVisualizer BitcoinVisualizer@noreply.github.com
Cc: Brown, Phil phil@mitre.org
Subject: Re: [BitcoinVisualizer] Not ready for prime time... (#17)

@philsie1https://github.com/philsie1 I've recently taken into restoring this as well myself with the newest version of neo4j, I'd love to have a look at the progress you made before going much further if you still have your changes?

—
You are receiving this because you were mentioned.
Reply to this email directly or view it on GitHubhttps://github.com//issues/17#issuecomment-201919381

[Rory-Powell]

Thanks a lot for the input! The batch inserter api has been much more efficient at adding large volumes to the database.

My own project was originally using encrypted blockchain.dat files that can be downloaded using a blockchain client, however I found that a lot of the information that is available from blockchain.info was missing and many parse errors were occurring, so I scrapped that and took inspiration from the code in this repo and implemented downloading the raw data using Springs rest template, which uses jackson under the hood. I downloaded a few months of data overnight and persisted some of it in batches of transactions using an approach like your own. Still some memory issues as I'm working with 8gb ram.

Now I face analysing the data to be more meaningful, with the aim of building a web app for searching per address. I will need to modify the save approach to be more meaningful to my application, but the initial results look promising. Here is an output path of 3 steps from a given address so far:

Repo can be found here, it's a spring boot application with currently server only code, but I'll be updating it with an interface shortly. Feel free to send any input or questions https://github.com/Rory-Powell/blockchain-analysis

[philsie1]

Looks nice (albeit busy). If you’re looking to make a useful tool then consider the types of use-cases beyond the casual transaction walker. Narrowing the field (filtering) will be highly valued; limiting by date ranges and amounts. Being able to selectively graph parts of the query results; i.e. taking query results and further filtering down to a small subset (by manual selection or parameter range) and THEN creating a graph (static or live) from that.

Just things to think about…

            phil

From: Rory Powell [mailto:notifications@github.com]
Sent: Thursday, March 31, 2016 6:03 PM
To: thallium205/BitcoinVisualizer BitcoinVisualizer@noreply.github.com
Cc: Brown, Phil phil@mitre.org
Subject: Re: [thallium205/BitcoinVisualizer] Not ready for prime time... (#17)

Thanks a lot for the input! The batch inserter api has been much more efficient at adding large volumes to the database.

My own project was originally using encrypted blockchain.dat files that can be downloaded using a blockchain client, however I found that a lot of the information that is available from blockchain.info was missing and many parse errors were occurring, so I scrapped that and took inspiration from the code in this repo and implemented downloading the raw data using Springs rest template, which uses jackson under the hood. I downloaded a few months of data overnight and persisted some of it in batches of transactions using an approach like your own. Still some memory issues as I'm working with 8gb ram.

Now I face analysing the data to be more meaningful, with the aim of building a web app for searching per address. I will need to modify the save approach to be more meaningful to my application, but the initial results look promising. Here is an output path of 3 steps from a given address so far:

[image]https://cloud.githubusercontent.com/assets/8755148/14192153/c3ed47f8-f793-11e5-8df3-a30924decd5d.png

Repo can be found here, it's a spring boot application with currently server only code, but I'll be updating it with an interface shortly. Feel free to send any input or questions https://github.com/Rory-Powell/blockchain-analysis

—
You are receiving this because you were mentioned.
Reply to this email directly or view it on GitHubhttps://github.com//issues/17#issuecomment-204149688