How can I analyse some tangles generated in the mainnet to compare with synthetic ones generated using a simulator?

Question

I’m a researcher

I’m trying to analyse some tangles generated in the mainnet to compare with synthetic ones I generated using a simulator

I have hard time understanding how to get a database. I’ve got two options here, either retrieve a database from https://dbfiles.iota.org/?prefix=chrysalis-mainnet/ or to use MQTT and listen to a node and build a dynamic tangle. The problem is I have no idea what is the structure of the stuff I get from either method. Could anyone help me understand it a bit better?

Just to make sure what I want to do is clear, I just want to be able to get a simple graph structure that I can parse in Python from a real database to analyse it

Thank you

Answer 1

IOTA full-nodes (such as HORNET and Bee) are able to start up from a more recent block/milestone instead of having to synchronize from genesis. This can be achieved by bootstrapping the IOTA full-node with recent snapshots. The files you find on https://chrysalis-dbfiles.iota.org/ are snapshots and serve as a starting point for the nodes. Snapshots only contain very specifc data - they don't contain all the messages of the tangle. Here you can find more information about what data snapshots contain and what how they work: https://github.com/luca-moser/protocol-rfcs/blob/local-snapshot-file-format/text/0000-local-snapshot-file-format/0000-local-snapshot-file-format.md I guess the best and most easy way is to subscribe to MQTT. It will give you all messages that are happening on the tangle. More information about the MQTT events can be found here: https://playground.asyncapi.io/?load=https://raw.githubusercontent.com/luca-moser/protocol-rfcs/rfc/node-event-api/text/0033-node-event-api/0033-node-event-api.yml The messages topic returns all the messages (serialized) that were attached to the tangle. The messages/referenced topic returns the actual message ids that were referenced by a milestone.

To build a simple graph structure from it you could use the message id as identifier for the vertex and link it with it's parents.

Also you could start a Hornet node for example, let it sync a while, and then dump everything with the node REST API. Starting at a message X you could follow the parents or the children of the message.

Probably easier than MQTT

So the options are:

• Dump message data from a full-node by the REST API. Starting at a message X you can traverse the tangle in both directions. Keep in mind full-nodes might not contain all the historic message data (depends on which snapshot the node was bootstrapped). Information about the REST API can be found here: https://editor.swagger.io/?url=https://raw.githubusercontent.com/rufsam/protocol-rfcs/master/text/0026-rest-api/rest-api.yaml
• Dump message data from a permanode (more message history on permanodes)
• Listen to MQTT topics to build a dynamic tangle, message per message

Credit goes to Vidal Attias and Samuel Rufinatscha