6

When viewing the logs I notice the totalTransactions value increases and decreases. Given data is constantly being added to the tangle, shouldn't this number constantly be increasing?

5

Log statement:

log.info("toProcess = {} , toBroadcast = {} , toRequest = {} , toReply = {} / totalTransactions = {}", getReceiveQueueSize(), getBroadcastQueueSize() , transactionRequester.numberOfTransactionsToRequest() , getReplyQueueSize(), TransactionViewModel.getNumberOfStoredTransactions(tangle));

To get a better understanding, lets follow the breadcrumbs and see what we can find. Let start with our entry method 'getNumberOfStoredTransactions' associated with our 'TransactionViewModel' class.


com.iota.iri.controllers.TransactionViewModel::getNumberOfStoredTransactions:

public static int getNumberOfStoredTransactions(final Tangle tangle) throws Exception {
    return tangle.getCount(Transaction.class).intValue();
}

We see here, the 'tangle' object method 'getCount' is called. Let's translate the parameter to understand what this method looks like.

com.iota.iri.storage.Tangle::getCount:

public Long getCount(Class<?> modelClass) throws Exception {  
    long value = 0;
    for(PersistenceProvider provider: this.persistenceProviders) {
        if((value = provider.count(modelClass)) != 0) {  
            break;
        }
    }  
    return value;  
}  

In viewing the statements above, we are simply contacting our persistence data store for the total count of 'com.iota.iri.model.Transaction' stored. Given current data store is rocksdb, lets view the 'count' method defined in the 'RocksDBPersistenceProvider' class.

com.iota.iri.storage.rocksDB.RocksDBPersistenceProvider::count:

public long count(Class<?> model) throws Exception {
    return getCountEstimate(model);
}

At a glance, we see we are simply calling a local class method named 'getCountEstimate'.

private long getCountEstimate(Class<?> model) throws RocksDBException {
    ColumnFamilyHandle handle = classTreeMap.get().get(model);
    return db.getLongProperty(handle, "rocksdb.estimate-num-keys");
}  

Finally getting to the meat of the 'totalTransactions' value means. Without getting into the details of the 'rocksdb' library, reviewing their FAQ ( https://github.com/facebook/rocksdb/wiki/RocksDB-FAQ ) we learn that:

Q: How to know the number of keys stored in a RocksDB database?

A: Use GetIntProperty(cf_handle, “rocksdb.estimate-num-keys") to obtain an estimated number of keys stored in a column family, or use GetAggregatedIntProperty(“rocksdb.estimate-num-keys", &num_keys) to obtain an estimated number of keys stored in the whole RocksDB database.

Q: Why GetIntProperty can only returns an estimated number of keys in a >RocksDB database?

A: Obtaining an accurate number of keys in any LSM databases like RocksDB is a challenging problem as they have duplicate keys and deletion entries (i.e., tombstones) that will require a full compaction in order to get an accurate number of keys. In addition, if the RocksDB database contains merge operators, it will also make the estimated number of keys less accurate.

As we followed the breadcrumbs, we learned the value shown in the log is an estimated figure and not absolute.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.