3

I have been trying to get my testnet node to fully sync for the last week. I have a EC2 M4.xlarge instance:

  • CentOS7 64bit
  • testnet iri 1.4.1.3
  • Oracle JDK 1.8.0.152
  • Xmx 5GB
  • neighbors with 5 community nodes that are fully sync'd

It takes about 4hours for the solid milestone tracker to start ticking over when I start from scratch i.e. deleting testnetdb and testnet.log folders.

Incidentally, I ran m4.large with Xmx3G and after a few hours, Solid milestone tracker thread crashed with GC overhead limit exceeded.

I'm looking for advice and guidance:

  • Is this more or less expected?
  • Where to look if I have a misconfiguration or sub-optimal setup?

Here is my iri.ini:

[IRI]
PORT = 14600
TCP_RECEIVER_PORT = 14700
UDP_RECEIVER_PORT = 14700
NEIGHBORS = 
IXI_DIR = ixi
HEADLESS = true
DEBUG = false
DB_PATH = /opt/iota/testnetdb
TESTNET = true

My systemd unit file:

[Unit]
Description=IOTA IRI full node
Requires=network.target
After=syslog.target network.target
[Service]
WorkingDirectory=/opt/iota
ExecStart=/usr/local/java/bin/java -Xmx5g -Djava.net.preferIPv4Stack=true -jar /opt/iota/iri-1.4.1.3.jar -c /opt/iota/iri.ini --remote
User=iri
[Install]
WantedBy=multi-user.target

Update I've given up on a 24/7 testnet node. Instead, I'm having success running a mainnet node using the same configs (of course with prod values, using mainnet jar, testnet=false, etc.). I'll attribute the success to a more fluid supply of reliable neighbors and downloadable sync'ed databases.

  • How would anyone point you to a misconfiguration without you sharing the configuration? – Helmar Dec 6 '17 at 6:56
  • Added systemd unit file and iri.ini. Holler if you think anything else can help. – 80chew Dec 6 '17 at 11:04
2

When running with Xmx3G you are limiting the memory to 3Gb. In Java, when memory consumption raises up to the limit : a process called the GC (Garbage collector) start to try to free memory (by releasing old objects not used anymore).

The problem with the GC is that (under some circumstances) it is a "stop the world" process. i.e. your app cannot do anything else while it is running: it freezes. Therefore, there is limitation on the duration of this "stop the world period". If GC is unable to free enough memory in the given period of time : it crashes.

Increasing Xmx to something higher may solve the problem or make it worse. It really depends on application design. The best thing to do is to carefully read the setup recommendations for running a fullnode. (there are many ways to tune the GC, i.e. many command line args that you can give to the JVM at start up so that your app will run smoothly).

When looking at source code: the default memory setup to run a full node is this :

-Xmx8g -Xms256m

So running it with -Xmx3G seems simply "not enough"

  • Yes, was thinking the same. But was trawling through slack and saw others having luck on OS's with 4GB – 80chew Dec 6 '17 at 11:07
  • 1
    @80chew It also seems to depend on whether your are syncing a huge Tangle from scratch or were part of the network before it reached this size. Because in the synced state IRI takes up much less RAM than while it is syncing. – cmpn Dec 6 '17 at 11:13

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.