IRI solid milestone tracker crash: GC overhead limit exceeded

Question

I have been trying to get my testnet node to fully sync for the last week. I have a EC2 M4.xlarge instance:

• CentOS7 64bit
• testnet iri 1.4.1.3
• Oracle JDK 1.8.0.152
• Xmx 5GB
• neighbors with 5 community nodes that are fully sync'd

It takes about 4hours for the solid milestone tracker to start ticking over when I start from scratch i.e. deleting testnetdb and testnet.log folders.

Incidentally, I ran m4.large with Xmx3G and after a few hours, Solid milestone tracker thread crashed with GC overhead limit exceeded.

I'm looking for advice and guidance:

• Is this more or less expected?
• Where to look if I have a misconfiguration or sub-optimal setup?

Here is my iri.ini:

[IRI]
PORT = 14600
TCP_RECEIVER_PORT = 14700
UDP_RECEIVER_PORT = 14700
NEIGHBORS = 
IXI_DIR = ixi
HEADLESS = true
DEBUG = false
DB_PATH = /opt/iota/testnetdb
TESTNET = true

My systemd unit file:

[Unit]
Description=IOTA IRI full node
Requires=network.target
After=syslog.target network.target
[Service]
WorkingDirectory=/opt/iota
ExecStart=/usr/local/java/bin/java -Xmx5g -Djava.net.preferIPv4Stack=true -jar /opt/iota/iri-1.4.1.3.jar -c /opt/iota/iri.ini --remote
User=iri
[Install]
WantedBy=multi-user.target

Update I've given up on a 24/7 testnet node. Instead, I'm having success running a mainnet node using the same configs (of course with prod values, using mainnet jar, testnet=false, etc.). I'll attribute the success to a more fluid supply of reliable neighbors and downloadable sync'ed databases.

Answer 1

When running with Xmx3G you are limiting the memory to 3Gb. In Java, when memory consumption raises up to the limit : a process called the GC (Garbage collector) start to try to free memory (by releasing old objects not used anymore).

The problem with the GC is that (under some circumstances) it is a "stop the world" process. i.e. your app cannot do anything else while it is running: it freezes. Therefore, there is limitation on the duration of this "stop the world period". If GC is unable to free enough memory in the given period of time : it crashes.

Increasing Xmx to something higher may solve the problem or make it worse. It really depends on application design. The best thing to do is to carefully read the setup recommendations for running a fullnode. (there are many ways to tune the GC, i.e. many command line args that you can give to the JVM at start up so that your app will run smoothly).

When looking at source code: the default memory setup to run a full node is this :

-Xmx8g -Xms256m

So running it with -Xmx3G seems simply "not enough"