Memgraph frequently hang when pumping data

I installed memgraph-2.3.0-1 on 3 Ubuntu 20.04.4 LTS servers (KVM virtual machine, 24CPUs, 64Gb Memory) by deb package with 1 MAIN instance + 2 REPLICA instances.
Using neo4j-java-driver (version 4.1.1) to puming data to memgraph.
Main instance frequently was hang when pumping data (all cpu cores are 100% and mgconsole can not connect to main instaince).
Check in log file no have informations.
Here is configuration file of memgraph:
–bolt-num-workers=16
–bolt-session-inactivity-timeout=300
–log-file=/var/log/memgraph/memgraph.log
–log-level=DEBUG
–query-cost-planner=true
–query-execution-timeout-sec=600
–query-max-plans=100
–query-vertex-count-to-expand-existing=100
–storage-properties-on-edges=true
–storage-recover-on-startup=true
–storage-snapshot-interval-sec=300
–storage-wal-enabled=true
–data-directory=/data2/memgraph
–isolation-level=SNAPSHOT_ISOLATION
–telemetry-enabled=true
Picture of CPU load when memgraph was hang.


What’s issue with main instance.

Interesting! Thanks for the details (seems like you have an advanced setup already :smile:) but it is hard to say what is going on…

In general, it would be great if we could access/download the representative dataset to recreate the issue? If no, you can maybe provide SSH access to the machine (feel free to send a direct message)?

If that’s also a no go, you can try to debug by yourself and report more details back :smile: Since htop is here (which is great), you can enable thread names to see more accurately what is consuming the resources, in htop that is possible to do by executing:

  • t (to toggle tree view)
  • F2 (for the setup menu)
  • Setup → Display options → Show custom thread names

Hopefully, after this you can see which Memgraph thread is consuming most of the resources, if that won’t show enough, we can compile in Debug/RelWithDebInfo mode and try to run under GDB and observe what is going on, but that’s quite expensive in terms of time.

ADDITIONAL QUESTION: Are you running replication on a single machine, why is that so, just for the sake of testing for now?

Looking forward to more info!

First of all, thanks for the feedback.
In htop that is possible to do by executing:

  • t (to toggle tree view)
  • F2 (for the setup menu)
  • Setup → Display options → Show custom thread names

Answer to the question “Are you running replication on a single machine, why is that so, just for the sake of testing for now?” . I running on three Ubuntu 20.04.4 LTS servers. 1 server for MAIN instance , 2 servers for replica instance. Because I want to test read-only on replica to make sure the data is synchronized in real time or not. My application needs to sync data in real time.
I will contact if there is any problem and have a question, why every time I restart memgraph, it loses replication configuration (both main and replica instainces)?

The htop seems regular, how many nodes and edges are in the graph?

Regarding replication, yes, we had a couple of huge bugs on the replication side which are fixed now (the config loss is one of the fixes → docs/changelog.md at release-2.4.0 · memgraph/docs · GitHub), we will publish the 2.4 early next week (we’ll start the release process today), stay tuned for the new binary!

Yes, replication is hang. Our database around 300 milion nodes, 200 milion edges, and growing will very fast.
Thanks.

Nice scale! We are currently developing a fully shared Memgraph, would you like to become a design partner?

we are testing for scalability, reliability, fault tolerance and query speed. Does memgraph have write on multi nodes ( mulit master instances)?

Not yet, but working on that quite intensively. At the moment, Memgraph has really good stats on a single machine. Soon, we’ll have a multi-shard implementation, happy to add your workload on the design (ofc if it’s a reasonable graph workload) side because in that case, we’ll actually test testing workload + scale → we’ll be able to continuously improve this particular workload. Scale is always tricky, everything breaks at a certain level.

Hi @duykhuong! Memgraph team just released 2.4 with a bunch of stuff, but most notably the fixes of replication that you also noticed :smile: More details are under changelog Changelog | Memgraph Docs Any feedback is welcome!

Thank @buda. We wil testing