Can't write snapshot after CSV import?

Hey there,

:bug: Bug Report

Description

Following the doc to import a csv, I encountered an error telling me that I couldn’t run the import as root. So I passed --user memgraph to run the import (volumes are properly filled):

docker run --user memgraph -v mg_lib:/var/lib/memgraph -v mg_import:/import-data   --entrypoint=mg_import_csv memgraph   --nodes /import-data/nodes_nobackslash.csv   --relationships /import-data/firstMedges.csv --skip-bad-relationships true

But memgraph fails to write the snapshot with this command.

Steps to reproduce

  1. Create container with volume: docker container create --name mg_import_helper -v mg_import:/import-data busybox
  2. Fill volume with docker cp
  3. Run the import helper (see command in description)

Expected behavior

Success, enjoy memgraph.

Actual behavior

Got a critical error after a successful import:

[critical] Assertion failed with message: 'While trying to open "/var/lib/memgraph/snapshots/20211025134732055815_timestamp_23887642" for writing an error occured: Permission denied (13)'

Your environment

Docker Desktop on macOS (fresh upgrade, all settings to defaults)

Let me know if you need extra details!

1 Like

Thank you for reporting this problem @mughetto ! :grinning_face_with_smiling_eyes:

While we try to replicate the behavior on our side, I have a potential solution you could try out.
Instead of using the --user memgraph option with the mg_import_csv command, try to use the same --user flag in the first step while creating the busybox container:

docker container create --user memgraph --name mg_import_helper -v mg_import:/import-data busybox

I have updated the instructions on the documentation guide so you can follow them along.

Good point will try this out!

Hey there,

passing --user memgraph when creating the busybox to fill the import volume doesn’t change the result. I still get an assert error.

Hi there mughetto,

I managed to replicate the bug on macOS and I’ll try to find a permanent fix.

Temporarily, you can change the snapshot repository to the one where the user running the docker process has access to, e.g. /Users/mughetto/repos/memgraph/ (be sure to create that folder beforehand).

Hope this helps! :sweat_smile:

1 Like

Ah great!

I was actually checking the rights on the dirs and indeed the snapshot directory doesn’t have the right for the memgraph user:

$ ls -alh /var/lib/memgraph/
total 36K
drwxrwxrwx 9 memgraph memgraph 4.0K Oct 25 10:44 .
drwxr-xr-x 1 root     root     4.0K Oct  5 05:01 ..
-rw-r----- 1 memgraph memgraph    0 Oct 25 10:36 .lock
drwxr-xr-x 2 memgraph memgraph 4.0K Oct 25 13:07 auth
drwxr-xr-x 2 memgraph memgraph 4.0K Oct 25 13:07 settings
drwxr-xr-x 2 root     root     4.0K Oct 25 10:36 snapshots
drwxr-xr-x 2 memgraph memgraph 4.0K Oct 25 13:07 streams
drwxr-xr-x 2 memgraph memgraph 4.0K Oct 25 13:07 telemetry
drwxr-xr-x 2 memgraph memgraph 4.0K Oct 25 13:07 triggers
drwxr-xr-x 2 root     root     4.0K Oct 25 10:36 wal

Re-reading your answer I have a doubt. The problem happens between the memgraph container that runs the import and the mg_import volume. Why would my local user be involved in the process? (especially with Docker Desktop)

1 Like

Here is the explanation of my previous comment, I’m not sure if this still holds, since osxfs was replaced.

I think the solution to this problem is simply changing a default docker setting:

  1. Go to docker desktop → settings → resources → file sharing → add new → /var. You can read more about it in the official docker documentation.
  2. Restart the docker engine.

We didn’t manage to reproduce on Catalina 10.15.7, but on BigSur 11.6 we had to add this setting.

Here is the complete process I did, once I changed the setting that worked:
Instead of using mg_import and mg_lib for the docker volume first argument (-v), I use an absolute path to the directory that is used as the memgraph lib/import directory, e.g. /Users/medo/repos/memgraph/mg_lib. Read more about it in the official docker documentation. That doesn’t mean it won’t work with creating volumes manually, but it can be a hassle.
Then I ran:

docker run -v /Users/medo/repos/memgraph/debug/mg_lib:/var/lib/memgraph -v /Users/medo/repos/memgraph/debug/mg_import:/bla --entrypoint=mg_import_csv memgraph/memgraph-platform --nodes /bla/people_nodes.csv

And it worked like a charm.

I hope this helps you! If you need more assistance feel free to continue the thread. Good luck! :slight_smile:

2 Likes

Ok I understand! For now I went a different route since I needed a machine with much more RAM than my laptop :smiley: So I ended up on a system that requires me to use Singularity which in turn has the “advantage” of mounting the local filesystem automatically ^^