Meilisearch index sync cause heap out error when MongoDB has high volume data

[maskinoshita]

What happened?

If I check the number of indexes in MongoDB and Meilisearch and they do not match, I restore the Meilisearch indexes from MongoDB, but in doing so I get a heap out error.

In my case, this happened when I exported mongodb data from LibreChat, which had accumulated data after a long period of operation, imported it into mongodb in another environment, and started Librechat. The number of indexes in mongodb and Meilisearch did not match and Index sync was executed, which caused a heap out error. (Please check the attached log)

Referring to the Index Sync code below, it seems that the heap is running out because all the data from mongodb and meilisearch are loaded in-memory.

LibreChat/api/models/plugins/mongoMeili.js

Line 43 in b22f1c1

static async syncWithMeili() {

Steps to Reproduce

1. Prepare mongodb with high volume data and dump
   mongodump --archive --gzip --uri ${MONGO_URI} > /tmp/mongodb.gz
2. Create a new LibreChat environment with small memory that easy to happen heap out
3. Restore mongodb in new environment
   mongorestore --drop --archive=/tmp/mongodb.gz --gzip --uri ${MONGO_URI}
4. Restart api container and fired Index sync, then heap out error will be occurred

What browsers are you seeing the problem on?

No response

Relevant log output

2024-08-18T13:18:34.014321836Z {"level":"info","message":"Connected to MongoDB","timestamp":"2024-08-18T13:18:34.014Z"}
2024-08-18T13:18:35.556894871Z {"level":"debug","message":"[indexSync] There are 782266 messages and 781908 indexed","timestamp":"2024-08-18T13:18:35.556Z"}
2024-08-18T13:18:35.557189445Z {"level":"debug","message":"[indexSync] There are 84064 convos and 84063 indexed","timestamp":"2024-08-18T13:18:35.557Z"}
2024-08-18T13:18:35.557514877Z {"level":"debug","message":"[indexSync] Messages out of sync, indexing","timestamp":"2024-08-18T13:18:35.557Z"}
2024-08-18T13:18:35.558372842Z {"level":"debug","message":"[indexSync] Convos out of sync, indexing","timestamp":"2024-08-18T13:18:35.558Z"}
...
2024-08-18T13:18:35.952324879Z {"level":"info","message":"Server listening on all interfaces at port 3080. Use http://localhost:3080 to access it","timestamp":"2024-08-18T13:18:35.952Z"}
2024-08-18T13:19:11.306281736Z 
2024-08-18T13:19:11.306338623Z <--- Last few GCs --->
2024-08-18T13:19:11.306348321Z 
2024-08-18T13:19:11.306356887Z [1:0x7de5fa3b3010]    38825 ms: Mark-Compact (reduce) 2044.6 (2085.2) -> 2044.4 (2083.2) MB, 584.24 / 0.00 ms  (+ 66.6 ms in 7 steps since start of marking, biggest step 45.3 ms, walltime since start of marking 662 ms) (average mu = 0.484, current mu = 0.[1:0x7de5fa3b3010]    39959 ms: Mark-Compact 2047.6 (2085.3) -> 2047.1 (2089.1) MB, 1118.23 / 0.00 ms  (average mu = 0.243, current mu = 0.014) allocation failure; scavenge might not succeed
2024-08-18T13:19:11.306377917Z 
2024-08-18T13:19:11.306389989Z 
2024-08-18T13:19:11.306398616Z <--- JS stacktrace --->
2024-08-18T13:19:11.306407092Z 
2024-08-18T13:19:11.306415277Z FATAL ERROR: Reached heap limit Allocation failed - JavaScript heap out of memory
2024-08-18T13:19:11.306578009Z ----- Native stack trace -----
2024-08-18T13:19:11.306595912Z

Screenshots

No response

Code of Conduct

• I agree to follow this project's Code of Conduct

[danny-avila]

what are your resources when doing this? is it 2 GB RAM?

I wonder if this can be alleviated by increasing the maximum heap size for Node.js using the --max-old-space-size flag alone.

Perhaps this can be revisited. It does use batch/bulk operations for the mongo side, at least.

LibreChat/api/models/plugins/mongoMeili.js

Lines 35 to 37 in b22f1c1

	* Note: This strategy does not use batch operations for Meilisearch as the `index.addDocuments` will discard
	* the entire batch if there's an error with one document, and will not throw an error if there's an issue.
	* Also, `index.getDocuments` needs an exact limit on the amount of documents to return, so we build the map in batches.

[maskinoshita]

Thanks for the quick response. I have allocated 4GB for the api container and 2GB is automatically allocated for --max-old-space-size.

My current pay-as-you-go plan for Azure Container Apps has a limit on the maximum memory size for each container and only allocates 4GB. Therefore, I cannot increase the --max-old-space-size any further.

We have tried the same with another large VM environment with 32GB memory, but in this case, Index Sync works because all data is on the heap. I think increasing the --max-old-space-size will also work.

I think the problem is that Index sync requires a large amount of memory depending on the capacity of MongoDB.

[pedrojreis]

That does make sense.
I'm also getting this issue as we increase our user base and they are processing files.
Our rag and LibreChat Deployments are not reaching the 3GB limits we setup (we will increase our limits).

@danny-avila do you think that setting env var NODE_OPTIONS with --max-old-space-size=8192 would work with the original image? I'm asking because we are setting it in Dockerfile and i'm not sure if the env var will override that option.

If you see any better way to attack this problem, let me know and I can pop a PR :)

[danny-avila]

Meilisearch is the root cause of this issue. After running into this situation myself, this is what helped the most (edited my compose file, the same can be done with the docker override file):

  meilisearch:
    container_name: chat-meilisearch
    image: getmeili/meilisearch:v1.12.6
    env_file:
      - .env
    environment:
      - MEILI_HOST=http://meilisearch:7700
      - MEILI_NO_ANALYTICS=true
      - MEILI_MAX_INDEXING_MEMORY=500MB # NEW
      - MEILI_MAX_INDEXING_THREADS=2 # NEW
      - MEILI_EXPERIMENTAL_MAX_NUMBER_OF_BATCHED_TASKS=5 # NEW

I may even put this in the default config. the only downside is that with loads of data, indexing may take a while.

Reference: https://www.meilisearch.com/docs/learn/indexing/ram_multithreading_performance

[pedrojreis]

Super helpful!
But even if the culprit is Meilisearch shouldn't we transform the setting as a variable somehow?
I believe that it allows LibreChat instances to grow vertically (super important in productive k8s instances), people may need a "super pod" with loads of memory and this kinda breaks it.

[Lazyshot]

We have also run into this issue where LibreChat crashes due to OOM. We have --max-old-space-size=12288 and memory up to 16GB. We also have up to 6 instances of LibreChat running. Another thing we've seen is that if timing isn't great, we could end up with multiple instances attempting to sync the indexes at the same time. It might be a good idea to use the existing redis to put a lock on this process.

As well, loading the entire collection into memory seems a bit extreme.