[BUG]: nv-ingest service RAM continuously increases and eventually fails

[alexliap]

Version

latest

Describe the bug.

I am using the ingest api to sequentially ingest batch of document and during this procedure the memory used by compose-nv-ingest-ms-runtime-1 is continuously increasing

i have deployed the whole blueprint using the docker compose files

Minimum reproducible example

Relevant log output

Full env printout

Other/Misc.

No response

Code of Conduct

• I agree to follow THIS PROJECT's Code of Conduct
• I have searched the open bugs and have found no duplicates for this bug report

[sumitkbh]

Nv-ingest parallelizes jobs to speed up ingestion and if you are uploading large number of documents it is expected to increase system memory usage. If you want to reduce memory usage you can try setting INGEST_DISABLE_DYNAMIC_SCALING to False but this will impact ingestion speed.

ref: https://gitlab-master.nvidia.com/chat-labs/OpenSource/rag/-/blob/main/deploy/compose/docker-compose-ingestor-server.yaml?ref_type=heads#L156

[alexliap]

I am ingesting sequentially batches of 16 pdfs, where each one is 2-3 pages max.
shouldnt Nv-ingest be freeing up memory after it is done with one batch? or at least not increasing?

[alexliap]

@sumitkbh The issue is solved btw, but i dont understand why. what changed with this variable?