Skip to content

Monitor, warn and/or prevent from high CPU/RAM consumption #13

@0237h

Description

@0237h

Problem

When starting data extraction on a large block range or on a job handling large amounts of data, the program will consume all the available memory, possibly rendering the host PC irresponsive if left for too long.

This hasn't been tested extensively but is visible with any sizable workload sent to the tool.

Proposition

Several safeguards could be put in place to prevent this (ranging from most amount of work to least):

  1. Monitor resources used, warn and possibly take actions (pause asyncio tasks, send processed blocks to post-process, save state, etc.) when certain threshold are attained.
  2. Just warn in case of high memory usage detected (so still include monitoring from the first option).
  3. Warn preemptively by judging the size of the workload at the beginning.
  4. Only warn in the documentation.

Metadata

Metadata

Assignees

Labels

enhancementNew feature or request

Type

No type

Projects

No projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions