Skip to content

Add memory guard-rails #203

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
baubakg opened this issue Apr 23, 2025 · 0 comments
Closed

Add memory guard-rails #203

baubakg opened this issue Apr 23, 2025 · 0 comments

Comments

@baubakg
Copy link
Member

baubakg commented Apr 23, 2025

We need two types of guard rails.

  • Global memory guard rails.
  • File level data guard rails

File level guard rails

If a file has too many lines we stop at an upper limit of entries.

See store the skipped/cut files in a shared location

We have the following properties:

  • PROP_LOGPARSER_FILEENTRY_LIMIT
  • PROP_LOGPARSER_FILESIZE_LIMIT
  • PROP_LOGPARSER_HEAP_LIMIT
  • PROP_LOGPARSER_MEMORY_LIMIT_PERCENTAGE
  • PROP_LOGPARSER_EXCEPTION_ON_MEMORY_LIMIT
Property Description Affects Scale Default Value
PROP_LOGPARSER_FILEENTRY_LIMIT Maximum number of entries to parse per file File parsing Count -1 (disabled)
PROP_LOGPARSER_FILESIZE_LIMIT Maximum file size in MB to parse File parsing Megabytes -1 (disabled)
PROP_LOGPARSER_HEAP_LIMIT Maximum heap size increase in MB before warning File parsing, FilterBy, Search, enrich, groupBy Megabytes -1 (disabled)
PROP_LOGPARSER_MEMORY_LIMIT_PERCENTAGE Maximum percentage of memory usage before warning File parsing, FilterBy, Search, enrich, groupBy Percentage -1 (disabled)
PROP_LOGPARSER_EXCEPTION_ON_MEMORY_LIMIT Whether to throw exception when memory limits exceeded Error handling Boolean false
baubakg added a commit that referenced this issue Apr 23, 2025
@baubakg baubakg mentioned this issue Apr 23, 2025
10 tasks
baubakg added a commit that referenced this issue Apr 24, 2025
baubakg added a commit that referenced this issue Apr 24, 2025
baubakg added a commit that referenced this issue Apr 24, 2025
* Issue #203 adding guard rails
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant