Skip to content

Commit dc01b6b

Browse files
authored
Memory issue (#206)
* Refactored code for enrichment * Issue #203 adding guard rails * changing command * fixing config * fixing config * fixing config * Fixing job * Fixing job * Fixing job * Fixing job * Fixing job * Fixing job * Fixing job * Fixing job * Adding tests for Gurad Rails * Fixed #203 added simple checks fr the memory * #203 Added guard rails * #203 Added guard rails * Updating TOC * Fixing unit test * Hardening unit tests * #203 renaming properties, removing the 'PROP_' prefix * Adding a report * Simplifying code * Adding ioException test * Renamed the error file and generalized it as a constant
1 parent a4c30bf commit dc01b6b

File tree

4 files changed

+169
-48
lines changed

4 files changed

+169
-48
lines changed

README.md

Lines changed: 30 additions & 25 deletions
Original file line numberDiff line numberDiff line change
@@ -56,20 +56,21 @@ The basic method for using this library is, that you create a definition for you
5656
- [File Entry Limitations](#file-entry-limitations)
5757
- [File Size Limitations](#file-size-limitations)
5858
- [Memory Limitations](#memory-limitations)
59+
- [Exporting Anomalies Report](#exporting-anomalies-report)
5960
- [Changelog](#changelog)
60-
- [1.11.3 (In-Progress)](#1113-in-progress)
61-
- [1.11.2](#1112)
62-
- [1.11.0](#1110)
63-
- [1.0.10](#1010)
64-
- [1.0.8.2](#1082)
65-
- [1.0.8](#108)
66-
- [1.0.7](#107)
67-
- [1.0.6](#106)
68-
- [1.0.5](#105)
69-
- [1.0.4](#104)
70-
- [1.0.3](#103)
71-
- [1.0.1](#101)
72-
<!-- TOC -->
61+
_ [1.11.3 (In-Progress)](#1113--in-progress-)
62+
_ [1.11.2](#1112)
63+
_ [1.11.0](#1110)
64+
_ [1.0.10](#1010)
65+
_ [1.0.8.2](#1082)
66+
_ [1.0.8](#108)
67+
_ [1.0.7](#107)
68+
_ [1.0.6](#106)
69+
_ [1.0.5](#105)
70+
_ [1.0.4](#104)
71+
_ [1.0.3](#103)
72+
_ [1.0.1](#101)
73+
<!-- TOC -->
7374

7475
## Installation
7576

@@ -549,23 +550,23 @@ As of 1.11.3 we have introduced a series of guard rails. These allow you to cont
549550

550551
The following table lists all available guard rail properties and their default values:
551552

552-
| Property | Description | Affects | Scale | Default Value |
553-
| ---------------------------------------- | ------------------------------------------------------ | ----------------------------------------------- | ---------- | ------------- |
554-
| PROP_LOGPARSER_FILEENTRY_LIMIT | Maximum number of entries to parse per file | File parsing | Count | -1 (disabled) |
555-
| PROP_LOGPARSER_FILESIZE_LIMIT | Maximum file size in MB to parse | File parsing | Megabytes | -1 (disabled) |
556-
| PROP_LOGPARSER_HEAP_LIMIT | Maximum heap size increase in MB before warning | File parsing, FilterBy, Search, enrich, groupBy | Megabytes | -1 (disabled) |
557-
| PROP_LOGPARSER_MEMORY_LIMIT_PERCENTAGE | Maximum percentage of memory usage before warning | File parsing, FilterBy, Search, enrich, groupBy | Percentage | -1 (disabled) |
558-
| PROP_LOGPARSER_EXCEPTION_ON_MEMORY_LIMIT | Whether to throw exception when memory limits exceeded | Memory Checks | Boolean | false |
553+
| Property | Description | Affects | Scale | Default Value |
554+
| ----------------------------------- | ------------------------------------------------------ | ----------------------------------------------- | ---------- | ------------- |
555+
| LOGPARSER_FILEENTRY_LIMIT | Maximum number of entries to parse per file | File parsing | Count | -1 (disabled) |
556+
| LOGPARSER_FILESIZE_LIMIT | Maximum file size in MB to parse | File parsing | Megabytes | -1 (disabled) |
557+
| LOGPARSER_HEAP_LIMIT | Maximum heap size increase in MB before warning | File parsing, FilterBy, Search, enrich, groupBy | Megabytes | -1 (disabled) |
558+
| LOGPARSER_MEMORY_LIMIT_PERCENTAGE | Maximum percentage of memory usage before warning | File parsing, FilterBy, Search, enrich, groupBy | Percentage | -1 (disabled) |
559+
| LOGPARSER_EXCEPTION_ON_MEMORY_LIMIT | Whether to throw exception when memory limits exceeded | Memory Checks | Boolean | false |
559560

560561
### File Entry Limitations
561562

562-
For whatever reason, you may want to set a limit on the number of entries you can extract from a file. This cab be done by setting the system property _PROP_LOGPARSER_FILEENTRY_LIMIT_ .
563+
For whatever reason, you may want to set a limit on the number of entries you can extract from a file. This cab be done by setting the system property _LOGPARSER_FILEENTRY_LIMIT_ .
563564

564565
When set, the log parser stops parsing after reaching the limit in a file, and moves to the next file. Whenever this happens we log a WARNING and add the skipped file to our internal list of issues.
565566

566567
### File Size Limitations
567568

568-
For whatever reason, you may want to set a limit on the number of entries you can extract from a file. This cab be done by setting the system property _PROP_LOGPARSER_FILESIZE_LIMIT_ .
569+
For whatever reason, you may want to set a limit on the number of entries you can extract from a file. This cab be done by setting the system property _LOGPARSER_FILESIZE_LIMIT_ .
569570

570571
When set, the we create a warning regarding the file size, and store it among the file size issues.
571572

@@ -575,13 +576,17 @@ Although we will not stop a process from executing due to memory issues, we prov
575576

576577
These limitations are set with the following System properties:
577578

578-
- _PROP_LOGPARSER_HEAP_LIMIT_ : Setting a limit above which we log these occurences.
579-
- _PROP_LOGPARSER_MEMORY_LIMIT_PERCENTAGE_ : A percentage of the occupied memory in reference to the max memory.
579+
- _LOGPARSER_HEAP_LIMIT_ : Setting a limit above which we log these occurences.
580+
- _LOGPARSER_MEMORY_LIMIT_PERCENTAGE_ : A percentage of the occupied memory in reference to the max memory.
580581

581-
We also have the possibility of throwing an exception iin the case of surpassing the memory rules. This is activated by setting the System property _PROP_LOGPARSER_EXCEPTION_ON_MEMORY_LIMIT_ to true.
582+
We also have the possibility of throwing an exception iin the case of surpassing the memory rules. This is activated by setting the System property _LOGPARSER_EXCEPTION_ON_MEMORY_LIMIT_ to true.
582583

583584
You can also call the memory guard rails in your own implementation by calling `ParseGuardRails.checkMemoryLimits()`. This will check both heap and memory percentage limits.
584585

586+
### Exporting Anomalies Report
587+
588+
We have the possibility of exporting the anomalies report. This is done by calling the method `LogData#exportAnomaliesReport(String fileName)`. If you do not give an argument `LogData#exportAnomaliesReport()` will export the anomalies to a file called anomalies.json.
589+
585590
## Changelog
586591

587592
### 1.11.3 (In-Progress)

src/main/java/com/adobe/campaign/tests/logparser/core/LogData.java

Lines changed: 6 additions & 5 deletions
Original file line numberDiff line numberDiff line change
@@ -37,6 +37,7 @@
3737
*/
3838
public class LogData<T extends StdLogEntry> {
3939

40+
public static final String STD_LOG_ERROR_ON_EMPTY_LOG_DATA = "No Log data to export. Please load the log data before re-attempting";
4041
protected static Logger log = LogManager.getLogger();
4142

4243
/**
@@ -418,7 +419,7 @@ public File exportLogDataToCSV() throws LogDataExportToFileException {
418419
.fetchEscapedTitle()
419420
+ "-export.csv");
420421
} else {
421-
log.warn("No Log data to export. Please load the log data before re-attempting");
422+
log.warn(STD_LOG_ERROR_ON_EMPTY_LOG_DATA);
422423
return null;
423424
}
424425

@@ -437,7 +438,7 @@ public File exportLogDataToCSV(String in_fileName) {
437438
if (l_firstEntry != null) {
438439
return exportLogDataToCSV(l_firstEntry.fetchHeaders(), in_fileName);
439440
} else {
440-
log.warn("No Log data to export. Please load the log data before re-attempting");
441+
log.warn(STD_LOG_ERROR_ON_EMPTY_LOG_DATA);
441442
return null;
442443
}
443444
}
@@ -484,7 +485,7 @@ public File exportLogDataToHTML(String in_reportTitle, String in_htmlFileName) {
484485
T l_firstEntry = this.fetchFirst();
485486

486487
if (l_firstEntry == null) {
487-
log.error("No Log data to export. Please load the log data before re-attempting");
488+
log.error(STD_LOG_ERROR_ON_EMPTY_LOG_DATA);
488489
return null;
489490
}
490491
return exportLogDataToHTML(l_firstEntry.fetchHeaders(), in_reportTitle,
@@ -548,7 +549,7 @@ public File exportLogDataToJSON() throws LogDataExportToFileException {
548549
return exportLogDataToJSON(l_firstEntry.fetchHeaders(),
549550
l_firstEntry.getParseDefinition().fetchEscapedTitle() + "-export.json");
550551
} else {
551-
log.warn("No Log data to export. Please load the log data before re-attempting");
552+
log.warn(STD_LOG_ERROR_ON_EMPTY_LOG_DATA);
552553
return null;
553554
}
554555
}
@@ -565,7 +566,7 @@ public File exportLogDataToJSON(String in_jsonFileName) throws LogDataExportToFi
565566
if (l_firstEntry != null) {
566567
return exportLogDataToJSON(l_firstEntry.fetchHeaders(), in_jsonFileName);
567568
} else {
568-
log.warn("No Log data to export. Please load the log data before re-attempting");
569+
log.warn(STD_LOG_ERROR_ON_EMPTY_LOG_DATA);
569570
return null;
570571
}
571572
}

src/main/java/com/adobe/campaign/tests/logparser/utils/ParseGuardRails.java

Lines changed: 42 additions & 10 deletions
Original file line numberDiff line numberDiff line change
@@ -9,11 +9,13 @@
99
package com.adobe.campaign.tests.logparser.utils;
1010

1111
import java.io.File;
12+
import java.io.IOException;
1213
import java.util.HashMap;
1314
import java.util.List;
1415
import java.util.Map;
1516
import java.util.Set;
1617

18+
import com.fasterxml.jackson.databind.ObjectMapper;
1719
import org.apache.logging.log4j.LogManager;
1820
import org.apache.logging.log4j.Logger;
1921

@@ -29,17 +31,17 @@ public class ParseGuardRails {
2931
protected static Map<String, Long> heapLimitations = new HashMap<>();
3032
protected static Map<String, Double> memoryLimitations = new HashMap<>();
3133

32-
34+
public static final String ANOMALY_REPORT_PATH = "./logParserAnomalies.json";
3335
public static long HEAP_SIZE_AT_START = MemoryUtils.getCurrentHeapSizeMB();
34-
public static int FILE_ENTRY_LIMIT = Integer.parseInt(System.getProperty("PROP_LOGPARSER_FILEENTRY_LIMIT", "-1"));
35-
public static long HEAP_LIMIT = Integer.parseInt(System.getProperty("PROP_LOGPARSER_HEAP_LIMIT", "-1"));
36+
public static int FILE_ENTRY_LIMIT = Integer.parseInt(System.getProperty("LOGPARSER_FILEENTRY_LIMIT", "-1"));
37+
public static long HEAP_LIMIT = Integer.parseInt(System.getProperty("LOGPARSER_HEAP_LIMIT", "-1"));
3638
public static double MEMORY_LIMIT_PERCENTAGE = Double
37-
.parseDouble(System.getProperty("PROP_LOGPARSER_MEMORY_LIMIT_PERCENTAGE", "-1"));
39+
.parseDouble(System.getProperty("LOGPARSER_MEMORY_LIMIT_PERCENTAGE", "-1"));
3840
protected static boolean EXCEPTION_ON_MEMORY_LIMIT = Boolean
39-
.parseBoolean(System.getProperty("PROP_LOGPARSER_EXCEPTION_ON_MEMORY_LIMIT", "false"));
41+
.parseBoolean(System.getProperty("LOGPARSER_EXCEPTION_ON_MEMORY_LIMIT", "false"));
4042

4143
protected static long FILE_SIZE_LIMIT = Long
42-
.parseLong(System.getProperty("PROP_LOGPARSER_FILESIZE_LIMIT", "-1"));
44+
.parseLong(System.getProperty("LOGPARSER_FILESIZE_LIMIT", "-1"));
4345
protected static int MEASUREMENT_SCALE = 1024 * 1024;
4446

4547
public static void reset() {
@@ -172,12 +174,42 @@ private static boolean hasReachedFileSizeLimit(long length) {
172174
public static Map<String, Set<String>> getAnomalyReport() {
173175
Map<String, Set<String>> report = new HashMap<>();
174176

175-
report.put("heapLimitations", heapLimitations.keySet());
176-
report.put("memoryLimitations", memoryLimitations.keySet());
177-
report.put("fileSizeLimitations", fileSizeLimitations.keySet());
178-
report.put("entryLimitations", entryLimitations.keySet());
177+
Map.of(
178+
"heapLimitations", heapLimitations,
179+
"memoryLimitations", memoryLimitations,
180+
"fileSizeLimitations", fileSizeLimitations,
181+
"entryLimitations", entryLimitations).forEach((key, map) -> {
182+
if (!map.isEmpty()) {
183+
report.put(key, map.keySet());
184+
}
185+
});
179186

180187
return report;
181188
}
182189

190+
/**
191+
* Exports the anomaly report to a JSON file
192+
* The file will be created if it doesn't exist, or replaced if it does
193+
* Only exports if there are anomalies to report
194+
*/
195+
public static void exportAnomalyReport() {
196+
exportAnomalyReport(ANOMALY_REPORT_PATH);
197+
}
198+
199+
/**
200+
* Exports the anomaly report to a JSON file at the specified path
201+
*
202+
* @param filePath The path where to save the anomaly report
203+
*/
204+
public static void exportAnomalyReport(String filePath) {
205+
Map<String, Set<String>> report = getAnomalyReport();
206+
if (!report.isEmpty()) {
207+
try {
208+
ObjectMapper mapper = new ObjectMapper();
209+
mapper.writeValue(new File(filePath), report);
210+
} catch (IOException e) {
211+
log.error("Failed to export anomaly report to {}", filePath, e);
212+
}
213+
}
214+
}
183215
}

src/test/java/com/adobe/campaign/tests/logparser/utils/ParseGuardRailsTest.java

Lines changed: 91 additions & 8 deletions
Original file line numberDiff line numberDiff line change
@@ -12,31 +12,43 @@
1212
import static org.hamcrest.Matchers.is;
1313
import static org.testng.Assert.assertThrows;
1414

15+
import java.io.File;
1516
import java.io.IOException;
1617
import java.nio.file.Files;
1718
import java.nio.file.Path;
19+
import java.util.List;
20+
import java.util.Map;
21+
import java.util.Set;
1822

23+
import org.hamcrest.Matchers;
1924
import org.testng.annotations.AfterMethod;
2025
import org.testng.annotations.BeforeMethod;
2126
import org.testng.annotations.Test;
2227

2328
import com.adobe.campaign.tests.logparser.exceptions.MemoryLimitExceededException;
29+
import com.fasterxml.jackson.databind.ObjectMapper;
2430

2531
public class ParseGuardRailsTest {
2632

2733
private Path tempFile;
34+
private static final String ANOMALY_REPORT_PATH = ParseGuardRails.ANOMALY_REPORT_PATH;
2835

2936
@BeforeMethod
3037
public void setup() throws IOException {
3138
ParseGuardRails.reset();
32-
39+
System.clearProperty("EXCEPTION_ON_MEMORY_LIMIT");
3340
tempFile = Files.createTempFile("test", ".log");
41+
// Clean up any existing anomaly report
42+
new File(ANOMALY_REPORT_PATH).delete();
3443
}
3544

3645
@AfterMethod
3746
public void cleanup() throws IOException {
3847
ParseGuardRails.reset();
48+
System.clearProperty("EXCEPTION_ON_MEMORY_LIMIT");
3949
Files.deleteIfExists(tempFile);
50+
// Clean up the anomaly report
51+
new File(ANOMALY_REPORT_PATH).delete();
4052
}
4153

4254
@Test
@@ -135,19 +147,18 @@ public void testCheckMemoryLimits_WhenHeapLimitReached() {
135147
assertThat("Should reach limit when heap limit is reached",
136148
ParseGuardRails.checkMemoryLimits(), is(true));
137149

138-
assertThat("Should have anomaly report", ParseGuardRails.getAnomalyReport().size(), is(4));
150+
assertThat("Should have anomaly report", ParseGuardRails.getAnomalyReport().size(), is(1));
139151
assertThat("Should have heap limitation", ParseGuardRails.getAnomalyReport().get("heapLimitations").size(),
140152
is(1));
141153

142-
assertThat("Should have memory limitation", ParseGuardRails.getAnomalyReport().get("memoryLimitations").size(),
143-
is(0));
154+
assertThat("Should not have a memory limitation",
155+
!ParseGuardRails.getAnomalyReport().containsKey("memoryLimitations"));
144156

145157
assertThat("Should have file size limitation",
146-
ParseGuardRails.getAnomalyReport().get("fileSizeLimitations").size(),
147-
is(0));
158+
!ParseGuardRails.getAnomalyReport().containsKey("fileSizeLimitations"));
148159

149-
assertThat("Should have entry limitation", ParseGuardRails.getAnomalyReport().get("entryLimitations").size(),
150-
is(0));
160+
assertThat("Should have entry limitation",
161+
!ParseGuardRails.getAnomalyReport().containsKey("entryLimitations"));
151162
}
152163

153164
@Test
@@ -179,4 +190,76 @@ public void testCheckMemoryLimits_WhenMemoryLimitReachedWithException() {
179190
ParseGuardRails.MEMORY_LIMIT_PERCENTAGE = 0.0; // Set limit to 0% to force reaching it
180191
assertThrows(MemoryLimitExceededException.class, () -> ParseGuardRails.checkMemoryLimits());
181192
}
193+
194+
@Test
195+
public void testExportAnomalyReport_WhenNoAnomalies() {
196+
ParseGuardRails.exportAnomalyReport();
197+
assertThat("Should not create file when no anomalies",
198+
(new File(ANOMALY_REPORT_PATH)).exists(), is(false));
199+
}
200+
201+
@Test
202+
public void testExportAnomalyReport_WhenHasAnomalies() throws IOException {
203+
// Create some anomalies
204+
ParseGuardRails.HEAP_LIMIT = 1;
205+
ParseGuardRails.HEAP_SIZE_AT_START = -20;
206+
ParseGuardRails.checkMemoryLimits();
207+
208+
ParseGuardRails.exportAnomalyReport();
209+
210+
// Verify file exists
211+
File reportFile = new File(ANOMALY_REPORT_PATH);
212+
assertThat("Should create file when anomalies exist",
213+
reportFile.exists(), is(true));
214+
215+
// Verify content
216+
ObjectMapper mapper = new ObjectMapper();
217+
Map<String, List<String>> report = mapper.readValue(reportFile, Map.class);
218+
assertThat("Should have heap limitations",
219+
report.containsKey("heapLimitations"));
220+
assertThat("Should have at least one heap limitation",
221+
report.get("heapLimitations").size(), Matchers.greaterThan(0));
222+
}
223+
224+
@Test
225+
public void testExportAnomalyReport_WhenFileExists() throws IOException {
226+
// Create initial file
227+
ObjectMapper mapper = new ObjectMapper();
228+
Map<String, Set<String>> initialData = Map.of("test", Set.of("data"));
229+
mapper.writeValue(new File(ANOMALY_REPORT_PATH), initialData);
230+
231+
// Create some anomalies
232+
ParseGuardRails.HEAP_LIMIT = 1;
233+
ParseGuardRails.HEAP_SIZE_AT_START = -20;
234+
ParseGuardRails.checkMemoryLimits();
235+
236+
ParseGuardRails.exportAnomalyReport();
237+
238+
// Verify file was replaced
239+
Map<String, Set<String>> report = mapper.readValue(new File(ANOMALY_REPORT_PATH), Map.class);
240+
assertThat("Should have replaced old content",
241+
report.containsKey("heapLimitations"), is(true));
242+
assertThat("Should not have old content",
243+
report.containsKey("test"), is(false));
244+
}
245+
246+
@Test
247+
public void testExportAnomalyReport_WhenIOExceptionOccurs() throws IOException {
248+
// Create some anomalies
249+
ParseGuardRails.HEAP_LIMIT = 1;
250+
ParseGuardRails.HEAP_SIZE_AT_START = -20;
251+
ParseGuardRails.checkMemoryLimits();
252+
253+
// Create a file that will cause an IOException when trying to write
254+
File reportFile = new File(ANOMALY_REPORT_PATH);
255+
reportFile.createNewFile();
256+
reportFile.setReadOnly();
257+
258+
// This should log an error but not throw an exception
259+
ParseGuardRails.exportAnomalyReport();
260+
261+
// Clean up
262+
reportFile.setWritable(true);
263+
reportFile.delete();
264+
}
182265
}

0 commit comments

Comments
 (0)