add env for max level of flattening allowed for events #1320
Conversation
WalkthroughA new configurable option, Changes
Sequence Diagram(s)sequenceDiagram
participant User
participant CLI
participant Options
participant JSONFlattener
User->>CLI: Provide --event-flatten-level or env var
CLI->>Options: Parse and store event_flatten_level
JSONFlattener->>Options: Access event_flatten_level
JSONFlattener->>JSONFlattener: Check nesting vs. event_flatten_level
JSONFlattener-->>CLI: Flatten or skip based on level
Poem
Note ⚡️ AI Code Reviews for VS Code, Cursor, WindsurfCodeRabbit now has a plugin for VS Code, Cursor and Windsurf. This brings AI code reviews directly in the code editor. Each commit is reviewed immediately, finding bugs before the PR is raised. Seamless context handoff to your AI code agent ensures that you can easily incorporate review feedback. Note ⚡️ Faster reviews with cachingCodeRabbit now supports caching for code and dependencies, helping speed up reviews. This means quicker feedback, reduced wait times, and a smoother review experience overall. Cached data is encrypted and stored securely. This feature will be automatically enabled for all accounts on May 30th. To opt out, configure 📜 Recent review detailsConfiguration used: CodeRabbit UI 📒 Files selected for processing (1)
🚧 Files skipped from review as they are similar to previous changes (1)
⏰ Context from checks skipped due to timeout of 90000ms (10)
✨ Finishing Touches
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. 🪧 TipsChatThere are 3 ways to chat with CodeRabbit:
SupportNeed help? Create a ticket on our support page for assistance with any issues or questions. Note: Be mindful of the bot's finite context window. It's strongly recommended to break down tasks such as reading entire modules into smaller chunks. For a focused discussion, use review comments to chat about specific files and their changes, instead of using the PR comments. CodeRabbit Commands (Invoked using PR comments)
Other keywords and placeholders
CodeRabbit Configuration File (
|
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
There was a problem hiding this comment.
Actionable comments posted: 1
📜 Review details
Configuration used: CodeRabbit UI
Review profile: CHILL
Plan: Pro
Cache: Disabled due to data retention organization setting
Knowledge Base: Disabled due to data retention organization setting
📒 Files selected for processing (4)
src/cli.rs(1 hunks)src/handlers/http/modal/utils/ingest_utils.rs(1 hunks)src/utils/json/flatten.rs(3 hunks)src/utils/json/mod.rs(2 hunks)
⏰ Context from checks skipped due to timeout of 90000ms (10)
- GitHub Check: Build Kafka aarch64-apple-darwin
- GitHub Check: Build Kafka x86_64-unknown-linux-gnu
- GitHub Check: Build Default x86_64-pc-windows-msvc
- GitHub Check: Build Default aarch64-apple-darwin
- GitHub Check: Build Default x86_64-apple-darwin
- GitHub Check: Quest Smoke and Load Tests for Distributed deployments
- GitHub Check: Build Default aarch64-unknown-linux-gnu
- GitHub Check: Build Default x86_64-unknown-linux-gnu
- GitHub Check: coverage
- GitHub Check: Quest Smoke and Load Tests for Standalone deployments
🔇 Additional comments (5)
src/cli.rs (1)
381-390: Well-structured configuration parameter for event flatteningThe new configuration parameter
event_flatten_levelis well-defined with appropriate documentation, command-line argument, and environment variable mapping. This will allow users to control the maximum depth of nested JSON structures during flattening operations.src/utils/json/mod.rs (2)
22-22: Import updated to reflect renamed functionThe import correctly reflects the renamed function from
has_more_than_four_levelstohas_more_than_max_allowed_levels, ensuring consistency with the changes made to the flattening logic.
46-46: Function call updated to use configurable flattening levelThe conditional check now uses the renamed function and passes the starting level value of 1 correctly. This change properly integrates the configurable flattening level instead of the previous hardcoded value.
src/utils/json/flatten.rs (2)
28-28: Added required import for PARSEABLE singletonAdded import for the
PARSEABLEsingleton which is needed to access the configurable flattening level.
319-334: Function renamed and implemented with configurable flattening levelThe function has been appropriately renamed from
has_more_than_four_levelstohas_more_than_max_allowed_levelsand its logic updated to use the configurableevent_flatten_levelparameter instead of the hardcoded value of 4. The documentation and recursion calls have been updated accordingly.This change improves flexibility by allowing administrators to configure the maximum nesting level based on their specific use cases and performance requirements.
env `P_MAX_FLATTEN_LEVEL` to control the maximum level of flattening allowed
default to 10
this is to ensure nested list type fields do not get created
eg. with current implementation of hard coded level of 4,
field gets created with data type -
```
{
"name": "Records_resources_accountId",
"data_type": {
"List": {
"name": "item",
"data_type": {
"List": {
"name": "item",
"data_type": "Utf8",
"nullable": true,
"dict_id": 0,
"dict_is_ordered": false,
"metadata": {}
}
},
"nullable": true,
"dict_id": 0,
"dict_is_ordered": false,
"metadata": {}
}
},
"nullable": true,
"dict_id": 0,
"dict_is_ordered": false,
"metadata": {}
}
```
after this change, data type changes to -
```
{
"name": "Records_resources_accountId",
"data_type": "Utf8",
"nullable": true,
"dict_id": 0,
"dict_is_ordered": false,
"metadata": {}
}
```
nikhilsinhaparseable
force-pushed
the
log-source-kinesis
branch
from
7d47b16 to
b7f8aff
Compare
env
P_MAX_FLATTEN_LEVELto control the maximum level of flattening allowed default to 10this is to ensure nested list type fields do not get created eg. with current implementation of hard coded level of 4, field gets created with data type -
after this change, data type changes to -
Fixes #XXXX.
Description
This PR has:
Summary by CodeRabbit
New Features
Improvements
Bug Fixes