Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

LLM-based PII redaction #585

Draft
wants to merge 1 commit into
base: main
Choose a base branch
from
Draft

LLM-based PII redaction #585

wants to merge 1 commit into from

Conversation

sarahyurick
Copy link
Collaborator

@sarahyurick sarahyurick commented Mar 12, 2025 •
edited
Loading

TODO:

  • Core implementation
  • Examples directory (and README)
  • Scripts directory (including README and pyproject.toml)
  • User guide
  • Unit tests

@sarahyurick
add async llm modifier and llm modifier
bbbf0bb 
Signed-off-by: Sarah Yurick <sarahyurick@gmail.com>
sarahyurick
sarahyurick commented Mar 12, 2025
View reviewed changes
Copy link
Collaborator Author

@sarahyurick sarahyurick left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This PR adds support for both an AsyncLLMPiiModifier and an LLMPiiModifier. Both versions exist to match existing functionalities, see: #136 and #527.

nemo_curator/modifiers/async_llm_pii_modifier.py
system_prompt: str = SYSTEM_PROMPT,
pii_labels: List[str] = PII_LABELS,
):
self.client = AsyncOpenAI(base_url=base_url, api_key=api_key)
Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I could use our existing AsyncOpenAIClient from here, although I do not see an immediate benefit to this. It seems to just be a wrapper.

nemo_curator/modifiers/llm_pii_modifier.py
system_prompt: str = SYSTEM_PROMPT,
pii_labels: List[str] = PII_LABELS,
):
self.client = OpenAI(base_url=base_url, api_key=api_key)
Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I could use our existing OpenAIClient from here, although I do not see an immediate benefit to this. It seems to just be a wrapper.

nemo_curator/modifiers/async_llm_pii_modifier.py
pii_entities_lists = await self._gather(tasks)
return pii_entities_lists

async def _gather(
Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Same _gather function as from here.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers
No reviews
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
1 participant
@sarahyurick