Add tsv-duplicates.py script

[goodmami]

This pull request adds a script for detecting potential duplicate lemmas in OMW .tab files. We can also use this PR to fix the duplicate issues.

There are 3 kinds of duplicates detected:

• underscores, e.g., canard colvert and canard_colvert
• case differences, e.g., Renaissance and renaissance
• diacritics, e.g., nanoséconde and nanoseconde

For all of the above, duplicates are only detected by normalizing the lemma forms within a single synset. There may be duplicate synsets, but the script does not test for these. But here's an example of two synsets that may be duplicates have redundant lemmas:

WARNING:tsv-duplicates:duplicate of 11640645-n: 'sequoia', 'séquoia'
WARNING:tsv-duplicates:duplicate of 11640898-n: 'sequoia', 'séquoia'

You can run the script as follows:

$ python scripts/tsv-duplicates.py --ignore-case --underscore --diacritics wns/{arb,fra,msa}/*.tab
wn-data-arb.tab duplicates	1502 synsets	3013 lemmas
wn-nodia-arb.tab duplicates	176 synsets	355 lemmas
wn-data-fra.tab duplicates	3821 synsets	7702 lemmas
wn-data-ind.tab duplicates	466 synsets	934 lemmas
wn-data-zsm.tab duplicates	366 synsets	732 lemmas
total duplicates	6331 synsets	12736 lemmas

It takes a variable number of paths, so you can check one at a time or many at once. The --verbose option will print a warning for every duplicate it finds (best when only checking a single .tab file).

[ekaf]

The two sequoia synsets are not duplicates: one is for the "tree" and the other denotes the "wood" (i.e. material to make furniture). The issue was discussed in OEWN some years ago.

[fcbond]

Hi, we are not saying that 11640645-n and 11640898-n are duplicates, but that within each one 'sequoia' and 'séquoia' are duplicates (or at least variants of the same word).

…

On Thu, 31 Oct 2024 at 10:32, Eric Kafe ***@***.***> wrote: The two *sequoia* synsets are not duplicates: one is for the "tree" and the other denotes the "wood" (i.e. material to make furniture). The issue <globalwordnet/english-wordnet#78> was discussed in OEWN some years ago. — Reply to this email directly, view it on GitHub <#33 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAIPZRXVOJRMINXEVF7YQJLZ6H2L7AVCNFSM6AAAAABQYCNM72VHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDINBZGQYTOMZVHE> . You are receiving this because you are subscribed to this thread.Message ID: ***@***.***>

-- Francis Bond <https://fcbond.github.io/>

[ekaf]

Thanks @fcbond,I should have made it clearer that I was responding to this opening comment above, where @goodmami expresses his sentiment that these two synsets might be duplicates, which they aren't:

There may be duplicate synsets, but the script does not test for these. But here's an example of two synsets that may be duplicates:

WARNING:tsv-duplicates:duplicate of 11640645-n: 'sequoia', 'séquoia'
WARNING:tsv-duplicates:duplicate of 11640898-n: 'sequoia',

[goodmami]

Apologies, @ekaf, my wording was imprecise and my example ill-chosen. @fcbond is correct that the script looks for near-duplicate lemmas within a synset and not for multiple synsets that are duplicates of each other (I've updated the issue text above to hopefully make this more clear). The idea is the TSV files have some lemmas that are only trivially different within the same synset and that we'd rather not keep all of them. This example might be more illustrative:

WARNING:tsv-duplicates:duplicate of 15277118-n: 'mortalite', 'mortalité'

If we look at all the lemmas for that synset, we see two others that are more interestingly different:

$ grep 15277118-n wns/fra/wn-data-fra.tab
15277118-n	fra:lemma	taux de mortalité
15277118-n	fra:lemma	mortalite
15277118-n	fra:lemma	morbidité
15277118-n	fra:lemma	mortalité

My guess is that the mortalite without diacritics is redundant and can be removed from the TSV file.

[ekaf]

@goodmami, rather than just redundant, "the mortalite without diacritics" is incorrect. But, as you wrote earlier, only native lexicographers can make such corrections:

I don't think we can get around having human annotators to fix the upper/lower case, diacritics, and plurals.

Maybe a spell checker could detect some incorrect forms, but any orthographic editing would need to be approached with great caution.

[goodmami]

I created #48 only for the scripts so we can move forward with that. The modifications to the data should happen in other PRs.

I think this PR should be closed without merging since the commits adding the scripts would cause conflicts. The manual modifications to the Icelandic wordnet could be cherry-picked for a new PR. Alternatively, we could repurpose this PR with a force-push that rebases without those commits. Let me know if you want help with either of those options.