Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Adding Acceptability judgment and SentEval tasks #1271

Merged
merged 6 commits into from
Feb 2, 2021

Conversation

zphang
Copy link
Collaborator

@zphang zphang commented Feb 1, 2021

Results using roberta-base / epochs=40, batch_size=16, lr=1e-5

  • acceptability_coord: 0.913
  • acceptability_definiteness: 0.942
  • acceptability_eos: 0.854
  • acceptability_whwords: 0.900

Results using roberta-base / epochs=3, batch_size=16, lr=1e-5

  • senteval_bigram_shift: 0.973
  • senteval_coordination_inversion: 0.857
  • senteval_obj_number: 0.965
  • senteval_odd_man_out: 0.743
  • senteval_past_present: 0.928
  • senteval_sentence_length: 0.876
  • senteval_subj_number: 0.954
  • senteval_top_constituents: 0.910
  • senteval_tree_depth: 0.681
  • senteval_word_content: 0.995

@codecov
Copy link

codecov bot commented Feb 1, 2021

Codecov Report

Merging #1271 (f083e81) into master (56ceae5) will increase coverage by 1.25%.
The diff coverage is 86.75%.

Impacted file tree graph

@@            Coverage Diff             @@
##           master    #1271      +/-   ##
==========================================
+ Coverage   47.21%   48.47%   +1.25%     
==========================================
  Files         149      163      +14     
  Lines       10877    11214     +337     
==========================================
+ Hits         5136     5436     +300     
- Misses       5741     5778      +37     
Impacted Files Coverage Δ
jiant/scripts/download_data/constants.py 100.00% <ø> (ø)
jiant/scripts/download_data/runscript.py 29.41% <0.00%> (ø)
jiant/tasks/evaluate/core.py 32.89% <ø> (ø)
...t/scripts/download_data/dl_datasets/files_tasks.py 7.69% <10.52%> (+0.12%) ⬆️
jiant/tasks/lib/senteval/base.py 77.96% <38.46%> (ø)
jiant/tasks/lib/acceptability_judgement/base.py 73.84% <73.84%> (ø)
jiant/tasks/lib/senteval/bigram_shift.py 95.65% <95.65%> (ø)
jiant/tasks/lib/senteval/coordination_inversion.py 95.65% <95.65%> (ø)
jiant/tasks/lib/senteval/obj_number.py 95.65% <95.65%> (ø)
jiant/tasks/lib/senteval/odd_man_out.py 95.65% <95.65%> (ø)
... and 25 more

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 56ceae5...f083e81. Read the comment docs.

@zphang zphang merged commit 84f2f5a into nyu-mll:master Feb 2, 2021
@zphang zphang deleted the acceptability branch February 2, 2021 03:36
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants