-
Notifications
You must be signed in to change notification settings - Fork 160
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Where can I obtain a generated dataset that includes an options column #76
Comments
@nanyyyyyy You would need to re-generate it and pass through an options column for relevant datasets. This could cost a bit of compute though. Alternatively you could isolate the options datasets and use a regex to extract them. Sorry, this data was intended primarily for training so we didn't pass that information along. Hope this helps though! |
Can you explain a bit about this? I want to include options and the exact template for generating each instance in the dataset. What are the detailed steps to achieve this? |
I haven't figured it out. sorry |
@nanyyyyyy @gao-xiao-bai So to generate all the templates and options alongside each example you would need to edit the preprocessors used for every task. One in particular is the To get the answer options you would do the same thing, passing through the |
This is super helpful. thanks a lot |
Thank you for your response. |
@nanyyyyyy @gao-xiao-bai were you guys able to figure this out? @shayne-longpre I must say it's a little weird not to include the options since FLAN paper evaluations are based on rank-classification with options, so it seems like a key thing to include. The data is appreciated nonetheless. |
Where can I obtain a generated dataset that includes an options column, which can be used for rank evaluation purposes? Thank you.
The text was updated successfully, but these errors were encountered: