eval generation path issue

[sia-cerebras]

Hello, I'm trying to use your eval pipeline.
I ran ./inference_pipeline.sh -b pubmedqa -c gpt2 -s 0 -m 0 -out_dir out_dir
After it is done with generation I get the following error:

Stored pubmedqa generations to the following path: ../benchmarks/generations/pubmedqa-gpt2.jsonl
Traceback (most recent call last):
  File "evaluate.py", line 475, in <module>
    main(args)
  File "evaluate.py", line 390, in main
    data = load_jsonl(path)
  File "evaluate.py", line 39, in load_jsonl
    with open(filename, 'r') as f:
FileNotFoundError: [Errno 2] No such file or directory: '../benchmarks/generations/pubmedqa/pubmedqa-gpt2.jsonl'

It looks like the generations are saved in: meditron/benchmarks/generations/pubmedqa-gpt2.jsonl
But eval looks for them in: meditron/benchmarks/generations/pubmedqa/pubmedqa-gpt2.jsonl'

[caocongfeng]

Hi @sia-cerebras have you dealt with this problem? I am facing the same issues right now.

[sia-cerebras]

Hi @sia-cerebras have you dealt with this problem? I am facing the same issues right now.

No, I ended up doing the evals with the Eleuther Eval Harness instead.