Non-matching results in the DDR dataset

[CristianoPatricio]

Hi there,

Congratulations on this awesome work.
I'm trying to reproduce the results of Table II and III of the paper, specifically the results for the DDR dataset. After running over the 10-folds with the default parameters from the config files, the results I obtained deviate a little bit from the reported ones in the paper. I expected my results match the line "CLAT(Ours)" of both Tables for the DDR-subset column. Is there any other configuration for the DDR dataset?

Thanks in advance.

Cristiano

[Sorades]

Thank you for your interest in our work and for taking the time to reproduce our results!

There is no other specific configuration for the DDR dataset. I suspect that the deviations might be due to the training-time intervention, which is a trick borrowed from CEM) that helps with test-time intervention. We observed in our experiments that this strategy can occasionally affect model performance. You can try setting training_int_prob to 0 to disable it and see if that helps align your results with the reported ones.