Having trouble with reproducing the results on MESSIDOR (zero-shot)

[zheangh]

Dear @jusiro ,

Thanks for releasing this well-organized repository. I cannot reproduce the results on MESSIDOR (zero-shot). What I have done is just downloading your pre-trained weights, downloading the MESSIDOR dataset, and running this command "python main_transferability.py --experiment 02_MESSIDOR --method zero_shot --load_weights True --domain_knowledge True --shots_train 0% --shots_test 100% --project_features True --norm_features True --folds 1 ".

However, the output is "Metrics: aca=0.490(0.000) - kappa=0.578(0.000) - macro f1=0.505(0.000)", which is different from the result in Table 3 of the paper (ACA=0.604). Do you see anything I missed? I really appreciate any help you can provide.

[jusiro]

Hello @zheangh,

Sorry for the late response.

I have runned again the experiment you are mentioning, and I am getting similar results to the ones we reported in the paper. Please, see the following output:

Transferability (fold : 1)
Pretrained weights: IMAGENET1K_V1
load model weight from: ./flair/modeling/flair_pretrained_weights/flair_resnet.pth
Zero-shot classification...
['no diabetic retinopathy', 'no microaneurysms']
['only few microaneurysms', 'mild diabetic retinopathy']
['many exudates near the macula', 'many haemorrhages near the macula', 'retinal thickening near the macula', 'hard exudates', 'cotton wool spots', 'few severe haemorrhages', 'moderate diabetic retinopathy']
['venous beading', 'many severe haemorrhages', 'intraretinal microvascular abnormality', 'severe diabetic retinopathy']
['preretinal or vitreous haemorrhage', 'neovascularization', 'proliferative diabetic retinopathy']
Extracting features (X / X Steps): 100%|██████████████████████████████████████████████████████████████████████████████████████| 436/436 [00:41<00:00, 10.41it/s]
Metrics: aca=0.59700 - kappa=0.764 - macro f1=0.605

Transferability (cross-validation)
Metrics: aca=0.597(0.000) - kappa=0.764(0.000) - macro f1=0.605(0.000)

Some small changes in the results can be explained by using updated libraries or seed-effect. However, the performance gap you are mentioning is too large. Right now I can't think of what might cause your problems in reproducing the results. Please let me know if there is any additional information that would help me reproduce your issue.

Kind regards.

[YuekaiXuEric]

Hi @jusiro @zheangh

I having the same reproducing issue as you got. I download the pretrained weight for the model and ran the command given in the repo

python main_transferability.py --experiment 02_MESSIDOR --method zero_shot --load_weights True --domain_knowledge True  --shots_train 0% --shots_test 100% --project_features True --norm_features True --folds 1

and got

Transferability (fold : 1)
Pretrained weights: IMAGENET1K_V1
load model weight from: ./FLAIR/results/flair_resnet.pth
Zero-shot classification...
['no diabetic retinopathy', 'no microaneurysms']
['only few microaneurysms', 'mild diabetic retinopathy']
['many exudates near the macula', 'many haemorrhages near the macula', 'retinal thickening near the macula', 'hard exudates', 'cotton wool spots', 'few severe haemorrhages', 'moderate diabetic retinopathy']
['venous beading', 'many severe haemorrhages', 'intraretinal microvascular abnormality', 'severe diabetic retinopathy']
['preretinal or vitreous haemorrhage', 'neovascularization', 'proliferative diabetic retinopathy']
Extracting features (X / X Steps): 100%|███████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 436/436 [02:21<00:00,  3.08it/s]
Metrics: aca=0.48900 - kappa=0.577 - macro f1=0.505

Transferability (cross-validation)
Metrics: aca=0.489(0.000) - kappa=0.577(0.000) - macro f1=0.505(0.000)

I pretty much got the results as @zheangh and have a large performance gap compare with table 3 in the paper. Is it possible to share more setting on dataset part?

Sincerely.