Significant performance drop of ILQL when using multi-GPU training

[mcao516]

🐛 Describe the bug

I was running experiments using the ILQL sentiment example code. When using a single A100 GPU, I got an evaluation score of 0.9286 after 1k steps of training. However, when I switched to multi-GPU training (2 A100s), after 1000 steps, I got a score of 0.692. I use the Huggingface accelerate. All hyper-parameters are the same. Any idea why this happens?

Multi-GPU training command:
accelerate launch --config_file accelerator_config.yaml examples/ilql_sentiments.py

Which trlX version are you using?

trlX==0.3.0

Additional system and package information

pytorch==1.13.0+cu116

[LouisCastricato]

Ouch, I'll have one of our engineers look into this Monday. Thanks. @reciprocated

[maxreciprocate]

Hi, the random seed wasn't seeding additional gpus properly, it was corrected with

• Set RNG seeds across multiple dependencies #113

Thanks for pointing this out and thanks Jon for fixing it!

https://wandb.ai/sorry/public/reports/Difference-between-seeds--VmlldzozMDQ5MDgy

The only difference in runs with the same config.yml on variable number of gpus is a different total batch_size (this variable functions as a micro batch size in the config). Accounting for that and dividing it by 4 for 4 gpus (128/4=32 batch_size) would give same curves:

https://wandb.ai/sorry/public/reports/Difference-between-num_gpus--VmlldzozMDQ5MjUw