The exact English pretraining data and Chinese pretraining data that are exact same to the BERT paper's pretraining data.

[guotong1988]

Any one know where to get them?
Thank you and thank you.

[adewynter]

This is a very good question.
I'd say that it might be hard to get 1-1 correspondence on the datasets, but you might want to start with the data used in the BERT version in Chinese and perhaps this RoBERTa implementation.
Note that I haven't used them myself so I cannot vouch for their completeness.

That being said, I'd say that for Bort specifically you'll want to focus on having a rich enough dataset. If you use a pre-trained model, use a "crazy" dataset that is not in the training data. For us, in English, it was UrbanDictionary.
The reason for this is because you want to be able to measure generalizability beyond what your pre-trained model has seen.

Finally, I'd point you to this question, too.

I will be leaving this question open in case anyone has better suggestions -- I'm curious myself!