DataFrame msgpack objects written from linux cannot be read on windows

[ruidc]

eg. on linux:

df2 = pandas.DataFrame([['100']], index=[1], columns=['v'])
df2.to_msgpack(r"/mnt/C/temp/p2.msgpack")

then on windows:

df = pandas.read_msgpack(r"C:\temp\p2.msgpack")
ValueError: Shape of passed values is (1, 1), indices imply (1, 2)

[jreback]

pls show versions. This IS tested in master for cross-platform compat. Its possible that you are hitting an old bug.

[ruidc]

0.16.2 tested

[ruidc]

and 0.14.2

[jreback]

pls try on master.

[ruidc]

where is pandas nightly these days?

[ruidc]

ah, found it, but stale for a year

[ruidc]

that's going to make it difficult for me to test windows reading.

[jreback]

hmm, this is only partially working, will mark as a bug.

[jreback]

cc @kawochen want to have a look

The reads cross platform work but are just returning a byte stream, so the tests don't fail. odd.
Every item is a KeyError, so need to add a check that we match certain items.

[ruidc]

it seems to be an issue between the index being picked up as dtype np.int32 on windows vs np.int64 on linux but i'm well out of my depth trying to dig into this

[kawochen]

OK I will take a look, but the cross platform issue itself should have been fixed on master already in pandas.io.packers.dtype_for.

[ruidc]

do you mean b0c14c1 ? can you reproduce my problem above?
If you can no longer reproduce since your fixes i'll have to find/merge all the related fixes as we need to still run 0.14.2 for compatibility reasons

[jreback]

@kawochen yes I think they are fixed on same-platform reading, but the cross-platform is returning odd results (and the tests dont catch this).

@ruidc If you have compat with 0.14.2 you are on your own. (it should work, and their are not too many fixes, but that will be tricky)

[ruidc]

... and actually seems to not just be the index, for some larger test cases, happens on the data too. Again forcing int64 dtype on the windows side seems to fix it for me.