Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Uncompressed data size mismatch (nodejs) still there in 3.7.1 - 32 Bit Issue with << #777

Open
JMS-1 opened this issue Aug 20, 2021 · 3 comments

Comments

@JMS-1
Copy link

JMS-1 commented Aug 20, 2021

I got a ZIP which includes a single file with more than 2GB in (decompressed) size. On decompression the error from the title is thrown. It clearly is some 32Bit signed/unsigned issue:

worker.on("end", function () {
            if (this.streamInfo['data_length'] !== that.uncompressedSize) {
                throw new Error("Bug : uncompressed data size mismatch");
            }
        });
        
this.streamInfo['data_length']
	4011341711 (0xEF18378F)
that.uncompressedSize
	-283625585 (0xEF18378F)

So my question is: is there a size limit of 2GB for individual files inside the ZIP (which by the way has been created with jszip 3.7.1 as well)?

Thanks

Jochen

@JMS-1
Copy link
Author

JMS-1 commented Aug 20, 2021

Problem is that << only reproduces 32 Bit Value, so

readInt: function(size) {
        var result = 0,
            i;
        this.checkOffset(size);
        for (i = this.index + size - 1; i >= this.index; i--) {
            result = (result << 8) + this.byteAt(i);
        }
        this.index += size;
        return result;
    },

will mess with the sign if the MSB is larger than 127. Better use multiplication

readInt: function(size) {
        var result = 0,
            i;
        this.checkOffset(size);
        for (i = this.index + size - 1; i >= this.index; i--) {
            result = (result * 256) + this.byteAt(i);
        }
        this.index += size;
        return result;
    },

FYI: * 256 will be safe up to 53 Bit (MAX_SAFE_INTEGER) which is around 9 PetaByte.

@JMS-1 JMS-1 changed the title Uncompressed data size mismatch (nodejs) still there in 3.7.1 - maybe misuse! Uncompressed data size mismatch (nodejs) still there in 3.7.1 - 32 Bit Issue with << Aug 23, 2021
@RubenGarcia
Copy link

We are also affected.

RubenGarcia pushed a commit to lexis-project/jszip that referenced this issue Sep 23, 2021
Stuk#777
which causes
Bug : uncompressed data size mismatch
Nicodemos234 pushed a commit to Nicodemos234/jszip that referenced this issue May 2, 2023
Stuk#777
which causes
Bug : uncompressed data size mismatch
MatheusrdSantos pushed a commit to cantoo-scribe/jszip that referenced this issue May 2, 2023
Stuk#777
which causes
Bug : uncompressed data size mismatch
mofojed pushed a commit to deephaven/web-client-ui that referenced this issue Aug 31, 2023
Fixes #1080. Fixes #1416 

Updates jszip and uses the internal stream helper instead of nodestream
which we would need a polyfill for. The progress on zip uploads is a bit
odd since we use the unzip progress. It seems papaparse loads more from
the stream while processing/before processing and can finish reading the
zip before it's done processing chunks.

Also, uploading the tables seems to be a blocking operation. Not sure if
that's an easy fix, but after the 50% mark on uploading a large zip,
there are some blocks on the main thread (marching ants freezes as an
indicator).

There is a ~2GB limit for JSZip it seems
Stuk/jszip#777
ponsea pushed a commit to ponsea/jszip that referenced this issue Feb 28, 2024
@aidoraide
Copy link

+1

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

3 participants