Streaming uncompression use-case

[pmqs]

I see that sozip stores the index data in an orphan local header that immediately follows the real compressed payload local header.

Whilst that orphan local header is invisible to most uncompressors, the streaming use-cases will find the hidden idx files.

The classic streaming unzipper is jar. Below is what I get when I use one of the example sozip files with jar t list the zip file contents.

$ jar --list -v <foo.zip
     3 Thu Jan 05 15:45:16 GMT 2023 foo
    40 Thu Jan 05 15:45:16 GMT 2023 .foo.sozip.idx

Another approach that has been used before is to store the index data in an extra field in either the local and/or central header. The big advantage of this approach is it is fully transparent to both the streaming & non-streaming use cases.

If the index data is stored in the central header it shouldn't be that much different from the approach you are using now.

If the index data is stored in the local header it gets more difficult to create the index without doing it in two steps. First step is compressing & maintaining the index. Second step is combining into a zip file.

Just a thought.

[rouault]

Another approach that has been used before is to store the index data in an extra field in either the local and/or central header.

I thought about that but there's a limitation to 64 KB for extra field content that files larger than 256 MB would reach.

If the index data is stored in the central header it shouldn't be that much different from the approach you are using now.

"jar" is kind of an exception. Most ZIP readers read files from the end.

[pmqs]

Another approach that has been used before is to store the index data in an extra field in either the local and/or central header.

I thought about that but there's a limitation to 64 KB for extra field content that files larger than 256 MB would reach.

Is the 256MB limit the max number of entries that would fit into your idx specification?

FYI, the prior art for using an extra field (0x5a4d) is https://www.artpol-software.com/ZipArchive/KB/0711101739.aspx

[rouault]

Is the 256MB limit the max number of entries that would fit into your idx specification?

yes, each entry is 8 byte large. Hence 65536 / 8 = 8192 entries. And with the default suggested chunk size of 32 KB : 8192 * 32768 = 256 MB. With a larger chunk size the index could fit in 65 KB, but a larger chunk size decreases the smoothness of working with a SOZip file, espacially when working with remote ZIP files

FYI, the prior art for using an extra field (0x5a4d) is https://www.artpol-software.com/ZipArchive/KB/0711101739.aspx

Interesting. This is the kind of prior art I was looking for one year ago when I designed SOZip and didn't spot.