NEW TOOL: 7-Zip Speed Optimizer: All of the Compression, None of the Slowness

[FenPhoenix]

Ah, 7-Zip: Loved for its high compression ratio, hated for its interminably slow performance in loaders. That's been the tradeoff since the very first version of FMSel in 2012. But I'm here to tell you it doesn't have to be that way. In fact, today I'm announcing the total eradication of 7-Zip slowness. Introducing:



When you pack your FMs with this tool, you get all the advantages of 7-Zip combined with scan performance that approaches traditional zip speeds.

To show just how fast it is, I took a set of 1309 FMs and repacked them into .7z format in two different ways.

The first is the Standard set, packed with 7-Zip 24.09 64-bit using LZMA2, compression level 9 (Ultra), 12 threads, and the rest of the settings left at default.
The second is the Optimized set, packed with 7-Zip Speed Optimizer (which uses 7-Zip 24.09 64-bit), LZMA2, compression level 9 (Ultra), and 12 threads.

The FMs were subjected to a full-set single-threaded scan benchmark. The benchmark was run on a Ryzen 5600, with the FMs located on an Aorus Gen4 NVME 2TB SSD (GP-ASM2NE6200TTTD).



2.5x faster. Not bad. But let's run the same benchmark in AngelLoader v1.10.0:



Over 11x faster. Now we're talking. But how does it do in FMSel*?



You're reading that right. Nearly 70 times faster.

So indeed, this is not just for AngelLoader: any 7z-supporting loader - past, present, or future - will benefit from this tool.

But what about file size? Does this spectacular speed increase come at the cost of enormous archives?

Absolutely not. Most FMs' size difference is a fraction of a percent, and none are more than a single-digit percent larger:



So have your cake and eat it too with 7-Zip Speed Optimizer. Download it today!

---

*I have no easy way to benchmark FMSel properly, so tests had to be run and timed manually. But the difference is so vast that it hardly matters: however you slice it, the standard set takes multiple minutes and the optimized set takes a handful of seconds.

[FenPhoenix]

How it works:
-----------------




A "block" is a solid segment of compressed data in which one or more files may be located. If a block contains multiple files, those files are compressed together as one long run of data, with no separation between files. This improves compression.

Importantly, blocks represent the boundaries of random access: If a file is located at the start of a block, then it can be decompressed immediately, whereas if a file is located some ways into a block, then all data before it in that block must be decompressed and thrown away before the file itself can be decompressed.

Thus, the basic idea is to place files loaders will want to read at the start of blocks so that they can be accessed without needing to decompress any data in front of them. However, splitting a 7z file into too many blocks results in a poor compression ratio. Therefore, care has been taken to balance these two competing concerns.

Files which are very small and may not all need to be read can each be placed in their own block, maintaining instant access to any of them while having negligible effect on compression ratio.

To maintain good compression ratios, .mis files (which are among the largest files in an FM) are packed in the "remaining files" block. However, the smallest used .mis file is placed at the start of the block for fast access. For Thief 1 and 2, a "used" .mis file means one that is specified in missflag.str and not marked "skip". For System Shock 2, all .mis files are considered "used". For Thief 3, no mission files need to be read to detect the game type, so this logic doesn't apply.

The reason for distinguishing "used" .mis files is that unused .mis files are sometimes dummy files that were created to placate DarkLoader or for other reasons. AngelLoader's game detector requires a validly structured .mis file in order to work, and it ignores unused .mis files for this reason. Some loaders may use alternative means of detecting valid .mis files such as "DEADBEEF" byte sequence detection, and will read the first valid .mis file they come across, rather than the smallest. However, the optimized layout works for them too: the fact that the smallest used .mis file is placed first in its block also means it's placed before any other .mis file in the entries list, so any loader taking the first valid .mis file it sees will automatically get the file that is both smallest and also at the start of its block.

The "main_x" images are placed in their own block because GarrettLoader used them as thumbnails when viewing FMs, and other loaders in the future may also want to do so. Furthermore, it was found that placing these images in their own block had negligible effect on 7z file size.

[Stefan B.]

Wow, that's fantastic!

When you first announced it, I wondered how it works. Thank you for the thorough explanation, it is brilliant!

[caqo]

Very nice tool. However I have a doubt about angeloader; If I want to convert all my missions to 7z, will I loose al ratings that I gave to the mission?. That's beacuse AngelLoader Will scan the missions again.

[FenPhoenix]

Hmm, yeah, both AngelLoader and FMSel would count the 7z version as a separate FM. I didn't think about that. I could add something to AngelLoader to handle that in some way, but that still leaves FMSel. Given that 7ZSO keeps the same name for the new 7z files in the batch convert, I guess a tool could be made to just go through a loader's ini file and replace .zip with .7z for all applicable archive names, or vice versa even if you wanted to go back to zips. I'm gonna be busy for the next couple days but I'll see what I can do after that.

[caqo]

A solution would be that FMData.ini file just care about the name of the file, not the extension. And warn if you have a duplicate file with another extension.

[FenPhoenix]

So this turns out to be more complicated than it sounds. It's not really possible to "not care" about an FM archive's extension. For example, when an FM is installed, an fmsel.inf file is created in the installed folder which looks like this (for example):

Name=Abandoned_Castle_V2
Archive=Abandoned Castle V2.zip

That file would have to be auto-changed for installed FMs that have had their archives changed from zip to 7z. That, of course, would confuse FMSel if the user ever switched back to using that.

Also, differential backups: For technical reasons, zip entries and 7z entries might have differing timestamps, which are used in the differential backup to detect changed files. Zip timestamps have a 2 second granularity, and I've also seen files' timestamps differ by 1 hour sometimes when repacked into 7z files, for some reason. Thus, if you change an installed FM from zip to 7z, differential backups might end up marking a bunch of files as changed when they weren't really.

There's also the matter that if AL finds like some_fm.zip, some_fm.7z, and some_fm.rar, it won't know which one it should use. It would have to ask the user, but should it be asking for potentially thousands of files? Or ask once "which format should take priority", assuming the user doesn't want to mix and match? I'm also not sure how easy or hard it would be to add this "extension replace" functionality to the FM finder, as the code in there is pretty hairy and I might run into unforeseen difficulties, or not, I'm not sure.

Of course, there's also the fact that FMSel treats 7z FMs as separate entries, and it would have to be modified and recompiled and added to updated TFix/T2Fix/Sneaky Upgrade versions, but even if that was done, everyone in the world would have to update their patches, which isn't gonna happen. So more or less, FMSel is gonna end up losing the FMs' metadata in the conversion from zip to 7z. There'd have to be an external tool that goes through its ini file like I said, for those who want to convert their collection for use with FMSel.

Honestly when I made the tool I was imagining FM authors would use it for new FMs, and I threw in the batch convert without thinking too deeply about it. I imagined that maybe FM site hosts would use it or something, and I ran the benchmark on the large 7z-converted set just to have a good sample size.

But you're right, if users are going to use this tool on their own collections, then losing all their metadata is not really tenable. I just have to figure out a workable way to solve that.

---

EDIT: So I guess you're saying FMData.ini entries should be like:

InstalledDir=some_fm
Archive=some_fm
etc.

And then whenever it needs to get the archive from disk, it will look for archives matching all supported extensions, and if more than one is found, it will alert the user and ask which to use?

That might be workable but I'd have to think about it and look at the code and make some tests.

There's also the matter of importing data from other loaders: they all have explicit archive extensions, so I'd have to figure out some way to deal with that too...

[caqo]

EDIT: So I guess you're saying FMData.ini entries should be like:

InstalledDir=some_fm
Archive=some_fm
etc.

No, I was thinking that if you use the tool to convert all your missions, then the tool should aswell modify the .ini file, changing just the extension of the filename in it, so, al the rest of the data will remains intact. But as you explained, maybe it is not as simple as that.

[FenPhoenix]

That was my first idea. It could do that, but you'd just have to make sure your old zips are moved out of the loader's FM archives folder, cause otherwise it would load in the old set of zips too (with blank metadata).

[caqo]

What about an option to delete the old successfully converted file, so you can have the files in the same Folder, and no need to change the Angel Loader FM folder paths in options.