How to properly install a Thief 3 FM

[snobel]

A short explanation: The Sneaky Upgrade is going to have a basic FM loader built in - the aim is to make FM playing just work, and nothing else. This loader also has the virtue that nothing needs to be copied or linked into your game folder in order to play an FM.

All the previous widescreen and tweaks stuff is now in a DLL, and I'd like to include enough functionality there, that any FM can be prepared and launched. Later I'll try to adapt the new "FMSel" to work with T3 or, if that's not feasible, make a simple FM loader frontend (SneakyLoader?) for this purpose.

Anyway, this is what I have from the old loader threads, please post comments and corrections. Questions will follow in another post:

FM installation: (Not considering quirks for individual FMs)

• Unzip the FM archive into an empty folder. (The FM root)
• For every *_patch.ini file in the FM root or the System folder, copy in the OM version of the file and patch it. The crude way is to prepend (not append) the patch to the original file, the better way is to go through its contents and set the corresponding values in the OM file individually
• For every *.cbt file in the Maps folder, reconstruct the .ibt from the OM assets. Normally done by calling DLLRebuildIBT.dll
• It may be necessary to rebuild the file SchemaMetafile_HardDrive.csc, so that it contains custom sounds defined by .sch files found in the Sounds folder. This is normally done by copying in the stock SchemaMetafile_HardDrive.csc, running Thief3SoundAddOn.exe from within the Sounds folder and renaming the resulting file to the original name. If this step is needed, Thief3SoundAddOn.exe must be supplied in the Fan Mission Extras folder.
• If you're going to use a separate savegames folder for each FM, copy in the User Options folder from the OMs. (Taking the locale into account)

Quirks: (These really should be fixed in the mission archives, not by the loader)

• User.ini needs renaming to User_patch.ini... (The Wizard)
• If there are any exe files in the FM's System folder, remove them
• If the locale is non-English, go through all .sch files looking for empty or missing lang_[locale] lines, and copy in the English text

Final step: (For completeness)

• The GL approach: For every file in the FM root and subfolders, copy or move it into the same location in the OM folder structure. Keep a list of copied-in files and created subfolders, so that they can be uninstalled later. If a file exists in the OM structure, save it somewhere so that it can be reinstated after uninstalling the FM
• The TFMM/ML approach: For every file in the OM root and subfolders, link it into the same location in the FM folder structure, creating subfolders as necessary - but only if the file doesn't already exist. T3Main.exe must be patched to ignore the ION_ROOT registry key.
• The SneakyLoader approach: All done

[snobel]

Questions:

1. SchemaMetafile_HardDrive.csc - as far as I can tell, only "All the World's a Stage" needs this rebuilt, and apparently it's not done automatically by the current loaders? Then the rule would probably be "if Thief3SoundAddOn.exe is in the FM Extras folder". Is there any other way to tell if the sounds must be added to the .csc, e.g. from the schema files?
   Having this step done by the loader is much preferable to including the rather large .csc file with the FM?
   
2. Generally it's not necessary to extract .ogg audio to .wav, neither mp3, because they are not used - right? But there's some talk in the old threads about some FMs having problems, so GL extracts ogg anyway. Anybody know any FMs which need this?
   
3. Apart from that one FM which has a corrupted .cbt file, has there been any problems with it, or has it gone out of fashion because the size of large missions is dominated by custom textures and movie files? It seems like a fast and easy way to get the file size down, at least for some FMs - "Thief's Rising" shrinks from 326 MB to 91 - or 57 with a little improvement of the scheme.
   
4. "Avoid spaces in the install path, because some FMs have problems." Anybody know any examples?
   
5. Apart from the quirk mentioned above, is there anything else locale related that needs to be done by the loader, e.g. to support Russian?

Edit: At this point the answers to the above seem to be:

1. No, you can't tell from the contents of the Sounds folder if SchemaMetafile_HardDrive.csc needs to be rebuilt
2. Presumably it's a misunderstanding, and .ogg files do not need conversion
3. The use of CBTs is still a sound concept, but it will depend on the content of the FM whether it's worth doing
4. It's probably a myth, or a conflation with other problems, that spaces in the install path are trouble
5. No - except that if the options.ini file needs to be copied for the FM, the locale must be accounted for, because the name of the containing folder is localized.

[Beleg Cúthalion]

@1: Since I cannot find any to-be-patched files in the FM archive of All the World's a Stage, I assume that the T3SoundAddOn is already a customized executable. I only know that I can use it to tweak the csc files myself, not that it gives me a customized patch, but apparently that's what it is. I have a 70kb executable of it on my HD.

The problem behind is that the FM contains a custom conversation and these can only be played from the csc schema archives. Their re-compilation is done via a Perl script but I cannot find it right now. Massimiliano once almost bit his teeth out trying to include new Mechanist voice sets. One of the problems I faced when trying to provide German voice acting support was that upon re-compilation the lip movements were gone. There could still be a workaround for FMs, like the one I was using to include custom music in my FM (you copy your sound files and schemas into the Sounds folder and run scripts on FM startup that play and stop these sounds, after that they can be used by the game's ambient schema properties and maybe even conversations...?!). But this doesn't help for the existing FM, so I fear you either have to back up the original csc file and execute the patch or decompile the patch file, extract the new material and hack the file yourself.

Further reading:
http://www.ttlg.com/wiki/Importing_Custom_Sounds

@2: I don't know any cases except for custom ogg files in the Sounds directory, but in this case they should be copied because the game accesses them. Still, don't know any back stories about other formats as replacements.

@3: Last time (e.g. Ziemanskye's Hearts and Minds) there were several crashes with shrinked ibt files, that's one reason why they came out of fashion. Custom ones or even altered originals (like in my FM with JohnP and altered JohnP textures) make things worse but in general it might be a good idea to re-think FM compression. The problem is that e.g. I'd have to ask the players to install JohnP's pack first in order not to have blurry vanilla textures next to a few new ones or altered high-res JohnP textures.

@5: Many FMs have Russian in the place of French. Since French translations have never been asked in my presence, you can either leave the mechanisms the way they are (i.e. fill English text in if there is no or just an empty lang_french tag) or maybe add something that detects if the FM contains proper French or Russian in its place.

[snobel]

Thanks for the links. I've found the perl script, which is great to have, because it can serve as a (too detailed) specification for the .csc files. There's also this incomplete spec on a now defunct Deus Ex site. I think that for now I'll just execute Thief3SoundAddOn if it's found, we can revisit .csc rebuilding if there's a need for conversations in a new mission.

About the missing lip sync thing - custom conversations will not have lip sync of course, but do you know if rebuilding the .csc with Thief3SoundAddOn will strip the lip sync data from existing sounds?

The wiki link was new, as I understand it you can use the (normally empty) ...memory.csc file to store custom sounds, with some limitations? But nothing is required from the mission loader to handle that?

It's weird that using the cbt scheme could cause a crash, I'll have a look at Hearts and Minds and see if I can reproduce it.

[Beleg Cúthalion]

About the missing lip sync thing - custom conversations will not have lip sync of course, but do you know if rebuilding the .csc with Thief3SoundAddOn will strip the lip sync data from existing sounds?

IIRC yes, but maybe we just screwed up some settings. After all it is an ugly DOS batch thingy. I don't think, however, that each original voice file had its own lip sync data. Don't you think this is done based on the file's volume and thus re-createable for custom sounds?

The wiki link was new, as I understand it you can use the (normally empty) ...memory.csc file to store custom sounds, with some limitations? But nothing is required from the mission loader to handle that?

I never tried it but apparently it provides a new schema archive that just has to be copied. The SoundAddOn, however, changes the Harddisk csc file if I'm not mistaken. Not overly big in size but already existant. I believe with just copying your sound files together with corresponding (and rather generic) schema files into ...\Sounds\ you can keep things simple, especially if, like GlasWolf says, conversations and ambient streams don't work anyway.

It might be worth, however, to see if this limitation to certain csc files is written down somewhere – and can be changed.

It's weird that using the cbt scheme could cause a crash, I'll have a look at Hearts and Minds and see if I can reproduce it.

This was back in the old GarrettLoader times, I only remember being one of the beta testers where it crashed during the loading process. I should try some older T3 FMs to see if it still happens, but on the other hand TFMM doesn't handle it at all, does it?

[snobel]

I don't think, however, that each original voice file had its own lip sync data. Don't you think this is done based on the file's volume and thus re-createable for custom sounds?

Unfortunately it looks like explicit lip sync data is used, so getting it to work with custom sounds is probably not realistic. But it should be possible of course, to add custom conversations without removing lip sync for existing sounds.

I believe with just copying your sound files together with corresponding (and rather generic) schema files into ...\Sounds\ you can keep things simple

Using discrete files is probably a better solution then.

This was back in the old GarrettLoader times, I only remember being one of the beta testers where it crashed during the loading process. I should try some older T3 FMs to see if it still happens, but on the other hand TFMM doesn't handle it at all, does it?

I believe both ML and TFMM handle .cbt files using ShadowSpawns rebuilder DLL, just like GL.

[gnartsch]

Questions:

4. Apart from that one FM which has a corrupted .cbt file, has there been any problems with it, or has it gone out of fashion because the size of large missions is dominated by custom textures and movie files? It seems like a fast and easy way to get the file size down, at least for some FMs - "Thief's Rising" shrinks from 326 MB to 91 - or 57 with a little improvement of the scheme.

5. "Avoid spaces in the install path, because some FMs have problems." Anybody know any examples?

@4: For Hearts and Minds there is also an uncompressed version availlable, which has the .ibt file instead of the .cbt. http://thiefmissions.com/m/HeartsAndMinds (54 MB) - which works without problems.
And since this is the only mission I know of which shipped with an .cbt version (which never ever worked for me), I wonder why there are still links around to that faulty one.

@5: Maybe this is just a rumor? Maybe the real problem is that it might not run on newer Windows versions (in particular on those with 64bit) due to the 'special' default install folder, like "C:\Program Files (x86)\" or even worse "C:\User\...\blabla\"?
A couple of old software (and games in particular) just refuse to work correct if they are installed in that default paths that Windows offers to the installer, and where Windows imposes higher restrictions for security purposes.
I was one of the guys who always suggested to avoid using paths with blanks - until I found out that the restrictions on these default-folders are the real problem (and in some cases the length of the paths that you get if you stick to the default - like in T2 where multiple directories are added as a single line in the install.cfg)

[snobel]

@4: For Hearts and Minds there is also an uncompressed version availlable, which has the .ibt file instead of the .cbt. http://thiefmissions.com/m/HeartsAndMinds (54 MB) - which works without problems.
And since this is the only mission I know of which shipped with an .cbt version (which never ever worked for me), I wonder why there are still links around to that faulty one.

I found the good version of Hearts and Minds on your website, where I also picked up a couple of the quirks above, so thanks for listing those. That cbt file is quite obviously corrupt, because it's a different size than the ibt in the good version - that should never happen of course.

@5: Maybe this is just a rumor? Maybe the real problem is that it might not run on newer Windows versions (in particular on those with 64bit) due to the 'special' default install folder, like "C:\Program Files (x86)\" or even worse "C:\User\...\blabla\"?
A couple of old software (and games in particular) just refuse to work correct if they are installed in that default paths that Windows offers to the installer, and where Windows imposes higher restrictions for security purposes.

That would be true for Thief 1-2, but I think T3 should work just fine out of Program Files ((x86) or not) as long as you don't try to move your SaveGames folder in there.

You may remember from the last round of testing that we wondered why Ion Launcher is allowed to write its log in the user's Documents folder, but the savegames are diverted to the public Documents? It turns out that there's an Application Compatibility database of files that have nasty things done to them - there's a tool that you can download from MS where you can see a list, and what's done in each case. For Thief 3 there are 'fixes' for the installer, T3.exe and T3Main.exe, but not Ion Launcher... In the case of T3Main, it's recognized by the filename and a couple of resource strings. If you change e.g. the File Description string with a hex editor your savegames folder goes back into your personal Documents folder...
Besides the very helpful messing up of the paths, the only other thing that's done to T3Main is that it runs in XP compatibility mode even if the properties dialog doesn't show it. (In the next patch release I intend to have the installer fix this stuff.)

[Beleg Cúthalion]

The latest version of str8g8's "The Bridge" has also cbt (and ned...?!) files included. This always worked for me.

That cbt file is quite obviously corrupt, because it's a different size than the ibt in the good version - that should never happen of course.

I thought the purpose of stripping textures from ibt files was to shrink their size.

[snobel]

The latest version of str8g8's "The Bridge" has also cbt (and ned...?!) files included.

Do you know where to find that? Is it a recent update?

I thought the purpose of stripping textures from ibt files was to shrink their size.

Yes and no. You don't remove anything, you just replace the stock textures with zeros. So the resulting cbt-file has the exact same size as the ibt. But when you zip up the mission, all those zeros collapse into nothing, and the resulting zip file becomes smaller.

[gnartsch]

Do you know where to find that? Is it a recent update?

The Bridge, v1.5 (Nov. 8, 2006)

[snobel]

Thanks. Turns out I had it already - I didn't remember any published mission having more than one cbt.

[gnartsch]

[*]Apart from that one FM which has a corrupted .cbt file, has there been any problems with it, or has it gone out of fashion because the size of large missions is dominated by custom textures and movie files? It seems like a fast and easy way to get the file size down, at least for some FMs - "Thief's Rising" shrinks from 326 MB to 91 - or 57 with a little improvement of the scheme.

Adsk1 is currently working on a revised version of "Thief's Rising".
If you could explain to him what needs to be done, then the update would be much smaller.
It is just right now (as I am working on the german translation for it) that I notized it being shipped with .ibt files.

[snobel]

Good idea, but I've no experience with the compression/optimizing side of things as they currently are - using GL etc. I've only used my own implementation for that. But I'll PM him the numbers, and if he's interested maybe one of the mappers can explain, or I can have a look at it.

Now that I'm revisiting the cbt stuff I'm considering making a small ibt/cbt utility - most of the code is already there - so maybe that can be an option too.

[Beleg Cúthalion]

If multiple missions contain the same textures, shouldn't zipping (or 7z..ing) up shrink them then, as you said in the cbt example?

[snobel]

Not with zip, because it compresses every file individually. 7zip will do much better.

I just did a couple of experiments to see how well modern compression programs can compress the non-cbt version of "Thief's Rising". For 7zip at its best the resulting archive was 196 MB, still more than double the cbt/zip version. And if you combine (improved) cbt and 7zip the archive is a little over 34 MB...

For fun I also tried freearc (supported only by ModernLoader?). That format is a bit exotic, but it is very impressive: 103 MB for the ibt version, quite close to the cbt/zip version. The cbt/freearc archive came out at 33 MB.

Overall, for this mission at least, cbt/zip or cbt/7zip seems to be the best choice, depending on whether you want to keep compatibility with GarretLoader. (GL doesn't support 7zip, right?)

[snobel]

The first two posts have been updated, based on the feedback. Also, a small status report may be in order:

Most of the loader functionality is in place. I have been struggling with the unzipping part (7z.dll being a bit of a pain), but since someone just dumped a perfectly good FM frontend (with source code!) on us, I'm just going to assume that it can be whipped into shape for T3. So the next step will be to rip out my own unzipping and dialog code. It will still be possible to play FMs without a frontend, but it's going to take a bit more work than originally intended.

After that, I'll add the property storage fixes together with some logging, so that we can find out how effective those fixes are. Then I'll try pushing an alpha release to the usual suspects 'round here, because some outside feedback would be good at this point.