To answer question 1
http://www.moreawesomethanyou.com/s...pic,8279.0.html
GPL'd source, and useful discussion there.

Thank you! Why didn't I just search Bowels of Trogdor for DBPF? I searched for "compress" and "compression" instead. Mustn't read MATY forums while sleep deprived...

Progress Report - Feb 14, 2009

done
- read in a package file
- decompress compressed stuff (thank you, benrq, for decompress)
- find the TXMTs and TXTRs
- decode the TXMTs
- figure out which TXMTs use which TXTRs

to do
- figure out which TXTRs to nuke and which to keep
- create an updated package header that reflects the changed number of resources
- create *correct* byte arrays for updated TXMTs with changed texture reference strings (string length likely changed, so big chunks will need to shift)
- recompress stuff (just use benrq's compress, no problem)
- create a new DIR that reflects the changed number of resources
- write out the new, updated, shrunken, recompressed package file, nuke the old one


Not terribly complex, just needs lots of attention to detail, very easy to screw this up with just a small mistake somewhere. Ah, fun with low level byte stream io. :D

Huge hugs for everyone who wrote the DBPF format wiki, and of course, benrq over at MATY for his port of zlib compress / decompress functions.

Taking a checksum, like an md5, should be sufficient. Store the resulting checksum of any TXTR read into memory somewhere, then anytime you read a new one, compare checksum to the ones you previously have, if you have any matches, it's a dupe, nuke it. Otherwise, store that one, continue.

Pescado - Thanks, I'll try that. Turns out I do need to check for dupes because of cases like this:

hair with headband or flower or whatever
- N hair groups all use texture A
- headband/flower/whatever uses texture B

alpha skirt (not fun to remove dupes for 30 recolors by hand...)
- skirt inside and outside both use texture A
- legs use texture B

We want just one copy of texture A, but without nuking texture B, so I have to check which TXTRs match and which don't. And of course, black hair with elder grey, but there, the TXMT names tell us which TXMT is elder, so its TXTR references are grey, and non-elder are black.

benrq's compression is the same scheme used in jfade's Compressorizor, yes?

Now, will this utility search for duplicates only within a single package, or will it be possible to search and kill duplicates across multiple packages so that all recolors of a hair can, for example, draw a veil or headband texture from a single recolor in a separate package?

yes

The current version is single package, but I plan to do cross-package scans later, once the single package version is done and thoroughly tested.

Multiple package scans would be nice for hair, and also for clothing. Lots of stuff I make uses the same bump map. I really should go back and make my old packages use the bump map in just one package, but I'm too lazy, so I'm writing this app to do it. Of course, there's a tradeoff there. You use less disk space, but you have to make sure you don't delete the one file that has the bump map for the 30 other files. Hm. Maybe I could stick the bump map in the mesh package? Could recolor files access it there?

I need to know which of these cases is more common:


CASE A: recolor with duplicate textures that REALLY ARE duplicates,
byte for byte, if you did a diff on the exported image files, it would say they're the same

I've seen Case A frequently, but mostly in clothing recolors. Does it occur often in hair? The duplicate texture remover I've been writing handles Case A just fine, removes extra TXTRs, updates TXMTs appropriately, will write out the new file (once I finish file writing).


CASE B: recolor with several textures that look like duplicates at first glance, but aren't
Upon extremely close inspection, they're jittered by one or two pixels so an automated comparison considers them different, even though for all intents and purposes, they're the same texture

I've seen Case B in some custom hair recolors I downloaded, but they were all by one creator, so perhaps its just that one person? In case B, my app thinks there are no duplicates. So I would have to write a UI that displays all the textures and lets the user select which ones should be treated as duplicates. That would be a pain in the butt. I need to know if its worthwhile or not. If its an infrequent case, people can just fix those files manually with SimPE. If its common, I would try to make the process less manual and tedious.

There is a NEWER version, with a GUI, in post #1

======================================================

Progress Report - Feb 17, 2009

***************************************
Alpha Software Now Available - TESTERS WANTED
***************************************

The alpha version of the extraTXTRremove program is now available for testing.
Please note that the alpha version is not the final, full featured version.
It is not meant for the general public yet, but rather for testers.

alpha version features
- Windows only
- command line interface
- new file is named "oldfilename_NO_DUPES.package" by default,
does NOT replace old file, unless you specifically tell it to
- works on a single package file at a time, not directories or multiple files

planned full version features (not yet done)
- Windows and Mac
- graphical user interface
- new file always replaces old file
- work on all files in a given directory, individually...
- ... or compare textures in several package files
- set TXMTs in several packages to use TXTR in one package
- optional ability to tell the program which textures should be treated as duplicates, for those special case hair recolors where the textures look identical, but aren't quite

Installation, Usage, Testing

- Download both the extraTXTRremove application and the Microsoft C Runtime Libraries.
- Unzip both *.zip files, put the contents in the same folder.
- To run the application, type the following at the command prompt:

./extraTXTRremove.exe FILENAME.package

If there are duplicates, the app will create a new, no dupes file with "_NO_DUPES" at the end of the filename, before ".package". Or type this and make both filenames the same if you want to overwrite the old file.

./extraTXTRremove.exe FILENAME_OLD.package FILENAME_NEW.package

The program will spew lots of debugging text at you, and then report either that there were no duplicate textures or create a new file without duplicate textures.

When testing, please let me know the following:
- Did my program create valid packages that work in game?
- If you look at the package with SimPE, did my program remove all the extra textures?
It should have, but you might see some that aren't exact duplicates.
Let me know if this is the case, ok?
- If something went wrong or didn't work, please give me the following info:
--- save the text my program printed to a text file
--- link or attachment with the recolor package file(s) you tested on
--- description of what went wrong, especially any error messages from the program, or Windows, the game, BodyShop, etc.

I don't expect this to harm your game in any way, but just to be safe,
test in an AnyGame, back up your files, blah blah standard disclaimer.

If anyone feels like testing this, thanks in advance.

*********************
DOWNLOAD - extraTXTRremove application (debug) (Feb 17 version)
DOWNLOAD - Microsoft C Runtime Libraries (debug)
*********************
.

Hmmm, I tried it out a couple of times, and I keep getting the message "The system cannot execute the specified program." :/ Running Windows XP Home SP3, fully patched, on an AMD Athlon 64, 1800 MHz (9 x 200) 2800+, for the record.

Amber - Do you have .NET? I'll probably need to give you MicrsoSoft redist dll's so the app will run. Stupid MicroSoft. Why can't the OS just ship with all the dll's users will need to run apps built with MicroSoft IDEs? 'Cause that would be easy and make sense, that's why. Grumble grumble...

I do have .NET, yes. Toss me a link and I can download those dll's.

Amber - I forgot to upload the MS CRT libraries. There is a link to them in post #10 now. Unzip the Microsoft file, stick the resulting folder in the same place where you put the extraTXTRremove.exe file, and it should run. Please let me know if it doesn't, thanks.

Feb 17 - user specified output filename, or default filename

download in post #10

Change from the Feb 16 version. Instead of the output file always being named "foo.package", you can now specify an output file name. It can rewrite the old input file if you want. Or if you don't specify an output filename, the output file will be named oldFileName_NO_DUPES.package.

-----------------------

To run the application, type the following at the command prompt:

./extraTXTRremove.exe FILENAME.package

If there are duplicates, the app will create a new, no dupes file with "_NO_DUPES" at the end of the filename, before ".package". Or type this and make both filenames the same if you want to overwrite the old file.

./extraTXTRremove.exe FILENAME_OLD.package FILENAME_NEW.package

Okay. Got it all downloaded and ran a hair through it as a test. As far as I can tell, it's working perfectly.

Amber - thanks bunches for checking

I'm working on the GUI and the ability to check for duplicates across multiple files. Will also continue to test on any and all duplicate txtr files I can find. This will probably take me a week or so...

No sweat. I'll run a few more files by it and let you know if I find any problems.

Update - March 01, 2009

New version, added a graphical user interface, see post #1.

Features not yet available in the 20090301 version, but planned for the future...
- Mac version
- look for duplicate across files, not just within individual files
- let users choose "nearly identical" textures, and treat them as duplicates

Well, I've run about half of my downloads folder through this program and haven't gotten a single error, so it seems to be working correctly.

Unfortunately, I haven't seen much of a difference in file size. Only a couple of my files were reported as having duplicate textures, and I only lost a few kb after duplicate removal. Either my files are cleaner than I thought or perhaps the program isn't as ruthless as I thought it would be.

Hopefully when you implement user-chosen "nearly identical" texture removal I'll see more of a file size difference. (Also, all my downloads are Compressorized if that has some effect.)

Thank you again for your work on this program. The GUI is very intuitive and the program itself works very quickly without any problems.

Forgot to mention: I like the Favorites feature. It makes navigating to my folders easier. Also, I was able to select multiple files at once without SHIFT or CTRL-clicking, but simply by clicking on the first file I wanted and dragging down. It would be nice if whole directories could be selected at once, rather than having to open each directory, select files, move up, open another directory, select files, etc.

wandersfar I appreciate the feedback, thanks. Of the files that did have dupes removed, do they all still work in game? They should, but if they don't, I need to know.

This program is not intended for most files. Most files do not have duplicate textures. You're only likely to see them if the file is a hair recolor, or a clothing recolor of a mesh with multiple mesh groups that use the same texture. The program has two main uses
- for creators, automatic dupe TXTR removal when they make hair or clothing with multiple mesh groups
- for downloaders, when they download hair / clothing from a creator that didn't bother to remove dupes, or didn't know how, whatever

Thanks for reporting "no errors", that's really good to hear. If you ever do get an error, feel free to report that too.

Glad the GUI makes sense. Made it similar to jfade's Compressorizer, since that makes lots of sense to me. The "favorites" feature is part of the FLTK gui library, so can't take credit for that, but glad you like it.

I agree that being able to select a directory (and all subdirectories too) would be quite handy. I'll try to add that in a future version.

I should have been more clear in my first post -- I ran my BodyShop downloads (specifically, hairs and clothing items) through the program, not my build/buy stuff. Two of my hair recolors showed up with duplicates and their file sizes were decreased from ~1mb to ~700kb.

I loaded up all the files I ran through the program in BodyShop and they're working fine. I will test them in game tomorrow. (UPDATE: Files are working in game, too.)

Thanks for the clarification.

I forgot to say, I plan to add "pick near identical textures" to remove because I have run into these in hair files. (sometimes this is deliberate and necessary, sometimes it just wastes space)

Just FYI, do NOT run this program on "mixed method" default skin files. I don't think you planned to, but just in case, don't. Such files come in pairs where one file has face textures, but no TXMTs. If it has duplicates, they might be removed, but the TXMTs in the OTHER FILE will not be changed, so the skin will break. I'll address this in a future version. I know there's a Maxis issue with face TXTRs and skins, so it may be that my program should just never be used on default skins.

I wonder, will this also pick up on unused textures in hair files?

I use the Wardrobe Wrangler to remove age groups like toddler, child, etc. that use Maxis hairs since the custom hair mesh doesn't have the appropriate resources for those ages. The thing is, I believe the WW doesn't actually delete the Maxis textures from the package files (since you can still see them in the texture preview window) but just makes it so they don't show up in the catalog.

Using your user-preview-select feature, will we be able to view those Maxis textures and delete them from there?

Ah, I wish I knew about the skin thing last night. As it happens, I did exactly that. I have a default skinset blend I made using the mixed method. I ran it through the program, but it didn't find any duplicate textures. (Zero textures were removed, old file size and new file sizes are the same.) I am using the default skins right now and they appear to still be working. I will replace the files with my backups, though, since this is unsafe.

wandersfar If the program says 0 duplicate textures, then it did not alter that file in any way. It just read, didn't write. Good thing you're making backups anyway, always good to do when testing.

The reason I'm concerned about default skins is I have seen mixed method packages that used the same face for male and female, or for two or more ages. The TXMTs for those face textures were in a different file, and since I don't yet compare across files, such a skin could get messed up.

I didn't know hair could have unused textures. I'll look into removing that, but don't expect that anytime soon. It might end up being a separate program that does other stuff with hair packages to clean them up.