OK, looks like this is fixed. I missed one of Deaderpool's fixes, so likely more folks are seeing this error than just azoresman. I didn't want to completely ignore the missing tuning, so the program will generate a user-friendly warning which can then be disabled in the extract options - something you will probably want to do if you have the expansion pack which is throwing this error.

Getting ready to package up and upload version 3.4.1 now.

ETA - It's uploaded and ready for consumption. Apologies for the mistake - and a big thanks to @azoresman for helping determine the cause of the crash.

Thank You for all your support

Fairly minor update tonight. I've added some improvements to the File Finder. Details are on the version history on the main description page. Attached below is a screenshot of the new grouping feature in action (you may also note that the items are sorted now).

Enjoy!

Played with two ideas I'd had in the back of my brain tonight after work. The first was in noticing that much of the conversion has an I/O bottleneck. So I converted the code from using temporary disk files to memory streams. Tremendous improvement in speed, nearly 40% faster.

Then, I thought, well now it's not I/O throttling the speed, it's just the CPU. So why not put those multiple CPU cores to work? So I tried launching each binary to XML conversion stage in a separate thread. Bam, another 10% speed improvement! Not as significant, due to the fact that most of the game packs finish very quickly as they're so small - so the bulk of the time is still spent converting the BG data. But 10% is nothing to laugh at!

The results are nice, on my quad core box the time for an extract dropped from around 3 minutes 10 seconds to 1 minute 37 seconds.

The code isn't really fully tested yet, and I do want to thread the final XML dump to the separate files as well. Maybe over the weekend I'll get version 3.5 out so everyone else can burn up their CPUs too(*).

(*) Just kidding, no CPUs were harmed in testing.

That sounds interesting =) -- right now it took 4 minutes 30 to extract the 1.15 stuff; I have a quad core as well so I guess the speed improvement would be quite significant. Looking forward!

Good ideas! More speed is always a plus!

Agreed - when something is slow, throw RAM and CPU cores at the problem! The screenshot below tells the resulting story. Even I was pretty impressed by the results!

I rewrote the threading stuff a bit tonight to be sure that once a ton of packs are out it limits the number of simultaneous threads to a reasonable number, and adding the threading for the final XML extract was a bit of a pain due to the way it was written. There's still a (relatively minor) bug to be squashed, so I can't really release it officially yet, but I can offer a "test version". Not even sure that I'll ever release the source for this version, it's so patched together in some ways to avoid a complete rewrite!

The bug? Just make sure to extract to an empty folder, as something with how I'm detecting the overwrite option is bugged. You'll get spammed by a bunch of "Overwite?" prompt windows, I think only one per thread as long as you check the "Don't tell me again" option in that dialog, but it's annoying to say the least. Should be a simple enough fix, but haven't really thought too much about how I want to implement it. Maybe I'll just force overwrite to be on always for a full extract.

Number of threads will default to 1/2 the detected logical CPUs on your machine - I assume most folks have hyperthreading so if VB .Net sees 8 logical CPUs, really only 4 of those are actual cores. Although it seems to be safe to pump it up to the number of logical processors if you want to exercise that right. It won't let you choose more threads than that, and it really wouldn't gain you much to do so.

Not sure if I'm going to bother making the threads update the main UI as it writes all the files like it previously did.

RAM usage is HIGH, expect it to use up around 500 Megs of RAM during the final stages of the extract. No idea what kind of performance someone with only 4 gigs of RAM will see with this.

There may be memory leaks, so I wouldn't advise doing a bunch of extracts one after the other without closing and reopening the program.

I haven't really tested the extracts from a single file options, although they should work okay... Really I've only tested the full extract with threading, but the extract from a single file had to be rewritten quite a bit as well. Should work. But dunno. Feel free to test it for me, although I will of course get around to it

The main thing though, the output, seems to be 100% correct. Comparing to previous extracts of the January XML shows the vast majority of files are 100% identical, and the files that have changed look to be perfectly valid changes.

In short, it pretty much works! For me at least. Your mileage may vary, and let me know if you get any nasty crashes or garbled output.

EDIT - Removed download due to major bug (see new message below)

Hey there Scumbumbo! It seems that Avast likes to block any program it doesn't know from starting. I was wondering why sometimes it ran and sometimes it would just not launch. Shutting down Avast's Real Time Protection temporarily allowed it to run properly. This also occurs to Sims4Studio's installer and I also have to shut down Avast to get it to run.

There should be a way to tell Avast to whitelist programs to allow it to let them run (while still hopefully virus checking them!)? I'll leave that as an exercise to those who use Avast. Although I use care in providing a virus-free program (I use ESET myself) there's no way I'd recommend whitelisting ANY program, even one I wrote, to skip virus checking.

Interesting info though!

So I see only two people downloaded the test version I posted yesterday, but since I can't tell who they were this is just a warning to you two - I found a MAJOR bug (and stomped it, of course). The cross-reference entries get garbled in places due to a race condition between threads. I got lucky and noticed it from comparing a known good "Reference - All XML Names" dump from a previous run to one produced by the version I uploaded last night. You can see some of the errors in the attached screenshot, the left pane is missing 191CE and the information that should be in 191CE overwrote 191D1 instead - then the information from 191D1 got put in as a duplicate copy of 191D4.

The extracted XML was still correct, except for the <!-- Comments -->

Anyway, I fixed the bug and everything else seems to be working great. Found some other minor bugs and fixed the overwrite dialog to make multiple threads wait for one dialog box to be answered before popping up another. The rename all files option is now slower than ever, but I can't imagine anyone ever wanting to use that anyway.

I still want to run a few more tests, but should have an official new version release later tonight yet.

All tests were successful, and comparing with an extract using the old version was a 100% match. 3.5.0 is now available for download.

Enjoy!

The source code is available as well, as hacked in as the threads are the code isn't THAT bad... or is it? Please route negative style comments to /dev/null... Helpful suggestions are, of course, welcome if you know .Net threading better than I!

Cool, thanks! I just re-extracted since I was curious: it finished in 1 minute 55 now, instead of 4:30 with the old version =).

@plasticbox - You may want to re-extract again soon. Version 3.5.1 is basically ready to go and has the "All That JAZZ" feature.

Only one concern, and you're probably the best person to ask - should the JAZZ XML files have a group number for the game pack, or should it always be zero? It's really a minor thing I suppose, but I would like to have the group number set for what people will need to include it in a package as an override, and I don't think I've ever seen a mod that overrides a JAZZ file other than from the base game, so I'm just plain not sure.

Right now I'm setting the group in the filename to zero on output, although I track the group number internally so they end up in the proper subfolder, e.g. "GP01\AsmJazz" so it would be easy to change that. If no one knows I'll just leave it as is until it's known that it must be one way or the other.

I only have one EP to check with, but in EP02/fullbuild0 all the JAZZ resources I see (type 0x02D5DF13) have a group 0x00000000.

Version 3.5.1 released and has an option to extract the ASM/JAZZ resources to an AsmJazz subfolder. These are also added to the cross-reference table and file index output, so you can easily find a resource once they are unpacked.

Sorry for the flurry of updates, I've been kind of in a "one more feature" mood and figured it was best to clean off my plate, so to speak, while I'm around again

Unless any bugs crop up, this should be the last update for a while.

Quad core, 4GB RAM - almost 50% improvement and it took 450MB of RAM.

Thank you.

I got tired of hitting '1' in the file finder and realizing I wasn't sure what the next number was I would be typing, waiting too long and the dialog getting bogged down with 30000+ results - having to wait 20 seconds for the stupid dialog to populate, then another 30 for it to depopulate once the new search terms are filtered... I had to fix it. I just had to.

The file finder will now restrict the results to 1000, which seems to keep performance very snappy while providing enough results that no reasonably valid search should get truncated. I suppose I could make that value configurable if someone is using this for searching for more than 1000 resources at a pop.

Thanks, great work.

When I try to run it, I get an error message:
"System.UnauthorizedAccessException: Access to the path 'C:\Reference - All Strings.txt' is denied."

Any idea how I can get around this? Thanks!

(Running Windows 10, if that matters.)

Edit: Never mind, fixed it!

With the latest patch, I ran into a problem with the S4_62E94D38_00000000_F1352E740C4710BA.xml containing a value of null for one attribute, and the Extractor choked on it. I was able to replace the null with 0 and extract it successfully, but thought I'd let you know.

Hey Scumbumbo,

I am not sure if you are aware of this issue so I thought I would post it. The extractor is not putting the ASMJazz subfolders in the correct EP/GP/SP folders. It is extracting them to seemingly random folders (for instance the ASMJazz files for Movie Hangout (SP05) end up in the Spa Day (GP02) folder). None of the ASMJazz files, except those for the BG, is going to the correct location. I fixed this by manually unpacking each EP/GP/SP manually, so it is not an insurmountable issue, just a little annoying. I am not sure if others have the same issue, but whatever I tried to do the unpacking always occurs like this.

I tested this with Single Combined Tuning File selected. Selected a .package file from a mod that I know cotains XML-files. Set a Destination folder and began extraction.

Nothing in the destination folder.
Ticked the Include JAZZ and an emty folder named AsmJazz is created in the destination folder.

A warning message shows up that there are No Tuning in Package and a Warning that says that Delta Build for FP01 contains no binary tuning.

Full Automatic Extract works though.
Is it supposed that this tool can extract XML-files from a single game mod package file? Why warnings from FP01 that I have not selected.

Edit:

Is not a single combined tuning file a number of XML-files in a package file? What am I missing?

Hi, is there a tutorial on how to use this program. I am new to modding, and would like to know how to if I can.

Thank you!

Thanks! But it seems I'm unable to extract the files from the expansions pack and game pack :/

Edit: Nevermind. It was because the game was running.