MTS has all free content, all the time. Find out how YOU can help to keep it running. Tell me how...

XML Extractor for The Sims 4

by scumbumbo Posted 28th Aug 2015 at 6:15 AM - Updated 8th Dec 2018 at 12:51 AM by scumbumbo : Version 3.6.2 - December 7, 2018
118 Comments / replies (Who?) - 20 Feedback Posts, 3 Thanks Posts
Picked Upload! This is a picked upload! It showcases some of the best talent and creativity available on MTS and in the community.
Update 12/07/2018 - Version 3.6.2
  • Updated pack names to add Get Famous
  • Added option to use "Sims 4 Studio" style file naming for the extracted XML

Update 09/30/2018 - Version 3.6.1
  • First, before the New Feature, just some Boring Enhancements
    • Numbers in XML that are not references to other actual XML files should no longer be commented with an incorrect XML reference name. For instance, just having the number 100000 for a buff cooldown will no longer show that as being a reference to PaintingTexture_Sml_IMP_High_01. This is also true for references to OBJD or icon resources that do not have actual XML names.
    • Old path information stored in the XML File Index is removed when loaded by the File Finder, so renaming or moving an extract folder will no longer make finding the actual files an irrelevant process.
    • The new XML File Index no longer includes irrelevant path information. One minor side effect is that an XML File Index produced by this and future versions of the program will NOT be able to be loaded by an older version of the program.
    • The decimal instance ID is now shown as a column in the File Finder.
    • Some rare bugs were found and stomped.
  • Feature for File Finder - Cross Reference Browser
    • The full extract process now produces an additional index file (XML Cross Reference.dat) which contains a cross-reference of all game XML. This index is loaded automatically by the File Finder when it is available.
    • Adds a Browse References selection to the right-click context menu in the file finder. Selecting this will open a cross-reference browser window with two lists. The left list shows all XML files which reference the current XML selection and the right list shows all XML files refered to by the current XML.
    • This allows you to easily find every XML file which references a particular XML resource without resorting to manually searching all the XML files using another program.
    • Click on an XML file in either of the reference lists to change the view to the chosen XML file.
    • Right-click on an XML file in either of the reference lists to view a context menu with options to show the selected file in the search window, view the file in Windows Explorer, or copy or edit the selected file.
    • If no cross-reference index file is available (for extracts generated by a previous version of the program) or if the right-clicked XML file in the search window does not have any references, the Browse References option will not be available.
    • The screenshots above show a few images of the Browse References usage.

Update 09/02/2018 - Version 3.5.8
  • Pretty minor update - just adds a clipboard monitor option to File Finder, but I find it quite useful (which of course is why I added it in the first place!)
    • Monitors the clipboard and immediately pastes any copied text into the search field
    • Option must be turned on in order function it is always off by default (you don't want the File Finder constantly searching for things when you're copying and pasting text).
    • Useful for quickly finding another game file while editing one - while in your editor simply highlight the XML name, or instance ID number in the file you are editing, press Ctrl-C and File Finder will immediately start searching.
    • For example, say you are editing the HaveBabyAtHospital interaction and you see the buff 27833, buff_Suppress_Visible_Motives referenced in it. Not sure what that does you want to take a quick look, simply highlight 27833 or buff_Suppress_Visible_Motives and press Ctrl-C, if the clipboard monitor is on File Finder will immediately search for it (no need to paste it into File Finder).
Update 08/16/2018 - Version 3.5.7
  • Added an option to "normalize" the output order of n and t attributes in XML elements.
    • Due to how the combined XML tuning is stored, sometimes between patches an XML element will have the n and t attributes switch from n-first to t-first, or vice-versa. This can be annoying.
    • Default option is normalization off which is the old behavior of generating the attributes in the order EAxis has given them to us.
    • If enabled you can choose either to always use n-first or t-first order.
    • If using XML Compare to check changes between older XML releases this will result in a huge number of XML changes.
    • Helpful Tip for those using XML Compare....
      • Use the normalization off this one last time to check for changes between the previous patch's XML and this latest update.
      • Once you have finished updating your mods, extract the XML again using the normalization ordering you prefer.
      • In the future, all patches will compare accurately due to the enforcement of the normalized order, so once you start using it leave that option on!

Description

This is a utility program to extract the game's XML files for use in creating mods. XML can be extracted from the installed game packages or a single combined tuning file. The program can also easily be used to extract a single binary tuning file from the game packages and convert that to a combined XML tuning file.

The program is designed to perform these functions quickly by only loading those parts of the game packages that are required for producing the XML and looking ahead to quickly build required lookup tables.

When extracting the XML in Full Automatic mode (the recommended and default mode) the game's strings and references to other XML resource names can be included in comments. This feature is not available when working with a single combined tuning file. If you have multiple game language packages installed, you can select which language you want the XML comments to use.

Also in Full Automatic, the program can optionally output three tab delimited text files with all strings and XML resource names. The two reference files are suitable for opening in a spreadsheet package for easy searching, and the XML File Index can be loaded into the companion XML File Finder program to quickly locate files for viewing and editing.

There are three naming conventions available for the final XML files:
  • Type, Group and Instance (TGI) for quick import into S4PE (e.g. S4_E882D22F_00000000_000000000001A2BC.xml)
  • TGI plus resource name, formatted to automatically create a _KEY Name when imported in S4PE (e.g. S4_B61DE6B4_00000001_000000000001C8C5_object_drink_Cider.xml)
  • Native TS4 naming format using the TGI (e.g. 0x00000000!0x00000000000029c3.0x2451c101)
  • Or just the resource name

Requirements

The program requires Windows XP or higher with the .Net Framework 4.5.1.

The program requires access to an installed copy of the game in order to extract the XML from the packages. To work on a machine that does not have the game installed you would need to copy the required packages, binary or combined tuning files to the PC you need to work on. These files cannot be provided by the author for copyright reasons.

Basic Usage
  1. Launch the program. This step is important!
  2. Select the Full Automatic Extract mode. This should be selected by default.
  3. The program should automatically locate your Sims 4 install folder. If not, you can browse manually for it (this setting is saved when you exit the program).
  4. Choose the options for the extract. Again, these options will be saved for the next time you use the program.
  5. Under Output Options, select the Destination Folder for the XML files. You should create a new empty work folder for your first time. Subfolders will automatically be created for each game pack (see screenshot).
  6. By default, the program will ask you before overwriting any files. You can also choose automatic overwrite at this time, which is useful if you're performing a full extract overr an existing extract (to avoid being prompted billions and billions of times). If you forget, you can select to "Stop Asking Me" when the overwrite dialog appears.
  7. If you choose to rename existing files, any file will be automatically renamed with (Copy #) appended to that file rather than overwriting. You will be warned if you choose this option as it is very slow if doing a full extract over an existing extract (30,000+ files would need to be renamed).
  8. Click the Begin Extracting button and wait. The program only updates it's user interface once for every file, so for the long steps it may appear to hang. Just wait and it will move on. The entire extraction process should take less than 3 minutes on a fast PC. Including strings, XML references and the file index will take only a few extra seconds!

File Finder Usage

In order to use the file finder, you must have extracted the full XML using the "Create XML File Index" option. This option is on by default.

If you installed using the MSI file, a hot key is registered with Windows to launch the File Finder application using the Ctrl-Alt-F keys. Note that this key sequence will not be active until Windows Explorer is restarted - the easiest way to do this is to logout and log back in after installing. You can change this key sequence if desired by editing the properties of the XML File Finder application in your Start menu.

The first time you launch the file finder, you will need to load an index file produced by the extractor. This file should be in the output folder of the XML extract in a text file called "XML File Index". Since the full path to the XML files is included inside the file index, if you move the XML files to another location on your hard drive you will have to recreate the index file using the extractor.

Once an index file is loaded, simply type the search terms you wish to look for in the Search For field. There is no need to hit Enter to search, if you pause typing for 1/2 second the search will being. You can continue typing to modify the search. Searches are not case sensitive.

So for instance, when looking for interactions dealing with the piano, you might start by typing "interaction". The program will find (currently) 7,720 matches, which is a bit much to look through, so add a space and type "piano" which yields 53 matches. You can keep adding terms until you find what you are looking for, so if you then added "pract" to the search terms you would see 6 matches.

The search can find files using portions of the TGI, so if you're looking for the 0x7B2E action (loot_Buff_Computer_HighScore_Typing) you could find this using the terms from the name, or by just typing "7b2e" (which also finds the interaction 0x17B2E).

The instance is also stored for searching in decimal form, so if you see the broadcaster type "129459" in an XML file which doesn't have comments, you can easily find that file by just typing "129459" into your search terms to find "broadcaster_Jealousy_ClubEncouraged".

Once you have a short enough list of files that you can find what you're looking for, you can simply right-click the file in the list to select either "Open in Windows Explorer" or "Open in XML Editor".

Version History
  • Version 3.3.1 - Initial release
  • Version 3.3.2 (August 29, 2015)
    • Added language choice for string comments if multiple language packages available.
    • Added native XML naming convention
  • Version 3.3.3 (January 14, 2016)
    • Bugfix: Missing or invalid XML extracts will get placed into an error directory rather than crashing
    • Bugfix: Empty XML attribute values will correctly generate an empty string on output
  • Version 3.4.0 (January 17, 2016)
    • XML Extractor can produce a file index when performing a full extract
    • Added XML File Finder program to load file index and quickly search for XML files
  • Version 3.4.1 (January 18, 2016)
    • Bugfix: Warn if a game package contains no binary tuning instead of crashing.
  • Version 3.4.2 (February 3, 2016)
    • Option to group results in File Finder by resource type
    • Results in File Finder are sorted by name
    • The input focus in the File Finder defaults to the search terms input
  • Version 3.5.0 (February 5, 2016)
    • Uses RAM instead of temporary files for the conversion process.
    • Can process each game pack in a separate thread to use multiple CPU cores.
    • Maximum number of simultaneous threads is easily selectable.
    • Added package name for SP05
  • Version 3.5.1 (February 7, 2016)
    • Option added to unpack ASM/JAZZ resources.
    • Exiting the program during a full extract is now (a slight bit) cleaner.
  • Version 3.5.2 (February 24, 2016)
    • Limit file finder will only show 1000 matches to keep the Windows dialog from getting bogged down with too many entries in the listview box.
    • Updated the two programs to use same version numbers, regardless of which program has changed, to keep versioning simpler.
  • Version 3.5.3 (November 8, 2017)
    • Name info for packs through GP05/EP04/SP13 are defined
    • Support for up to SP20 (TGI group number 0x00000029)
    • Core code will support group numbers up to 0x0000003F if future packs are created (only the XML resource in sources will need updating)
    • AsmJazz resources are extracted to proper folders when a full extract is performed
  • Version 3.5.4 (January 3, 2018)
    • Fixes the resource type ID used for extracting relationship bit (relbit) resources.
  • Version 3.5.5 (March 20, 2018)
    • Name info for packs through GP06/EP04/SP14 are defined
    • File Finder
      • Improved index file load times substantially.
      • Multiple search results selectable.
      • Double-click a selected search result to launch XML editor.
      • Copy to Folder feature added to right-click menu.
      • Mark search results if the XML file has been deleted.

Additional Credits
  • Velocitygrass for the original XML Extractor and source code.
  • SimGuruModSquad for obtaining permission to share the game's PackedXmlDocument source code with me.
  • Too many people to mention by name at Mod The Sims who have tested and given suggestions. But thanks to all of you, feel free to visually insert your name here!
  • Caphyon Ltd. for the freeware Advanced Installer used to create the install file.
  • Microsoft for the Visual Studio Express IDE for .Net.

Operating System: Windows
Utility Type: Miscellaneous
Plugin Type: Not Applicable

You must have the expansion or stuff packs listed below installed to use this custom content. Please see the post text for any exceptions.

Sims 4
Key: - File was updated after upload was posted
Filename - Tip: You can click the magnifying glass to see the archive contents Size Downloads Date
XML Extractor for The Sims 4 (MSI Installer).zip | Installer for the XML Extractor

Size: 282.0 KB · Downloads: 10,292 · 8th Dec 2018
282.0 KB 10,292 8th Dec 2018
XML Extractor for The Sims 4 (EXE Only).zip | The EXE files only if you dont want an installer

Size: 162.7 KB · Downloads: 10,135 · 8th Dec 2018
162.7 KB 10,135 8th Dec 2018
TS4 Combined XML Extractor (VS 2015 Source Code).zip | Visual Basic source code for the project (v 3.6.2)

Size: 268.4 KB · Downloads: 1,828 · 8th Dec 2018
268.4 KB 1,828 8th Dec 2018
Don't forget to Thank the creator after downloading! Use the Thanks button located above the screenshots above.
Basic Download and Install Instructions:
1. Download: Click the File tab to see the download link. Click the link to save the .rar or .zip file(s) to your computer.
2. Extract: Use WinRAR (Windows) to extract the file(s) from the .rar or .zip file(s).
3. Cut and paste the file(s) into your Mods folder
  • Windows XP: Documents and Settings\(Current User Account)\My Documents\Electronic Arts\The Sims 4\Mods\
  • Windows Vista/7/8/8.1: Users\(Current User Account)\Documents\Electronic Arts\The Sims 4\Mods\
Need more information?
  • Anyone can use both .rar and .zip files easily! On Windows, use WinRAR.
  • If you don't have a Mods folder, just make one.
 
118 Comments / Replies (Who?) - 87 Feedback Posts, 26 Thanks Posts
Hide Thanks Posts for this thread (Show only feedback)
Page 2 of 5
Lab Assistant
#26 Old 5th Dec 2015 at 12:36 AM
Broken in 64-bit patch
Test Subject
#27 Old 5th Dec 2015 at 12:39 AM
It's throwing an error at me too. Scumbumbo's been gone less than a month, hopefully it's not a permanent disappearance, as we don't currently have an alternative.
Field Researcher
#28 Old 5th Dec 2015 at 2:59 PM
Quote:
Originally Posted by jaybud4
It's throwing an error at me too. Scumbumbo's been gone less than a month, hopefully it's not a permanent disappearance, as we don't currently have an alternative.


I have the error too but it still extracts binary files, which can be extracted by velocitygrass' tool : http://modthesims.info/t/534316. I'm not sure everything is extracted correctly, but still better than nothing in the meantime
Forum Resident
#29 Old 5th Dec 2015 at 7:27 PM
I did the same thing.. but I had to convert the bnry file to xml before I could use velocity's tool

When an engineer says that something can't be done, it's a code phrase that means it's not fun to do.
Forum Resident
#30 Old 5th Dec 2015 at 7:49 PM
Similar to the method Neia mentioned.... here are the amateur hacky steps I took to get the files.

Use Scumbumbo's tool to extract the BNRY files to a folder.
Start with Full automatic Extract
You will get an error. But the binary files will be there.
exit the tool

Using the tool again, do single combined tuning file and browse to the binary file in your folder.

S4_62E94D38_00000000_F1352E740C4710BA.bnry (Base Game)
This will create the combined XML it takes some time... so just wait it out.

Now use Velocitygrass's tool to convert that combined XML into a filed structure.

When an engineer says that something can't be done, it's a code phrase that means it's not fun to do.
Mad Poster
#31 Old 5th Dec 2015 at 9:37 PM
I have patched Scumbumbo's code so it no longer throws an error. The problem was, the new expansion package file doesn't include any XML yet (resource 0x62E94D38). There was no error-catching in his code to work-around that. I imagine, next week when the expansion is released, they'll throw an updated package out there that has this resource so the XML for the new expansion can be unpacked as well.

For now, if you want to use his patched program, I have put it up on my Dropbox.

Source code: here
Just the Executable: here
Forum Resident
#32 Old 5th Dec 2015 at 11:18 PM
Sweet! Thanks!

When an engineer says that something can't be done, it's a code phrase that means it's not fun to do.
Pettifogging Legalist!
retired moderator
#33 Old 5th Dec 2015 at 11:36 PM
Last I heard, scumbumbo isn't gone, he's just very busy. I put a note in the first post that points to post #31 (Deaderpool patch). Thanks!

Stuff for TS2 · TS3 · TS4 | Please do not PM me with technical questions – we have Create forums for that.

In the kingdom of the blind, do as the Romans do.
Forum Resident
#34 Old 5th Dec 2015 at 11:40 PM
That's great news =)
All of your work is greatly appreciated Scumbumbo!
Don't leave us! well maybe for a little fallout.. but after that come back!

When an engineer says that something can't be done, it's a code phrase that means it's not fun to do.
Mad Poster
#35 Old 8th Dec 2015 at 4:19 PM Last edited by Deaderpool : 11th Dec 2015 at 4:39 PM.
I'm getting a crash after the Get Together update today since they added the new XML files for that update. I'm in the process of doing a debug on it and see if it's something else simple to fix. If so, I'll let plasticbox know when I post an update to it here so he can do his linking magic.

Edit: Seems like there's some sort of problem with the packed-XML files in the Sims 4 distribution. I get XML errors when I try to validate it...a missing tag or something. Not really something I'd be able to release a "patch" or anything to fix since it is a problem with the files themselves.

Edit: Edit: Ok, I figured-out the problem. Somewhere during the extraction of the binary file from the package and then the decompression of the XML from the binary, a null-character was being inserted into the final combined XML file. That caused an exception that prevented the individual XML files from being extracted. My theory is that it is in a localization of one of the dictionary entries and if I did an extraction without the text strings, it would have probably worked ok.

Anyway, I added some code to clean-up the XML files before running the individual XML extraction from the combined XML that removes any null characters. Seems to work fine and I can now see the EP02 XML files. Also, I added the Spooky Stuff and Get Together to the drop-down of Add-ons in the utility.

There are two zips attached. One includes the source files, the other is just the executable.

Thanks!
Download - please read all instructions before downloading any files!
File Type: zip XML Extractor for The Sims 4 - Includes Source.zip (634.7 KB, 83 downloads)
File Type: zip XML Extractor for The Sims 4.zip (125.5 KB, 216 downloads)
Test Subject
DELETED POST
9th Dec 2015 at 4:56 AM
This message has been deleted by Hadron1776.
Field Researcher
THANKS POST
#36 Old 10th Dec 2015 at 2:07 AM
Quote:
Originally Posted by Deaderpool
I'm getting a crash after the Get Together update today since they added the new XML files for that update. I'm in the process of doing a debug on it and see if it's something else simple to fix. If so, I'll let plasticbox know when I post an update to it here so he can do his linking magic.

Edit: Seems like there's some sort of problem with the packed-XML files in the Sims 4 distribution. I get XML errors when I try to validate it...a missing tag or something. Not really something I'd be able to release a "patch" or anything to fix since it is a problem with the files themselves.

Edit: Edit: Ok, I figured-out the problem. Somewhere during the extraction of the binary file from the package and then the decompression of the XML from the binary, a null-character was being inserted into the final combined XML file. That caused an exception that prevented the individual XML files from being extracted. My theory is that it is in a localization of one of the dictionary entries and if I did an extraction without the text strings, it would have probably worked ok.

Anyway, I added some code to clean-up the XML files before running the individual XML extraction from the combined XML that removes any null characters. Seems to work fine and I can now see the EP02 XML files. Also, I added the Spooky Stuff and Get Together to the drop-down of Add-ons in the utility.

The files are still located where they were previously, they've just been updated:
Source code: here
Just the Executable: here

Thanks!


I really wish to give you a hug. Thanks! :D
Test Subject
THANKS POST
#37 Old 12th Dec 2015 at 2:21 AM
Bless you, DP.
Instructor
#38 Old 13th Dec 2015 at 10:54 AM Last edited by Nukael : 13th Dec 2015 at 12:59 PM.
Whenever extracting, it uses one of the Asian languages as strings in the XML's and the Reference - all Strings (I don't know whether it's Japanese or Korean) and not the English version.

EDIT: It seems to be using the Chinese string file, which is the first string file in order. So maybe the extractor is just using the first string file it comes across? After deleting ALL string files, except the US English one, I'm getting English strings.

This space intentionally left blank...
Mad Poster
#39 Old 14th Dec 2015 at 11:22 PM
Quote:
Originally Posted by Nukael
Whenever extracting, it uses one of the Asian languages as strings in the XML's and the Reference - all Strings (I don't know whether it's Japanese or Korean) and not the English version.

EDIT: It seems to be using the Chinese string file, which is the first string file in order. So maybe the extractor is just using the first string file it comes across? After deleting ALL string files, except the US English one, I'm getting English strings.

Probably. I doubt when Scumbumbo wrote it that he had multiple string files. I have Spanish and German, but they were not used instead of English. It's probably only certain ones like the Asian one you saw.
Lab Assistant
DELETED POST
24th Dec 2015 at 7:08 AM
This message has been deleted by Reptonplay.
Deceased
Original Poster
#40 Old 14th Jan 2016 at 5:14 AM
Quote:
Originally Posted by Deaderpool
Probably. I doubt when Scumbumbo wrote it that he had multiple string files. I have Spanish and German, but they were not used instead of English. It's probably only certain ones like the Asian one you saw.


Thanks for the patch @Deaderpool !!

I'll update the download. Not sure when I'll get it posted - probably not tonight, but in the next few days at least. The string language selection was added in v3.3.2, but you used v3.3.1 for your base - so I'll merge your changes into the latest version.

If there are any other fixes that are needed for this, speak up quickly while I'm here
Deceased
Original Poster
#41 Old 14th Jan 2016 at 8:13 AM
Version 3.3.3 has been uploaded fixing the identified bugs.
Test Subject
THANKS POST
#42 Old 14th Jan 2016 at 11:39 AM
Thanks for the update Scumbumbo!
Mad Poster
#43 Old 14th Jan 2016 at 9:38 PM
Quote:
Originally Posted by scumbumbo
Thanks for the patch @Deaderpool !!

I'll update the download. Not sure when I'll get it posted - probably not tonight, but in the next few days at least. The string language selection was added in v3.3.2, but you used v3.3.1 for your base - so I'll merge your changes into the latest version.

If there are any other fixes that are needed for this, speak up quickly while I'm here

All that I remember were the ones I already commented on and you merged in (or fixed better!) so nope, sounds good to me!
Pettifogging Legalist!
retired moderator
#44 Old 15th Jan 2016 at 6:07 AM
One can also extract binary JAZZ with this btw (I don't remember any more since when exactly that is binary; I think they changed it at the same time as XML):

* Find and export the JAZZ you want to extract with s4pe , and rename it to .bnry;
* Use the "Single Combined Tuning File" option to extract it;
* Since this won't contain any whitespace, use e.g. the "Pretty Print - with line breaks" option in npp (PLugins > XML Tools) or something similar in whatever other text editor you have to format it.

The game still reads JAZZ in XML form just fine, Maxis only deliver it as binary for performance reasons.

Stuff for TS2 · TS3 · TS4 | Please do not PM me with technical questions – we have Create forums for that.

In the kingdom of the blind, do as the Romans do.
Deceased
Original Poster
#45 Old 17th Jan 2016 at 6:56 PM
I must apologize for updating this so quickly after the v3.3.3 release fixing that bug, but I thought of a really nice new feature. Version 3.4.0 adds a file finding program which, I don't know about you, but I'm going to love!
Instructor
#46 Old 17th Jan 2016 at 11:17 PM
Hi @scumbumbo,

It's not doing automated full extraction for me.
Used to do. It throws an error, but hitting continue it extracts the compressed binary. Then it just stays there with no CPU usage at all.
Next I had to manually extract from the compressed binary (single file) it produced a (no white spaces) combined XML.
Next I had to run that XML again on single pack extract and it produced the the individual sub folders and XML files...

Finally I will do a debug next time and send you.

Thanks for the updates and new search tool which i didn't tried yet bu will soon.
Thanks Deaderpool for helping .
Deceased
Original Poster
#47 Old 17th Jan 2016 at 11:56 PM
Quote:
Originally Posted by azoresman
It's not doing automated full extraction for me.... I will do a debug next time and send you.


Yes, I'll definitely need more information as it's working fine for me. If you can let me know what exception is being thrown and where that'd be a minimum, but any additional information you can provide like which install version you're using (MSI, exes or compiled from sources) and what option settings you were using for the full extract would be helpful.
Instructor
#48 Old 18th Jan 2016 at 12:03 AM
Quote:
Originally Posted by scumbumbo
Yes, I'll definitely need more information as it's working fine for me. If you can let me know what exception is being thrown and where that'd be a minimum, but any additional information you can provide like which install version you're using (MSI, exes or compiled from sources) and what option settings you were using for the full extract would be helpful.


Just sent you PM with error code

Using latest 3.4.0 EXE didn't installed, Yesterday I tried 3.3.3 and it did the same error.

All options are selected, Strings ENG_US for output TGI+Names and overwrite in on.

I have updated Op.sys today just in case.
Page 2 of 5