I am so excited for this mod, thank you for all the hard work you done so far on this one. There was a person already who tried to eliminate/reduce lag by allowing the game to use 64bit features, but no updates on that one for years. Most likely he abandoned that project. So I really hope all your time and energy will come to fruition soon. This mod will be life changing for us sims 3 fanatics.

Yay! I finally finished the core program!

Well, technically, it's not really done yet, as it hasn't been tested on many package files, so it needs to be tested first. Although I have tested it on one huge 1GB file and it managed to convert that correctly!
It took about two minutes to convert that one GB file I mentioned.

I'm pretty drained right now so I'll continue tomorrow.

EDIT: Oh wow, noticed something in the code that could be optimized. The one GB file that took 2 minutes now takes 1 minute now! I've also reduced the memory usage from the first version when converting a 1GB file from 3 gigabytes of RAM to only 1GB.

I'm on a roll today!

(bounces excitedly).... I've had my lurking, beady eye on this thread since it popped up and am really pumped to read about the progress thus far....

Yep, also excited. I still have to record some of my footage in slow motion just in case the game is feeling laggy at the moment. And slow mo is a big pain in the ass because I then have to manually add certain sound effects back in. And the whole thing could look a bit unnatural. And it takes more time and disk space. And not everything scales properly with the slow motion cheat enabled.......needless to say, plenty of reasons to avoid it.

I discovered a bug when converting large package files. Since normally, the FullBuilds for TS3 and TS4 are around 800 to 900 megabytes, it would convert to just under 2GB when processed. However, TS Medieval's FullBuilds are already close to the maximum of 2GB, so when converted, they'd reach around 3.5GBs.

Whenever a package file goes over the 2GB limit, the game slows down to a crawl, and if it's over 3GB, it outright crashes the game.

This is an easy fix, as I also encountered this in the first version of the tool. All I need to do is to split up the package files into multiple smaller ones.

Of all the dumb crap I've tried and considered, it's never even occurred to me that 32-bit means there's limitations on file sizes as well. Even though I know for a fact 32-bit systems can't handle files larger than 4 gigs! That's not just Windows, though, is it, that's TS3 as well. I don't know how much difference it makes beyond that but would you say it matters, when comparing files that are smaller than a gigabyte? I don't see 100 8-megabyte files loading quicker than 1 800 megabyte file, but there's got to be a sweet spot.

I experimented with changing how much a file gets split early on. However, I only tested the package file splitter on TS4 package files, where the tool has a less noticeable effect.
I'm going to go out on a limb and say that 100 eight megabyte files could be potentially much faster than 1 800 megabyte file.

I'm going to handwave it a lot because I had three paragraphs written and it was way too complex of an explanation. So, enjoy this simplified explanation!

You can only read from three sources, TS3 memory, Windows cache, and the HDD/SSD.

The game almost never caches actual packages in Windows cache because (I believe?) each file has to fit completely entirely in memory for it to be cached. However, if you split the 800 MB file into 100 eight megabyte files, then instead of Windows caching the whole 800 MB package if you you frequently use one hair from it, you'd only have to cache the eight megabytes of data the hair actually resides in (assuming that the hair is all located completely in only one split package).

I hope that makes sense! I'll experiment to try and see if my theory is correct. I'm going to drink more caffeine now because this is a genuinely exciting thought I tossed out early on that might be useful.
See ya!

Yes, that'd be true, wouldn't it, that by splitting up the packages the game will most likely load in less unnecessary data. I think I like where this is going, but since the FullBuilds still need to be accessible I don't think it'd be wise to split them as small as 8 megs. 50 maybe, or 80, or 100. It'll still have a similar effect.

you should see the opposite, open/lock/close routine will add additional overhead (though 100 files will be a minimal one), one could argue that searching the one big file for particular value (or "pointer") may be slower, but it will be slower only if you assume that you do not need search these little chunks of files at all. Also, there's no reason why the one big file could not be read in parts. And in reality: that is how it's usually working. Until someone is incompetent or insane and open&read file blindly, as long as there's still free memory to fill. And then the program dies.

I can't wait ). Thanks for all your hard work!

After I made that Third Person Mod video a while back, I immediately wanted to do another one but the world was too laggy, so I shelved it. I think I'll turn it into a bit of a mod review like the other one.

I implemented the package splitter, started converting The Sims Medieval's files, started the game... and it still crashed. I inspected the code a couple more times, and fixed some major bugs, which lead to the game actually not crashing on startup! However, there was still one problem. The game still wouldn't read the package file.

I switched over to TS3, converted DeltaBuild0, and placed the converted file in the Mods folder. If the game read the package file correctly, it should instead use the resources in the converted DeltaBuild0, as it has a higher priority. This means that it should not read the DeltaBuild0 in the Program Files at all.

I opened Process Monitor (procmon) which is a program that has the ability to show you what files a program is reading, and what it read specifically. I noticed an oddity. It was opening the converted DeltaBuild0 in the mods folder, reading the first 96 bytes, and closing it. It did this a large amount of times per second. This meant that something was wrong with the package file I was creating.

After a day of debugging, I noticed that I wasn't updating one part of the package correctly, which is the size of the actual index. I didn't update it from the original package file, so it was smaller than the new file's index size.

Long story short, the index and resource datas were correct, it was just the package file header, the first 96 bytes that had an incorrect index size, so the actual index data was not read fully, causing the game to think it was damaged.

This should be the final bug to fix if there aren't any more that pop up. We're getting closer to the finish line!

Yay! It's officially working!!

Here's a video showing off CAS with the same packs installed as before.

https://youtu.be/MMfwCkspNMo

Needs more optimizations though. It currently takes forever to convert all the objectCache/Full/DeltaBuild files. Even with all four cores of my computer being used it takes so long that as of right now, I'd recommend that you take a nap while it converts the game files.

Good thing with the rebuilt version, I can just swap out the "conversion" part of the program, and I don't have to worry about breaking the parts that read and write package files! Optimization should be take a lot less time now.

Oh that's brilliant. Do you have a date for when it's going to be out? I've got an episode coming out on the 27th or 28th and it'd be nice to immediately start doing the video review on this mod.

Hmm.. this thread is interesting..

Sorry to jump in.., I don’t play TS3 anymore after a while but I have tried what’s in this subject before quitting the game, maybe what I understood about how the game reads and executes its database from package files can contribute to this thread.

First things that needs to be understood in this subject, reducing loading time and lags (or stuttering) are two different things, let’s say we managed to reduced TS3 loading time in under 1 minute (and yes it’s achievable with all EPs and SPs installed and ton of mods) from main menu to the saved (or new) game fully started, it doesn’t mean the game won’t lag at some point, by being able to reduced TS3 loading time it will only eliminate lags (or stuttering) that caused or related to how the game reads and executes them, the majority of lags and stuttering were caused by the complexity of other things like routing and etc as what Nitromon has mentioned in post #17

At the time I was still active playing the game, I did considered about making the same thread (as someone contacted me and asked me about the subject) but then I changed my mind, the reason why I changed my mind was because it may violates the terms of use TS3 set by EA.

The personal experiments I did in reducing loading time was based on TS3 1.67, to understand what the 1.67 super patch does to the game and to understand the concept how the game reads and executes its package files, we only need the BG and the 1.67 super patch (we can add the other EPs and SPs later on).

Now let’s start with the BG stand alone installation, do not add anything or patching it, run the game (with the assumption that FPS Limiter were also installed) and write down its loading time, this loading time is the best reference to what time we can achieve when everything is installed.

Notice that with the BG stand alone installation, there’s no Deltabuild0.package file, there’s only Fullbuild0.package file, and if we scan this BG stand alone installation with Dashboard, there’s no conflicts no duplicates and no corrupts, so in other word, in a perfect world, this TS3 BG stand alone installation is the “ideal” TS3 loading time and gameplay or what was originally expected when TS3 was designed.

Then come the first EP WA and TS3 BG needs patch to link the EP to the game, the game files installation structure was completely changed, and this is the 1st change, the 2nd change was started with SP HE and followed by AM EP FL SP LN EP and OL SP, starting from GE EP it was changed again to the final installation structure model which wraps up by the super patch 1.67.

So there are 3 stages that needs to understand how the WA patch works, then the AM FL LN and OL patches works, and the 3rd and final model of the patch from GE to super patch 1.67, and in my experiment I modified all the file installation structure format as in the GE EP onward starting from HE SP.

Now let’s go back to TS3 BG stand alone and patch it to any patch version, for this purpose, let’s just patch the BG with 1.67 super patch without adding any EPs and/or SPs, once the patch completed, now we can see there’s Deltabuild0.package file added to BG (in addition to Fullbuild0.package)

Now if we run this TS3 BG (patched to 1.67) installation, its loading time it’s already slower than the “ideal” time, and if we scan this TS3 BG (patched to 1.67) folder with Dashboard, we will find conflicts duplicates and corrupts, and these are the ones that slowing its loading time.

So to make TS3 loads faster, we need to get rid those conflicts duplicates and corrupts, and that can be done with the help of S3PE, once they’re cleaned, the loading time will close to the “ideal” stand alone TS3 BG, and the rest with all the EPs and SPs are affecting TS3 BG the same way with or without those conflicts duplicates and corrupts cleaned up or not.

So each EP and SP has its own Fullbuild(<number>).package file, and the one that linked them to BG is the Deltabuild0, so the more EPs and/or SPs added, the bigger the Deltabuild0 file size become, and in the end it will be wrapped up with the super patch 1.67

To clean up those corrupts conflicts and duplicates, just copy Deltabuild0 and Fullbuild0 to the same folder and scan them with Dashboard, then break both of them with S3PE into some small pieces type of files to make it easier to sort them out, remove all duplicates that in Fullbuild0, this will eliminate all conflicts between the two, the corrupts one are in the Deltabuild0 with empty data or no entry in it, if all of these were cleaned up, then these two won’t slow down the loading time of the game anymore.

The same thing when EPs and SPs were added, they need to be cleaned up as well, do it 1st with WA as they’re launched, once they’re cleaned up, modify the file structure installation so it’s in the same format as the GE EP and onward.

The only thing that might still slow down its loading time after all has been cleaned up is the size of both Deltabuild0 and Fullbuild0 after they’ve been cleaned up, so in theory and practice, taking some of them out is making the game load faster, but if we put the things that taken out from them into mod package file and place it in mod directory, it doesn’t make the game loads faster but in fact slowing it down (even after they’ve been cleaned up).

So my solution was to take out a lot of the things that I don’t want to use anymore with the help of object cloning tool and remove them with S3PE, this will make both of my Deltabuild0 and Fullbuild0 smaller in size, and then I can add any object mod from the mod integrated into Deltabuild0, this will make the loading time faster, I only left core mod and scripts mod in the mods folder.

There’s nothing to change to the Fullbuild1 and Fullbuild2 package files, because they are contains with texture and audio files only.

Once all these steps are done, back them up and then replace the ones located in the game installation files directory. Keep both of the originals and the cleaned up ones as backups in separate place before placing it to the game installation directory. Test and write down the new modified TS3 loading time, any new future mod files that would be added can be added to Deltabuild0 as long as they are objects so it won’t compromise the new faster loading time.

So those are the concept of my experiments to improve TS3 loading time the last time I played it, and in addition to make it even faster, I put them to RAMDisk, and that’s the max I can get, but still not fixing the lags or stuttering as they are caused by other things mostly and that would be another subject.
I don’t know much how this would contribute as the thread progressed and obviously I don’t play the game anymore, but this thread really interested me to play again

PS: I don’t know what tools that being used by the OP, I only use S3PE, Dashboard, Object Cloner and Process Monitor, I didn’t convert anything but removed some and combined with S3PE, I’d suggest whenever using S3PE working on DeltaBuild0 and Fullbuild0, always place them in RAMDisk before processing, it’s a hard work for HDD and/or SSD.

Huh. I don't know how you got that information, but you seem to have misunderstood the way patching works with TS3 (and in a way, TS4 too.)

The DeltaBuilds having duplicates and empty files is completely intended. The "duplicates" are simply newer versions of files in the FullBuilds, while the empty files are files that have been deleted from the FullBuilds.
The part about FullBuild0 being different from FullBuilds 1 and 2 is wrong too. There's nothing different about FullBuild0. You could extract all TS3 files from the packages and toss them into random package files and it'd still work. (Yes, I did that while testing the tool, lol)

The reason why DeltaBuilds get larger with each update is because with each passing update, the more stuff changes. Like, if EA updated an animation or texture, it'd be placed in the DeltaBuilds, with the same instance id. The program you're using assumes that anything with the same instance id is a duplicate, which is very useful for mods, but not very helpful for the Delta/FullBuilds as the DeltaBuilds job is to have the same files from the FullBuilds but up-to-date.

Yes everything in Deltabuilds is newer and some are add on to Fullbuild, but in terms of reducing loading time, this duplicates are affecting the lowest time it can achieve, I don't know if the old ones in Fullbuilds were being paused by the game when Deltabuilds are there (or are they're being reads both way), but if the goal is to reduce the loading time then these things matter.



Yes of course it would still works, and this is actually where I was worried if it would violate the TS3 term of use set by EA, for example I replaced all the taxis with Ford Crown Victoria (made by FreshPrince) for testing purpose as well as other cars with realistic cars, the texture for the original taxis textures were located in Fullbuild1, all the audios are in the Fullbuild2, just extract them to check them yourself, to place Ford Crown Victoria as a built in car game (not as a mod), I need to replace all the taxis files in Deltabuild0, remove the ones in Fullbuilds0, replace textures in Fullbuilds1 and use the original taxi audio in the Fullbuild2

Now with TS3 BG stand alone but patched with 1.67, any fullbuild(<number>) from any EPs or SPs can be integrated to either Deltabuild0 or Fullbuild0, so this can be misused to pirating the game, just by downloading superpatch 1.67 and patch the game, anyone can get copy of fullbuild(<number>) package file and merged it to BG and they won't need to buy any SPs anymore (and maybe EPs as well), and that's why I didn't continue with the idea.

I'm no lawyer but as sure as hell they violate a basic law when they release a broken product. What EA was doing is at least morally wrong. Does it really matter if you violate their TOS when you do the work they should have done ages ago? It's not like they care anyway.

This is a company who brazenly admitted that they rely on modders to fix bugs instead of them doing it themselves...they have no standards whatsoever

I am pretty sure that in my country, I am allowed to modify any product that I own, as long as it is for personal use. No matter what terms of use are made up by a company. This goes for any offline game too. I know that this goes for any EU country at least. Don't know about other countries.

I used to use my old XBox console to stream movies and run linux on it. Microsoft has no say in that, legally. I only recently replaced that with a Raspberry PI. But my point is, I could turn TS3 into an accounting management system and EA can cry all it wants about it. Not that they would, or even would know about it, but I like the idea of them crying somehow.

At this point I doubt EA would even care about these sorts of technicalities.

I don't like saying "my country" because I'm not a heavily accented Eastern European, but yeah, in my country, once you buy shit, it's pretty much yours. They can tell you you're not allowed to do this or that, but that's usually pending a lawsuit from some consumer rights organization at which point it turns out that Silicon Valley doesn't actually have any say in how things really work. Not in the physical world, anyway, because Steam just told me I can't gift a certain TS3 pack to a friend in the USA because of a regional price difference. The EU introduced a law last year which makes it mandatory for EU online retailers to sell each product for each price to each EU country. I guess Steam can legally restrict sales outside of the EU, though Steam has a long and rich history of skirting our laws.

In my understanding about these things called digital properties or licensing, anything that came from them as we bought them (and it’s in our possession), but yet it’s still counts as their properties, the only freedom we have is the Mods and/or Saved Games Folders.

The way I see the problem, the actual loading time problem is located in how they manage allocating them when placing them, if we take example like the Windows Updates, the old ones are either removed or replaced or stored somewhere else so it can also be useful for rollback purpose, but that’s not the case with TS3 whenever new EPs or SPs came out with their patch (update), anyone could argue it may have its own purpose.. but at the cost of loading time, now just imagine how Windows Updates outcome would look like if they were managed like these patches?

So to fix the loading time there’s no other way than altering their digital properties, I’m not a programmer but I do understand some basics programming like how program reads its database, unless some advance modders with advance programming skills would make like a core mod that alter and tell the the game how it would read its installation files, like pause or skip reading at these duplicates section or go to somewhere else etc from Mods folder, then the only way around is by manipulating the installation files so the game would skip them (because they don't exist anymore).

So this digital properties term has many interpretations, in one side it is our property because we bought it and it’s in our possession, on the other side it also considered as the other way around, the one thing that made it our property because we bought it as the right to use, but we still can’t use any way we want it, that’s why it’s confusing how it really works.

I still kept all of my altered Deltabuilds and Fullbuilds installation files in my external HDD, anytime I want to install the game back, I replace all the original ones with the altered ones.

The thing is if I make a step by step tutorial how to altered TS3 structure installations files, it’s not the same as making a mod and places them in mods folder, we may have freedom in making mods, but it’s not the same thing as altering the game installation files.

And if I upload and share my altered Deltabuilds and Fullbuilds files, not only they are big size files then it also might be considered as sharing a pirated copy, and when I realized about what I’ve been messing with, that’s the time I decided that I’m done with TS3 and just let my daughter to play with it, that’s why I just keep them to my own personal usage.

Nah, it's actually really simple. I paid for it, it's mine. Fight me. I'm a glass is half full kind of guy, a "everything's legal until it's not" sort of person who considers morality a better guidebook than whatever dumb laws there are this week. If something's illegal it may be for one of two reasons: either it was decided this or that is socially unacceptable, or somebody in a suit paid enough money to make it illegal.

@PapaEmy OK, if you're dealing with the legal semantics of a situation like game installation files, technically you don't "own" any part of it. Even discs are just a license to install and use the product, which refers to the game itself. That much is obvious. But many people get up in arms about "game rights". If people actually read TOS agreements, it makes it clear skimming over all of the legal jargon that you don't "own" it, you have a license which allows you to use it.

Now, with that said, anytime you run into trouble with this is usually because of one of two things.
1) Porting assets from one game to another, such as Bethesda games. EA seems not to care if people do this or doesn't care enough to take action. T2to3to4 conversions happen all the time.
2) Reverse-engineering the game's code, which would involve modifying the game executable itself. You might get warned about modifying game installation files "voiding your warranty" so to speak, but unless you are actually taking the game exe itself apart, it's really not a big deal.

So honestly, since it's obvious that EA has washed its hands of TS3 anyway, and doesn't seem to care about fixing their own games to work properly, a tool handling the Full/Deltabuilds hardly sounds like the worst thing in the world.