Home | Download | Discussion | Help | Site Map | New Posts | Sign in

Latest Site News

Theme Seasons and Celebrations - posted on 1st Oct 2017 at 1:52 PM
Replies: 71 (Who?), Viewed: 29535 times.
Page 1 of 3
Lab Assistant
Original Poster
#1 Old 30th Aug 2014 at 1:34 AM
Default Converting pyo files to py files
Greetings. Anyone know how to convert compiled python in pyo files to py files so they can be read in the python 3 shell?
Advertisement
Pettifogging Legalist!
staff: moderator
#2 Old 30th Aug 2014 at 1:48 AM Last edited by plasticbox : 1st Sep 2014 at 1:41 PM.
Moving to Modding Discussion.

Also, from the EA forums :
Quote:
Originally Posted by SimGuruModSquad
One of the reasons we chose python is because it's pretty much an open book. Unfortunately we cannot release our raw source code. So pyo->py is an exercise for the reader, we know how clever you guys are. As I have mentioned previously, we have documentation coming but it will not cover this topic. As you guys figure things out more and have specific questions on the python side of things I will do my best to give you the info you need.


ETA, also from EA (08-26):

Quote:
Originally Posted by SimGuruModSquad
Out of the Mods dir, the game will load Python zips and loose python files. Loose files are meant for iterating while developing and zips are meant for distribution. Zips should contain compiled python for performance. The game itself contains .pyo files.

As for documentation, we have no immediate plans to release info on all our python APIs since there are a lot of them. So some disassembly is still going to needed, sorry. But we will be looking for feedback on what specific APIs you would like more information on and will do what we can to accommodate information requests. The game also puts a lot more things in tuning than Sims 3 did so it's possible a number of mods you would traditionally do as script will be done via tuning and not code. But that said, we know script mods will be needed for things you want to do and I will be here to help with that.

As you have discussed, adding code instead of replacing Maxis code is going to be the best way to author mods. Although in the past week we realized that pure additions has a bug, and an upcoming game update will fix that. Can't give a firm time frame right now other than "soon", but it will be fixed quickly so you guys aren't forced to override our code and make potentially brittle mods.

Stuff for TS2 TS3 TS4 | Please do not PM me with technical questions we have Create forums for that.

In the kingdom of the blind, do as the Romans do.
Instructor
#3 Old 30th Aug 2014 at 8:54 AM
Quote:
Originally Posted by jtravers88
Greetings. Anyone know how to convert compiled python in pyo files to py files so they can be read in the python 3 shell?

I was just googling for this answer. The best I could find was this post: http://stackoverflow.com/questions/...y-from-pyo-file
I don't know why they didn't include some of the source python code for key modding sections. I noticed there is a "key" file in most of the folders with .pyo files. Maybe it's useful for decompiling?
Sesquipedalian Pisciform
staff: retired moderator
#4 Old 30th Aug 2014 at 9:31 AM
There is this on github: https://github.com/gstarnberger/uncompyle

Not sure how we can use it though.

More downloads by Leesester, BoilingOil and others at Leefish.nl | My Stuff at Leefish.nl | LeeFish RSS | Sims4 News Blog | TumblinLeefish
Inventor
#5 Old 30th Aug 2014 at 12:08 PM
I found http://sourceforge.net/projects/easypythondecompiler/ which uses https://github.com/zrax/pycdc for the Python 3 stuff and it states it supports Python 3.3.

There's also https://code.google.com/p/unpyc3/

I haven't checked either of them, though, so I don't know how/if they'd work on the game files.
Lab Assistant
Original Poster
#6 Old 30th Aug 2014 at 4:02 PM
Easy Python Decompiler gives the message it couldn't complete after only 20 lines of declarations.

unpyc3 only accepts py and pyc files. after changing it to accept pyo files, it errors on an import called marshal.

bad marshal data. unknown type code.
Test Subject
#7 Old 31st Aug 2014 at 1:04 AM Last edited by TheHologramMan : 31st Aug 2014 at 5:13 AM.
unpyc3.py failed to work for me with Python 3.3. I found pycdc independently and it crashes on most of the sims 4 python files I've tried usually with something like "Unsupported opcode: LOAD_BUILD_CLASS".

Edit: It does seem like most of the generate protobuf pyo files do decompile with pycdc sufficiently that I suspect someone can easily deconstruct .proto files. They included the FileDescriptor and which to my recollection has the complete compiled proto file and then the generated class as well if that does not help.

Edit 2: I fixed a local copy of unpyc3.py and it does a much better job of reversing the python 3.3 files for me.
Test Subject
#8 Old 31st Aug 2014 at 7:35 AM Last edited by RadioactiveMike : 31st Aug 2014 at 8:13 AM.
Quote:
Originally Posted by TheHologramMan
unpyc3.py failed to work for me with Python 3.3. I found pycdc independently and it crashes on most of the sims 4 python files I've tried usually with something like "Unsupported opcode: LOAD_BUILD_CLASS".

Edit: It does seem like most of the generate protobuf pyo files do decompile with pycdc sufficiently that I suspect someone can easily deconstruct .proto files. They included the FileDescriptor and which to my recollection has the complete compiled proto file and then the generated class as well if that does not help.

Edit 2: I fixed a local copy of unpyc3.py and it does a much better job of reversing the python 3.3 files for me.


Brilliant! Your fix is working like a charm. (Somewhat at least, still tracking through some stuff)
Lab Assistant
Original Poster
#9 Old 31st Aug 2014 at 5:45 PM
Quote:
Originally Posted by TheHologramMan
unpyc3.py failed to work for me with Python 3.3. I found pycdc independently and it crashes on most of the sims 4 python files I've tried usually with something like "Unsupported opcode: LOAD_BUILD_CLASS".

Edit: It does seem like most of the generate protobuf pyo files do decompile with pycdc sufficiently that I suspect someone can easily deconstruct .proto files. They included the FileDescriptor and which to my recollection has the complete compiled proto file and then the generated class as well if that does not help.

Edit 2: I fixed a local copy of unpyc3.py and it does a much better job of reversing the python 3.3 files for me.


I'll try it today. Thanks!
Test Subject
#10 Old 31st Aug 2014 at 7:02 PM
Quote:
Originally Posted by jtravers88
I'll try it today. Thanks!

No problem. I made some additional fixes to handle statements not quite working so grab the latest as I'll probably make some additional fixes but don't really plan on support any length of time. Any pull requests are welcome but I'm not the most active github user so don't take offense if is ignored for a while.

This tool is by no means perfect but the original author did a fairly good job in reproducing source so should work for us. Currently trying to get continue statements inside loops working reasonably well and there is quite a bit that needs more investigation but I'm down to about 73 files which still throw exception or assert. Not sure if the other files are any good but its a start.
Lab Assistant
Original Poster
#11 Old 31st Aug 2014 at 9:22 PM
Quote:
Originally Posted by TheHologramMan
No problem. I made some additional fixes to handle statements not quite working so grab the latest as I'll probably make some additional fixes but don't really plan on support any length of time. Any pull requests are welcome but I'm not the most active github user so don't take offense if is ignored for a while.

This tool is by no means perfect but the original author did a fairly good job in reproducing source so should work for us. Currently trying to get continue statements inside loops working reasonably well and there is quite a bit that needs more investigation but I'm down to about 73 files which still throw exception or assert. Not sure if the other files are any good but its a start.


Works well. I'm finding that the aspirations.pyo is causing a number of errors in the ImportStatement class though. I've got a couple fixed. Now 'alias' is not defined. I see no other references in the file so I'm not sure how to declare it.

I've added the following. Now aspirations.pyo decompiles. But you would know better than I if it is correct.

class ImportStatement(PyStatement):
--> alias = ""
--> precedence = 100
def __init__(self, name, level, fromlist):
self.name = name
--> self.alias = name
self.level = level
self.fromlist = fromlist
self.aslist = []
def wrap(self, condition=True):
if condition:
return "({})".format(self)
else:
return str(self)

I added the 3 lines with the arrows and the wrap definition which I copied from another class.
Test Subject
#12 Old 1st Sep 2014 at 2:27 AM Last edited by TheHologramMan : 1st Sep 2014 at 7:18 AM.
I incorporated those changes and made a bunch of other changes. Somewhat nervous about the while loop stuff since that is very hackish but it was already like that I just cannot tell if I broke anything easily. There are still a few error scenarios but most of the sim4 specific files now generate code though there are still obvious errors throughout but this is good enough for most coding modders to at least get started I think.

Edit: Something of note that really has my interest is the core\shared_commands\reloader_commands.py file which seems to be a hot reloader for scripts. Now that is sexy. If there is one single feature that justifies using python that is it. I basically could never get excited about sims 3 modding because debug cycle times were just so incredibly huge. Hot reload of scripts would just be so convenient that I have a hard time believing EA would allow such a thing.
Lab Assistant
Original Poster
#13 Old 1st Sep 2014 at 1:27 PM
The following code was added to the ImportStatement class after the init.

def wrap(self, condition=True):
if condition:
return "({})".format(self)
else:
return str(self)

I'll grab your latest and add this wrap fnc. Please add this to your version when time permits. And I'll have to research that hot loader concept. I'm new to python so...
Test Subject
#14 Old 1st Sep 2014 at 2:56 PM Last edited by TheHologramMan : 1st Sep 2014 at 10:59 PM.
I added the wrap statement to the base class already before your comment. Still cannot sort out the issues with the import statement generating incorrectly. The alias is supposed to be set as a result of the assignment in the bytecode after the import but that code is not triggered. Not that it would necessarily help in the current code generator as that is also not getting called at correct time.

Right now mostly function annotations that use the extended arg qualifier are breaking the most and those could be useful in understanding meta programming capabilities so I'll see if I can fix that but I'm having difficulty contriving examples that reproduce the problem as it does not happen at all in the base python libraries.

I'm not going to stess too hard over that right now as annoying as it is because the generated files should hopefully just be documentation but Intellij IDE would features would better with it. Any modifications of the core files will likely be done through some sort of monkey patching rather than straight editing.

The Hot Reloading features seem to require the __debug__ flag to be set globally in the scripts. That I presume will be set by the game when it configures the environment. Hopefully that is not too hard to enable in game. The way the python classes are loaded it does not look like its easy to enable after the initial load but nothing that couldn't be worked around.

Edit: Made what is likely final updates to the unpyc3.py script as the game approaches the unlocking time. I fixed the import x.y.z as w statement and the attributes issue though did not fix any of the advanced features like Function Signature Objects which I don't really understand too well. Also noticed there are some places where tuples should exist but don't and loops should exist but don't but they seem to be fairly rare.
Lab Assistant
#15 Old 1st Sep 2014 at 11:43 PM
Hello. So how can i make a scripting mod for the game? I decompyle the pyc file, then modify it and add it to a zip (with the py file inside) in mod folder?
Test Subject
#16 Old 2nd Sep 2014 at 12:27 AM
Quote:
Originally Posted by IceM
Hello. So how can i make a scripting mod for the game? I decompyle the pyc file, then modify it and add it to a zip (with the py file inside) in mod folder?


No, that won't work.

As an update of the current state, the files currently aren't being 100% accurately decompiled. Even when that is done we will need to be able to recompile the files(which may or may not be easy). Alternatively we could just learn how to patch current files And ignore the parts we couldn't decompile correctly.

Even with that done we will need to be able to load the files into the game. That could be its own brand of headache as we still don't know the purpose of the key file in the zip folder. (checksum? quasi-digital signature?)
Test Subject
#17 Old 2nd Sep 2014 at 12:31 AM
Quote:
Originally Posted by IceM
Hello. So how can i make a scripting mod for the game? I decompyle the pyc file, then modify it and add it to a zip (with the py file inside) in mod folder?


The decompiled pyo files are really more for documentation than anything needed for production. Basically we have a substantial chunk of the source code available to understand how things are done but shouldn't be needed to actually develop mods.

I'm not actually sure what we would normally mod with these but in Sims 3 NRaas and other modders extensively patched the core code to improve the game. It seems like a lot of custom interactions would be done via this means as well.

In practice I've yet to figure out how files will be loaded as the loader seems to expect either full source code or all zip files and nothing in between. At the moment looks like we might have to use some files which are not shipped such as lib.zip, debug.zip, tests.zip, build.zip to set up the script loading. By the way, the folder of interest is the Data\Simulation folder I think. I hope not and that I'm missing something or that they will patch it quickly after release. It looks like it was supposed to search a "Scripts" folder and load anything in there but that only happens when not loading from archive files and I'm assuming the shipping game loads from archive files.
Lab Assistant
#18 Old 2nd Sep 2014 at 1:34 AM
Information gathered from the game files and from SimGuruModSquad tells us exactly where to place the files at least. A folder called "Scripts" should be placed in the "Mods" folder and there we can place either loose py files or zip files containing pyo files. Compiling py files to pyo can be done with python as usual. What we need to know is what is needed to get the game to run the scripts and maybe how to make a key file if it is needed.
Lab Assistant
#19 Old 2nd Sep 2014 at 1:37 AM
Quote:
Originally Posted by Fogity
Information gathered from the game files and from SimGuruModSquad tells us exactly where to place the files at least. A folder called "Scripts" should be placed in the "Mods" folder and there we can place either loose py files or zip files containing pyo files. Compiling py files to pyo can be done with python as usual. What we need to know is what is needed to get the game to run the scripts and maybe how to make a key file if it is needed.


we probably need to find some event handler to call the scripts whenever something happens i dunno, Tbh i am kinda new to py, but i have coded in c++ and C# and Lua so it will be easy for me to adapt to it.

its probably some event handler and i hope we can modify or override maxis functions like ppl did in sims 3
Eminence Grise
#20 Old 2nd Sep 2014 at 2:42 AM
Just as a history refresher -- for anybody who wasn't doing script modding at TS3 release When TS3 came out, it was initially thought that if you wanted to do anything beyond XML tuning, you'd have to replace the core files. That was the basis on which Pescado and a few others started on core modding. The issue was, of course, that the system wasn't modular, since you can only replace the core with one thing at a time. If you used some of Pescado's changes, you had to use ALL of them, and couldn't use other mods which modified the same core file.

Then a couple of months in, two guys named Lemmy and Captain Binky turned up and started work on a story progression mod using a technique called "script injection", where they were able to use EA scripts to call their own. They left to go work on their own game (with some success! http://store.steampowered.com/app/1..._237_151__103_4 ), and passed their code on to Twallan. The Script Injection technique was picked up by many other script modders as well and the rest is history.

My point here is just that script modding may not be something that gets going tomorrow... although with the help of SimGuruModSquad around it should be faster than last time around

And, you can parlay your Sims modding experience into becoming a successful indie gamer!
Fat Obstreperous Jerk
#21 Old 2nd Sep 2014 at 5:14 PM
It's very unlikely you're going to get a pyo->py conversion that can be recompiled and work. In TS3, it was never practical to attempt to decompile the source completely, either. Any core changes had to be done by disassembly rather than decompilation, and edited in MSIL. The difference is the decompiling attempts to return the code back to a sourcecode format, disassembly simply attempts to extract the bytecode into an assembly format, which will not be anything like the original Python.

Grant me the serenity to accept the things I cannot change, the courage to change the things I cannot accept, and the wisdom to hide the bodies of those I had to kill because they pissed me off.
Lab Assistant
#22 Old 2nd Sep 2014 at 5:16 PM
None of this is making sense to me. I have downloaded python 3.4, and the little icons for the .pyo files have changed. Would somebody clarify how to open them?
Fat Obstreperous Jerk
#23 Old 2nd Sep 2014 at 5:18 PM
Quote:
Originally Posted by cwurts00
None of this is making sense to me. I have downloaded python 3.4, and the little icons for the .pyo files have changed. Would somebody clarify how to open them?
You can't. PYOs are compiled, and optimized, Python files. This means that much of the original structure has been destroyed and the file will likely never be able to be decompiled, modified, and then recompiled. Any attempts to edit major files will be limited to disassembly-only or even raw bytecode hacking.

Grant me the serenity to accept the things I cannot change, the courage to change the things I cannot accept, and the wisdom to hide the bodies of those I had to kill because they pissed me off.
Lab Assistant
#24 Old 2nd Sep 2014 at 6:30 PM Last edited by cwurts00 : 2nd Sep 2014 at 6:40 PM.
OMG I'm a genius! I just opened it in Notepad!

Edit: I opened the key in notepad, as well, but it was all in Chinese/Japanese script. With Google Translate, I came up with this:

Kanzhuiqinggui Bu  Rong ٹ⦐ᰒ 샞 ꐁ I 씄슻  toot obstruct ᑯ Hen mace Jin ꍻ 죩 䋰 Ȗ panic ሖ 䊬 نجم 멺 barrel Ting Ji ⃮ ๬ 쩟 You ᣓ 䦅 Tan 쮬 㟽 ⮀ 넮 ꉮ 렣  䧺 㧊 낔 ꟗ 묟 ꜟ off tendons 㓍 뺋 Mo Xiao Qin Yi  냣 Ⳋ omen 㷟 ᪒ფꙫ 䑅 feast 㱥 䯋 ꎑ☣ 뤫갚 كم embankment 㬜 ୎ 뙍 Wu Mi Group (1) Chan 뽥 㯃 턄괹 Fu Ⳉ 䞶 퇻 ䷄ Yang ᦴ ⋋⡃ Su ꇘ 텳 You Nagi 촺 툈쒢 ꛹ Yong-Liang Chiu ھࣟꖽ ૅ 졞  Er 㚉

It appears to have something to do with bowel movements.
Instructor
#25 Old 2nd Sep 2014 at 7:23 PM
Quote:
Originally Posted by Srikandi
Just as a history refresher -- for anybody who wasn't doing script modding at TS3 release When TS3 came out, it was initially thought that if you wanted to do anything beyond XML tuning, you'd have to replace the core files. That was the basis on which Pescado and a few others started on core modding. The issue was, of course, that the system wasn't modular, since you can only replace the core with one thing at a time. If you used some of Pescado's changes, you had to use ALL of them, and couldn't use other mods which modified the same core file.

Then a couple of months in, two guys named Lemmy and Captain Binky turned up and started work on a story progression mod using a technique called "script injection", where they were able to use EA scripts to call their own. They left to go work on their own game (with some success! http://store.steampowered.com/app/1..._237_151__103_4 ), and passed their code on to Twallan. The Script Injection technique was picked up by many other script modders as well and the rest is history.

My point here is just that script modding may not be something that gets going tomorrow... although with the help of SimGuruModSquad around it should be faster than last time around

And, you can parlay your Sims modding experience into becoming a successful indie gamer!

Wow I didn't know they did Project Zomboid. I love that game, so much freedom and exploration. A zombie survival sandbox, with multiplayer. I'd like to see Project Zomboid and a Sims game mashed together in a new hybrid :D
Page 1 of 3
Back to top