notnite / blog / ps3

One week modding the PlayStation 3

Alternatively, one week spent compiling RPCS3

posted 2024-08-13

I recently bought a PlayStation 3. In 2024. No, this isn’t a joke.

I was tired of emulating games on RPCS3 and figured that it was an interesting avenue for homebrew. Despite the saying that the PS3 “has no games”, there were a lot of interesting games I wanted to play, and I wanted to write code for it given the PS3 architecture is more similar to a supercomputer than a video game console.

After consulting the lovely ConsoleMods.org buying guide, I ended up picking up a 21XX series Slim model. I installed Evilnat 4.91 CFW on it and put an old 1TB SSD laying around in it (yes, it’s a bit of a waste since SATA 1, but it is an improvement no matter what).

It’s a lovely little console, and I’ve been enjoying it heavily (Skate 3 has consumed my life), and I wanted to talk about the week I’ve spent learning about it.

Initial setup & first impressions

Even though I am two months older than the PlayStation 3, I never actually owned one before. Growing up, I had a Wii (a quite early model!) and a Xbox 360 E (the final revision that looks sorta like a Xbox One), but I didn’t really play a lot of unique games on there. I played a lot of Mario Kart Wii and the LEGO games with my dad, Minecraft with my school friends, and Portal 2 as the first singleplayer game I beat; but I didn’t explore much of the library in the seventh generation of consoles.

I’ve homebrewed my Wii and played around with it before, but I didn’t find it that interesting. I haven’t played a game on my Xbox 360 in years (all Xbox modders reading this post: get to work on those hypervisor exploits), just turning it on to look around once or twice. (Also, my Xbox 360 once had its internal storage corrupt and I lost all my saves, so that doesn’t particularly motivate me…)

So, what is the console like? What is homebrewing it like? Well, I turned on the console for the first time and set it up. The previous owner updated it to the latest version and factory reset it before I purchased it, so I was already on 4.91 and ready to hack it. I followed the ConsoleMods.org guide, knowing that I verified the console’s serial number before purchasing it, and used PS3 Toolset to dump the NAND & patch the flash.

This step is very easy to do - you just have to go to a website on your PS3, watch in horror slash amazement as it somehow scans process memory of your PS3, and then hit a few buttons. Then, to install CFW, you just plug in a USB drive and install the firmware from it! A very handy feature of the PS3.

I also grabbed webMAN afterwards, and then it was pretty much done. Some thoughts on the actual PS3, though:

Baby’s first homebrew

There are two toolchains people use in the PS3 community for making homebrew:

The commits on these repositories already look a little sad. Commits are few and far between and usually just fixes that were PR’d. But let’s not lose hope!

ps3toolchain is designed for a Unix-like environment, but I use Windows on my desktop. I could’ve tried it through MinGW or used WSL, but my MinGW install is currently broken and my WSL doesn’t have internet (average day on my computer), so I just got out the laptop that is so laptop.

It failed building the first try because of my Python install lacking a library. One line removal from my Nix config and sudo dnf install python3 later (this is what I get for using Nix), it seemed to just work. I made a little shell script to set up the environment variables and launched CLion through a terminal.

Now, uh… this is where things go downhill. CLion was doing a bad job at understanding the toolchain and Makefile. I wanted to switch to a better build system, so I tried looking for CMake. There’s a PR for it, but the executables built with it don’t run.

Great. I tried making the CMake project myself, and ran into the same issue. Also tried Meson instead, and same issue. After a quick necropost I got “I did not, sadly” in response to asking if this was fixed. The PR remains closed as of today.

This is a rocky start, but these are quality of life features, not absolute dealbreakers. With a normal Makefile, the programs still compile, so I guess I’ll just write code with a partially broken IDE.

The samples built and ran on RPCS3, but I didn’t care enough to try them on console. Now, let’s start talking about what I really wanted to focus on: modding games!

What the hell is an EBOOT, anyway

It seems like most developers hang out on a forum called PSX-Place. I don’t want to make an account for it, but browsing around and looking at the dates of posts already gives me the feeling I’m walking into a place far before my time. It’s the same kind of feeling I got playing FINAL FANTASY XI - this place, and the golden ages of its existence, was far before I even knew of it.

Luckily, I didn’t need to swamp around in here for very long to figure out what I was doing. My good friend Emma works on a mod for Rock Band 3 called RB3Enhanced and she maintains a fork of it for experimental PS3 support. She gave me some pointers on the PS3:

PS3 game modding is primarily done through SPRX files. Developers will build a SPRX that gets loaded into the game process and then writes into executable memory to hook functions and do fun stuff.

Now, next question: how do you load a SPRX?

The long road of patching executables

I did what I usually do for answering these questions: opening a search engine. The first result was a post asking this question with zero replies. I found lots of posts from the 2010s, usually kids modding various shooter games and making tutorials (text or YouTube video). I saw a reference to a certain “SPRX ELF Builder” tool. (Let me make this very important that I do not endorse this tool and it’s a random closed source executable packed with UPX DO NOT USE THIS HOLY SHIT)

This tool works by taking in a decrypted EBOOT.BIN (usually named EBOOT.elf), doing some unknown closed-source magic on it, and then outputting a patched version. This patched version would load a .sprx from the PS3 hard drive using the path the user input into it. But how does it actually work? Well, time to reverse engineer it, I guess.

I ended up downloading it in a VM and opening it in IDA (sorry to Vector 35 for not using the Binary Ninja license I got a few months ago). As said above, it was packed with UPX, so I unpacked it and looked inside. It seemed to not have many interesting strings, and it was hard to follow the Qt signals for the buttons, so I tried just running it and see what it does.

I saw it create a file named “temporary_file.exe” and then freeze up. …What the hell? I opened that executable in IDA, and it turns out… this GUI is just a wrapper for someone else’s tool! This is part of the PS3 scene that annoys me - so many things taken and built on top one another, and along the way that credit and source gets lost.

I ran it from the command line, picking a random EBOOT from my PS3, and saw the string “Error: gcc build string not found”. …What???

I tried it on Rock Band 3, since Emma informed me it worked there. I ended up opening the original and patched versions in IDA and 010 Editor (a hex editor) to compare. It turns out that this program does two things:

Clever, but cursed. It relying on the GCC version string like that was not great, because a handful of games I tried it on didn’t work because there wasn’t a version string.

This tool obviously had flaws, and it seemed like it was my only option. This left me with only one idea: What if I made my own patcher?

Local idiot butchers explanation of ELF file format

So. Let’s learn ELF files. (I used Wikipedia as a reference for this, by the way).

The file starts with a header that describes the target machine, architecture, endianness, and more. We can assume most of these values won’t ever change on the PS3, as all games target the same platform. There is also the fields for the offset and size of the program and section headers, which we’ll get into later.

After this, there is the program header table. This contains a list of chunks in the executable file that describe how they should be loaded - location in the file, what address it should appear at in memory, size, etc.

Then, the sections. Sections are just chunks of arbitrary data with some flags - usually there is a section for code, a section for data, and whatever else the compiler feels like adding.

Finally, there is the section header table, which defines names and areas of sections (sorta like the program header). This information is pretty much useless at runtime, and is mainly used for linking, from my understanding.

If we wanted to add custom code to this, we’d need to make our own section. The biggest problem is the program header would need to increase in size to add the new entry, shifting everything in the process. However, I mentioned that the main header contains the offsets of the program/section headers. These can actually be anywhere in the file (unless the parser of the ELF file is bugged), as long as the offset in the main header points to it.

Shifting everything in the executable may be possible to implement, but we can simply move the program header to the end of the file, and fill where it was originally with zeroes to not move anything.

I tried using the Rust library object and the Python library LIEF, but the former didn’t want to relocate the program header and the latter made RPCS3 fail to read the executable.

So, guess what: we’re doing it all from scratch, baby!

Creating SPRXPatcher

I ended up making my own tool, called SPRXPatcher, in C#. It’s under the MIT License and has worked with every game I’ve tested it with. But first, I want to talk about how it works.

As explained above, I’m adding my own section, then moving the program header to the end of the file and filling where it originally was with zeroes. The code I use is some assembly that Emma wrote for me, and it takes the address of the SPRX path and does the required syscalls to load it into memory.

The goal is pretty simple: assemble that code into shellcode, paste that shellcode into the executable at a certain address, put the inputted SPRX path next to the code in that section, and then modify the entrypoint to jump to that shellcode.

The ELF header contains the address of the entrypoint, but on the PS3 it’s actually an entry in the table of contents. Reading that gives us the “real” address of the entrypoint, and we can write out some instructions to jump there:

public uint[] BuildJump(uint address) {
    var upper = (ushort) (address >> 16);
    var lower = (ushort) (address & 0xFFFF);
    return [
        0x3D600000 | (uint) upper, // lis r11, upper
        0x616B0000 | (uint) lower, // ori r11, r11, lower
        0x7D6903A6,                // mtctr r11
        0x4E800420                 // bctr
    ];
}

This takes more instructions than a bl but allows us to reach the code no matter what, since there is a maximum jump limit (more on that later). It just writes the address into the r11 register, moves r11 into the counter register, and then branches to the counter register.

I then save all the registers to the stack, load the SPRX, restore the registers, run the instructions I overwrote in the entrypoint, and jump back. This does have a pretty big issue that if the first few instructions in the entrypoint are a branch, it may fail, but I haven’t ran into yet so I have no reason to fix it right now.

After the shellcode, I directly stick the string of the SPRX path there, so it’s all in the same section. I spent quite a while getting the ELF file to properly parse and export, and then getting the shellcode to work, but after about two days of effort: it works! To my knowledge, this is the first SPRX injector-patcher-thingy in recent times, and also maybe the first open source one? Unsure about that last one.

I tested this with RB3Enhanced, both in emulator and on a real console, and it worked. I was impressed it worked on a real console, given that relocating the program header is a known hard spot for ELF file loading.

Unfortunately, after this long, I didn’t actually have much energy to mod any games, but I did write a small hooking library which I want to talk about.

Hooking functions with only 60 wasted instructions

In some games I tried modding, hooks would randomly fail and crash. Why? Well, it turns out that the executables of some games were so large that it would try and jump more than it could for its maximum size, and then underflow and jump backwards (into 0xFFFF____ space). I joked about solving this with a Backwards Long Jump - making a section in the executable allocated at 0xFFFF0000 and then doing a bctr into my SPRX - but it was too much effort.

I ended up taking a look at how RB3Enhanced does its hooking (and asked Emma), and found out a few things I need to know:

The TOC was solved in RB3E with making a “stub function” that handled setting it, but it did a weird hack by copying the game’s TOC to solve having to fix it. I wanted to do a little better, so I would need two stub functions for this scenario. I came up with a very verbose, but functional hooking system:

The link register is saved and restored so that we can make our way back to the caller. I store the link register right after the return instruction. This is not thread safe, and a very verbose way of doing this, but it seems to be stable for my use cases.

Hooks also sometimes need to call the original function, though. For this, I made a second stub:

I ended up needing to fix the branches here, as I said earlier in the section about SPRXPatcher. I did this by making four instructions for each instruction we overwrote, so sixteen instructions total. In the scenario I detected a branch, I would branch to that extra space and use the four instructions to bctr to the branch. This is a very wasteful approach, but again, it functions!

I ended up writing a few macros for this:

DETOUR(function_name, int /* return type */, void* self /* zero or more arguments */)
{
    LOG("Meow! :3");
    return function_name_orig(self);
}

void init_hooks() {
    HOOK(/* function addr */, function_name);
}

Pretty nice!

Closing thoughts

I enjoy the PS3 a lot more than I thought I would. My girlfriend has been spending a lot of time playing games on it, while I’ve been sitting here bashing my head against PowerPC assembly. What a cute dynamic.

I want to give a lot of thanks to the PS3 Developer Wiki for a lot of knowledge, PSL1GHT and ps3toolchain (even if it hurt me internally to set up), Emma for her infinite wisdom, and a lot of my other nerd friends (a special shoutout to husky and Aly for helping me with some shellcode). I also stumbled upon libpsutil, which seems neat, but their hooking library obviously won’t work for me. :P

Even though this scene is kind of fading away slowly as I speak, it makes me happy to know there are still a handful of passionate people around. People who want to build stuff for the curiosity of it, not because they see free games and go WOOOOOOO. While searching for information on how to make SPRX files, I found this Reddit post. To quote:

I don’t know if I will. My intention was theoretical research. I was going to write stuff to gain knowledge and improve documentation. But - I’m sad to say this - the scene is in a very sorry state. Everyone is happy with how it is, because copied games can be run and that seems to be enough for most. So I’m really losing motivation.

I cannot even find generic high-level information, such as what SPRX plugins are loaded by, and into what processes. The only good source of information is psdevwiki, but it mostly contains reverse engineered specifications, which are nice to have as a reference while you’re writing stuff, but are completely useless to get something up and running.

It is absurd that there is so much homebrew software around written by a lot of different people, but all resources regarding how it was developed are already lost, after a few years of scene inactivity. I even tried going down the road of contacting a few active sceners, but some of the main communities are either offline or are not accepting new registrations.

So much lost knowledge! It breaks my heart.

This comment resonated with me a lot. It was made four years ago, and yet it still stands true. I call out to those in software development communities: please, please, please, make sure your tools are available and archived. Make sure your methods and knowledge are preserved. It lowers the barrier to entry, and most importantly, it makes sure the second generation of your community can still press on.

A console as old as me still brought me a lot of enjoyment this day, thanks to the friends I know who were willing to help me on my adventure. I am thankful that, even though the scene rots away in silence, there is still opportunity to learn and create and explore.