Modding executables

[Phasma Felis]

I've been looking at the various modded .EXE replacements for the first X-Com game, like the 2xSAI graphic enhancer and the ET mod. How exactly are these made? I'm interested in making some of my own modifications to the game--specifically, I'd like to add the soldier-loadout-memory feature from xcomutil to the 2xSAI patch--but I'm not sure where to begin.

 

For what it's worth, I'm a fairly experienced programmer, with professional experience in C# and college work in C++ and Java, so I do have some clue what I'm doing. It's reverse-engineering the existing executable and figuring out what does what that I'm clueless on.

[Bomb Bloke]

2xSAI graphic enchancer? Where's that?

 

I'm afraid folks who can reverse engineer are few and far between. It's not overly hard to learn, but it requires quite a bit of time and patience. If it was me, I'd just write a program that executes between the geoscape and tactical engines, which assigns weapons at the beginning of combat and notes down the final configuration at the end.

[Phasma Felis]

2xSAI graphic enchancer? Where's that?

Man. I was starting to wonder if anyone was reading this forum.

 

Now, where did I get that 2xSaI thing from...ah, here it is. This is another reason I want to make my own mods, there's too many people making excellent mods and then burying them in the Tech Support section of some obscure forum where all of a dozen people ever see them. If you're not familiar with it, 2xSaI is a fast image filter designed for gaming that doubles the horizontal and vertical resolution of an image in an intelligent, good-looking way.

 

Note that the patch there does a bunch of other things as well--read the rest of the thread for more info.

 

I'm afraid folks who can reverse engineer are few and far between. It's not overly hard to learn, but it requires quite a bit of time and patience. If it was me, I'd just write a program that executes between the geoscape and tactical engines, which assigns weapons at the beginning of combat and notes down the final configuration at the end.

That was my original idea, but doesn't the Windows version combine the Geoscape and Tactical executables?

 

I think XComUtil separates them again, but it's not compatible with mods like the 2xSaI patch...which is yet another reason I want to roll my own, so I can make features like that work together. Sigh. Maybe I'll try pestering the authors of the individual mods for help.

[Bomb Bloke]

Woah. I never thought I'd see the day someone made an edit like THAT to the EXE!

 

And it further surprises me to see how long ago it was done.

 

*BB is blown away*

 

Anyway, this has all got me feeling constructive, so I scripted up a quick program to copy the splitting effect of XcomUtil. Which means programs I release in the future will no longer need that to be installed.

 

I ran it over Mok's EXE, and hey presto, it worked! It seems his scaler code is tacked onto the end of the file, and Scott's edits don't seem to conflict. Note that f0dder's patch does conflict - The graphics get rather garbled if you try to run his loader on Mok's (already fixed) version. In fact, I'd use Mok's EXE all the way if my computer was just a little bit faster!

 

Attached to this post is a split version of the scaled EXE file. Also included in the archive is the program I used to split it. It's nothing complex, but since it was rather tedious to write (all those offsets!), it might save others some time.

 

I took a look through the XcomUtil batch file, and it seems that you can disable the use of f0dder's patch by deleting "patch.dll" from your game directory. So if you extract this archive, and delete that file, you should be set to go.

 

I've also sent a message off to Mok asking permission to add his mod to our files section. Here's hoping he's still around.

Mok__s_2xSaI_EXE___Split_.zip

[Zombie]

Heh BB, would it be possible to write up a program like that to recombine a split executable? I have been looking for something like that for a while. Even talked to Scott T Jones over the weekend about this but I'm not sure if he understood what my intentions were.

 

- Zombie

[Bomb Bloke]

I tried emailing Scott once. The mails bounced back. Have never managed to get in contact with the guy.

 

Recombining the EXE is an interesting proposition. In order to create the split, you simply create two copies of the original, and then edit them so that they work the way you want them to. That is to say, you don't actually split them, you just create two near identical copies which act like the original Dos executables (which were indeed two completely different files).

 

Hence recombining them is tricky... Either one of the "split" files can be converted back to it's original state without too much trouble. But merging them together? Hmm.

 

What I'm guessing is that you've edited the alien stats in a "split" EXE, and now want to turn in back to a single file version. Well, I'd recommend editing the single file to start with (as all the offsets in concern are in the same place), but I can see why you wouldn't have done that - The logger requires the split files.

 

That said, I could write a program that takes both "split" EXEs, compares them against the original file, then creates a new one keeping all changes it detects. Then I just patch this new file so that it is no longer "split", and that should do the job. I'll do that now.

[Phasma Felis]

I ran it over Mok's EXE, and hey presto, it worked! It seems his scaler code is tacked onto the end of the file, and Scott's edits don't seem to conflict.

Sweet! Never thought it'd be THAT easy. I take it that it's just a matter of making three copies of the original with just the relevant bits in each, and making them call eachother somehow? How is the info passed between Geoscape and Battlescape?

 

In fact, I'd use Mok's EXE all the way if my computer was just a little bit faster!

Heh, that's what I've been doing, along with some sound and music mods. Is it just the 2xSaI stuff that makes it run too slow for you? I think there's another version buried in that thread that does everything else but 2xSaI. You might try that one. In fact, if you do, could you split it too and post it here?

 

Well, I've got a rather ambitious mod project in mind, which you've just made much easier with your splitter--thanks very much! If I ever get it working, I'll be sure to post it here. First step is building that loadout tracker, which probably shouldn't be too difficult. I think I've seen a byte-by-byte breakdown of the save file format around here somewhere...

[Bomb Bloke]

It creates two copies, one that serves as the geoscape engine and one that works as the tactical engine.

 

This is how the game originally functioned before the CE version was released, two seperate programs that were called as needed via a batch file. They communicated by way of error codes (within the batch file), and by dumping files in the MISSDAT folder.

 

If you don't have an example of the batch, XcomUtil provides one similar to the original, as does my own mod.

 

The best place to start looking for game data info is the UFO-Wiki-pedia. Check this page specifically to see the format of the object data records, but read through that first link to get an understanding of which files you want to work with when, and how the two engines share game data.

 

Of course, XcomUtil's loadout feature should be fully compatible with the 2xSaI mod now that it's split, so unless you really want to make your own version, I'd recommend trying that out...

 

It is the 2xSaI scaler that slows my machine down. Given that that's the only feature I want from the mod, I'll put it aside in case I ever get a more capable computer.

 

I polished the splitter, and it can now be executed via a batch file. The merger is also complete, and both are now packed together in our files section.

[Phasma Felis]

Thanks for all the info. It'll be a big help.

 

Of course, XcomUtil's loadout feature should be fully compatible with the 2xSaI mod now that it's split, so unless you really want to make your own version, I'd recommend trying that out...

I do, actually. It bugs me that several of my favorite mods (a) aren't compatible with eachother and (b) require a fair bit of hacking to work, so I thought I'd see about combining them into one big patch with a user-friendly front end, both as a community service and an educational programming exercise.

 

Well, I may use XComUtil anyway, for the various other features it offers. We'll see. Still have a fair bit of work to do on everything else, though.

[Zombie]

I polished the splitter, and it can now be executed via a batch file. The merger is also complete, and both are now packed together in our files section.

This is awesome!

 

You are truly a god Bomb Bloke! It's simple to use, very effective and you can bet I'll put this to good use. Thanks! (x2)

 

- Zombie

[Karstedt]

I too would be interested in figuring out how some of the mods work. Particularly because I'd like to fix the proximity grenades in the ET mod, and also strip out the video stuff so I can use D3DWindower with it.

 

What/where is Mok's exe? I don't think I've seen that one yet.

 

Bomb Bloke, do you think you could add the ability to compare exe's and report (IE list what offsets are different) the changes? A decent hex editor would probably do that for me, but I don't have one.

[Phasma Felis]

What/where is Mok's exe? I don't think I've seen that one yet.

I linked to it up in the third post.

[Bomb Bloke]

To compare files, I use CompDAT. However, this isn't something I use often - I usually write a new tool from scratch to deal with whatever type of file I'm working with at the time.

 

I couldn't find any referrences to CompDAT online (can't for the life of me remember where I got it from), so it's now in our files section. Remember, the Windows calculator can convert hex values to decimal for you.

[Zombie]

I needed to use your merger/splitter programs today for some tests on a split CE executable. Ever try running either of them through DOS? You can't, as this error shows up: '_EXE' is not recognized as an internal or external command, operable program or batch file. DOS probably doesn't like the EXE part as that is a reserved command, so I ended up renaming both. "EXEMerge" and "EXESplit" work fine but I just used the normal Bomb Bloke nomenclature: "bbMerge" and "bbSplit". The re-merged executable name is fine, but it would be nice if it didn't contain an underscore for ease of typing.

 

Also (and this is a minor gripe), but I'd much rather have a split file named "Geoscape.exe" instead of "GeoScape.exe". It's not absolutely necessary, but it would stick to the same naming conventions used by the game. (In fact, there is nothing wrong with naming it "Geoscape.exe" as in the CE version those files don't even exist, so its not like it will be overwriting anything important). Something to think about I guess.

 

- Zombie

[Bomb Bloke]

I'm not sure that EXE is reserved, but I do know what the problem is. Remember that when you type DOS commands, every word counts as a seperate parameter. That is to say, _EXE Split.bat has a space in it, so if you type that in it thinks you're trying to run "_EXE" (which doesn't exist) with the parameter "Split.bat".

 

The solution is to type it with quotes, so that it's recognised as all part of the one file name, though renaming them of course works just as well.

 

When you say you're running it through DOS, do you mean the Windows Command Prompt? Windows 2000 and XP finally introduced tab completion - If you type in _EXE and hit tab, the rest of the file name will be filled in for you, complete with quotes if needed. Keep hitting tab if the wrong file name turns up, it'll scroll through the possibilities.

 

I'd assume DosBox has similar functionality (as the feature is nothing new, especially in the Linux world), but I doubt you're using that, as the Java VM most likely doesn't function there.

 

You can also use this when filling in parameters, like when you type "cd ". Dead handy, as my UFO folder is called "X-Com I - UFO - Enemy Unknown (Gold)" and I can't be bothered to change it.

 

If you're dealing with real DOS, however, it's quite a different kettle of fish - Long file names and spaces just aren't supported. As a general rule, you take the first six characters of the file name - excluding spaces - and add "~1" to the end. For example, "_EXE Split.bat" becomes "_exesp~1.bat". If there are multiple file names with the first six characters the same, the first will have the "~1" suffix, the second "~2", and so forth. If in doubt, the "dir" command will show you the DOS representations that are required.

 

If you don't like my camel text names, you wait until you see what my map editer has done to your save files! However, it's easy enough to "fix" in the case of the executables.

 

Right click "_EXE Merge.bat" and select "Edit" from the context menu. Add lines to the bottom which read:

 

del ""

ren "_ReMergedUFO.exe" ""

 

And then save the change.

 

Edit the "_EXE Split.bat" batch in the same way, but instead add the lines:

 

ren "geoscape.exe" "geoscape.exe"

ren "tactical.exe" "tactical.exe"

 

The case of file names is only really cosmetic, when you go to run the things you don't need to bother with your shift key. The original DOS version of UFO uses all upper case filenames, the CE version uses lower case ones, and I try to stick with camel text. It all works the same.

 

Anyway, when you run these batch files in future, they'll auto-rename the files after performing their main tasks.

[Zombie]

I normally run all of your stuff with simple double click on the icon and it works fine. In this case, I noticed an error and couldn't read it as it flashed by the screen so fast. So I went to the command prompt and tried to run it again to see what the problem was (turns out I forgot to copy the bb_tact folder over but this part wasn't determined until I renamed the split batch file). There's really no reason to run the programs through DosBox as it doesn't recognize java and runs fine under normal methods.

 

Quotes? Yeah I totally forgot about that one. (And that tab key is a great tip by the way). Thanks for the code, both work (and look) great now. So basically the batch files function somewhat like a command prompt? (I never realized this until I looked at the code just now).

 

- Zombie

[Bomb Bloke]

Yep, exactly. They're text files that contain a list of commands. You run the batch, you execute all the commands in sequence.

 

Although there's a lot you can do with them, a simple example is when you have a large amount of commands to run and don't want to sit there and do them one at a time... You type all the commands into a batch, run it, and then relax while all the processing is done without further user interaction.

 

Batch files do support some additional complexity, in that you can use a limited amount of flow control and pass parameters around.

[NKF]

A good example for a semi-complex batch file would be the original batch file that runs UFO itself. It features simple looping by way of a goto and labels. XComutil's replacement batch file has some minor conditional program flow controls in it as well.

 

I used to 'streamline' my UFO batch file by removing the bits that weren't necessary. Like black.exe. Doesn't really help much.

 

- NKF

[Zombie]

That's fascinating stuff guys! I should really look at the old Dos X-COM files some more.

 

- Zombie

[hatfarm]

I wrote CompDAT. I made it so I could check the saved files and what exactly changed when I did one thing. I'm glad people use it, it's pretty handy for stuff like that. I wrote it with VB6, if you'd like the code I can try to find it, though I had a harddrive failure and lost a lot of my X-COM stuff (including all my notes).

 

-Hat

[Zombie]

I could see this coming in handy as I have been comparing file differences with DOS commands which isn't the best.

 

Thanks hatfarm! I updated the file description to reflect you are the author.

 

- Zombie

[Bomb Bloke]

Gah! I completely forgot to deal with that, and I read this ages ago! Sorry!