Decompiling Black Mesa Maps

I had some significant reservations about posting on this topic publicly, but I’ve concluded that I will not be the only person ever to realize it could be done and more to the point I’m offering a workaround for something (namely, providing up-to-date sources for all all areas of the game) that the developers have stated they themselves plan to do but are understandably too busy to bother with at the moment. That said, please be polite in your decompiling adventures.

The information below is out-of-date. Check lower in the thread for a version of the decompiler made by Flavrans which does not need the workaround described below.

Getting a Decompiled Map [/size]

  • The actual maps which Black Mesa runs need to be extracted from the vpk files like any other game asset. This can be done by dragging bms_maps_dir.vpk over the vpk.exe program stored in the game’s ‘bin’ folder.

  • You will need ata4’s bspsrc utility. The current best (only?) place to get it is from the releases section of the author’s github page.

  • Unzip and run bspsrc.bat.

  • Leave the advanced compilation options as defaults.

  • Add as many maps as you want and decompile. The program will throw a number of “invalid vector” errors.

  • This will produce decompiled VMFs with some bad data in them that will cause Hammer to crash if opened directly.
    Making A Decompiled Map Functional[/size]

  • In order to make the VMFs you created up above work, you will need to open them in a text editor like Notepad and remove a block of bad data.

  • The data you are looking for are a bunch of spurious lighting origins lumped together near the bottom of the file. They take the form of blocks like this:
    [tt]entity[/tt]
    [tt]{[/tt][tt] “id” “”[/tt][tt] “classname” “info_lighting”[/tt][tt] “targetname” “sprp_lighting_”[/tt][tt] “origin” “”[/tt][tt]}[/tt]
    The coordinates in the origin are occasionally something absurd like “0 0 103161719131765890000000000”- it’s these Very Large Numbers that break Hammer when the map is loaded. However, all of these lighting origins will cause the props they are connected to to light improperly, and will likely be in a place that will cause map leaks, so there’s nothing to be gained by only deleting the weird ones.

  • There are typically several hundred of these entites (Gasworks has nearly two thousand!) but since they are all in a row and the decompiler never includes any “legitimate” lighting origins you can just find the first one, find the last one, select everything in between, and delete.

  • The map can now be opened in Hammer, but I find that the reconstructed optimization (areaportals, occluders, and hint brushes) almost universally have an extremely negative effect on both compile time and in-game performance, and may cause compile errors. So, I typically use the ‘auto’ visgroups to hide everything except for those three categories, select the entire map area, and delete all of it. This makes a map that is barely playable and will take a long time to compile, but it’s better than nothing and you can fill in at least basic optimization yourself.

  • Very rarely other elements may be decompiled in a way that causes errors; it’s typically possible to just delete them and make new ones that look the same.

  • If you’re looking to update one of the maps like the outdoor ones that only have had new static props added since the official VMF release, you are probably better off just using hammer’s “entity report” or visgroup functionality to delete every prop_static from the official map, select every prop_static from a decompiled map, and copy them over. This will let you use the new props with the original optimization.

so the vmf’s that were release some time ago in the mapsrc folder don’t get redownloaded if you had to do a clean install of the game ?

They most certainly do, the thing is they are not the most recent ones- they are from before the STU update.

The current version of bspsrc can’t properly read the static prop structure in Black Mesa’s BSP format. The version of the static prop structure in Black Mesa’s BSP format is identified as V11, which isn’t supported by bspsrc, and as a result, it falls back to reading the static prop structure as a V4 structure. This works for reading some of the static prop data, but fails from the point where the two structure versions are different and it will interpret the rest of the data incorrectly. That’s what’s causing the erroneous data like the spurious lighting origins and invalid numbers that cause Hammer to crash.

I’ve made some modifications to the source code of bspsrc to add support for Black Mesa’s V11 static prop structure, and I think it’s working perfectly now. Decompiled maps can be opened in Hammer without crashing it, and the lighting origins should be correct and other settings should be present in the static prop data. But I need to test it some more. There’s one piece of data in the static prop structure that I’ve ignored, and that’s the DiffuseModulation struct member. This corresponds to the “rendercolor” and “renderamt” settings in the .VMF. Hammer will apply the color to the static prop in the 3D view, but you can’t edit the settings in the prop’s properties, and it doesn’t appear to be rendered in-game.

If anyone wants to try/test my version of bspsrc, it’s available for download below along with the modified source code.

EDIT 2017-07-22:

Uploaded new version. Changes:

  • Fixed buggy handling of V10 structs. Cleaned up V11 struct code so that it now extends V10.
    Other notes:

BM’s V11 static prop struct also supports prop lightmaps settings, but they don’t appear to be handled by BM’s vbsp.exe, i.e., the flag that enables it and the Lightmap Resolution X and Y don’t get written to the .BSP.

EDIT 2017-12-31:

New version. Changes:

  • Added support for the newer V11 prop_static structure format found in the Black Mesa December 2017 Update while maintaining support for the older V11 structure found in surface-tension-update(cu3) and halloween-update(cu4), as well as the V10 structure found in the crossfire-update(cu2) (and likely in older releases). The new structure contains an additional (32-bit) integer (m_FlagsEx) which use is currently unknown, and this data will therefore not be processed by BSPSource.
  • Changed fallback mechanism for unknown static prop lumps so that it falls back to v4 only if the static prop lump version is < 10. If it’s >= 10 it will fall back to v10.
  • Added code to sort the list in the BSP format combobox alphabetically for easier selection.
  • Updated file pattern for Black Mesa in the app database to include ‘i’ maps, eg., bm_c3a2i, and added some new entities to improve auto detection of current and future Black Mesa map files.
  • Added support for displacement surface flags (No Physics Collision, No Hull Collision, No Ray Collision) based on a patch by jason-e.

Download[/size][/size][/size]
bspsrc_1.3.23_bmfix_r4.zip | Source code (zip)

Previous version:
bspsrc_1.3.23_bmfix_r3.zip | Source code (zip)
[/size]

Ummm for some reason bspsrc_1.3.23 won’t work, even the bmfix version. As soon as I hit the “decompile” button, the program crashes and leaves an error log. Anyone else getting this problem? :frowning:
Oddly enough, it worked just fine for me over the summer. Maybe I’m forgetting to do a step…

Does it crash with any .bsp, or just one specifically? Which version of the Java runtime is installed on your system? (Click the Windows Start Menu button and type “About Java” and then click it/press Enter to see the installed version number.) Perhaps an update broke it. The version I’m using is 8u151, but the latest is 9.0.1. If you are on 9.0.1, then you could try downgrading to 8u151 and see if that works (available here).

EDIT: I tried running bspsrc 1.3.23 (orig. and bmfix) under JRE-9.0.1, and didn’t experience any crashes (decompiling some stock BM maps). So, that’s probably not it.

Any .bsp file. I’ve installed the latest version of java. For some reason, About Java isn’t working right now (I’ll check it again later) but I think it’s 9.0.1. So yeah I’ll just try downgrading. Thanks!

FYI, the December Update (aka. the Xengine Update) broke BSPSource again, including my version which included a fix to handle Black Mesa’s static prop lump.

The format of the static prop lump has been changed again, thus causing the same problem as before. I’m looking into it and I’ll see if I can patch BSPSource again to handle the new format if/when I find some time.

I’ve built a new version of bspsrc-bmfix which adds support for the new V11 prop_static structure. See my first post in this thread for download links and changelog.

So, the static prop lump found in the December update map files is still identifying itself as version 11, like in cu3 and cu4, but has grown by four bytes (from 76 to 80 bytes) compared to the earlier version of the structure. The four additional bytes are used by a new struct member called m_FlagsEx (a 32-bit integer), which I have no idea what is being used for. In the few maps I’ve checked it’s just set to 0 and there’s nothing in the .fgd’s that might shed some light on what it’s supposed to be used for. Perhaps it’s something to do with the upcoming improved prop lighting feature (as mentioned in the December update announcement).

I’ve just made sure that BSPSource is able to detect the format and able to read the rest of the relevant data from the structure properly.

Yes, the decompilation would appear to finish successfully, but you would see the following messages in the errors & warnings box:

[warning] BspFileReader: Static prop struct size mismatch: expected 80, got 76 (using DStaticPropV11) [warning] BspFileReader: Falling back to static prop v4This means that static props have been read incorrectly, and will have some properties that will likely be wrong. Most if not all static props will have an invalid lighting origin defined, even if they are not supposed to have a lighting origin, and those that are supposed to, will have lighting origin coordinates filled with invalid numbers that might crash Hammer upon opening the .vmf. This has now been fixed in my new build of bspsrc-bmfix.

This doesn’t work for me. When i launch the .bat, an error pops up saying "Windows cannot find ‘javaw’ ".

You need to download and install the Java SE Runtime Environment:

https://www.oracle.com/technetwork/java/javase/downloads/jre10-downloads-4417026.html

The modified version of bspsrc I posted in this thread was built in a Java 8 environment. If there’s a problem with running bspsrc under JRE 10, uninstall the JRE 10 and try the JRE 8 (8u171) instead:

https://www.oracle.com/technetwork/java/javase/downloads/jre8-downloads-2133155.html

Solved, but new problem. After the unpacking (valve side), i cant find the files to de-compile (3rd party side). As soon as vpk finishes, it closes.

VPK should dump the unpacked folder into the same directory as the VPK. For instance, bms_maps_dir.vpk will make a folder called bms_maps_dir right next to itself.

I have released a new unofficial build of BSPSource built from the codebase in my forked repository on github.

You can download it from the releases section of my repository: https://github.com/flavrans/bspsrc/releases

This version is based on the latest commit by the original author from December 10, 2018. I have merged a number of patches by a github user named gitRihi into it, which add several fixes and improvements. Two of of note are: 1) a rewrite of the areaportal and occluder mapping code, 2) fix for blocklight tool textures, which would appear as nodraw textures in previous versions.

In addition to the aforementioned patches, there are also a few minor fixes of my own. One of the fixes ensures support for static prop lumps across all versions of Black Mesa. Another fix worth mentioning is a feature that fixes the alignment of tool textures. All tool textures will be realigned on their brush faces (by emulating Hammer’s face alignment method.) This fixes the issue with some tool textures appearing weirdly scaled and distorted in the decompiled map. The reason this happens is because the texture axis (UV) information is not preserved in the BSP, and what is stored in the BSP often results in invalid or nonsensical UVs. This is not surprising as texture alignment on tool brushes is completely irrelevant to the engine.

A full changelog is on the release page.

It is also worth noting that with this version of BSPSource, the decompiled bm_c2a5a and bm_c2a5h maps no longer causes Hammer to crash upon loading the VMF’s.

1 Like

Founded in 2004, Leakfree.org became one of the first online communities dedicated to Valve’s Source engine development. It is more famously known for the formation of Black Mesa: Source under the 'Leakfree Modification Team' handle in September 2004.