Sometimes you have a very good idea of what's in a binary file, you might even have the source code for a reader and writer. This tutorial guides you through the steps to write the layout code to view HHA (Handmade Hero Asset) files with BEdit.
This is written for version 0.0.2 of the viewer. To see what version you have you can use bedit.exe -version, if the option isn't recognized the version you have is 0.0.1.
The file format we're investigating here is HHA version 0 and I'm using the handmade_file_formats.h source code from day 221. The asset files for version 0 can be found in handmade_hero_legacy_art.zip. You don't need access to the files to follow this tutorial but if you want to try it out yourself you do.
Follow the law
Some file formats are protected by license. Before viewing or attempting to reverse-engineer a file make sure you have the right to do so.
We will be using handmade_file_formats.h in this wiki entry, that file has a notice (C) Copyright 2015 by Molly Rocket, Inc. All Rights Reserved.
I have written permission by Molly Rocket, Inc to share code from handmade_file_formats.h in this tutorial.
Copy and paste
Let's start by creating hha.bet, this file will contain the definition of the file type written by us in the layout language.
If you have access to the source code of Handmade Hero, you can see that the file format is mostly specified in handmade_file_formats.h. The layout language of BEdit is very similar to C, so it's somewhat easy to just copy-paste what we need directly.
After copy-pasting the entire content of handmade_file_format.h to hha.bet we can run and see what errors we get.
> bedit.exe -layout hha.bet -data v0_hhas\intro_art.hha
Invalid tokens in layout file:
1: #if !defined(HANDMADE_FILE_FORMATS_H)
~
75: #define HHA_CODE(a, b, c, d) (((uint32)(a) << 0) | ((uint32)(b) << 8) | ((uint32)(c) << 16) | ((uint32)(d) << 24))
~
77: #pragma pack(push, 1)
~
81: #define HHA_MAGIC_VALUE HHA_CODE('h','h','a','f')
~
84: #define HHA_VERSION 0
~
180: #pragma pack(pop)
~
182: #define HANDMADE_FILE_FORMATS_H
~
183: #endif
~
The number on the left show the line number of the error.
Replace preprocessor directives
BEdit does not have a preprocessor. Replacing the #pragma pack(push, 1) and #pragma pack(pop) is very easy, since all structs are assumed to be tightly packed already we can just delete those lines. The include guard is also not needed so we delete that one too.
If your file format assumes C struct alignment you must manually add padding members when translating the types.
The macro HHA_CODE(a, b, c, d) takes 4 1-byte integers and produces a 4 byte integer. This functionality is inbuilt in BEdit as string literals.
Delete all #define and #pragma instances in the code and add
1 2 3 4 5 | enum { HHA_VERSION = 0, HHA_MAGIC_VALUE = "hhaf", }; |
to the top of the file. Let's run again and see what errors we get.
>bedit.exe -layout hha.bet -data v0_hhas\intro_art.hha
83: u32 MagicValue;
~~~
ERROR Expected type.
Replace scalar types
C scalar types are not supported in BEdit since they have an unknown size and byte order. An unsigned 32-bit integer displayed in decimal is defined using u(4), a signed 64-bit is defined s(8) and a 32-bit float is f(4).
We can at this point go through all instances of u32 and replace them by u(4), but easier (and less typing) is to add typedefs.
1 2 3 4 | typedef u(4) u32; typedef u(8) u64; typedef f(4) r32; typedef u(4) bitmap_id; // NOTE(Jens): We could also do `struct bitmap_id { u32 Value; };` |
Let's run again and see what we get.
>bedit.exe -layout hha.bet -data v0_hhas\intro_art.hha
177: union
~~~~~
ERROR Expected type.
Replace union
C types are static, BEdit types are not. A struct in BEdit may depend on the contents of the data file. As such, unions are not supported (this may change in the future).
For now, let's replace the union with "untyped" bytes, this is done by defining a scalar with raw.
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 | struct hha_asset { u64 DataOffset; u32 FirstTagIndex; u32 OnePastLastTagIndex; /* TODO(Jens): We'll soon see how to get this behavior back. union { hha_bitmap Bitmap; // sizeof(hha_bitmap) == 4*4 hha_sound Sound; // sizeof(hha_sound) == 3*4 hha_font Font; // sizeof(hha_font) == 5*4 }; // sizeof union is 20 bytes */ raw(20) DataHeader; }; |
Let's run again.
>bedit.exe -layout hha.bet -data v0_hhas\intro_art.hha WARNING No members exported, this can be due to missing 'layout' or because type had no members.
Specify the layout
Currently we have defined a bunch of types but we haven't told BEdit which type is the one that is the file. We do this with the layout keyword. The file starts with hha_header so in the bottom of the file, add
1 | layout hha_header; |
and run again.
>bedit.exe -layout hha.bet -data v0_hhas\intro_art.hha hha_header 0h MagicValue 1 717 659 752 04h Version 0 08h TagCount 109 0Ch AssetTypeCount 21 10h AssetCount 55 14h Tags 44 1Ch AssetTypes 916 24h Assets 1 168
The left column indicates the location in the file where the member is, middle is member name and right side is value.
It's not very helpful to see data like MagicValue in decimal form so let's change that.
Tweak scalar display
BEdit types in general have scalar-type, size and radix specifiers. You can also use string for ASCII strings. To see MagicValue as a 4-byte string change it to string(4) MagicValue;, if you want to see it as hexadecimal (in byte order) use raw(4) MagicValue;. Tags, AssetTypes and Assets are locations in the file, since addresses are displayed on left side as hexadecimal let's change the header to
1 2 3 4 5 6 7 8 9 10 11 12 13 14 | struct hha_header { string(4) MagicValue; u32 Version; u32 TagCount; u32 AssetTypeCount; u32 AssetCount; u(8, hex) Tags; // hha_tag[TagCount] u(8, hex) AssetTypes; // hha_asset_type[AssetTypeCount] u(8, hex) Assets; // hha_asset[AssetCount] }; |
and check the output
>bedit.exe -layout hha.bet -data v0_hhas\intro_art.hha hha_header 0h MagicValue "hhaf" 04h Version 0 08h TagCount 109 0Ch AssetTypeCount 21 10h AssetCount 55 14h Tags 2Ch 1Ch AssetTypes 3 94h 24h Assets 4 90h
We can confirm that the MagicValue and Version is what we expected, but this visual confirmation can only take us so far.
Add asserts
BEdit evaluates the types when it has the data, this enables us to write things like assertions.
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 | struct hha_header { string(4) MagicValue; assert(MagicValue == HHA_MAGIC_VALUE); u32 Version; assert(Version == HHA_VERSION); u32 TagCount; u32 AssetTypeCount; u32 AssetCount; u(8, hex) Tags; // hha_tag[TagCount] u(8, hex) AssetTypes; // hha_asset_type[AssetTypeCount] u(8, hex) Assets; // hha_asset[AssetCount] }; |
If we run it again we will get the same result since the assertion is not being triggered, but now if we try to load something from v2_hhas folder we'll get
>bedit.exe -layout hha.bet -data v2_hhas\intro_art_v2.hha
hha_header
0h MagicValue "hhaf"
04h Version 2
92: assert(Version == HHA_VERSION);
~~~~~~
Assertion triggered!
Specify entire file
At this point we see the header, but how about the rest of the file? Members in BEdit structs don't have to be next to each other, you can specify the absolute address of them. We can modify the hha_header to include members but I personally prefer to create a new type, let's call it hha_file
1 2 3 4 5 6 7 8 9 10 | struct hha_file { hha_header Header; @(Header.Tags) hha_tag Tags[Header.TagCount]; @(Header.AssetTypes) hha_asset_type AssetTypes[Header.AssetTypeCount]; @(Header.Assets) hha_asset Assets[Header.AssetCount]; }; layout hha_file; // Remember to remove `layout hha_header` |
The address specifier @(...) specifies the absolute address of the member in the file. If a member does not have an address specifier it starts where the previous member ended.
Now we see (I have removed some entries to make it less verbose)
>bedit.exe -layout hha.bet -data v0_hhas\intro_art.hha
hha_file
hha_header (.Header)
0h MagicValue "hhaf"
04h Version 0
08h TagCount 109
0Ch AssetTypeCount 21
10h AssetCount 55
14h Tags 2Ch
1Ch AssetTypes 3 94h
24h Assets 4 90h
hha_tag (.Tags[0])
2Ch ID 0
30h Value 0.000000
hha_tag (.Tags[1])
34h ID 5
38h Value 1.000000
...
hha_asset_type (.AssetTypes[0])
03 94h TypeID 0
03 98h FirstAssetIndex 0
03 9Ch OnePastLastAssetIndex 0
hha_asset_type (.AssetTypes[1])
03 A0h TypeID 0
03 A4h FirstAssetIndex 0
03 A8h OnePastLastAssetIndex 0
... (2nd to last entry of AssetTypes has non-zero data)
hha_asset (.Assets[0])
04 90h DataOffset 0
04 98h FirstTagIndex 0
04 9Ch OnePastLastTagIndex 0
04 A0h DataHeader 0x0000000000000000000000000000000000000000
hha_asset (.Assets[1])
04 B4h DataOffset 3 148
04 BCh FirstTagIndex 1
04 C0h OnePastLastTagIndex 3
04 C4h DataHeader 0x29090000800400000000003F0000003F00000000
...
If you are having trouble seeing the entire output you can pipe it to file or go to console settings and increase the buffer size.
One thing to notice is Tags.ID is suppose to be an enum, same for AssetTypes.TypeID. In C you can't specify the size of an enum, hence Casey specified them as fixed-width integer types instead. BEdit does support sized enums so let's replace those member types with name_of_enum(size_in_bytes). While we're at it, let's also change hha_asset.DataOffset to be hexadecimal.
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 | struct hha_tag { asset_tag_id(4) ID; r32 Value; }; struct hha_asset_type { asset_type_id(4) TypeID; u32 FirstAssetIndex; u32 OnePastLastAssetIndex; }; ... struct hha_asset { u(8, hex) DataOffset; u32 FirstTagIndex; u32 OnePastLastTagIndex; /* TODO(Jens): We'll soon see how to get this behavior back. union { hha_bitmap Bitmap; // sizeof(hha_bitmap) == 4*4 hha_sound Sound; // sizeof(hha_sound) == 3*4 hha_font Font; // sizeof(hha_font) == 5*4 }; // sizeof union is 20 bytes */ raw(20) DataHeader; }; |
Output now looks like
>bedit.exe -layout hha.bet -data v0_hhas\intro_art.hha
hha_file
hha_header (.Header)
0h MagicValue "hhaf"
04h Version 0
08h TagCount 109
0Ch AssetTypeCount 21
10h AssetCount 55
14h Tags 2Ch
1Ch AssetTypes 3 94h
24h Assets 4 90h
hha_tag (.Tags[0])
2Ch ID Tag_Smoothness
30h Value 0.000000
...
hha_asset_type (.AssetTypes[0])
03 94h TypeID Asset_None
03 98h FirstAssetIndex 0
03 9Ch OnePastLastAssetIndex 0
...
hha_asset (.Assets[0])
04 90h DataOffset 0h
04 98h FirstTagIndex 0
04 9Ch OnePastLastTagIndex 0
04 A0h DataHeader 0x0000000000000000000000000000000000000000
hha_asset (.Assets[1])
04 B4h DataOffset C 4Ch
04 BCh FirstTagIndex 1
04 C0h OnePastLastTagIndex 3
04 C4h DataHeader 0x29090000800400000000003F0000003F00000000
...
Conclusion
Wait what? It ends now? But we're just getting started!
We have just gotten started and we'll continue in the next part! But at this point you can already specify a lot of different file formats, download the command line reader if you haven't already and play around with it.
Key take-aways from this wiki entry is, when you have a file format already in C:
- Follow the law
- Copy and paste structs and enums to bet-file
- Replace preprocessor directives
- Replace scalar types
- Replace any unions (more on this next part)
- Specify the layout
- Tweak scalar display
- Add asserts (can't hurt at least)
- Specify the entire file
In the next part we'll figure out how to deal with that union, and we'll see how dynamic BEdit layout language actual is.
Edited by Jens
on