SUBSIM Radio Room Forums - View Single Post

TheDarkWraith · 10-18-11, 06:34 PM

Maybe for people like Obelix you might find this interesting:

Like I posted above there are two new menu entries in Granny Viewer for the Undine: SelectionSets and Layers. I have no clue what their purpose is but regardless of purpose I need to be reading these from the file. So let's dig into this.....

Let's take a look at the first dozen or so pointers and see what's going on. Remember that part of dumb luck that I spoke about some posts back? It deals with how I decoded the pointers. I looked for patterns in them and noticed that every one whose second word (4 bytes) is a 0x6 usually denoted a section, change, or something different. So I set those as one type of pointer. Pointers that referenced embedded strings were type str and those pointers who in turn referenced those pointers were type str_reference. I also defined a type as pointer_data and pointer but I'd have to go back and look in my code to see what determines each. Point is I made a wild guess and it worked. Dumb luck to the extreme. You have the early versions of the source code that shows how I denoted what type of pointer they are. The really funny thing is that I had to include a hack to get some of the pointers to work correctly. And this hack worked also

Looking at the debug output of the first dozen or so pointers shows:

[1572] Reading file header data starting at 0x0
[1572] Magic string=)ÞlÀº¤S+%õ·¥öfâî 0x10
[1572] Start of file data file offset (from file beginning)=0x1C8 0x14 where the file list header starts
[1572]
[1572] Reading section 0 (arttoolinfo) data starting at 0x68
[1572] Unknown=0x0 0x6C always 0x0
[1572] Section file offset=0x468C 0x70 where this section starts in the file
[1572] Compressed size=61184 0x74 compressed size and uncompressed sizes should equal for SH5 GR2s
[1572] Uncompressed size=61184 0x78 compressed size and uncompressed sizes should equal for SH5 GR2s
[1572] Alignment=0x4 0x7C what the alignment boundary is in the file
[1572] Start of data fileoffset=0xBD9C 0x80 start of data
[1572] Start of data fileoffset2=0xBD9C 0x84 start of data
[1572] Pointers file offset=0x1C8 0x88 where the pointers for this section are located
[1572] Number of pointers=1467 0x8C
[1572] End of pointers file offset=0x468C 0x90
[1572] Unknown2=0x0 0x94
[1572] Current position in file is 0x94
[1572]
[1572] Reading section 1 (vertex_data) data starting at 0x94
[1572] Unknown=0x0 0x98 always 0x0
[1572] Section file offset=0x1358C 0x9C where this section starts in the file
[1572] Compressed size=1368640 0xA0 compressed size and uncompressed sizes should equal for SH5 GR2s
[1572] Uncompressed size=1368640 0xA4 compressed size and uncompressed sizes should equal for SH5 GR2s
[1572] Alignment=0x20 0xA8 what the alignment boundary is in the file
[1572] Start of data fileoffset=0x14E240 0xAC start of data
[1572] Start of data fileoffset2=0x14E240 0xB0 start of data
[1572] Pointers file offset=0x1358C 0xB4 where the pointers for this section are located
[1572] Number of pointers=0 0xB8
[1572] End of pointers file offset=0x1358C 0xBC
[1572] Unknown2=0x0 0xC0

(snip)

[1572] Current position in file is 0xC0
[1572]
[1572] File info offset at 0x8 (0x4694)
[1572] Number of textures=5 0xC (0x4698)
[1572] Number of materials=11 0x14 (0x46A0)
[1572] Number of skeletons=1 0x1C (0x46A8)
[1572] Number of vertex datas=20 0x24 (0x46B0)
[1572] Number of tritopologies=20 0x2C (0x46B8)
[1572] Number of meshes=20 0x34 (0x46C0)
[1572] Number of models=1 0x3C (0x46C8)
[1572] Number of trackgroups=0 0x44 (0x46D0)
[1572] Number of animations=0 0x4C (0x46D8)
[1572] Number of unknown=0 0x54 (0x46E0)
[1572] Current position in file is 0x46E4
[1572] Pointer 0 is type str with offsetinsectiondata of 0x5C (0x46E8) and unknown of 0x0 and offsetinsectionvalue of 0xC378 (0x10A04)
[1572] Pointer 0 updates the pointer to ArtToolInfo:FromArtTooFromName with pointer to embedded string for it

So what do we know? From file header the first section starts at 0x1C8 in the file. Section 0 tells us that the section file offset is at 0x468C and start of pointers is at 0x1C8. So pointers come before data in this file (first thing that tripped me up - all the other GR2 files I've read had data before pointers). There are 1467 pointers for this section and the end of them resides at offset 0x468C. End of pointers equals start of section offset - Check. Let's double check this: file header said first section starts at 0x1C8. Section 0 Pointers start at 0x1C8. Check. There are 1467 pointers that contain 3 words (4 bytes) so 1467 * 3 * 4 = 0x44C4. Start of pointers 0x1C8 + size of pointers 0x44C4 = 0x468C. Section 0 said section file offset is at 0x468C. Check. Section 0 says size of this section is 0xEF00. Section 1 says it starts at 0x1358C. Start of section 1 0x1358C - start of section 0 0x1C8 = 0x133C4. This difference 0x133C4 - size of pointers 0x44C4 = 0xEF00. Check. So with the basic checks out of the way we can proceed.

Ha!! Just had a revelation while typing this! I couldn't figure out what to do with the:
[1572] Start of data fileoffset=0xBD9C 0x80 start of data
[1572] Start of data fileoffset2=0xBD9C 0x84 start of data

Those numbers (0xBD9C) were voodoo to me. But it just hit me after telling you all about the above how to use it! Funny how things work like that. If I take the section start offset 0x468C and add 0xBD9C to it I should be at start of embedded strings (0x10428). Now why I don't have to add the start of section (0x1C8) to this is just a quirk of the GR2 file I guess.....let's take a look:

00010428 41 72 74 54 6F 6F 6C 49 6E 66 6F 00 46 72 6F 6D ArtToolInfo.From
00010438 41 72 74 54 6F 6F 6C 4E 61 6D 65 00 41 72 74 54 ArtToolName.ArtT
00010448 6F 6F 6C 4D 61 6A 6F 72 52 65 76 69 73 69 6F 6E oolMajorRevision
00010458 00 41 72 74 54 6F 6F 6C 4D 69 6E 6F 72 52 65 76 .ArtToolMinorRev
00010468 69 73 69 6F 6E 00 ision.

Yep, confirmed. Cool. Now have to change my code in the way I was retrieving these embedded strings......Thinking back the section start offset is a physical location in the file not an offset. I didn't know if the 0xBD9C was an offset or physical address. Now I know it's an offset. So everything makes sense now as to why I didn't have to add 0x1C8 to the result. Now I have to go and confirm this with other GR2 files....Confirmed. Works like a charm!

[1572] Pointer 0 is type str with offsetinsectiondata of 0x5C (0x46E8) and unknown of 0x0 and offsetinsectionvalue of 0xC378 (0x10A04)
[1572] Pointer 0 updates the pointer to ArtToolInfo:FromArtTooFromName with pointer to embedded string for it

So how do I know this? Let's look at the pointers in a hex editor. Open up the undine GR2 with hex editor. Now section 0 said pointers start at 0x1C8. So let's navigate to 0x1C8.

000001C8 5C 00 00 00 00 00 00 00 78 C3 00 00 \.......x...

the first word (0x5C000000) is what I call offsetinsectiondata. The second word (0x00000000) is what I call unknown. The third word (0x78C30000) is what I call offsetinsectionvalue. You have to remember that these are OFFSETS not physical locations. To get physical location you have to add the section's file offset to it. Those not familiar with reading the Intel hex format will think that the 78C30000 is just that. Incorrect. Intel hex format is little-endian style so you have to read from right to left. The value is actually 0x0000C378 if you wanted to type it into windows calc to get decimal value (50040d). Great, so what does this mean? 0x5C could mean anything right? Yes it could BUT let's set the hex editor at 0xC378 + section 0 section file offset (0x468C). You should have hex editor at 0x10A04. Here's what it shows:

00010A04 33 44 20 53 74 75 64 69 6F 20 4D 41 58 00 44 6F 3D Studio MAX.Do
00010A14 63 75 6D 65 6E 74 43 6F 6E 74 65 6E 74 73 00 48 cumentContents.H
00010A24 65 61 64 65 72 73 00 30 eaders.0

Those not familiar with C or assembly won't recognize the significance of the 00 bytes. Those 00 bytes are null termination characters. They signify the end of a string in C and assembly. If we put this all together then pointer 0 is type of string pointer (str) that points to the embedded string 3D Studio MAX. So some will say big deal, it points to a string. This IS a big deal. It tells us much more than that!!! Fire up RAD Game tools GrannyViewer and open up the undine GR2 file in it. Click the file list tab at the bottom. In the box for undine right click and select View In Detail. What do we see? We see some entries for ArtToolInfo, ExporterInfo, etc. Once again some will say so what? Let's take a deeper look at this. Look at ArtToolInfo once again. Notice the struct * to the left of ArtToolInfo? Those not familiar with C++ won't recognize the significance of the * symbol. This means a pointer to a struct. What this tells me is that if I click the (click to view sub-structure) of ArtToolInfo it's going to tell me the layout for that struct. Let's click it. Hot damn, what do you know

We have a perfect definition of this structure. So where does this pointer 0 play into this? Well remember it pointed to an embedded string. The first entry in this new screen in granny is a pointer to a character array (C++ syntax for a string) and what do you know it says "3D Studio MAX". Kinda strange how that is the same value as to what pointer 0 points at isn't it? I'd be willing to bet then that if we set the hex editor at pointer 0's offsetinsectiondata + section 0 section file offset we will find this struct in the file. Let's see. Set hex editor to 0x5C + section 0 section file offset 0x468C. This makes 0x46E8. Let's see what we find:

000046E8 00 00 00 00 0A 00 00 00 00 00 00 00 40 00 00 00 ............@...
000046F8 CD CC CC 3D 00 00 00 00 00 00 00 00 00 00 00 00 ...=............
00004708 00 00 80 3F 00 00 00 00 00 00 00 00 00 00 00 00 ...?............
00004718 00 00 80 3F 00 00 00 00 00 00 00 00 00 00 00 00 ...?............
00004728 00 00 80 BF ....

Jackpot! The first 4 bytes are 0x0 because they are where the pointer to the embedded string "3D Studio MAX" resides. In the programming world pointers are usually 32 bits (4 bytes - 8 bits to a byte). GrannyViewer says next entry should be an int32 (4 bytes) and it describes the ArtToolMajorRevision. Well 0x0A000000 is 10 decimal. GrannyViewer says ArtToolMajorRevision is 10. Voila. That is why I say the GrannyViewer is an SDK of sorts. It tells you everything you need to know to read the GR2 file along with the pointer data. It defines the structures (for the most part, sometimes you have to stumble you way 4-8 bytes here or there to find a item) and within those structures it tells you the data type for each structure member.

If you skip 0x28 bytes from the last 'entry' for ArtToolInfo you will be at ArtToolInfo:ExtendedData

ocumentContents:Headers in GrannyViewer. It's this skipping of 0x28 - 0x10 (these 0x10 are known) bytes that is new to Undine that I have to figure out how to read. These 0x18 bytes are the SectionSets (2) and Layers(3) entries in GrannyViewer ArtToolInfo:ExtendedData:

0000472C 00 00 00 00 00 00 00 00 00 00 00 00 02 00 00 00 ................
0000473C 00 00 00 00 03 00 00 00 00 00 00 00 00 00 00 00 ................
0000474C 97 00 00 00 00 00 00 00 ........

Now I have a feeling that there is only 0x10 new bytes defined because: 0x4 bytes number of SectionSets (02000000), 0x4 bytes for pointer to those SectionSets in file (00000000), 0x4 bytes for number of Layers (03000000), and 0x4 bytes for pointer to those Layers in file (00000000). Grand total is 0x10 new bytes....I'm guessing here but am usually correct. This also means an assumption I made about where ArtToolInfo:ExtendedData

ocumentContents resides in the file is incorrect and I have to re-evaluate this.

The reading of the file header and sections I couldn't figure out because GrannyViewer didn't 'show' anything about them. It wasn't until Privateer released his first template for 010 editor that they were revealed. That was the missing piece of the puzzle I needed to put everything together

More to come......

10-18-11, 06:34 PM	#113
TheDarkWraith Black Magic Join Date: Jun 2007 Posts: 11,962 Downloads: 147 Uploads: 5	Maybe for people like Obelix you might find this interesting: Like I posted above there are two new menu entries in Granny Viewer for the Undine: SelectionSets and Layers. I have no clue what their purpose is but regardless of purpose I need to be reading these from the file. So let's dig into this..... Let's take a look at the first dozen or so pointers and see what's going on. Remember that part of dumb luck that I spoke about some posts back? It deals with how I decoded the pointers. I looked for patterns in them and noticed that every one whose second word (4 bytes) is a 0x6 usually denoted a section, change, or something different. So I set those as one type of pointer. Pointers that referenced embedded strings were type str and those pointers who in turn referenced those pointers were type str_reference. I also defined a type as pointer_data and pointer but I'd have to go back and look in my code to see what determines each. Point is I made a wild guess and it worked. Dumb luck to the extreme. You have the early versions of the source code that shows how I denoted what type of pointer they are. The really funny thing is that I had to include a hack to get some of the pointers to work correctly. And this hack worked also Looking at the debug output of the first dozen or so pointers shows: [1572] Reading file header data starting at 0x0 [1572] Magic string=)ÞlÀº¤S+%õ·¥öfâî 0x10 [1572] Start of file data file offset (from file beginning)=0x1C8 0x14 where the file list header starts [1572] [1572] Reading section 0 (arttoolinfo) data starting at 0x68 [1572] Unknown=0x0 0x6C always 0x0 [1572] Section file offset=0x468C 0x70 where this section starts in the file [1572] Compressed size=61184 0x74 compressed size and uncompressed sizes should equal for SH5 GR2s [1572] Uncompressed size=61184 0x78 compressed size and uncompressed sizes should equal for SH5 GR2s [1572] Alignment=0x4 0x7C what the alignment boundary is in the file [1572] Start of data fileoffset=0xBD9C 0x80 start of data [1572] Start of data fileoffset2=0xBD9C 0x84 start of data [1572] Pointers file offset=0x1C8 0x88 where the pointers for this section are located [1572] Number of pointers=1467 0x8C [1572] End of pointers file offset=0x468C 0x90 [1572] Unknown2=0x0 0x94 [1572] Current position in file is 0x94 [1572] [1572] Reading section 1 (vertex_data) data starting at 0x94 [1572] Unknown=0x0 0x98 always 0x0 [1572] Section file offset=0x1358C 0x9C where this section starts in the file [1572] Compressed size=1368640 0xA0 compressed size and uncompressed sizes should equal for SH5 GR2s [1572] Uncompressed size=1368640 0xA4 compressed size and uncompressed sizes should equal for SH5 GR2s [1572] Alignment=0x20 0xA8 what the alignment boundary is in the file [1572] Start of data fileoffset=0x14E240 0xAC start of data [1572] Start of data fileoffset2=0x14E240 0xB0 start of data [1572] Pointers file offset=0x1358C 0xB4 where the pointers for this section are located [1572] Number of pointers=0 0xB8 [1572] End of pointers file offset=0x1358C 0xBC [1572] Unknown2=0x0 0xC0 (snip) [1572] Current position in file is 0xC0 [1572] [1572] File info offset at 0x8 (0x4694) [1572] Number of textures=5 0xC (0x4698) [1572] Number of materials=11 0x14 (0x46A0) [1572] Number of skeletons=1 0x1C (0x46A8) [1572] Number of vertex datas=20 0x24 (0x46B0) [1572] Number of tritopologies=20 0x2C (0x46B8) [1572] Number of meshes=20 0x34 (0x46C0) [1572] Number of models=1 0x3C (0x46C8) [1572] Number of trackgroups=0 0x44 (0x46D0) [1572] Number of animations=0 0x4C (0x46D8) [1572] Number of unknown=0 0x54 (0x46E0) [1572] Current position in file is 0x46E4 [1572] Pointer 0 is type str with offsetinsectiondata of 0x5C (0x46E8) and unknown of 0x0 and offsetinsectionvalue of 0xC378 (0x10A04) [1572] Pointer 0 updates the pointer to ArtToolInfo:FromArtTooFromName with pointer to embedded string for it So what do we know? From file header the first section starts at 0x1C8 in the file. Section 0 tells us that the section file offset is at 0x468C and start of pointers is at 0x1C8. So pointers come before data in this file (first thing that tripped me up - all the other GR2 files I've read had data before pointers). There are 1467 pointers for this section and the end of them resides at offset 0x468C. End of pointers equals start of section offset - Check. Let's double check this: file header said first section starts at 0x1C8. Section 0 Pointers start at 0x1C8. Check. There are 1467 pointers that contain 3 words (4 bytes) so 1467 * 3 * 4 = 0x44C4. Start of pointers 0x1C8 + size of pointers 0x44C4 = 0x468C. Section 0 said section file offset is at 0x468C. Check. Section 0 says size of this section is 0xEF00. Section 1 says it starts at 0x1358C. Start of section 1 0x1358C - start of section 0 0x1C8 = 0x133C4. This difference 0x133C4 - size of pointers 0x44C4 = 0xEF00. Check. So with the basic checks out of the way we can proceed. Ha!! Just had a revelation while typing this! I couldn't figure out what to do with the: [1572] Start of data fileoffset=0xBD9C 0x80 start of data [1572] Start of data fileoffset2=0xBD9C 0x84 start of data Those numbers (0xBD9C) were voodoo to me. But it just hit me after telling you all about the above how to use it! Funny how things work like that. If I take the section start offset 0x468C and add 0xBD9C to it I should be at start of embedded strings (0x10428). Now why I don't have to add the start of section (0x1C8) to this is just a quirk of the GR2 file I guess.....let's take a look: 00010428 41 72 74 54 6F 6F 6C 49 6E 66 6F 00 46 72 6F 6D ArtToolInfo.From 00010438 41 72 74 54 6F 6F 6C 4E 61 6D 65 00 41 72 74 54 ArtToolName.ArtT 00010448 6F 6F 6C 4D 61 6A 6F 72 52 65 76 69 73 69 6F 6E oolMajorRevision 00010458 00 41 72 74 54 6F 6F 6C 4D 69 6E 6F 72 52 65 76 .ArtToolMinorRev 00010468 69 73 69 6F 6E 00 ision. Yep, confirmed. Cool. Now have to change my code in the way I was retrieving these embedded strings......Thinking back the section start offset is a physical location in the file not an offset. I didn't know if the 0xBD9C was an offset or physical address. Now I know it's an offset. So everything makes sense now as to why I didn't have to add 0x1C8 to the result. Now I have to go and confirm this with other GR2 files....Confirmed. Works like a charm! [1572] Pointer 0 is type str with offsetinsectiondata of 0x5C (0x46E8) and unknown of 0x0 and offsetinsectionvalue of 0xC378 (0x10A04) [1572] Pointer 0 updates the pointer to ArtToolInfo:FromArtTooFromName with pointer to embedded string for it So how do I know this? Let's look at the pointers in a hex editor. Open up the undine GR2 with hex editor. Now section 0 said pointers start at 0x1C8. So let's navigate to 0x1C8. 000001C8 5C 00 00 00 00 00 00 00 78 C3 00 00 \.......x... the first word (0x5C000000) is what I call offsetinsectiondata. The second word (0x00000000) is what I call unknown. The third word (0x78C30000) is what I call offsetinsectionvalue. You have to remember that these are OFFSETS not physical locations. To get physical location you have to add the section's file offset to it. Those not familiar with reading the Intel hex format will think that the 78C30000 is just that. Incorrect. Intel hex format is little-endian style so you have to read from right to left. The value is actually 0x0000C378 if you wanted to type it into windows calc to get decimal value (50040d). Great, so what does this mean? 0x5C could mean anything right? Yes it could BUT let's set the hex editor at 0xC378 + section 0 section file offset (0x468C). You should have hex editor at 0x10A04. Here's what it shows: 00010A04 33 44 20 53 74 75 64 69 6F 20 4D 41 58 00 44 6F 3D Studio MAX.Do 00010A14 63 75 6D 65 6E 74 43 6F 6E 74 65 6E 74 73 00 48 cumentContents.H 00010A24 65 61 64 65 72 73 00 30 eaders.0 Those not familiar with C or assembly won't recognize the significance of the 00 bytes. Those 00 bytes are null termination characters. They signify the end of a string in C and assembly. If we put this all together then pointer 0 is type of string pointer (str) that points to the embedded string 3D Studio MAX. So some will say big deal, it points to a string. This IS a big deal. It tells us much more than that!!! Fire up RAD Game tools GrannyViewer and open up the undine GR2 file in it. Click the file list tab at the bottom. In the box for undine right click and select View In Detail. What do we see? We see some entries for ArtToolInfo, ExporterInfo, etc. Once again some will say so what? Let's take a deeper look at this. Look at ArtToolInfo once again. Notice the struct * to the left of ArtToolInfo? Those not familiar with C++ won't recognize the significance of the * symbol. This means a pointer to a struct. What this tells me is that if I click the (click to view sub-structure) of ArtToolInfo it's going to tell me the layout for that struct. Let's click it. Hot damn, what do you know We have a perfect definition of this structure. So where does this pointer 0 play into this? Well remember it pointed to an embedded string. The first entry in this new screen in granny is a pointer to a character array (C++ syntax for a string) and what do you know it says "3D Studio MAX". Kinda strange how that is the same value as to what pointer 0 points at isn't it? I'd be willing to bet then that if we set the hex editor at pointer 0's offsetinsectiondata + section 0 section file offset we will find this struct in the file. Let's see. Set hex editor to 0x5C + section 0 section file offset 0x468C. This makes 0x46E8. Let's see what we find: 000046E8 00 00 00 00 0A 00 00 00 00 00 00 00 40 00 00 00 ............@... 000046F8 CD CC CC 3D 00 00 00 00 00 00 00 00 00 00 00 00 ...=............ 00004708 00 00 80 3F 00 00 00 00 00 00 00 00 00 00 00 00 ...?............ 00004718 00 00 80 3F 00 00 00 00 00 00 00 00 00 00 00 00 ...?............ 00004728 00 00 80 BF .... Jackpot! The first 4 bytes are 0x0 because they are where the pointer to the embedded string "3D Studio MAX" resides. In the programming world pointers are usually 32 bits (4 bytes - 8 bits to a byte). GrannyViewer says next entry should be an int32 (4 bytes) and it describes the ArtToolMajorRevision. Well 0x0A000000 is 10 decimal. GrannyViewer says ArtToolMajorRevision is 10. Voila. That is why I say the GrannyViewer is an SDK of sorts. It tells you everything you need to know to read the GR2 file along with the pointer data. It defines the structures (for the most part, sometimes you have to stumble you way 4-8 bytes here or there to find a item) and within those structures it tells you the data type for each structure member. If you skip 0x28 bytes from the last 'entry' for ArtToolInfo you will be at ArtToolInfo:ExtendedDataocumentContents:Headers in GrannyViewer. It's this skipping of 0x28 - 0x10 (these 0x10 are known) bytes that is new to Undine that I have to figure out how to read. These 0x18 bytes are the SectionSets (2) and Layers(3) entries in GrannyViewer ArtToolInfo:ExtendedData: 0000472C 00 00 00 00 00 00 00 00 00 00 00 00 02 00 00 00 ................ 0000473C 00 00 00 00 03 00 00 00 00 00 00 00 00 00 00 00 ................ 0000474C 97 00 00 00 00 00 00 00 ........ Now I have a feeling that there is only 0x10 new bytes defined because: 0x4 bytes number of SectionSets (02000000), 0x4 bytes for pointer to those SectionSets in file (00000000), 0x4 bytes for number of Layers (03000000), and 0x4 bytes for pointer to those Layers in file (00000000). Grand total is 0x10 new bytes....I'm guessing here but am usually correct. This also means an assumption I made about where ArtToolInfo:ExtendedDataocumentContents resides in the file is incorrect and I have to re-evaluate this. The reading of the file header and sections I couldn't figure out because GrannyViewer didn't 'show' anything about them. It wasn't until Privateer released his first template for 010 editor that they were revealed. That was the missing piece of the puzzle I needed to put everything together More to come...... Last edited by TheDarkWraith; 10-18-11 at 11:40 PM.