[WIP] Almagest. Strict Import for SH5.

[Jeff-Groves]

It's not important to limit the output since the TDW importer does round it down.
Not even sure He actually checks Normals from the Main object file to the AO object file.

[gap]

Originally Posted by Jeff-Groves

OK. Let's talk about hard edges.
What is a hard edge? I'll start with how SH3 and SH4 handles them.
Say you make a Box. You NEED to have each side as a separate object, so to speak, to avoid the Games built in smoothing. That's been a well known for years.

Exactly!

Originally Posted by Jeff-Groves

How do you do that? In 3D Max I select the face for a wall and detach.
That produces a separate object as I suspect the Blender option to rid hard edges does.

That's the hard way, but depending on the model complexity this process can take a long time and it is not perfect. I give you an example. Let's say that we want to model a prism with a tear shaped base. Only one of its longitudinal edges is supposed to be hard, but there is no way you can have only one hard edge by detaching faces manually. This is where Blender Edge Split modifier comes in handy. It can detach even a single edge within an otherwise perfectly smooth surface, and it does that in one click

[gap]

Originally Posted by Jeff-Groves

It's not important to limit the output since the TDW importer does round it down.
Not even sure He actually checks Normals from the Main object file to the AO object file.

I can't say for sure, but since the only use of the AO object is providing a secondary UV map, it makes sense that only UV coordinates are read from file. All the other information should be ignored. It is also my understanding that TDW's GR2 importer assumes faces in the AO object being in the same order as in the main object. Thence the buggy AO mapping if face order is not exactly the same in the two objects.

I know 3dsMax supports multiple UV sets for the same object. This is the best way to ensure that, once exported, main and AO objects are identical in all the respects but UV coordinates, and I bet devs exploited this feature to ease their job.

[gap]

Yesterday I have had an insight into a couple of files, and I definitely see some problems.

This is a quick comparison of the two files:



Vertices (v)

• As expected, 3D vertex coordinates are identical in number, order and value between the two files.

Texture vertices (vt)

• For each file, there are more UV vertices than 3D vertices. This is expected because when you unwrap a closed surface for texturing it, you need to "cut" it in pieces so to make it as flat as possible and to avoid texture distortions. As a consequence texture vertices along the boundaries of "cropped" UV regions are duplicated.
  
  
• There is a mismatch in number of texture vertices between the two files, the main object having a bigger number of UV coordinates than its UV2 copy. This is probably not good for TDW's importer, and it might be the consequence of me cutting further some UV regions when creating the main UV map from the AO one.

Vertex normals (vn)

• Vertex normals are identical in number, order and value between the two files. Again, this was expected as the main and AO objects have the same 3D geometry.
  
  
• There are more vertex normals than vertex coordinates. That should be the consequence of my models having hard edges in place. Since the vertices along those edges are considered as part of two or three surfaces (each pointing in its own direction) rather than one smooth surface, there is a need for the shared vertices to have more than one normal.
  If I had removed hard edges before exporting the models, I am sure that the number of normals would have equated the number of vertices, and if I split the hard edges (as I did with the Blender exports), there would be as many vertices as the current number of normals.

Faces (f)

• There is an equal number number of face definitions between the two files. Again, this is the direct consequence of the two objects being identical in their 3D geometry, thus having the same number of faces.
  
  
• In a triangulated model each face definition is composed of three triplets of numbers, each triplet corresponding to one of the three vertices composing each face. Within each triplet, the first number should be the index to each of the 3D vertices connected to form a triangle; the central member of each triplet should be for texture vertices, and the third one for vertex normals.
  
  
  • Between my two models, there is a partial mismatch in face order relative to the first (v) and third (vn) indices of each triplet. This mismatch might be caused by the fact that the main model has three materials assigned - one for the main texture, one for the railing and one for the rigging - whereas the AO model got only one material. Ungrouping materials in the AO model might have scrambled face order relative to the main model, since faces of the latter model are ordered on file depending on the material they use.
    
    
  • There is a total mismatch of texture vertex indices, i.e. the second numbers in each triplet of face definitions. This mess might have been caused by the removal of "redundant" materials from the AO model (as above), but also by the cutting of some edges in the UV map of the main model (as explained before, there are more vt coordinates in the main model than there are in the AO model).

If Almagest could obviate to my mistakes, sorting out texture coordinates and faces in one click, that would be terrific. I realize that setting up the needed automated routines might be more complicated than it sounds though. Alternatively, I could try to manual edit my AO model in the hope to correct at least in part the problems above.
It that doesn't work, redoing the main UV mapping on the base of the AO model is still an option. That should not be a complicated task either. A little boring but not too complicated. For this particular task I think I would resort to Blender.
Blender's advantage is that, like Max, it supports multiple UV sets, and it got advanced and configurable shader setting. Very useful for previewing the combined effect of diffuse, specular, normal and AO maps. Unfortunately I am not as proficient in Blender as I am in Wings3D, but learning it better would be an investment for future projects.

[Jeff-Groves]

All that stuff gets sorted automagically.


The nice thing about how Granny stores the files is that We can INSERT stuff that does NOT load! Extract a 3D model from a GR2 file and look at it!
You'll see repeats of verts or textures that NEVER actually get called by any Face!

So I balance the counts to match the largest of the input.
In your case? V and VT will equal VN.
The faces will be sorted which moves the VT placement of textures in the obj file.
Thus one set of Faces that point to properly sorted information!
Now since We NEVER call the extra information? It never gets loaded!

I did try to point all this out to TDW several times. But I'm not sure he caught the idea so went with his way of doing the Loose Import.
That, as We all know, causes HUGE increases in file size. Not only that but EACH line needs to be loaded to display properly in Game.

[gap]

Originally Posted by Jeff-Groves

All that stuff gets sorted automagically.

I like your automagic lol

Originally Posted by Jeff-Groves

The nice thing about how Granny stores the files is that We can INSERT stuff that does NOT load! Extract a 3D model from a GR2 file and look at it!
You'll see repeats of verts or textures that NEVER actually get called by any Face!

So I balance the counts to match the largest of the input.

Originally Posted by Jeff-Groves

In your case? V and VT will equal VN.

Well, in the case we are discussing, vn's are not an actual problem.
As I said, as far as hard edges are removed or split before GR2 import, vn's count will match v's count

Originally Posted by Jeff-Groves

The faces will be sorted which moves the VT placement of textures in the obj file.
Thus one set of Faces that point to properly sorted information!

Doing this shouldn't require the addition of any extra information either. The good thing about my otherwise buggy AO model is that its vertices are in the same order as in the AO model, so we can easily track down which vt corresponds to which v in both files and, from there, which vt in one file corresponds to which vt in the other file. What is actually needed is sorting vt coordinates in the AO object so that they are in the same order as used by face definitions in the main model. All the other lines in the AO object can be identical to the ones of the main model.

I actually thought of a way for doing that in Excel, but it will work only for this specific set of files. Doing that spreadsheet generic would involve more work than it is probably worth. Nonetheless, while you work on a proper object sorter, I might give my stop-gap Excel tool a try... I am just curious to see my model in Blender with the two UV maps and connected textures applied, but I can't import the secondary UV map until the mismatch is fixed... (and I can't create a decent LOD model).

Originally Posted by Jeff-Groves

I did try to point all this out to TDW several times. But I'm not sure he caught the idea so went with his way of doing the Loose Import.
That, as We all know, causes HUGE increases in file size. Not only that but EACH line needs to be loaded to display properly in Game.

Well, we should admit that these concepts are not so easy to understand even if someone explains them to you one hundred times. Moreover one I can grasp the basics of someone else's discoveries, but putting them in practice would be another matter. TDW is brilliant at math, programming and debugging. He demonstrated his talents in many ways, but probably 3D geometry wasn't his piece of cake. In all honesty I can't blame him for that. I myself have a hard time trying to figure out the public domain and well known waveform format, le alone the granny one lol

P.S: a big limit to performing only strict import with TDW's tool, will be vertex count. I am not even sure I can fit my puffer in the template given by the stock fishing boat...

[Jeff-Groves]

I'm not criticizing TDW's work. It's BRILLIANT!
And the whole thing is kind of complicated if one does not have a full understanding of both the GR2 file storage and how obj files can be manipulated to match the GR2 format.

I'll give you a quick example you can do.
Cut say half the Faces in any object file and paste them to the top of the faces section of any obj file.

So if you have 500 faces? Cut 250 and paste them at the top so you still have all 500.

Then copy and paste say half your verts at the bottom of the verts section.
Open the file in any 3D program.

[Jeff-Groves]

If you read back through this thread?
i've already told you HOW to adjust file sizes!
Use the Loose import to grow or shrink things!

It's going to take some work to determine how big a file to import under the loose setting to shrink or grow but it does work!

[gap]

Originally Posted by Jeff-Groves

I'm not criticizing TDW's work. It's BRILLIANT!
And the whole thing is kind of complicated if one does not have a full understanding of both the GR2 file storage and how obj files can be manipulated to match the GR2 format.

I'll give you a quick example you can do.
Cut say half the Faces in any object file and paste them to the top of the faces section of any obj file.

So if you have 500 faces? Cut 250 and paste them at the top so you still have all 500.

Then copy and paste say half your verts at the bottom of the verts section.
Open the file in any 3D program.

To the best of my understanding of the obj format, face order within a mesh doesn't really matter (it matters in our case though), and the redundant vertices would be isolated vertices. The obj format doesn't support isolated vertices though. Depending on the program they might be ignored or the might show up, but they would be considered a topological error. Probably they wouldn't survive after a save/reload cycle.

Originally Posted by Jeff-Groves

If you read back through this thread?
i've already told you HOW to adjust file sizes!
Use the Loose import to grow or shrink things!

It's going to take some work to determine how big a file to import under the loose setting to shrink or grow but it does work!

Good to know! I had missd that piece of information, sorry

[Jeff-Groves]

Redundant information is ignored with most programs.
And you can see that by exporting from a GR2 and then saving again in a program if you have optimize on save.

Now how can you have 16150 faces but still have 25665 V, VT, VN?
That's because the GR2 files SKIP some garbage information!

It's simple math! 9515 lines are ignored.
And that is directly from the NAGC_C2Appalachian GR2!

I'm exploiting the way Granny works! I don't give a crap about Wings, Blender, nor 3D Max and how they may react once the files are run through Almagest.

[gap]

Originally Posted by Jeff-Groves

Redundant information is ignored with most programs.
And you can see that by exporting from a GR2 and then saving again in a program if you have optimize on save.

Now how can you have 16150 faces but still have 25665 V, VT, VN?
That's because the GR2 files SKIP some garbage information!

It's simple math! 9515 lines are ignored.
And that is directly from the NAGC_C2Appalachian GR2!

I'm exploiting the way Granny works! I don't give a crap about Wings, Blender, nor 3D Max and how they may react once the files are run through Almagest.

Okay, looking forward to your progress with the automagic, almighty Almagest

[Jeff-Groves]

Nice thing about the AO mapping is that it don't care about multiple textures for the main object file.
Now Almagest will determine WHICH faces structure it needs to use.
Does it use the AO faces structure or the Main Object faces structure?
It needs to scan the files, see if it's multi textured and choose the right way to sort things.

A single mapped main object? It may be better to use the AO face structure.
Multi mapped main object? Probably better to use that face structure THEN maybe fake some Faces.

It's kind of like the Street Game where you try to follow the balls under cups!
Everything has to be manipulated at just the right time, in the right order, to pass for magic!

[gap]

Originally Posted by Jeff-Groves

Nice thing about the AO mapping is that it don't care about multiple textures for the main object file.
Now Almagest will determine WHICH faces structure it needs to use.
Does it use the AO faces structure or the Main Object faces structure?
It needs to scan the files, see if it's multi textured and choose the right way to sort things.

A single mapped main object? It may be better to use the AO face structure.
Multi mapped main object? Probably better to use that face structure THEN maybe fake some Faces.

It's kind of like the Street Game where you try to follow the balls under cups!
Everything has to be manipulated at just the right time, in the right order, to pass for magic!

In the meanwhile I finished processing my AO model so that its texture vertices are sorted in the same way as in the main model.

First, I sorted AO faces so that they matched faces in the main obj file. For accomplishing this task I set my spreadsheet to compare vertex triplets in both files, as I knew that one triplet in one file corresponded with one triplet, and only one, in the other file.
Matching face order in both files gave me information on which UV2 vt corresponded with which vt in the main object (the second numbers in each triplet), so I updated vt's in the AO model according to the order used in the main model.

Now I need to copy the sorted AO vt's in main model's obj files, and I need to check in a 3D editor if I didn't make any mistake. If, as I hope, everything is in order I will send to you the new UV2 file so you can test your program on it too

[gap]

Originally Posted by gap

Now I need to copy the sorted AO vt's in main model's obj files, and I need to check in a 3D editor if I didn't make any mistake. If, as I hope, everything is in order I will send to you the new UV2 file so you can test your program on it too

It works!

There are only a few errors in the "new" UV2 map which I mostly know how to correct

[Jeff-Groves]

There you go!
You looked behind the curtain and saw the Magic.



Now you can hand edit the rest of the file to be proper for GR2 format.
Just copy and paste V or VT or VN's to MATCH which ever is highest in counts!
You could also just add with 0's. So V 0.0 0.0 0.0 will work. Your just filling space.
Since no FACE calls them? You'l never see them!



I do suggest using Blender files as they are way smaller but produce the same results! Plus you get the modifier you like.

Not sure WHY the Wings files have so many Normals compared to the Blender files?