[WIP] Almagest. Strict Import for SH5.

[Mister_M]

Originally Posted by gap

Yesterday I have had an insight into a couple of files, and I definitely see some problems.

This is a quick comparison of the two files:



Vertices (v)

• As expected, 3D vertex coordinates are identical in number, order and value between the two files.

Texture vertices (vt)

• For each file, there are more UV vertices than 3D vertices. This is expected because when you unwrap a closed surface for texturing it, you need to "cut" it in pieces so to make it as flat as possible and to avoid texture distortions. As a consequence texture vertices along the boundaries of "cropped" UV regions are duplicated.
  
  
• There is a mismatch in number of texture vertices between the two files, the main object having a bigger number of UV coordinates than its UV2 copy. This is probably not good for TDW's importer, and it might be the consequence of me cutting further some UV regions when creating the main UV map from the AO one.

Vertex normals (vn)

• Vertex normals are identical in number, order and value between the two files. Again, this was expected as the main and AO objects have the same 3D geometry.
  
  
• There are more vertex normals than vertex coordinates. That should be the consequence of my models having hard edges in place. Since the vertices along those edges are considered as part of two or three surfaces (each pointing in its own direction) rather than one smooth surface, there is a need for the shared vertices to have more than one normal.
  If I had removed hard edges before exporting the models, I am sure that the number of normals would have equated the number of vertices, and if I split the hard edges (as I did with the Blender exports), there would be as many vertices as the current number of normals.

Faces (f)

• There is an equal number number of face definitions between the two files. Again, this is the direct consequence of the two objects being identical in their 3D geometry, thus having the same number of faces.
  
  
• In a triangulated model each face definition is composed of three triplets of numbers, each triplet corresponding to one of the three vertices composing each face. Within each triplet, the first number should be the index to each of the 3D vertices connected to form a triangle; the central member of each triplet should be for texture vertices, and the third one for vertex normals.
  
  
  • Between my two models, there is a partial mismatch in face order relative to the first (v) and third (vn) indices of each triplet. This mismatch might be caused by the fact that the main model has three materials assigned - one for the main texture, one for the railing and one for the rigging - whereas the AO model got only one material. Ungrouping materials in the AO model might have scrambled face order relative to the main model, since faces of the latter model are ordered on file depending on the material they use.
    
    
  • There is a total mismatch of texture vertex indices, i.e. the second numbers in each triplet of face definitions. This mess might have been caused by the removal of "redundant" materials from the AO model (as above), but also by the cutting of some edges in the UV map of the main model (as explained before, there are more vt coordinates in the main model than there are in the AO model).

If Almagest could obviate to my mistakes, sorting out texture coordinates and faces in one click, that would be terrific. I realize that setting up the needed automated routines might be more complicated than it sounds though. Alternatively, I could try to manual edit my AO model in the hope to correct at least in part the problems above.
It that doesn't work, redoing the main UV mapping on the base of the AO model is still an option. That should not be a complicated task either. A little boring but not too complicated. For this particular task I think I would resort to Blender.
Blender's advantage is that, like Max, it supports multiple UV sets, and it got advanced and configurable shader setting. Very useful for previewing the combined effect of diffuse, specular, normal and AO maps. Unfortunately I am not as proficient in Blender as I am in Wings3D, but learning it better would be an investment for future projects.

Originally Posted by gap

What I can tell you for sure is that if the main and the UV2 model are identical in all the respects except vt coordinate values, there is a 100% probability of success. With this idea in mind I have created an Excel spreadsheet which sorts vt coordinates for us. It is a rather crude tool, it requires a lot of manual data filtering, cutting and pasting; any mistake will cause wrong results, but if used correctly it does its job just fine.

If you are interested into testing it, I can upload it somewhere and post its link in the Almagest thread where it belongs more than here.

Couldn't be the problem coming from the fact that sub-objects (composing the final exported "diffuse object") are combined in a different order than in the "AO object" ?

If your excel sheet can solve the problem, let's try it. Just explain to me how this works.

[Jeff-Groves]

I know from reading the other thread that a question was raised about how the information on mapping is stored in a dat file.
A dat with only 1 map channel is stored in a simple block of data.
When We add say an AO channel? A TMAP section is added to the dat.
I'll have to refresh my memory on how the TMAPs actually work since it's been so long since it had to be done by hex editing. (Pre 2007!)
BUT! Only faces and texture coords are saved to the TMAP sections if I am recalling correctly.

I actually still have a program from 'back in the day' that exports and imports the different TMAP channels. So one could change or add just a TMAP and not have to re-import the complete obj like S3D does.

[Jeff-Groves]

One thing I don't do is try to 'Guess' what went wrong.
Send me the files and I'll look at them.
I've probably forgotten more then most people know.
Not trying to be a smarmy butt head but I have been around the block more then once.

[Mister_M]

Originally Posted by Jeff-Groves

One thing I don't do is try to 'Guess' what went wrong.
Send me the files and I'll look at them.

Here are .obj files which don't render correctly in game with the AO map : https://www.mediafire.com/file/ut1ss...O_map.zip/file

We are trying to understand why...

[Jeff-Groves]

Well that was easy.
I compared the main obj and the AO obj in 010.
Your vertices do not match between the Main Hull and AO Hull.
(I suspected that but as I said. I'll not play a guessing game)
So anything else is going to be All screwed up on import!

What S3D does when importing the Main diffuse object and the AO object is this!

IF number counts are the same for the vertices? It ignores what those are in the AO object and only imports the faces and texture coords!

So basically S3D IS importing garbage you created.
That is the flaw AND the opening in S3D that I am working to fix!

[gap]

Originally Posted by Jeff-Groves

Well that was easy.
I compared the main obj and the AO obj in 010.
Your vertices do not match between the Main Hull and AO Hull.

If you are talking about my puffer model vertices in the main model do match vertices in the AO model, confirmed by WimMerge comparison and by the fact that my spreadsheet (which supposes an exact match in vertex order) generates a correct output

I think you meant texture vertices. Didn't you?

Originally Posted by Jeff-Groves

IF number counts are the same for the vertices? It ignores what those are in the AO object and only imports the faces and texture coords!

So basically S3D IS importing garbage you created.
That is the flaw AND the opening in S3D that I am working to fix!

in an obj file 3D faces are denoted by the vertices composing them. So in a triangulated model you need three numbers to denote a face, e.g. 3 31 15 (this is: vertices # 3, 31 and 15 in the list of vertices are connected to compose a face).

This is for non-UV-mapped objects. For UV-mapped objects you must add three more numbers. Following the previous example: 3/10 31/7 15/25.
This line tells us that vertices #3, #31 and #15 in the vertex list are connected to compose a face in the 3D space, and that vertices #10, #7 and #25 in the texture vertex list compose a face in the UV space, where vt #10 corresponds to v #3, vt #7 to v #31 and vt #25 to v #15.

The errors I and Mister_M have been experiencing lead me to think - but this is only my guessing - that face definitions are written in dat/gr2 files only once, that secondary texture coordinates are stored in the same order as main UV coordinates, and that neither S3d nor GR2E read all the information stored in secondary obj file's face definitions in order to sort texture coordinates appropriately.

[gap]

Originally Posted by Mister_M

Hull1.obj :
2018 f
868 vt

Hull1-uv2.obj :
2018 f
2729 vt

It already seems problematic...

Oh my

I was expecting some unbalance in vt count between the two files, but such a huge unbalance was really unexpected.

Did you mess with secondary UV map? I mean: cutting edges which were originally joined or vice-versa?

And since you are at it, can you check that vertices (v lines) are sorted in the same order in the UV2 obj file as in the main obj file? That's a quick comparison in WinMerge

[gap]

Originally Posted by Mister_M

Couldn't be the problem coming from the fact that sub-objects (composing the final exported "diffuse object") are combined in a different order than in the "AO object" ?

Possible, but that's not the main suspect in my list of possible culprits.

What for sure can mess up the order of sub-objects (and faces) is material assignation. The main model and the AO model should have materials with similar naming (Wings3D sorts them in alphabetical order) assigned to corresponding faces. If, for example, a group of faces in the main model is given a material whose name is let's say 'wood', the equivalent faces in the AO/UV2 object should be given a material named 'wood_AO', 'wood_2', or something similar.

Originally Posted by Mister_M

If your excel sheet can solve the problem, let's try it. Just explain to me how this works.

Yes, it can. As I said, by using the output of my spreadsheet we can create an AO/UV2 obj file which is the exact copy of the main obj file, exception made for the values of texture coordinates.

Before I upload it, I need to be sure that it has enough rows for processing your model. Can you tell me the number of f and vt lines in the obj files you are processing? You can easily count them by pasting their content in an empty Excel document and by using the data filtering options or the COUNTIF function.

Originally Posted by Jeff-Groves

I'll have to refresh my memory on how the TMAPs actually work since it's been so long since it had to be done by hex editing. (Pre 2007!)
BUT! Only faces and texture coords are saved to the TMAP sections if I am recalling correctly.

Thank you for your inputs Jeff!

Does that mean that faces and texture coordinates can be in a different order in the secondary map channel than in the primary one?

Originally Posted by Jeff-Groves

I actually still have a program from 'back in the day' that exports and imports the different TMAP channels. So one could change or add just a TMAP and not have to re-import the complete obj like S3D does.

in future this tool might come in handy

[Mister_M]

Originally Posted by gap

Can you tell me the number of f and vt lines in the obj files you are processing?

Hull1.obj :
2018 f
868 vt

Hull1-uv2.obj :
2018 f
2729 vt

It already seems problematic...

[QUOTE=Jeff-Groves;2788565]Well that was easy.
I compared the main obj and the AO obj in 010.
Your vertices do not match between the Main Hull and AO Hull.

Ah ! You replied while I was writting this...

Originally Posted by Jeff-Groves

So basically S3D IS importing garbage you created.

Sorry, you mean : garbage that Wings3D created...

Originally Posted by Jeff-Groves

That is the flaw AND the opening in S3D that I am working to fix!

You are trying to fix garbage created by Wings3D when exporting ?

[Jeff-Groves]

Well. In the dat file format you CAN HAVE different counts on that information!
But the VERTS MUST BE EXACTLY THE SAME!

I'd suspect Wings re-ordered the Verts placement in the exported obj file thus faces and textures are screwed.

And You are the one that used Wings. It didn't to it by itself!
LOL!

[Mister_M]

Originally Posted by Jeff-Groves

Well. In the dat file format you CAN HAVE different counts on that information!
But the VERTS MUST BE EXACTLY THE SAME!

I'd suspect Wings re-ordered the Verts placement in the exported obj file thus faces and textures are screwed.

And You are the one that used Wings. It didn't to it by itself!
LOL!

I don't create garbage by myself...

[Jeff-Groves]

Just playing with you Mate.


It is strange to many that somethings do not work as expected with programs like S3D or TDW's program for SH5.

Both missed things and based their codes on false assumptions.
Sometimes that was a Ego thing on their parts to NOT listen to suggestions.
Sometimes it was a lack of understanding EXACTLY how the import target is constructed!

S3D used ALOT of information from the Grey Wolves in it's construction.
TDW's program aslo credits a Grey Wolf as soon as you open it!
(Guess WHO privateer is!)