[TEC] Let's grab the ID's

[Jeff-Groves]

A BIG problem has always been ID conflicts when doing ANY mod!
Sure. There's a small chance for duplicate ID's but it DOES and HAS happened!
I'm going to tell you how to get the ID's and put them into a SEARCHABLE data base! Then any new Unit or whatever can be checked against that to reduce conflicts.

I only work with 010 Editor so it's based off that. It's only a few bucks and any serious Modder needs it in their tool as I'll be writing Scripts and such to assist.

Step 1 in the process......
Run IDduplicatefinder and index a folder.
Minimize That.
Open 010 and under FILE, Open Process.
Scroll untill you find IDduplcatefinder and click on it. Open as read only.
Save that as a hex document like process(0).dat.

Step 2 in the process......
Now open IDduplicate finder and get the first ID in the scrollable list.
If its say 199D6FC7A8007495 (May be different!) that needs to be typed into a txt document for further use.
Go to the LAST ID and do the same.

Step 3 in the process......
Go ahead and close 010 at this time and close IDdulpicatefinder. You have the file save of course so we will work with that until I write scripts for you lazy Gits.

Step 4 in the process......
Open your browser (And if you are wondering what that is? STOP NOW!)
Go to https://www.browserling.com/tools/text-to-hex
Pasted your 1st and last saved ID's into that converter to get the hex values. (One at a time! That's one ID complete. Not one entry at a time!)
Copy and save the output of each!

Step 5 in the process......
If you've made it this far? Congratulations! You should have something like below saved.
31 39 39 44 36 46 43 37 41 38 30 30 37 34 39 35
You need to modify that like below.
31 00 39 00 39 00 44 00 36 00 46 00 43 00 37 00 41 00 38 00 30 00 30 00 37 00 34 00 39 00 35
Those will be our search strings in 010.

(I know at this point your saying " This all seems pretty complected!" Suck it up ButterCup! I'll write scripts for you when I get the time. So pull your diapers up and put Big Boy boots on and stop the whimpering.)

Step 6 in the process......
Using 010, open the saved process file. (You did save one right?)
Select Search, find and paste in the first of those strings you saved and make sure your set for Hex Bytes.
Delete everything above that!
Do the same for the last string and delete EVERYTHING below that!

You should end up with all ID's now. Save the file!

Step 7 in the process......
Time to start making this into a usable and searchable data base.

In 010 Select Search, Replace. Options needs to be set for Match Whole Word, Maximum WildCard length 2.
Find Hex Bytes 00
Leave Replace blank. Once the Replace is done? Select ALL then Copy As Text Area.
Make a new text document and paste into that. Now that IS searchable. But.......

Step 8 in the process......
Let's cut this file down and get rid of the Hex notions shall We?
I use open Office so here's what I do.
Open a new Spreed Sheet. Go to Insert, Sheet From File, select your save and make sure spaces is your delimiter.
Delete all columns except the full ID and save as a cvs.
Once saved? Rename it to whateverYouwant.txt

Step 9 in the process......
Pop the Top off your favorite Beer and enjoy the fact you actually made it this far!!!

Step 10 in the process......
Start posting about how the instructions confused you because you used Step 9 (Multiple times!) before Step 2!
Then post about how I should have written the Scripts before hand so you could have skipped Steps 2 through 8.
Never admitting You used Step 9 multiple times before ever really trying this.


(Admittedly posted after practicing Step 9 multiple times. Maybe I should make that Step 1?)

[Jeff-Groves]

I COULD get really techie and remove duplicate ID's you'll find in the list.
If you need to know WHY those are there? Just ask and I'll explain.
This work flow can be done on SH3 and SH4.
SH5? I'm working on breaking the HASH used.

ID conflicts should never be an issue from this day forward!

Given the process captured? I can track EVERY conflict you introduce!

[propbeanie]

The IDs ~love~ to be grabbed, but they're just like that... all those permutations of possible IDs that the numbering system has, and they all seem to be like humans, and want to keep up with the Jones and Smith families, and congregate on the town square, all looking the same...

A certain s7rikeback has been digging deep into the bowels of the FotRSU mod with this very approach, and has made HUGE headway. Some of the "returns" from eliminating conflicts slows things up a bit (less sharing of assets), but others have more than made up for that (less sharing of assets ). The FotRSU mod is quite large compared to what it was, and loads faster than before, quicker to the mission, and better at Saves and later Loads than previously measured and noted. A really BIG difference is when the Museum goes to load, in that over a minute (almost "and a half") has been shaved from its load time, and there are more ships planes and subs to look at now. We consider this "mandatory" work, but it does take quite a bit of patience to work through. Thankfully, s7rikeback has plenty of that. I am not certain if he has "catalogued" the IDs, but that does sound like an excellent idea.

Now for a shameless plug: If not for Jeff-Groves, we would most likely have never delved into this aspect of "modding". We were content to "Re-Map IDs" and use the "Random" button, under the assumption ("assumption!!! never 'assume' nothing!") that S3D would pick an unused set of numbers - HAH! every time we change an ID "here", there ends up being a conflict over "there"... so it is a continual battle. But kudos to Jeff-Groves for the impetus and tools to do this! Of course, he came from the GWX team, so he is good people... plus he's SubSim through and through...

[Jeff-Groves]

I'm gonna wait and see WHO takes a good look at the process capture.
IF they look close? You can capture WHERE every one of the ID's is!
What file and what position in that file!
Talk about getting REAL detailed!

[propbeanie]

You must mean when one clicks on the "hit", as in

Originally Posted by s7rikeback

Duplicated Id :1254C8704732486B
C:\Users\s7rikeback\Desktop\New folder\Terrain\Locations\Pearl_Sub_Base.dat at : 2758
C:\Users\s7rikeback\Desktop\New folder\Terrain\Locations\Pearl_Sub_Base.dat at : 2283

Notice though, that I are not skilled enough at this, and have to use the services of the knowledgeable one...

[s7rikeback]

Originally Posted by Jeff-Groves

A BIG problem has always been ID conflicts when doing ANY mod!
Sure. There's a small chance for duplicate ID's but it DOES and HAS happened!
I'm going to tell you how to get the ID's and put them into a SEARCHABLE data base! Then any new Unit or whatever can be checked against that to reduce conflicts.

I only work with 010 Editor so it's based off that. It's only a few bucks and any serious Modder needs it in their tool as I'll be writing Scripts and such to assist.

Step 1 in the process......
Run IDduplicatefinder and index a folder.
Minimize That.
Open 010 and under FILE, Open Process.
Scroll untill you find IDduplcatefinder and click on it. Open as read only.
Save that as a hex document like process(0).dat.

Step 2 in the process......
Now open IDduplicate finder and get the first ID in the scrollable list.
If its say 199D6FC7A8007495 (May be different!) that needs to be typed into a txt document for further use.
Go to the LAST ID and do the same.

Step 3 in the process......
Go ahead and close 010 at this time and close IDdulpicatefinder. You have the file save of course so we will work with that until I write scripts for you lazy Gits.

Step 4 in the process......
Open your browser (And if you are wondering what that is? STOP NOW!)
Go to https://www.browserling.com/tools/text-to-hex
Pasted your 1st and last saved ID's into that converter to get the hex values. (One at a time! That's one ID complete. Not one entry at a time!)
Copy and save the output of each!

Step 5 in the process......
If you've made it this far? Congratulations! You should have something like below saved.
31 39 39 44 36 46 43 37 41 38 30 30 37 34 39 35
You need to modify that like below.
31 00 39 00 39 00 44 00 36 00 46 00 43 00 37 00 41 00 38 00 30 00 30 00 37 00 34 00 39 00 35
Those will be our search strings in 010.

(I know at this point your saying " This all seems pretty complected!" Suck it up ButterCup! I'll write scripts for you when I get the time. So pull your diapers up and put Big Boy boots on and stop the whimpering.)

Step 6 in the process......
Using 010, open the saved process file. (You did save one right?)
Select Search, find and paste in the first of those strings you saved and make sure your set for Hex Bytes.
Delete everything above that!
Do the same for the last string and delete EVERYTHING below that!

You should end up with all ID's now. Save the file!

Step 7 in the process......
Time to start making this into a usable and searchable data base.

In 010 Select Search, Replace. Options needs to be set for Match Whole Word, Maximum WildCard length 2.
Find Hex Bytes 00
Leave Replace blank. Once the Replace is done? Select ALL then Copy As Text Area.
Make a new text document and paste into that. Now that IS searchable. But.......

Step 8 in the process......
Let's cut this file down and get rid of the Hex notions shall We?
I use open Office so here's what I do.
Open a new Spreed Sheet. Go to Insert, Sheet From File, select your save and make sure spaces is your delimiter.
Delete all columns except the full ID and save as a cvs.
Once saved? Rename it to whateverYouwant.txt

Step 9 in the process......
Pop the Top off your favorite Beer and enjoy the fact you actually made it this far!!!

Step 10 in the process......
Start posting about how the instructions confused you because you used Step 9 (Multiple times!) before Step 2!
Then post about how I should have written the Scripts before hand so you could have skipped Steps 2 through 8.
Never admitting You used Step 9 multiple times before ever really trying this.


(Admittedly posted after practicing Step 9 multiple times. Maybe I should make that Step 1?)

Hello Jeff,

This is a very good guide, I have followed it using StocksH4 Air folder as a test bed.



First ID
Step 2 = AA0295AC084117E7
Step 4 = 41 41 30 32 39 35 41 43 30 38 34 31 31 37 45 37
Step 5 = 41 00 41 00 30 00 32 00 39 00 35 00 41 00 43 00 30 00 38 00 34 00 31 00 31 00 37 00 45 00 37

Last ID
Step 2 = 1AFSE4D85F2070E9
Step 4 = 31 41 46 53 45 34 44 38 35 46 32 30 37 30 45 39
Step 5 = 31 00 41 00 46 00 53 00 45 00 34 00 44 00 38 00 35 00 46 00 32 00 30 00 37 00 30 00 45 00 39

Step 7:
https://1drv.ms/t/s!AsNqeJIusLGBnBV1...01tUX?e=kJJlhi

Step 8:
https://1drv.ms/x/s!AsNqeJIusLGBnBJcPyDIN13u9DET?e=ybSUaF

------------------------
Step 8 Addtion for MS Office 365 users.

Open new spreadsheet, go to Data tab, select “From Text/CSV” tab, select your save, make sure spaces is your delimiter, select “load”
Delete all columns except the full ID and save as a cvs.
Once saved? Rename it to whateverYouwant.txt

[s7rikeback]

Originally Posted by propbeanie

You must mean when one clicks on the "hit", as in




Notice though, that I are not skilled enough at this, and have to use the services of the knowledgeable one...

In MS Office 365.
We can load our database,which was created from above.
Select the ID column, go to Home Tab > and select Conditional Formatting button, select "Highlight Cell Rules > then Duplicate Values, by default this shows up as red...
And we get this when go up or down the list of ID's....



Which only tells us, we have an ID conflict, no idea where at this point... without using the IDDuplicator, which would to re-index the files and then scan for duplicates, which can take hours on a large mod, you then have to view the output log to find the ID's above to understand what file is the issue. Or manually do each line in the output window at a time.

I have not yet found Jeff's link trail from 010 / database to the file of concern, still looking very hard at the details...

[Jeff-Groves]

Look!!!



Now there are a bunch of NON-ID assigned entries in that whole section.
Gonna have to figure out a way to filter those out. BUT that would create a more comprehensive Date Base.
Remember that SOME ID's are repetitive legitimately!
EnvColors dat files for example, have the same ID's but only one dat is loaded at a time!
Some ID's are calls to other files! But show as duplicates.
Take a look at any Location dat file and you'll see the Placement controller! Notorious for popping Duplicates!
But those are ALL legit usages!

Once good data bases are created? Simple to Script 010 to allow us to enter an ID and have it scan that.
We tell it to create a random ID and if it finds one exists? Re do.
All this would be done in Memory using Arrays and simply checking through the Arrays.
You could do it with Vectors but they use more memory.


Thinking about this? I probably HAVE 90% of the Code written in C++ that could be changed into a Stand Alone program so one would not need 010.

[Jeff-Groves]

Originally Posted by propbeanie

We were content to "Re-Map IDs" and use the "Random" button, under the assumption ("assumption!!! never 'assume' nothing!") that S3D would pick an unused set of numbers - HAH! every time we change an ID "here", there ends up being a conflict over "there"... so it is a continual battle.

Coders KNOW the Random functions have flaws! C++, C#, don't matter!
S3D does have a seed function (Good Idea) Bad approach with no cross check for existing ID's?
But his intention was never to SCAN every file with ID's.
I'd call it Las Vegas ID creation!
Now it's not a bad idea to create a random ID. Just a bad idea to NOT CROSS CHECK with a data base.

[propbeanie]

"Random", as in the same numbers consistently in the same order each time it runs... lol

[Jeff-Groves]

You know Las Vegas suffered from the random function in some Gaming Machines at one time.
A smart Guy can predict the numbers that should come up based on several factors.
My random function will cross check existing numbers from a Data base and re-do a random ID number. So NO CHANCE of a Las Vegas number!

[Jeff-Groves]

While looking for something totally different I stumbled on a way to get all the ID's faster AND easier!

It's SysExporter v1.77
https://www.nirsoft.net/utils/sysexp.html

Run IDDuplicateFinder and index like normal.
Once it's got the list in the left window, open SysExporter and grab the ID's!
Save as tabbed type txt and Bob's your Uncle!!!


Now this is a very simple little program so.........
Don't MAKE ME get my Crayons out to draw you a picture!

[vickers03]

thanks, i'm working on a mod with many new ID's and this will be very useful.

[Jeff-Groves]

I'm thinking of codeing a scanner that would read the saved ID's and compare a list of new ID's We create to insure no duplicates. Unless I can find something on the netz that does that already.



SobolSoft has a nice one. It's $20 bucks for the License.
Gonna test a version and see if I like it first.