Trying to dev a Java M2 lib&tools

[spik96]

Hey !

As my C converter was a bit of a nightmare to maintain I initially decided to rewrite it in Java.
Then I thought I may use this opportunity to make something better and more useful on the long term.
I'm now building something really simple to use in a well-known language.
It's based on WC3Utils by Dr Super Good from The Hive Workshop, the WoW Wiki, and many other sources.
It recognizes an M2 format based on its magic number (typically MD20) and its version (such as 264 for WotLK, but you could put -1 to disable this and load chunked files like Warcraft 3 MDX files for example).
I heavily used reflection to make basic classes, such as ArrayRef (n and offset) and AnimationBlocks easy to use.

Now adding another format is really easy.
Example : for many years some people were scared about parsing the preWotlk formats because the AnimationBlock is really different (one timeline instead of multiples timelines). Some thought you had to create lot of new methods and there was a minefield of bugs.

LK 3.3.5 :
   private short interpolationType;
   private short globalSequence;
   private ArrayRef<ArrayRef<Integer>> timestamps;
   private ArrayRef<ArrayRef<T>> values;
BC 2.4.3 :
   private short interpolationType;
   private short globalSequence;
   private ArrayRef<Integer> timestamps;
   private ArrayRef<T> values;

That's it. Done. It writes itself, it reads itself, this is all you have to change (and add&remove a few fields in other places). You do not need to recreate other reading/writing functions, you do not need one for each type like in C as there is genericity, it's just fast and simple. You do not need arrays of arrayrefs, read them from the stream, then read each n&ofs to finally reach the real data... the class does it for you. Just put the ArrayRef at the correct place in your declaration, and call its unmarshalling method, and done. A "getContent()" will give you your real Array. Or an Array of Arrays if that's needed.
It is not meant to be performant, just easy to read and easy to adapt when Blizzard format changes.
And it should be fast to add hooks to perform any task you want.

The backbone is the ArrayRef class : http://pastebin.com/47pG7Qqx
Still work in progress.

I will put this on Github soon. If anyone wants to help me, just post here.

[xJKz]

And 4.x, 5.x, 6.x ?

[spik96]

Yes, even 7.x could be added fast once the documentation is written by the dataminers. I'm starting with 3.x as it's the most documented, then I'll just have to implement the changes. At the exact same time small conversion methods could bridge versions between them and so the custom models one made years ago could be brought to newer versions. That's just all concept and small part of codes right now of course, but well, as the section says, that's what I'm working on.

[schlumpf]

WoD should be documented equally well.

[spik96]

WoD should be documented equally well.

Well, I thought some things were still missing in particles & .bone files, but after all it may be finished.
I wish there was a way to quickly view changes in the formats on the wiki. Right now you just have to read carefully to find mentions of extensions. Some kind of tag system in the wiki editor would be nice ([LK-WoD] by default, then restrictions to specify [Cata-MoP] for example).

[Namok]

As always, I wish you good luck Spik !

[spik96]

Almost finished the parser. Some things stay obscure :
About Ribbons

Code: [Select]

struct M2Ribbon
{
 ....
  uint32_t nTextures;                 // Number of referenced textures.
  uint32_t ofsTextures;               // Offset to the referenced textures.
 ...
} ribbons[];

Anyone knows the structures this is pointing to ?

I already started the particles so, if I don't have too much work irl, I should be able to read M2 WoD, MoP, Cata, WotLK, BC, Vanilla.
Then I'll add a few converting functions (in the way of convertUp() and convertDown() methods evolving structures one extension up or down ^^) because I personally need that.
After that I will work on parsing Legion models.
Finally I will have to make methods to call that a proper lib and not just a parser.

[schlumpf]

http://wowdev.wiki/index.php?tit ... ldid=19228

[spik96]

Thanks ! I see you also switched the blend references from the imprecise "integer" mention to uint16, really useful, I thought they were uint32. The devil is in the detail.

[spik96]

Almost finished WotLK BC and Cataclysm, but the particles make me struggle a bit.
LK:
http://www.wowdev.wiki/index.php ... articleOld
0x168 : Undefined structure "CRange". At the moment I assume it's a 2 float pair.
0x170 : There is a blank. That's a mistake or this number really exists ?

Cataclysm:

In addition to these two parameters, "ParticleType" and "HeadOrTail" got reused as
fp_2_5 multiTextureParamX[2];

I'm sorry, I'm not the best in English at all, I wonder if reused mean replaced in the M2ParticleOld structure (type changed) or do you mean it's not used/gone from the old structure and the fields are now located at the end of the new structure and with their new type ? I hope I'm clear.

[schlumpf]

http://www.wowdev.wiki/index.php ... 6.4.135.53

[spik96]

0x168    CRange    twinkleScale    {float: min, max}
0x174    float    Drag    Air resistance, see below.

The size of a structure beginning at 0x168 and ending at 0x174 would be 3*4 bytes. twinkleScale is only 2*4 bytes so whether the offset has become wrong or there is really 4 bytes missing.
I know I'm pointing stupid details but I think some bytes wrong can f*ck up many things very quickly. I hope you're not wasting too much time.

[schlumpf]

Oh. Yes. Guess I shouldn't do this stuff when drunk and on mobile.

I don't know the purpose of that field. You may want to try find one in the history of the wiki.

[spik96]

This was changed from float scale[3] to twinkleScale[2] and a blank the 25 october.
https://wowdev.wiki/index.php?title=M2/ ... ldid=19103

Maybe the twinkleScale is in fact three floats (x, y, z), it would make sense.
Except if you saw something pointing it is not the case somewhere. You probably changed it to twinkleScale[2] for a good reason.

[schlumpf]

Twinkle scale is based on the client code and correct.