uop file format (updated: 1.7.07)

Diskussion und Informationen über UO:KR
Nachricht
Autor
Arahil
Moderator (RunUO)
Beiträge: 437
Registriert: 15 Jan 2004 09:32
Wohnort: Wien

uop file format (updated: 1.7.07)

#1 Beitrag von Arahil » 01 Jul 2007 16:56

since the old format definition contained a bug concerning the file offsets (there is no offset value in the index block header) here is the newest one. if you got more infos please contact me via this forum

Code: Alles auswählen

UOP Fileformat ( aka Mythic Package )
---------------------------------------
credits: SENE, Kelon, Dantalion, Arahil

(Every Offset value is its first evidence as example)
All values are stored in Little Endian sequence, as usual in x86 architecture.
Compression method is DEFLATE using zlib.

sizeof(DWORD) = 4
sizeof(QWORD) = 8

[1] - General Format Header (sizeof: 40bytes )
Byte(23) 0x0 - 0x17 -> Containing general file headers (Version etc.)
DWORD? 0x18 -> Amount of contained files/indexes
byte(12) -> Unknown gibberish

[2] - Index Block Header (sizeof: 24bytes)
There can be multiple index blocks, they are splitted into chunks.
DWORD 0x28 -> Amount of contained files in this index, max 100/0x64
QWORD 0x2c -> Offset to the next index block header OR Zero
When a index block doesn't contain 100 index definitions, it will be padded with nulls

	[3] - FileIndex Definitions (sizeof: 34bytes )
	QWORD 0x34 -> Offset to start of Data Block of this file
	DWORD 0x3c -> Length of Data Block header (usually 0x0C)
	DWORD 0x40 -> Lenght of compressed data
	DWORD 0x44 -> Size of decompressed file
	QWORD 0x48 -> UNKNOWN-1 (maybe filename?)
	DWORD 0x50 -> UNKNOWN-2 (maybe CRC?)
	WORD  0x54 -> Separator? (always 0x0001)
	...this repreats, until all FileIndexes are processed

	[4] - Data Block/File (sizeof: 12+Lenght bytes)
	DWORD 0xd7c -> separator, start of Data ( WORD[2] 0x0008 0x0003 )
	QWORD 0xd80 -> UNKNOWN, possibly a CRC
	BYTE(Lenght) 0xd88 -> compressed data
	...this is repeated until all Files from FileIndexes are processed

repreat until next Index Block=0.


Pseudocode:

[1] - General Format Header (sizeof: 40bytes )

while ( repreatindex ) do
	[2] - Index Block Header (sizeof: 12bytes)

	while ( indexfilenumber~=indexfilecounter ) do
		[3] - FileIndex Definitions (sizeof: 34bytes )
	end
	while ( indexfilenumber~=indexfilecounter ) do
		[4] - Data Block/File (sizeof: 8+Lenght bytes)
	end
end

MagnusBr

#2 Beitrag von MagnusBr » 02 Jul 2007 20:13

Hey man, any news about the tool? :)

Arahil
Moderator (RunUO)
Beiträge: 437
Registriert: 15 Jan 2004 09:32
Wohnort: Wien

#3 Beitrag von Arahil » 02 Jul 2007 21:52

undergoing the third rewrite at the moment ;)
since i'm pretty busy with the rest of my rl you'll surely have to wait some time. to really pack those files again we need to do some investigation as well (checksum and so on).

MagnusBr

#4 Beitrag von MagnusBr » 03 Jul 2007 17:31

Thanks :)

If I can help by any ways, let me know.
I think I'm a little more interested to start some research on UOP Mods than in server itself :P

Trick

#5 Beitrag von Trick » 23 Aug 2007 22:16

I'm working on myp-packs from warhammer online. The packet format is almost the same.
warhammer online uses a mft.myp which includes a lot of records like

Code: Alles auswählen

ph="6160c295" sh="ffa03bbe" ul="880" ct="1" cl="716" ft="ffffffff" ml="89" mc="249da95c" t="45548c51" 


ph, sh, mc are apparently hash values of file names and I repeatable found them again in the fileindex definitions. Due to that I guess that UNKNOWN-1 and UNKNOWN-1 are 3 DWORD values with filename hashes or something similar (folder names?).

this leads to:

Code: Alles auswählen

[3] - FileIndex Definitions (sizeof: 34bytes ) 
   QWORD 0x34 -> Offset to start of Data Block of this file 
   DWORD 0x3c -> Length of Data Block header (usually 0x0C) 
   DWORD 0x40 -> Lenght of compressed data 
   DWORD 0x44 -> Size of decompressed file 
   DWORD 0x48 -> some (filename? folder?) hash 
   DWORD 0x52 -> some (filename? folder?) hash 
   DWORD 0x56 -> some (filename? folder?) hash 
   WORD  0x5a -> Separator? (always 0x0001) 
   ...this repreats, until all FileIndexes are processed 
Don't know if this us usefull for you. At least I'm not more than a beginner with this stuff. :)

Edit:
Within my myp-packs I got some data blocks that can not uncompress via zlib.
zlib returns -3 (data error).
However these data packs are starting with 0x78 0xC9 ... which means that they are compressed with zlib? Or not?

SiENcE

#6 Beitrag von SiENcE » 25 Aug 2007 00:02

i thought that warhammer uses the same fileformat, because i found GUI bitmaps from warhammer in uo:kr. I thing they have forgotten this.

could you give me a file or an link for a myp file from warhammer? Or where do i get the beta?

Do you tried this tool: http://uodev.de/viewtopic.php?t=4856 ?

Trick

#7 Beitrag von Trick » 25 Aug 2007 00:29

I'm sorry ... I can't. The Beta is closed atm.
The file format definitely is different from uokr.

- general fomat header is 512 bytes (but mostly padded with null).
- index block header and fileindex definitions are similar.
- data block header is 136 bytes and fairly different

Due to this the uo-unpacker won't work I guess. I wrote a new unpacker for my war myp's.

Trick

#8 Beitrag von Trick » 28 Aug 2007 07:47

Update:

Code: Alles auswählen

[1] - General Format Header (sizeof: 40bytes ) 
Byte(12) -> Containing general file headers (Version etc.) 
WORD -> General Format Header Length (in warhammer-myp's 0x0200)
Byte(6) -> unknown1 (0x00 in all myp's I've seen so far) 
WORD -> unknown2 0x0064 ... except in warhammer-mft.myp (0x03E8)
WORD -> unknown3
DWORD? -> Amount of contained files/indexes 
byte(remaining format header) -> Unknown4 gibberish 

Code: Alles auswählen

[3] - FileIndex Definitions (sizeof: 34bytes ) 
   QWORD 0x34 -> Offset to start of Data Block of this file 
   DWORD 0x3c -> Length of Data Block header (usually 0x0C) 
   DWORD 0x40 -> Lenght of compressed data 
   DWORD 0x44 -> Size of decompressed file 
   DWORD 0x48 -> some hash(filename? extension?) 
   DWORD 0x52 -> some hash (filename? extension?)
   DWORD 0x56 -> some hash (folder? because in warhammers-mft.myp this is 0x00 and in all other myp's this is ~ 0x00)  
   WORD  0x5a -> Compressed flag (0x0001 = compressed; 0x0000 = uncompressed) 

Code: Alles auswählen

   [4] - Data Block/File (sizeof: length data block header + Lenght bytes) 
   WORD 0xd7c -> some flag (0x0003 in uokr-myp's, 0x0004 in warhammer-myp's) 
   WORD       -> offset to file data from this position (in warhammer-myp's  
                  this is 133 and sometimes 132. so the data block header 
                  has no fixed size)
   QWORD      -> UNKNOWN, possibly a CRC 
                 (don't think so because this value isn't unique for every data 
                 block header. if this is a crc it has to be unique unless there are 
                 to equal files inside a myp ... or not?)
   BYTE(Lenght) 0xd88 -> compressed data 
imho there are no such thing as separators in myp's at all.
furthermore i found myp's without any data block headers. guess this is defined somewhere inside the general format header (Unknown4? this one is different from myp's with data block headers).

Arek Maldazaar

#9 Beitrag von Arek Maldazaar » 05 Sep 2007 19:54

Code: Alles auswählen

DWORD 0x48 -> some hash(filename? extension?)
   DWORD 0x52 -> some hash (filename? extension?)
   DWORD 0x56 -> some hash (folder? because in warhammers-mft.myp this is 0x00 and in all other myp's this is ~ 0x00)  
I don't beleive these are hash, how would they get the filenames from it. I think there are just encoded strings somehow. Could anyone do research on it, I failed at it. I have 12 bytes and appropriate filename, but I'm unable to get the algorhytm

Trick

#10 Beitrag von Trick » 05 Sep 2007 22:45

That's quite easy: The don't use real filenames at all. :D

See http://www.zezula.net/en/mpq/namebreak.html for more information. It's about Blizzards mpq files ... but I guess it's the same with Mythics myp's.

It's just a guess of course. Maybe these parts are encoded strings. But myself I don't believe in this.

My newest theory is that these value are hash table offsets and not the real hashes.

Arek Maldazaar

#11 Beitrag von Arek Maldazaar » 06 Sep 2007 14:45

Yep, 12 bytes for a string is too few... Well, if they're hash values, we're fucked up, 'cause we don't know the algorhytm... What about disassembling KR client to find out? :)

Trick

#12 Beitrag von Trick » 06 Sep 2007 17:06

If these values are hashes I guess they use md or sh hashes. So thats easy.

But ... at least DWORD 0x48 can't be a hash, because it's a growing number.

All three values can be recovered in mft.myp (which imho describes all game files).

Example for a file dataset in mft.myp:
ph="6160c295" sh="ffa03bbe" ul="880" ct="1" cl="716" ft="ffffffff" ml="89" mc="249da95c" t="45548c51"

I found DWORD 0x48 in PH, DWORD 0x52 in SH, and DWORD 0x56 in MC.

ML definitely is the the Data Block Header length. All other values are unknown.
SH and PH sounds like xx Hash ... but again this is a guess.

BTW: Is there a mft within KR?

Arek Maldazaar

#13 Beitrag von Arek Maldazaar » 07 Sep 2007 06:50

Well, what that would leave 8 bytes (0x52 and 0x56) to be hash. Does not change anything on the fact, we don't know the algorhytm how to get it.

Imho there isn't anything like mft, I can write you filenames of all uops...

The good thing is, I am able to get names of few files (those packed inside!), actually what I am able to give you is: a file, its index block header (where these bytes are) and its real name (let's say the actual string they got the bytes from). Do you think we would be able to get the hash algorhytm?

I did some research on 0x56, it seems to be same for like 3-7 files, then it changes and is same again. That may look like they had a hell lot of directories, but then again: I tried this with the map and every block of map 64x64 has different 0x56....

E4e

#14 Beitrag von E4e » 11 Sep 2007 17:36

If someone is interested in WAR MYP, he can message me on the forum

SiENcE

#15 Beitrag von SiENcE » 12 Sep 2007 11:04

you should post your extract tool for war myp on: http://forum.xentax.com/

would be nice to see it there.

Antworten