Jump to content
Search In
  • More options...
Find results that contain...
Find results in...
Death Egg

Which source ports all plan to implement UDMF?

Recommended Posts

KBDoom will eventually implement UDMF. Has anyone considered a specification for compressing UDMF? Not a general Huffman-based method. I'm talking about tokenizing common keywords, while leaving custom keywords as-is. It seems it could be a relatively simple algorithm, with decent space-saving properties. It might even parse faster. Just a thought.

I'm just trying to gauge interest and support for such an idea.

Share this post


Link to post
kb1 said:

Has anyone considered a specification for compressing UDMF?

Not specifically. UDMF compresses extremely well in zips, so the idea is to use zips. Several ports can load zipped files already so...

The UDMF specs were designed to be as simple as possible to implement; that's why it doesn't feature, for example, nested blocks. Putting some sort of lump-specific compression (rather than generic archive compression like with a pk3) would complicate the specs needlessly.

Share this post


Link to post
printz said:

But still, why not just use the ZDoom specials in a common namespace?


ZDoom (and derivatives) has some specials I can't see Eternity supporting any time soon (or ever), such as: Player_SetTeam, Team_Score, Team_GivePoints, SetGlobalFogParameter, FS_Execute, Sector_SetPlaneReflection, Sector_SetContents, etc.

Even some specials that we should support we don't atm.

Share this post


Link to post
Gez said:

Not specifically. UDMF compresses extremely well in zips, so the idea is to use zips. Several ports can load zipped files already so...

The UDMF specs were designed to be as simple as possible to implement; that's why it doesn't feature, for example, nested blocks. Putting some sort of lump-specific compression (rather than generic archive compression like with a pk3) would complicate the specs needlessly.

Not lump-specific compression, "tokenizing" of keywords, similar to how old dialects of BASIC tokenized keywords to normalize syntax, shrink size, and speed up execution. Something like IE's MHTML web page tokenizing. Again, just getting some feedback on an idea. Thanks.

Share this post


Link to post
kb1 said:

Not lump-specific compression, "tokenizing" of keywords, similar to how old dialects of BASIC tokenized keywords to normalize syntax, shrink size, and speed up execution. Something like IE's MHTML web page tokenizing. Again, just getting some feedback on an idea. Thanks.

Why spend time on implementing an inefficient compression method when standard deflate is much better and easier to add? You can literally add deflate decompression by including one extra source file (miniz) into your project and 10 lines of code to decompress the lump.

As for speed gains, if the tokenization part is slow there's something very wrong in how the tokenizer is implemented. There are algorithms there that allow this to be done in a single for loop over all the characters.

Share this post


Link to post
kb1 said:

KBDoom will eventually implement UDMF. Has anyone considered a specification for compressing UDMF?


No. It's useless. It only complicates matters and the speed gain is insignificant. Loading the largest available map on my computer (ZDCMP2) takes 0.4 seconds to parse in GZDoom, but then loading all the needed data takes twice as long.

Here's the timing I got:

https://forum.zdoom.org/viewtopic.php?p=733709#p733709

And parsing the keywords is the least to worry, even then parsing the numeric values into actual binary values is by far the largest chunk of the parse time.

dpJudas said:

As for speed gains, if the tokenization part is slow there's something very wrong in how the tokenizer is implemented. There are algorithms there that allow this to be done in a single for loop over all the characters.


Correct. ZDoom uses its existing Name class to tokenize the UDMF input and that thing is already very efficient at converting a string into a token that can then be used in a 'switch' statement for dispatching.
On the other hand, tokenizing the UDMF itself adds a completely new layer of complexity to any tool that needs to write the format and the decision to go pure text was to a large part motivated by making this task as easy as possible, even if it means some minor efficiency loss. Overall, the most costly operations that occur with UDMF are not related to parsing (see my table), the second most expensive part is indeed loading the data from the HD, and zipping helps a lot more here than pre-tokenization. UDMF normally compresses to ca. 1.5x the size of binary maps.

Altazimuth said:

ZDoom (and derivatives) has some specials I can't see Eternity supporting any time soon (or ever), such as: Player_SetTeam, Team_Score, Team_GivePoints, SetGlobalFogParameter, FS_Execute, Sector_SetPlaneReflection, Sector_SetContents, etc.

Even some specials that we should support we don't atm.


Some of these are not even in ZDoom but Zandronum exclusives, and Sector_SetContents is deprecated anyway, as its only purpose is to load the few Vavoom maps out there. There's no need to make those part of a unified namespace.

Share this post


Link to post
Graf Zahl said:

[Compressing UDMF is] useless. It only complicates matters and the speed gain is insignificant.

What about the gain of reduced filesize?

EDIT: I've just read Gez's post, but still.

Share this post


Link to post

It would shorten the uncompressed filesize but make very little impact on compressed size, because the keywords appear so frequently in the text that they are normally compressed away easily.

Share this post


Link to post

If size on the disk is your major concern, you can take a pre-existing compression algorithm and then modify it so that it is optimized for UDMF. For example you could take deflate and use an alternative fixed huffman tree that is optimized for UDMF's text, however you would need to write new utilities just to compress and decompress it. Maintaining these utilities would increase cost and complexity of programs, and in most cases mod authors would not be using it anyway because it just complicates things.

Share this post


Link to post
RestlessRodent said:

Maintaining these utilities would increase cost and complexity of programs, and in most cases mod authors would not be using it anyway because it just complicates things.



And that's why I wouldn't even think about considering it. Just look at the history of compressed nodes how such things go. Only after I added an uncompressed version to ZDBSP they gained some traction.

Share this post


Link to post
Chu said:

3DGE got UDMF recently, but it is still WIP.


Ooh that's exciting! I wish you the best of luck in getting it in.

On a different note, I have updated the EE UDMF spreadsheet so it now lists all the properties I could find, their type, what they bind to, origin, and which ports support them. Check the "Properties" tab.

Share this post


Link to post

LOL who cares... :D:D:D Why did you even list Vavoom? It only copied what it got from ZDoom.

In any case, if I find some time this evening I'll make a list of ACS functions so that this part can be checked as well.

Compiling these 3 lists into a spec shouldn't be too hard then.

Share this post


Link to post

I'm glad to see UDMF finally starting to get some traction.

Now that's a motivation to start creating more editor configs.

Share this post


Link to post
Graf Zahl said:

LOL who cares... :D:D:D Why did you even list Vavoom? It only copied what it got from ZDoom.

In any case, if I find some time this evening I'll make a list of ACS functions so that this part can be checked as well.

Compiling these 3 lists into a spec shouldn't be too hard then.


I tend to inject some levity into my work, helps me enjoy myself a bit more; this usually manifests itself into jokey comments.

You'll find EE's ACS functions listed here. I also recently added compat for a couple line specials that had different versions when called from ACS, and they're here.

Share this post


Link to post

So, developer side, should I add anything special to the new common namespace? Or simply treat the common namespace the same as Eternity's? While also filtering out undefined stuff, sure, hopefully.

Share this post


Link to post

Can't say yet. I think all that will be needed is filtering out some of the action specials and namespace-check the UDMF properties and maybe a few other items. ACS cannot be filtered anyway in case you load an add-on that's using other things.

Share this post


Link to post

I just ran through the ACS functions. I compiled a list of all that appear to be in both engines but this is hardly something conclusive, because Eternity's ACS VM is so radically different that I can't see which feature, aside from the listed functions is supported and which isn't. I'm leaving this part out for now.

Where does it translate the ACS bytecode into what the VM runs on? I couldn't find because looking for 'behavior' is rather futile.

Share this post


Link to post
Graf Zahl said:

LOL who cares... :D:D:D Why did you even list Vavoom? It only copied what it got from ZDoom.


Hey now, 3DGE is still relevant!! ;-)

Should I replace Vavoom with Eternity then? Or another standard?

Share this post


Link to post

If disk space is a concern, you could just use file system compression, which is transparent to applications. NTFS on Windows supports it, HFS+ on Mac OS X supports it, several file systems (eg, btrfs and zfs) on Linux support it.

Share this post


Link to post
Chu said:


That wasn't aimed at you. Do go there and do check the properties tab:

Altazimuth said:

On a different note, I have updated the EE UDMF spreadsheet so it now lists all the properties I could find, their type, what they bind to, origin, and which ports support them. Check the "Properties" tab.

Share this post


Link to post

I'm working on the UDMF support. It's been coming along pretty well. If you use the latest from the repo, you have:

Support for uncompressed GL v1/v2/v3 znodes.

Sector properties:
heightfloor
heightceiling
xpanningfloor
ypanningfloor
xpanningceiling
ypanningceiling
xscalefloor
yscalefloor
xscaleceiling
yscaleceiling
rotationfloor
rotationceiling
gravity
texturefloor
textureceiling
lightlevel
lightcolor
fadecolor
special
id
desaturation

Linedef properties:
id
v1
v2
special
arg0
arg1
arg2
arg3
arg4
sidefront
sideback
blocking
blockmonsters
twosided
dontpegtop
dontpegbottom
secret
blocksound
dontdraw
mapped
passuse
repeatspecial
playeruse
impact

Sidedef properties:
offsetx
offsety
offsetx_bottom
offsety_bottom
offsetx_mid
offsety_mid
offsetx_top
offsety_top
scalex_bottom
scaley_bottom
scalex_mid
scaley_mid
scalex_top
scaley_top
texturetop
texturebottom
texturemiddle
sector

Vertex properties:
x
y
zfloor
zceiling

Thing properties:
id
x
y
height
angle
type
skill1
skill2
skill3
skill4
skill5
ambush
single
dm
coop
friend
standing
strifeally
translucent
invisible
dormant
class1
class2
class3

Note that not every property is complete. For example, while sector specials are read, only a few are supported just yet. That should expand rapidly. I've been spending the last few days on the important visual properties - scaling, rotation, light color, fog, desaturation, vertex slopes... those all should be working fine now.

Share this post


Link to post
Chilly Willy said:

I'm working on the UDMF support. It's been coming along pretty well. If you use the latest from the repo, you have:

But which repo, if you don't mind me asking?

Share this post


Link to post

What linedef format? Doom or Hexen? Although all ports support the Doom format, editor support is virtually non-existent, if I am not mistaken.

Share this post


Link to post

As far as I can tell 3DGE uses Hexen special #'s for its UDMF format, but not all have been assigned, yet.

Although I may just not be clear where you're coming from in asking that.

Share this post


Link to post

If you use the doom namespace, you have all the doom specials. If you use hexen/zdoom/zdoomt namespace, you get the hexen/zdoom specials... which are not all supported yet. That's one of the things we'll be working on this weekend - getting all the most commonly used specials going. Note that we don't plan of full zdoom specials or properties, just the common ones needed to make it easier on modders.

Share this post


Link to post

Create an account or sign in to comment

You need to be a member in order to leave a comment

Create an account

Sign up for a new account in our community. It's easy!

Register a new account

Sign in

Already have an account? Sign in here.

Sign In Now
×