Does UDMF allow custom blocks?

Doomworld Forums : Powered by vBulletin version 2.2.5

Doomworld Forums > Classic Doom > Source Ports > Does UDMF allow custom blocks?

Author

All times are GMT. The time now is 05:55.

printz
CRAZY DUMB ZEALOT

Posts: 8193
Registered: 06-06

http://www.doomworld.com/eternity/e...tuff/udmf11.txt

In UDMF, apart from the big five (thing, linedef, sidedef, sector, vertex), does the standard allow you to use your own block names? Is there a recommended naming convention like the user_ prefix is for fields? Do the compliant editors and ports tolerate and ignore them, or do they spawn errors and delete them on next save?

__________________
Available for download: Automatic Wolfenstein - Wolfenstein 3D bot
MBF configuration for WHackEd2

07-26-12 21:50 #

Profile || Blog || PM || Homepage || Search || Add Buddy

IP || Edit/Delete || Quote

Graf Zahl
Why don't I have a custom title by now?!

Posts: 7479
Registered: 01-03

There's really no point in custom blocks. ZDoom does not use them and I think that any port supporting custom data should put that into a separate lump belonging to the map to not let it get into the way.

Don't forget that each tool supporting UDMF would need the ability to handle these custom blocks and that's just too much to ask.

07-26-12 22:13 #

Profile || Blog || PM || Email || Search || Add Buddy

IP || Edit/Delete || Quote

Maes
I like big butts!

Posts: 10313
Registered: 07-06

Graf Zahl said:
Don't forget that each tool supporting UDMF would need the ability to handle these custom blocks and that's just too much to ask.

That's whay UDMF should be in XML format. ;-) That way, application schemas and namespaces could be used and...hmm...never mind...

07-26-12 22:25 #

Profile || Blog || PM || Homepage || Search || Add Buddy

IP || Edit/Delete || Quote

printz
CRAZY DUMB ZEALOT

Posts: 8193
Registered: 06-06

I'm asking as a developer, so I know to tell my level editor that the world is not divided into five hard-coded categories, and any extra entry classes are valid to load.

Graf Zahl:
Don't forget that each tool supporting UDMF would need the ability to handle these custom blocks and that's just too much to ask.

I plan to simply use tree views for any user-defined block, no hard-coded gimmicks unless a port really demands them. I don't really care if other editors delete this data or quit in error, I only care that I'm compliant to UDMF, because it doesn't state too much about this topic. I do care if Doom ports bang in error though, because maps exist to be played :(

__________________
Available for download: Automatic Wolfenstein - Wolfenstein 3D bot
MBF configuration for WHackEd2

Last edited by printz on 07-26-12 at 23:02

07-26-12 22:52 #

Profile || Blog || PM || Homepage || Search || Add Buddy

IP || Edit/Delete || Quote

Graf Zahl
Why don't I have a custom title by now?!

Posts: 7479
Registered: 01-03

Maes said:

That's whay UDMF should be in XML format. ;-) That way, application schemas and namespaces could be used and...hmm...never mind...

... and each program using UDMF still had to keep the data around somewhere.

07-26-12 23:08 #

Profile || Blog || PM || Email || Search || Add Buddy

IP || Edit/Delete || Quote

Maes
I like big butts!

Posts: 10313
Registered: 07-06

Graf Zahl said:
... and each program using UDMF still had to keep the data around somewhere.

Wrong. In pure Web Development philosophy, the "data" you're talking about should be distributed along with said XML-UDMF files, or at least be given an URI in the XML header. The programs using it should first and foremost be able to parse XML, apply XSD and execute XSLT, the rest is just details. Remember: extensible, human readable, flexible ;-)

...yeah, fuck that. In the end it's easier to go with a fixed-specs binary.

07-26-12 23:22 #

Profile || Blog || PM || Homepage || Search || Add Buddy

IP || Edit/Delete || Quote

DaniJ
Senior Member

Posts: 1958
Registered: 08-03

Although the spec doesn't state specifically that custom blocks are allowed, it does however say that not-understood data should be tolerated (provided its syntactically correct) by implementing applications.

To me this says:

If I write a UDMF editor I must respect unknown data in the file and include it in its original, unadulterated form* should I emit derivatives of said file.

If I write a UDMF consumer (e.g., a source port) then I must ignore all not-understood data and proceed normally.

* Keeping that data around isn't much of an issue; because we can't touch it we don't even need to store it - its in the original file :p

07-27-12 00:27 #

Profile || Blog || PM || Search || Add Buddy

IP || Edit/Delete || Quote

Quasar
Moderator

Posts: 5237
Registered: 08-00

That was the intent of the standard, yes - that custom blocks be allowed. I believe there's language in there that allows translating utilities such as editors to possibly discard such information if they don't understand it, but I'd have to check the text :P

07-27-12 15:02 #

Profile || Blog || PM || Email || Homepage || Search || Add Buddy

IP || Edit/Delete || Quote

DaniJ
Senior Member

Posts: 1958
Registered: 08-03

Perhaps the spec could be updated to clarify that? Ambiguity in the standard spec has the potential to consequently result in implementation diversification due to varying interpretation.

07-27-12 15:25 #

Profile || Blog || PM || Search || Add Buddy

IP || Edit/Delete || Quote

Maes
I like big butts!

Posts: 10313
Registered: 07-06

Kinda like VRML was supposed to be a completely portable and user-extensible language for describing 3D worlds? Only that in practice you were lucky if your program could read a subset of it (e.g. point pixel data for a 3D scanner software), let alone being able to perfectly parse and render everything....

07-27-12 15:39 #

Profile || Blog || PM || Homepage || Search || Add Buddy

IP || Edit/Delete || Quote

Quasar
Moderator

Posts: 5237
Registered: 08-00

My last suggestions for a new revision of the UDMF spec (mainly, making a slight revision to the allowed formats for floats to better match behavior of the C strtod function's allowed inputs), was largely poo-pooed, particularly by CodeImp who seems to want to treat the standard as final forever no matter what needs to develop.

This is particularly sad for Strife, considering we've discovered some new things about its map format due to the Chocolate Strife project (and btw the 25%fg linedef flag IS used in at least one IWAD map - The Factory: Manufacturing, on an entrance to a secret area). If we cannot add those items as part of the standard Strife namespace, then existing Strife maps cannot be translated in UDMF accurately.

If there's any chance of cooperation from external parties, I'd be more than happy to produce a draft for a new UDMF revision.

There are quite a few things about UDMF that I am unhappy with, but it's too late to change. For example I now feel that it is far too verbose, particularly with requiring fluff such as = signs and semicolons. These just bloat the data even further. UDMF maps are much larger than they should be IMHO.

07-27-12 17:41 #

Profile || Blog || PM || Email || Homepage || Search || Add Buddy

IP || Edit/Delete || Quote

Gez
Why don't I have a custom title by now?!

Posts: 9278
Registered: 07-07

Quasar said:
My last suggestions for a new revision of the UDMF spec (mainly, making a slight revision to the allowed formats for floats to better match behavior of the C strtod function's allowed inputs), was largely poo-pooed, particularly by CodeImp who seems to want to treat the standard as final forever no matter what needs to develop.

I think CodeImp doesn't want to have to change DB2 much. But given that there are other people working on DB2 and its branches now (boris, Kaiser, MXD), he might want to let things squeak by.

I'm personally favorable to updating the standard with the necessary features to losslessly express Strife maps.

Quasar said:
There are quite a few things about UDMF that I am unhappy with, but it's too late to change. For example I now feel that it is far too verbose, particularly with requiring fluff such as = signs and semicolons. These just bloat the data even further. UDMF maps are much larger than they should be IMHO.

ZDCMP2's TEXTMAP lump is about 20 megabytes. :)

Of course I'm not sure that map would fit in one of the existing binary formats at all. It's pretty huge.

07-27-12 18:24 #

Profile || Blog || PM || Search || Add Buddy

IP || Edit/Delete || Quote

Graf Zahl
Why don't I have a custom title by now?!

Posts: 7479
Registered: 01-03

Gez said:

I think CodeImp doesn't want to have to change DB2 much. But given that there are other people working on DB2 and its branches now (boris, Kaiser, MXD), he might want to let things squeak by.

Let's be clear about one thing:

If the standard is extended it means that all tools need to be extended as well. Otherwise the extension is worthless.

So no, I'm strictly against adding new constructs of any kind. The standard allows adding additional data as extra lumps which is far more versatile and far more robust with regards to tools. They don't have to be parsed and can simply be ported over.

Gez said:

I'm personally favorable to updating the standard with the necessary features to losslessly express Strife maps.

What features do you miss?

07-27-12 18:55 #

Profile || Blog || PM || Email || Search || Add Buddy

IP || Edit/Delete || Quote

Gez
Why don't I have a custom title by now?!

Posts: 9278
Registered: 07-07

Graf Zahl said:
What features do you miss?

What Quasar said:

Quasar said:
This is particularly sad for Strife, considering we've discovered some new things about its map format due to the Chocolate Strife project (and btw the 25%fg linedef flag IS used in at least one IWAD map - The Factory: Manufacturing, on an entrance to a secret area). If we cannot add those items as part of the standard Strife namespace, then existing Strife maps cannot be translated in UDMF accurately.

07-27-12 19:07 #

Profile || Blog || PM || Search || Add Buddy

IP || Edit/Delete || Quote

DaniJ
Senior Member

Posts: 1958
Registered: 08-03

If I can't losslessly convert all id tech 1 maps to equivalent UDMF namespaces then the specification undermines its whole purpose. Not revising the standard namespaces to include all data necessary to define Strife maps makes those namespaces pretty much unusable/worthless.

I have to say that removing a few '=',';' characters is pretty much pointless. If this was XML then I would understand wanting to remove the markup but the inclusion of such short syntactic grammars does not contribute much to the "bloat" of the format.

The verbosity of the format is inherent in the design of the "big five" elements themselves, which are pretty much the original id tech 1 data constructs transposed directly to keyname = value statements. One only has to look at the Vertex{} definition to understand how much bigger a UDMF format map is compared the original binary map format.

Still, I don't want to drag it all up again, I made my position clear during the initial format specification discussion(s).

As far as UDMF support in Doomsday is concerned, we intend translation only and certainly won't be promoting it as a distribution format.

20 megabytes for a single map's geometric data is frankly ridiculous.

07-27-12 19:09 #

Profile || Blog || PM || Search || Add Buddy

IP || Edit/Delete || Quote

printz
CRAZY DUMB ZEALOT

Posts: 8193
Registered: 06-06

DaniJ said:

20 megabytes for a single map's geometric data is frankly ridiculous.

Yeah but it can be compressed into a ZIP or 7Z and read that way by the port...

__________________
Available for download: Automatic Wolfenstein - Wolfenstein 3D bot
MBF configuration for WHackEd2

07-27-12 19:23 #

Profile || Blog || PM || Homepage || Search || Add Buddy

IP || Edit/Delete || Quote

DaniJ
Senior Member

Posts: 1958
Registered: 08-03

You still have to decompress it, which usually means allocating a "large" chunk of memory. One of the design considerations during the initial specification was that the format should not need a two stage parser and that one could stream-read. If I have to decompress it before I even begin parsing then what was the point of making that a design consideration?

Furthermore, simply throwing a generic compression algorithm at this data doesn't actually solve the verbosity problem. At best all that will happen is that recurring strings and values will be folded and whitespace RLE'd.

07-27-12 19:34 #

Profile || Blog || PM || Search || Add Buddy

IP || Edit/Delete || Quote

Quasar
Moderator

Posts: 5237
Registered: 08-00

Firming up the statement that custom blocks are allowed is not a change. It's not my fault the ZDoom team misinterpreted that part of the standard. After all, to me, this is very clear:

code:
Compliant parsers will ignore all unknown keywords in global assignments, 
block-level assignments, and block headers. Compliant parsers should attempt
to preserve as much of such information as is possible by using a flexible
mapping such as hashing.

What's an unknown block header? It's a top level entity starting a definition with braces, as clearly and unambiguously defined by the grammar:

code:
block := identifier '{' expr_list '}'

07-27-12 20:46 #

Profile || Blog || PM || Email || Homepage || Search || Add Buddy

IP || Edit/Delete || Quote

Graf Zahl
Why don't I have a custom title by now?!

Posts: 7479
Registered: 01-03

Gez said:

What Quasar said:

Adding new properties is a non-issue. More critical is adding more syntax.

DaniJ said:

20 megabytes for a single map's geometric data is frankly ridiculous.

20 megabytes is nothing in today's computing environment. BTW, ZDoom could theoretically stream-read the data from a compressed file. But what would be the point? Why make your life more complicated when in the end it saves nothing whatsoever. The map will need much, much more memory anyway after fully being set up.

07-27-12 22:33 #

Profile || Blog || PM || Email || Search || Add Buddy

IP || Edit/Delete || Quote

Blzut3
Member

Posts: 394
Registered: 06-04

DaniJ said:
One of the design considerations during the initial specification was that the format should not need a two stage parser and that one could stream-read.

Not sure if you're trying to say that you can't stream read UDMF, but I'm able to read UDMF (and similar formats) in a single pass with a strictly left-to-right parser.

07-27-12 23:07 #

Profile || Blog || PM || Email || Homepage || Search || Add Buddy

IP || Edit/Delete || Quote

DaniJ
Senior Member

Posts: 1958
Registered: 08-03

Graf Zahl said:
20 megabytes is nothing in today's computing environment.

Singularly perhaps. However when you think that a theoretical 32 map megawad would be around 640MB(!) you have to question the validity of that format.

BTW, ZDoom could theoretically stream-read the data from a compressed file.

With some compression algorithms, perhaps, however the irony is that the algorithms most suited to compressing this data can't be stream-read.

But what would be the point? Why make your life more complicated when in the end it saves nothing whatsoever. The map will need much, much more memory anyway after fully being set up.

Really?? ZDoom needs more than 20MB just for the runtime representation of the map geometry?

Blzut3 said:
Not sure if you're trying to say that you can't stream read UDMF, but I'm able to read UDMF (and similar formats) in a single pass with a strictly left-to-right parser.

My meaning was pretty clear, if I have to decompress that data using a large buffer then the fact the format itself can be stream-read is somewhat redundant.

07-27-12 23:34 #

Profile || Blog || PM || Search || Add Buddy

IP || Edit/Delete || Quote

printz
CRAZY DUMB ZEALOT

Posts: 8193
Registered: 06-06

Graf Zahl said:

Adding new properties is a non-issue. More critical is adding more syntax.

What are you and Quasar talking about regarding Strife, exactly? Quasar's post was fuzzy on details. Does Strife need new syntax or only new fields?

20 megabytes is nothing in today's computing environment

Oh yes it is. A single file is "nothing", but 30 of these? They become needlessly hard to handle especially through slow connections (not necessarily to Internet). Good thing it can be compressed.

__________________
Available for download: Automatic Wolfenstein - Wolfenstein 3D bot
MBF configuration for WHackEd2

07-27-12 23:34 #

Profile || Blog || PM || Homepage || Search || Add Buddy

IP || Edit/Delete || Quote

Quasar
Moderator

Posts: 5237
Registered: 08-00

Nobody's suggested new syntax so far AFAIK. Strife needs at least the 25% translucency linedef flag, and could stand to have some corrected alternate field names added for things that were previously misinterpreted (this isn't as critical obviously).

Custom blocks have always been allowed by the spec. Saying they aren't now is what would constitute a change to the syntax.

07-28-12 00:07 #

Profile || Blog || PM || Email || Homepage || Search || Add Buddy

IP || Edit/Delete || Quote

Graf Zahl
Why don't I have a custom title by now?!

Posts: 7479
Registered: 01-03

printz said:

Oh yes it is. A single file is "nothing", but 30 of these? They become needlessly hard to handle especially through slow connections (not necessarily to Internet). Good thing it can be compressed.

A binary level of that size (if it could be saved, that is, wouldn't be any smaller compressed. Lots of data needs lots of space, nothing can be done about it.

Uncompressed size is a non-issue in my book because you wouldn't ever transmit a file that way.
#

07-28-12 00:35 #

Profile || Blog || PM || Email || Search || Add Buddy

IP || Edit/Delete || Quote

Gez
Why don't I have a custom title by now?!

Posts: 9278
Registered: 07-07

DaniJ said:
20 megabytes for a single map's geometric data is frankly ridiculous.

Things aren't geometry.

But on the geometry side, that same map managed to get a ZNODES lump (uncompressed format binary format) of 4.5 megabytes. It is a big map. A full-contingent megawad of 32 maps like that one would give Eternal Doom a run for its money.

07-28-12 00:58 #

Profile || Blog || PM || Search || Add Buddy

IP || Edit/Delete || Quote

DaniJ
Senior Member

Posts: 1958
Registered: 08-03

I was only working with the figure you gave me with no idea as to how that was distributed among the various data lumps and object types.

The majority of a map's data is geometric. Given you now say that the node set constitutes 4.5 megabytes, I'm even more inclined to think that the geometry forms the lions share of the TEXTMAP lump.

07-28-12 01:10 #

Profile || Blog || PM || Search || Add Buddy

IP || Edit/Delete || Quote

Ladna
Junior Member

Posts: 245
Registered: 04-10

I guess there's a number of issues here:

- File Size/Read Buffer Size

You can use stream compression. ZLIB and LZO are well-suited for this purpose, and while they don't reach the insane ratios that LZMA (for example) does, they still go a long way (ZLIB is 3:1 on average).

- Memory Usage

This is just what happens when you remove limits, kind of UDMF's fault but not really.

- Lump Caching

It's probably not so hard to do [byte index of 1st chunk | byte offset of lump | chunk count] instead of [byte index of lump] if you're trying to keep memory usage down by purging cached lumps when you reach a given memory threshold. Stream compression tends to emphasize decompression speed and low CPU usage, so the additional read time is probably negligible, but hard numbers would be nice here.

- Extensions

While what I think probably doesn't matter much, I agree w/ Quasar that the standard should be extended to at least allow Strife to be represented by UDMF. That just doesn't seem that difficult to me, but I'm only guessing.

07-30-12 19:56 #

Profile || Blog || PM || Email || Search || Add Buddy

IP || Edit/Delete || Quote

CodeImp
Moderator

Posts: 1495
Registered: 12-03

Ladna said:
I guess there's a number of issues here:

- File Size/Read Buffer Size

You can use stream compression. ZLIB and LZO are well-suited for this purpose, and while they don't reach the insane ratios that LZMA (for example) does, they still go a long way (ZLIB is 3:1 on average).

- Memory Usage

This is just what happens when you remove limits, kind of UDMF's fault but not really.

- Lump Caching

It's probably not so hard to do [byte index of 1st chunk | byte offset of lump | chunk count] instead of [byte index of lump] if you're trying to keep memory usage down by purging cached lumps when you reach a given memory threshold. Stream compression tends to emphasize decompression speed and low CPU usage, so the additional read time is probably negligible, but hard numbers would be nice here.

All these are no issues IMO. Memory is not an issue today and (as has been said) the UDMF format (compressed or not) is streamable in case you want to save memory.

What does need discussion (IMO) is the Strife support. If this is only some field additions (which Quasar mentioned above) then I don't see the issue. And I definitely don't see why I would have ever been against it (what Quasar said). Link the post where I said that so I can back it up with reason, because I doubt I would be against something without any decent reasoning.

What I am quilty of is the lack of preserving the unknown "block headers"... Doom Builder doesn't keep any information outside the 'big five' (but note that it DOES preserve all unknown fields inside the big five blocks even when it doesn't know what the fields are, these appear in the 'Custom field' in the user interface). I have noted this and I think I should some day make a solution to keep all unknown blocks to write them back when saving. This is the UDMF guideline and it makes perfectly sense, so yea, noted.

07-30-12 21:42 #

Profile || Blog || PM || Homepage || Search || Add Buddy

IP || Edit/Delete || Quote

All times are GMT. The time now is 05:55.

Doomworld Forums > Classic Doom > Source Ports > Does UDMF allow custom blocks?

Show Printable Version |

Email this Page |

Subscribe to this Thread

Forum Rules:

You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

HTML code is OFF
vB code is ON
Smilies are OFF
[IMG] code is ON

< Contact Us - Doomworld >