Doomworld Forums - Textual Map Format: Non-Democratic Discussion

Doomworld Forums : Powered by vBulletin version 2.2.5

Doomworld Forums > Classic Doom > Source Ports > Textual Map Format: Non-Democratic Discussion

Pages (2): [1] 2 »

Author

All times are GMT. The time now is 13:02.

Quasar
Moderator

Posts: 2203
Registered: 08-00

This is a new thread to non-democratically discuss details of a new attempt to define a port-independent textual map format. The syntax of the format is defined as follows:

code:
translation_unit := global_expr_list
global_expr_list := global_expr global_expr_list
global_expr := block | assignment_expr
block := identifier integer '{' expr_list '}'
expr_list := assignment_expr expr_list
assignment_expr := identifier '=' value ';'
identifier := [A-Za-z0-9_]+
value := integer | float | quoted_string | unquoted_string
integer := [+-]?[1-9]+[0-9]* | 0[0-9]+ | 0x[0-9A-Fa-f]+
float := [+-]?[0-9]+'.'[0-9]*([eE][+-]?[0-9]+)?
quoted_string := "([^"\\]*(\\.[^"\\]*)*)"
unquoted_string := [^{}();"'\n\t ]+

In case you can't follow BNF, you can have global assignments, and global blocks. There are no comments. Blocks are started with a keyword identifier and a numeric index. Here's an example of a minimally defined entity:

code:
linedef 0 { tag = 1; }

Current entities targeted for definition in the format include linedefs, sidedefs, things, sectors, and vertices. Undecided are how and if to specify blockmap, reject, and BSP, or to leave the construction of these resources to implementing ports at runtime. The basic specification will be a superset of the Hexen map format, providing those fields that were available in Doom and Hexen.

Compliant parsers will ignore all unknown keywords in global assignments, block-level assignments, and block headers. Compliant editors are not, however, required to maintain any such unknown information, since its continuing correctness cannot be assumed once the map has been edited (an implementing editor may choose to treat such unknown information as a parsing error, or to otherwise warn the user that information may be lost).

Information pertaining to thing types, thing specials, thing flags, line specials, line flags, and sector specials must be represented in a textual format. The text map format standard overseer (myself) will maintain an index of all such specials/types and will assign them to implementing source ports on request. Namespaces will be supported within thing type, thing special, line special, and sector special lists for those types/specials which do not conform to any previously existing standard. For example:

Line specials defined by Doom are in the unqualified namespace.
Line specials defined by Hexen are in the unqualified namespace.
Line specials defined by ZDoom are in namespace ZD.
Line specials defined by Eternity are in namespace EE.

An assignment involving such a namespace would appear as follows:

linedef 0 { special = EE:ExtraDataControlSpecial; }

Implementing source ports may zero any such fields which use values in namespaces they do not support. Implementing editors can easily have this information provided in their configuration. For both map reading and writing, the recommended implementation is to hash names to internal linedef type numbers and vice versa as necessary. Implementing editors may find it unnecessary to deal with numeric values at all and may prefer to work only with the implementation-independent string representation.

Justification:
Creating namespace-qualified names allows semantic treatment of all such values. For example, Eternity and ZDoom need not have matching internal linedef numbers for lines which implement the same feature. Each port can map the implementation-independent names to the internal special which most closely emulates the intended behavior. Mere numbers do not allow this sort of flexibility, and require extensive disambiguation in order for a unified cross-port list to be created.

The same argument applies to flags. 0x4000 means only what a port defines it to mean, whereas the textual word FRIEND can be given a standardized meaning and can be resolved to any internal binary flag, or even ignored by ports which do not support the feature of friendly monsters.

Flag specification would match the BEX/EDF preferred representation for flag lists, as in the following example:

code:
thing 245 { flags = EASY|NORMAL|HARD|AMBUSH; }

Storage within WAD files:
The issue of wad lump names must be circumvented for two reasons: Older editing tools will disassemble any map that contains non-standard lumps, and source ports will be unable to detect the format of such a map without inspecting its individual lumps. In order to achieve this, the new map format must have a different lump layout, as follows:

(HEADER) (any name from 1 to 8 characters long, usually zero length)
TEXTMAP (ASCII text lump containing all standard data for this map)
... (Any other lumps)
ENDMAP (Required end marker lump, zero length)

Implementing editors and source ports can now distinguish a textual map from a traditional binary map by testing the name of the first directory entry following the map header. Implementing resource editors will be capable of keeping all lumps for the map, even non-standard ones, together by properly scanning for the ENDMAP entry. Non-implementing resource editors will not recognize the sequence of lumps as a map, and are thus less likely to interfere with the order or presence of the lumps. Use of non-implementing resource editors is not recommended, however.

------------------------------------------------

Comments? Ideas? I will consider most ideas. But as the topic says, this is largely non-democratic. The basic syntax is definitely not up for debate except to whatever extent CodeImp requests changes. I am most interested in hearing from other source port authors on the practicality of various parts of the specification and what if any problems might arise. I am especially uncertain of what should be done with the BSP, blockmap, and reject resources. There are already at least three different BSP tree representations, for example, and accounting for all of them within the standard is really not possible. Leaving the resources in their original binary forms is possible, but not preferred due to limitations inherent to their formats (maximum number of segs, maximum BSP tree depth/size, blockmap limit, etc).

01-14-08 21:07 #

Profile || Blog || PM || Email || Homepage || Search || Add Buddy

Report to mod || IP || Edit/Delete || Quote

Graf Zahl
Forum Legend

Posts: 4194
Registered: 01-03

Sounds good.

And since you already decided on a multi-lump setup, I think nodes should just be handled as before, i.e. inserted as binary lumps after the map data. This data doesn't need to be edited so I see no problems here as long as it allows storing GL nodes without the hassle that's caused by GLBSP's way to do it.

Just one question: Are there any places where you would allow numerical representations of information that isn't numerical by nature?

Obviously of most concern here are

- line specials
- sector types
- thing numbers
- any kinds of flags (if I read this correctly, you won't allow this one, right?)

One more thing: Should this allow fractional vertex positions? This has always been one issue that bothered me while editing micro detail in some maps.

01-14-08 21:24 #

Profile || Blog || PM || Email || Homepage || Search || Add Buddy

Report to mod || IP || Edit/Delete || Quote

esselfortium
Senior Member

Posts: 1823
Registered: 01-02

Awesome to see that you two can agree on this. I know SlayeR and CodeImp aren't incredibly interested in supporting things as complex as this specifically for Eternity Engine, since it's not currently a very popular port for modding (yet), but if this was supported and encouraged by ZDoom/GZDoom, I imagine the likelihood of its implementation in Doom Builder 2 and Slade 3 being much higher. (In addition to the possibility of other sourceports adding support for it.)

Graf Zahl said:
One more thing: Should this allow fractional vertex positions? This has always been one issue that bothered me while editing micro detail in some maps.

I agree...this is the kind of thing that I can see being brushed off as unnecessary under the guise of "why would you ever want to make detail in units smaller than 1 unit? that's already way too small", but when working with diagonal lines that need to intersect each other, it's often impossible to do with the current 1-unit minimum size limit without having the lines become noticeably bumpy, innaccurate, and ugly where they intersect.

__________________
sl4.poned.com: Free Esselmusic
Esselmusic: The Album
Knee-Deep in KDiZD: Relive KDiZD, in doom2.exe: (WiP Page) (Doomworld Thread)

01-14-08 23:00 #

Profile || Blog || PM || Email || Homepage || Search || Add Buddy

Report to mod || IP || Edit/Delete || Quote

Quasar
Moderator

Posts: 2203
Registered: 08-00

Graf Zahl said:
Sounds good.

And since you already decided on a multi-lump setup, I think nodes should just be handled as before, i.e. inserted as binary lumps after the map data. This data doesn't need to be edited so I see no problems here as long as it allows storing GL nodes without the hassle that's caused by GLBSP's way to do it.

Just one question: Are there any places where you would allow numerical representations of information that isn't numerical by nature?

Obviously of most concern here are

- line specials
- sector types
- thing numbers
- any kinds of flags (if I read this correctly, you won't allow this one, right?)

One more thing: Should this allow fractional vertex positions? This has always been one issue that bothered me while editing micro detail in some maps.

I'm really unsure about allowing numeric values. The idea is to increase interoperability, allowing any port to load and at least display a map even if everything in it doesn't work. Numbers carry no contextual information, and thus you can only interpret them relative to your own feature set. For example, EE's 3DMidTex line flag value is not a line flag for the same purpose in ZDoom. But ZDoom wouldn't be able to tell that by looking at the binary value. If on the other hand it sees "3DMIDTEX" as a flag value, it can either 1) remap that to its own method of determining 3D line clipping if/when it has one, or 2) ignore it completely and have lines that are passable (not ideal but you can still explore the map, nothing crashes or malfunctions in unpredicable ways).

It would also complicate the parser to allow both value types for the fields in question. While not a major issue, CodeImp really wants this to be as simple as possible.

Fractional vertex coordinates absolutely should be allowed, even if inputting such locations is not supported by the editor. This has always been needed for better node building (Doom's BSPs are mathematically incorrect in their placement of seg splits, leading to phenomena such as slime trails).

01-15-08 01:01 #

Profile || Blog || PM || Email || Homepage || Search || Add Buddy

Report to mod || IP || Edit/Delete || Quote

Graf Zahl
Forum Legend

Posts: 4194
Registered: 01-03

Quasar said:

I'm really unsure about allowing numeric values. The idea is to increase interoperability, allowing any port to load and at least display a map even if everything in it doesn't work. Numbers carry no contextual information, and thus you can only interpret them relative to your own feature set. For example, EE's 3DMidTex line flag value is not a line flag for the same purpose in ZDoom. But ZDoom wouldn't be able to tell that by looking at the binary value. If on the other hand it sees "3DMIDTEX" as a flag value, it can either 1) remap that to its own method of determining 3D line clipping if/when it has one, or 2) ignore it completely and have lines that are passable (not ideal but you can still explore the map, nothing crashes or malfunctions in unpredicable ways).

Good to see that we agree on this. I absolutely don't like the idea of doing it - except maybe thing numbers. Otherwise we'd need to define a complete list of all things in all games that all ports would have to adhere to - and with that I have my doubts. ZDoom for example can't just change its names because they are accessible from various control lumps and scripts already so it'd result in a secondary set of names if there were differences.

It would also complicate the parser to allow both value types for the fields in question. While not a major issue, CodeImp really wants this to be as simple as possible.

Understandable. So numbers for flags, line specials and sector types are definitely out. That's good.

Fractional vertex coordinates absolutely should be allowed, even if inputting such locations is not supported by the editor. This has always been needed for better node building (Doom's BSPs are mathematically incorrect in their placement of seg splits, leading to phenomena such as slime trails).

For the nodes I wouldn't even bother to store any related information in text format - including the added vertices. It'd only create needlessly large files and would increase parsing time.

01-15-08 08:11 #

Profile || Blog || PM || Email || Homepage || Search || Add Buddy

Report to mod || IP || Edit/Delete || Quote

DaniJ
Forum Regular

Posts: 887
Registered: 08-03

I personally don't hold out much hope for something like this. No matter how I look at it, I keep coming back to the same conclusion; it will turn into an arms race whereby the port that requires feature X first gets to dictate its high level logic and users will begin to demand exacting compatibility when functionality is similar (again resulting in more racing). Non-democratic? Count me out in that case.

Furthermore, what about engines that are trying to diversify themselves from just DOOM/Heretic/Hexen/Strife?

Including BLOCKMAP, BSP and REJECT data in this seems wrong to me. They do not add additional data that is not implied by the core map data lumps (Doomsday for example would ignore all of them anyway). Is this supposed to be a new map format or just a replacement for the existing lumps using a textual representation?

I don't expect I'll be rushing to support this in Doomsday to be honest as it doesn't mesh with our goals imo.

Last edited by DaniJ on 01-15-08 at 12:45

01-15-08 12:40 #

Profile || Blog || PM || Search || Add Buddy

Report to mod || IP || Edit/Delete || Quote

Graf Zahl
Forum Legend

Posts: 4194
Registered: 01-03

I agree that node data should not be part of the text lump - but there's no reason not to think about it at all.

My suggestion would be to make GL nodes mandatory (if nodes are present) for any new map format anyway. They are a complete superset of normal nodes and every software renderer can use them with minimal code additions to filter out minisegs in a few places (like the BSP based sight checking code.) Allowing regular nodes is pointless IMO. Any node builder which wants to be able of handling a new map format would have to be changed and the only ones in active development and with flexible enough code are GLBSP and ZDBSP which both can create GL nodes.

But again: Nodes in text format - no, thank you!

01-15-08 12:55 #

Profile || Blog || PM || Email || Homepage || Search || Add Buddy

Report to mod || IP || Edit/Delete || Quote

fraggle
Super Moderator

Posts: 4355
Registered: 07-00

One thing to bear in mind: one of the advantages of defining a textual level format is that you can store it in a version control repository and do diffs and merges (imagine two people able to edit a level simultaneously and later merge their changes together automatically, using standard diff/merge tools). However, if half the lines get renumbered every time you delete a line, that advantage is lost.

It would be very useful to make the id values for lines, vertices, etc. persistent across saves. What I mean is, when you create line #6, for example, it will always be line #6, even if line #5 gets deleted. Don't stay tied to the "array index" system of ids that the original Doom level format has.

Considering it's a textual format, there's no reason to worry about limits on the number of these ids any more. You might even consider making the lines have names rather than numeric ids (even though such names would probably be automatically assigned numerically, eg. "line57").

Last edited by fraggle on 01-15-08 at 13:12

01-15-08 13:05 #

Profile || Blog || PM || Email || Homepage || Search || Add Buddy

Report to mod || IP || Edit/Delete || Quote

DaniJ
Forum Regular

Posts: 887
Registered: 08-03

In Doomsday 1.9.0 we do not use the original BSP, BLOCKMAP or REJECT data for anything other than checking for legacy DOOM.exe special effects. Our internal map representation is also evolving away from the original DOOM data structures and thus these lumps will be pretty much useless to us sooner or later anyway, even if we did want to use them.

It is my opinion that in order to maximize compatibility; map data should be stripped back to the absolute minimum in order to define it.

01-15-08 13:05 #

Profile || Blog || PM || Search || Add Buddy

Report to mod || IP || Edit/Delete || Quote

Yagisan
Mini-Member

Posts: 51
Registered: 01-06

I'm simply not going to support a text based level format at this time.

I've appropriated the Doomsday maploader for my engine, and you'd need to supply some reference tools for non-win32 systems before I'd give it a second thought.

With my Doomsday hat on, I'm unconvinced why we should change direction to support this map format, when we no longer use the old DooM format internally.

01-15-08 13:26 #

Profile || Blog || PM || Homepage || Search || Add Buddy

Report to mod || IP || Edit/Delete || Quote

Graf Zahl
Forum Legend

Posts: 4194
Registered: 01-03

With ZDoom the map format limitations are becoming increasingly problematic because a lot of information that's supposed to be part of the map structures has to be added with indirect methods. That doesn't make creating more complex maps easy. And considering that Eternity's Extradata is a different method to work around the same limitations I think these 2 ports have the biggest interest in a new map format.

However, since you apparently are moving in a completely different direction with Doomsday it's up to you if you want to support it or not. It's obvious that you have completely different design goals than ZDoom and Eternity. Most other ports will stay much closer to the original game though and have no interest in said generalization that appears to stand in your way regarding a new map format. What we need is a way to make the features easier to access so stripping down the map format is the complete opposite of what this is for.

@Quasar: Do you see any chance to continue this discussion in a more private place where some details can be worked out between people who are supportive of this idea?

@Fraggle: That idea is actually very good. Tracking version differences can always be a major hassle and this way it'd become significantly easier. Of course it'd also require additional work in the editor which I see as the biggest obstacle with this. The editor developer has to be willing to implement this.

Last edited by Graf Zahl on 01-15-08 at 13:37

01-15-08 13:28 #

Profile || Blog || PM || Email || Homepage || Search || Add Buddy

Report to mod || IP || Edit/Delete || Quote

DaniJ
Forum Regular

Posts: 887
Registered: 08-03

I kind of resent that last remark Graf. Just because I am not supportive of the idea doesn't mean I will try my up most to derail its development. Although we won't be supporting the format in Doomsday, that does not mean I don't understand the reasoning behind its conceptualization or the need for it from the POV of ZDoom/Eternity. Nor does that mean I cannot be objective about it. I would argue that my involvement might actually benefit development due to impartiality.

Still, its no skin off my nose, I've got enough work of my own to worry about.

01-15-08 13:41 #

Profile || Blog || PM || Search || Add Buddy

Report to mod || IP || Edit/Delete || Quote

Yagisan
Mini-Member

Posts: 51
Registered: 01-06

Graf Zahl said:
Most other ports will stay much closer to the original game though and have no interest in said generalization that appears to stand in your way regarding a new map format.

The only port that comes to mind as staying true to the original is Chocolate DooM. - You have this issue because you don't want to stay true to the original limits.

Graf Zahl said:
@Quasar: Do you see any chance to continue this discussion in a more private place where some details can be worked out between people who are supportive of this idea?

How about you leave it out in the open, this sounds like you can't stand criticism.

As a non-implementor, I do think storing this in a .wad file is asking for trouble. If I were you, I'd want to put it in it's own container.

01-15-08 13:51 #

Profile || Blog || PM || Homepage || Search || Add Buddy

Report to mod || IP || Edit/Delete || Quote

Graf Zahl
Forum Legend

Posts: 4194
Registered: 01-03

I want a discussion about the format not about the usefulness of the format. This has nothing to do with your opinion about this matter If you two are not interested, fine. That's really not my problem. But I'd still like to have a place where the details can be discussed between the people who want to make this happen.

01-15-08 14:41 #

Profile || Blog || PM || Email || Homepage || Search || Add Buddy

Report to mod || IP || Edit/Delete || Quote

Yagisan
Mini-Member

Posts: 51
Registered: 01-06

Graf Zahl said:
I want a discussion about the format not about the usefulness of the format. This has nothing to do with your opinion about this matter

I feel a contradiction here. I have given my opinion on the format, yet you are not interested. I'll quote again what you seem to have missed.

Yagisan said:
As a non-implementor, I do think storing this in a .wad file is asking for trouble. If I were you, I'd want to put it in it's own container.

I think you asked for yes-men, not a discussion. Now, why do you want to stick this into the .wad format, when it would be better served in it's own container ?

01-15-08 14:48 #

Profile || Blog || PM || Homepage || Search || Add Buddy

Report to mod || IP || Edit/Delete || Quote

Graf Zahl
Forum Legend

Posts: 4194
Registered: 01-03

Don't forget that this comes from a port developer who still uses WAD as the only container format for his port. And I have to agree that doing it differently will inevitably cause more problems than it will solve.

I don't have a problem with it and neither has ZDoom since it already uses WADs as a container-within-a-container for levels, even when using Zips.

At least this way it's easy to add map related data as additional lumps without convoluted schemes.

01-15-08 15:42 #

Profile || Blog || PM || Email || Homepage || Search || Add Buddy

Report to mod || IP || Edit/Delete || Quote

Quasar
Moderator

Posts: 2203
Registered: 08-00

DaniJ said:
I personally don't hold out much hope for something like this. No matter how I look at it, I keep coming back to the same conclusion; it will turn into an arms race whereby the port that requires feature X first gets to dictate its high level logic and users will begin to demand exacting compatibility when functionality is similar (again resulting in more racing). Non-democratic? Count me out in that case.

Furthermore, what about engines that are trying to diversify themselves from just DOOM/Heretic/Hexen/Strife?

Including BLOCKMAP, BSP and REJECT data in this seems wrong to me. They do not add additional data that is not implied by the core map data lumps (Doomsday for example would ignore all of them anyway). Is this supposed to be a new map format or just a replacement for the existing lumps using a textual representation?

I don't expect I'll be rushing to support this in Doomsday to be honest as it doesn't mesh with our goals imo.

This was designed with the idea of preventing an "arms race" by providing each developer with their own private namespace. This makes you free to do whatever you want within that space. As for "high level logic," there simply isn't any. There are not plans to add scripting or EDF-style data specification in the future. It's strictly a map format to which arbitrary fields can be added. As for what fields become part of the map standard, that is something that does have to be debated and decided upon. But I see no reason for anything that anyone needs to not be implemented. Ports will ignore fields they do not understand, and this does not add any significant overhead even if there are a million such keywords with respect to one port or another, because ignoring them all is done with the same bit of code.

The "non-democratic" part is meant to pertain to debates about the basic syntax and the wad lump organization. Suggesting that this should be stored in a container other than a wad (or anything else that Doom Builder supports) is self-defeating and would make this entire thing pointless. The last time we tried this, it died because of stuff like that, and I'm simply not going to entertain it this time around. Some things must be fixed in stone.

And most importantly, it has to become something that CodeImp will implement, and will be ABLE to implement with minimum trouble. This means, among other things:

1) No alternate file formats. The file format to be used isn't actually part of this specification anyway. It only assumes that the archive being used has a well-ordered directory, which unfortunately excludes straight zip files (but not wads inside zip files). Basically, if you can do a "GetNextLump" operation on the archive, it is supported by this specification.

2) No elaborate lump naming schemes. You should be able to name a map whatever you want. Doom Builder should not have to worry about breaking a map by renaming it, or having to respect some kind of correspondence between map lumps and their names.

3) Absolutely no external index lumps. CodeImp will absolutely not refer to any other lump in the wad to find or interpret any part of a map. He has made this perfectly clear. This is part of the problem with Eternity's ExtraData (although I have offered a solution to that issue already, one that may yet appear in the next release of EE).

4) No new entities. It will be restricted to defining the types of entities already used in Doom maps. There will not be blocks for scripts, thing types, etc., even though if this were a from-scratch format for a new game engine, such things would be logical and extremely useful to be able to include. They're outside the scope of this project, however.

I explicitly asked for the input of port authors and I intend to consider all of what is being said here. Expect the specification to see some major revisions. Particularly if/when we get any feedback from CodeImp, as he's the most important factor here.

Graf Zahl: Actually you do have a point as far as the thing names go... the problem is that it doesn't end up achieving the level of interoperability that I want. Example: EE and ZDoom call their particle fountains different things, even though they have the same doomednums and more or less identical behavior. Under the current scheme, we would end up with two identifiers, one in EE's namespace and one in ZDoom's namespace. Now, it would certainly be possible for implementing ports to cross-hash such names, but this requires a separate entry in some kind of lookup array for every such thing you want to remap. In time that would grow to be very large and convoluted. I fear the same thing may actually end up applying to all such names... and so I may need to go back to the drawing board on the entire names thing :/

One advantage of having names as at least an option however is the instant ability for Doom Builder to support custom-defined things by name. For example, if a User namespace were defined, and Eternity were to see User:MyNewBaron, it would interpret "MyNewBaron" as an EDF-defined thingtype name and would spawn an object of that mobjtype. This would work similarly for ZDoom's DECORATE.

The same thing cannot be said for doomednums because of the overlap they suffer between source ports. For example, Eternity's EDF reserves the doomednum range from 20000 to 32767 for user-defined thingtypes, whereas DECORATE seems to not define any such reserved range. So if a DECORATE thing in a ZDoom-oriented map in this format were to use the doomednum 7022, for example, it would in Eternity spawn a thingtype from Heretic (7xxx is the range EE uses for things translated from other games). Whereas User:DecorateThing would mean nothing to EE unless it were defined in a corresponding EDF, and it could spawn an "Unknown" object to denote that something unsupported is in that location.

The only real alternative to names is to create the disambiguated set of numbers I previously mentioned, and hash them to the "real" numbers when loading the map in a source port. This would require a giant and ever-expanding lookup table, however, and wouldn't have the benefit of supporting user-defined thing types since there is no way to account for them in the standard.

EDIT: Actually there's still a third alternative, and one I don't love. The third alternative is to go back to numbers, and to indicate in the map (using a global assignment statement) what source port the map is intended for. Then, an implementing port could:

1) Reject the map entirely when it sees that the intended port does not match. Seems kinda rude to me.

2) Process the map to remap any things/lines/sectors that don't match its own implementation -- ie, translating the map into a format it understands. This may result in an unplayable map, but it will contain the minimum number of glitches possible. This requires a lookup table for translating the numbers for each port you want to support.

3) Load the map naively and let everything malfunction (because after all, it's not designed for this port). This is what all ports currently do.

As for "non-Win32" reference tools, any code I write to deal with this map format will be 1) either public domain, BSD, or LGPL in license, and 2) completely portable with no system-specific code and no dependence on code from any particular Doom engine or editor. So I don't really see the point of asking for non-Windows-specific stuff when nothing being discussed here is Windows-specific in the first place :)

Last edited by Quasar on 01-15-08 at 16:35

01-15-08 16:23 #

Profile || Blog || PM || Email || Homepage || Search || Add Buddy

Report to mod || IP || Edit/Delete || Quote

Graf Zahl
Forum Legend

Posts: 4194
Registered: 01-03

My 2 cents:

Quasar said:
The "non-democratic" part is meant to pertain to debates about the basic syntax and the wad lump organization. Suggesting that this should be stored in a container other than a wad (or anything else that Doom Builder supports) is self-defeating and would make this entire thing pointless. The last time we tried this, it died because of stuff like that, and I'm simply not going to entertain it this time around. Some things must be fixed in stone.

Agreed. And the format as defined is certainly good enough so I'm in full agreement that no more discussion about the syntax is needed.

1) No alternate file formats. The file format to be used isn't actually part of this specification anyway. It only assumes that the archive being used has a well-ordered directory, which unfortunately excludes straight zip files (but not wads inside zip files). Basically, if you can do a "GetNextLump" operation on the archive, it is supported by this specification.

Again I fully agree. WAD is good enough for this and the ENDMAP closing lump makes this very easy to use - unless you insist on declaring WAD obsolete. I can remember the discussion about storing maps in ZDoom Zips. It was the same person (i.e. DaniJ) who had issues with it. It should be pointed out that he was the *only* person having issues with it.

2) No elaborate lump naming schemes. You should be able to name a map whatever you want. Doom Builder should not have to worry about breaking a map by renaming it, or having to respect some kind of correspondence between map lumps and their names.

Anything else would be counterproductive considering that several ports can handle arbitrary level names already. There's no need to make it more complex as the current definition.

3) Absolutely no external index lumps. CodeImp will absolutely not refer to any other lump in the wad to find or interpret any part of a map. He has made this perfectly clear. This is part of the problem with Eternity's ExtraData (although I have offered a solution to that issue already, one that may yet appear in the next release of EE).

... and they should not be needed with a proper format.

4) No new entities. It will be restricted to defining the types of entities already used in Doom maps. There will not be blocks for scripts, thing types, etc., even though if this were a from-scratch format for a new game engine, such things would be logical and extremely useful to be able to include. They're outside the scope of this project, however.

Pity. However, since the container design is open enough there's nothing that would prevent an implementation in a secondary lump after the main map data so it's not a serious issue.

Stuff about thing names/numbers

That's the same problem I am having with this whole mess.
I feared that this would be the one area that still requires some serious thinking...

01-15-08 16:52 #

Profile || Blog || PM || Email || Homepage || Search || Add Buddy

Report to mod || IP || Edit/Delete || Quote

DaniJ
Forum Regular

Posts: 887
Registered: 08-03

I lol'd.

The very first post after Quasar's well written definition as to what it does/doesn't/won't do includes a statement of intent to deviate from the proposed "standard". I'm sorry Quasar but I fear that for all your good intentions of this being truly port indifferent, that it will actually last all of five minutes.

It has been made perfectly clear to me, that the opinions/wishes/needs of some port authors will be completely disregarded in this discussion (as has been the case every time a new format proposal has come up).

I've already stated my position as far as Doomsday is concerned. This does not suit our needs and thus most likely it won't be supported.

So Graf, you can feel free to ignore anything I have said/will say on this topic.

01-15-08 17:38 #

Profile || Blog || PM || Search || Add Buddy

Report to mod || IP || Edit/Delete || Quote

Graf Zahl
Forum Legend

Posts: 4194
Registered: 01-03

DaniJ said:
I lol'd.

The very first post after Quasar's well written definition as to what it does/doesn't/won't do includes a statement of intent to deviate from the proposed "standard". I'm sorry Quasar but I fear that for all your good intentions of this being truly port indifferent, that it will actually last all of five minutes.

So far nothing definitive has been decided. Doesn't that mean that potential issues should not be brought up and discussed?

It has been made perfectly clear to me, that the opinions/wishes/needs of some port authors will be completely disregarded in this discussion (as has been the case every time a new format proposal has come up).

If you show a certain unwillingness for constructive discussion don't be surprised if your opinion is left out.

I've already stated my position as far as Doomsday is concerned. This does not suit our needs and thus most likely it won't be supported.

Fine. A new map format doesn't require Doomsday. You have made clear where you stand. Why don't you leave this to the people who *are* interested in seeing a new map format happen? With you I get the feeling you have a personal interest in making it fail.

(And no, I won't ignore your posts if they continue to be in this tone)

01-15-08 18:11 #

Profile || Blog || PM || Email || Homepage || Search || Add Buddy

Report to mod || IP || Edit/Delete || Quote

CodeImp
Moderator

Posts: 1084
Registered: 12-03

Shit, this was posted here yesterday and I'm already late! Don't post rash and fast people, sleep over it a night before you post :P :P

Ok, too much to comment on all of it, but I have been talking with Quasar on IRC on and off about this, so here is where I stand;

1) Do not define the namespace for each and everything, just define it once at the top like this:

code:
namespace = EE;

Maps are always designed with one sourceport in mind (or a lower level sourceport such as vanilla doom, boom, zdoom, etc. that is inherently supported by others) and if one sourceport wants to support the actions of another, it can just look at the namespace at the top and interpret/remap the actions appropriately.

2) No names! Because of the original doom map format, all sourceports and editors are based on the numbers and defining names for all of the numbers is a cumbersome task and not needed at all. As long as a namespace is defined, the numbers are NOT ambiguous and any sourceport and editor can interpret them properly.

EDIT: Oh, Quasar said numbers for bitfield flags (such as Thing flags) are a problem because he can't store them all in a 16 bit value or something (? don't ask) so I think we can agree on using names for bitfield flags only (it is acceptable as long as it stays a small number of names and not like 50 different ones per sourceport)

A little point on the base syntax; do we support weird constructs as values, like your first namespace example "special = EE:ExtraDataControlSpecial;" ? I just described why I disagree with this, so it is ONLY AN EXAMPLE HERE. It would make life easier for any program reading this if the values stay simple and logical. This example shows actually 2 values: namespace and special. If other sourceports are going to use other kinds of weird constructs, parsers may be confused and parse incorrectly. I think it should be clearly defined that each 'variable' only can be set to a 'primitive type' such as an integer, a float or a string in which case the string has no multiple values. For example, a string like customtype = MyCustomMonster; is acceptable, it is ONE value that refers to something, but customtype = mydatafile.dat:MyCustomMonster; is foolish IMO.

__________________
Doom Builder 2 is coming!

Last edited by CodeImp on 01-15-08 at 19:27

01-15-08 18:34 #

Profile || Blog || PM || Homepage || Search || Add Buddy

Report to mod || IP || Edit/Delete || Quote

Graf Zahl
Forum Legend

Posts: 4194
Registered: 01-03

CodeImp said:

[quote]
1) Do not define the namespace for each and everything, just define it once at the top like this:
code:
namespace = EE;
Maps are always designed with one sourceport in mind (or a lower level sourceport such as vanilla doom, boom, zdoom, etc. that is inherently supported by others) and if one sourceport wants to support the actions of another, it can just look at the namespace at the top and interpret/remap the actions appropriately.

That should work in most case. I'm assuming that there won't be that much ambiguity and what's left should not be too hard to handle

2) No names! Because of the original doom map format, all sourceports and editors are based on the numbers and defining names for all of the numbers is a cumbersome task and not needed at all. As long as a namespace is defined, the numbers are NOT ambiguous and any sourceport and editor can interpret them properly.

You mean for thing types, line specials and sector types?

For thing types namespaced editor numbers should work. Same for Doom-format linedef actions (if they should be supported at all.)
Hexen linetypes normally already have a name so that could be used. With sector types we have to be careful because the sector type is a mixture of a type and a bit mask. I always found this organisation less than optimal and maybe it's best to separate it so that maintenance becomes a bit easier for both editing and loading in the engine.

EDIT: Oh, Quasar said numbers for bitfield flags (such as Thing flags) are a problem because he can't store them all in a 16 bit value or something (? don't ask) so I think we can agree on using names for bitfield flags only (it is acceptable as long as it stays a small number of names and not like 50 different ones per sourceport)

Normally it should stay small - but it should be flexible enough to allow later addition of new flags if there's some need for it.

However, about line flags in Hexen format there's one thing that should be changed: Take out the trigger type and make it a separate field. If that one remains a bit field in the flags it will seriously limit future options.

A little point on the base syntax; do we support weird constructs as values, like your first namespace example "special = EE:ExtraDataControlSpecial;"
...

Simple motto: Keep it as simple as possible! I'm strictly against complex references in a map. If some port developer absolutely needs such convoluted stuff better force it into external control lumps the editor doesn't have to bother with. It also makes it easier to ignore for other ports.

How about defining how the current map features are to be represented. Then we'd have a start we could work with.

01-15-08 21:34 #

Profile || Blog || PM || Email || Homepage || Search || Add Buddy

Report to mod || IP || Edit/Delete || Quote

CodeImp
Moderator

Posts: 1084
Registered: 12-03

Graf Zahl said:
about line flags in Hexen format there's one thing that should be changed: Take out the trigger type and make it a separate field. If that one remains a bit field in the flags it will seriously limit future options.

I agree, that one definitely has to become a seperate field.

Graf Zahl said:
Simple motto: Keep it as simple as possible! I'm strictly against complex references in a map. If some port developer absolutely needs such convoluted stuff better force it into external control lumps the editor doesn't have to bother with. It also makes it easier to ignore for other ports.

Good, I hope Quasar agrees as well :) But I was just thinking: this still is the same problem when using names for bit flags, because they essentially are one big string that has yet to be parsed after the whole format has been parsed. However, if we strictly define how bit flags are to be formatted, then I suppose I can live with this as a single exception on the rule.

__________________
Doom Builder 2 is coming!

01-15-08 22:21 #

Profile || Blog || PM || Homepage || Search || Add Buddy

Report to mod || IP || Edit/Delete || Quote

DaniJ
Forum Regular

Posts: 887
Registered: 08-03

@Graf
I've already attempted to leave this discussion several times now. However, when people post troll'ish half-truths it makes it difficult to do so.

Its pretty much impossible for me to write this post now, as you have suddenly decided to start arguing my points back at me (seriously, wtf?).

Me opposed to constructive criticism?

Graf Zahl said:
@Quasar: Do you see any chance to continue this discussion in a more private place where some details can be worked out between people who are supportive of this idea?

Anyway, you are now trying to make me do what I said I wouldn't do (derail the thread) so that makes me think that in fact, it is you who have a vested interest in this failing.

"Right. That's the motion to get on with it passed with uh, one abstention. I now propose we go without further ado. May I have a second, therefore?"

01-15-08 22:32 #

Profile || Blog || PM || Search || Add Buddy

Report to mod || IP || Edit/Delete || Quote

CodeImp
Moderator

Posts: 1084
Registered: 12-03

Graf Zahl said:
How about defining how the current map features are to be represented. Then we'd have a start we could work with.

Proposal for the basic format This works for both Doom and Hexen type maps. You understand that everywhere you read "int", "float" or "string" it is supposed to have a value of that type instead. For the strings I have given the bit flags in comment (although comment here is just for the example, I don't really think comments are required in the map format, no sane person will make a map with a text editor anyway)

Not a big issue, but the reason why I didn't choose "s1" and "v1" is because I rather be clear about what my variables represent, hence "frontside" and "startvertex" instead.

__________________
Doom Builder 2 is coming!

Last edited by CodeImp on 01-15-08 at 23:05

01-15-08 22:53 #

Profile || Blog || PM || Homepage || Search || Add Buddy

Report to mod || IP || Edit/Delete || Quote

Quasar
Moderator

Posts: 2203
Registered: 08-00

Suggestions:

1) In Eternity, the activation parameters, which are stored in the "extflags" field accessible via ExtraData, include the following flags:

* CROSS
* USE
* IMPACT
* PUSH
* PLAYER
* MONSTER
* MISSILE
* REPEAT
* 1SONLY

This allows a single line to be activated by any combination of thingtypes, methods, etc. I understand that ZDoom does not currently support this. How can it be handled?

2) Z height should remain relative, as it was in Hexen. Otherwise, changing a sector floor height will require updating all z-positioned things inside the sector. Either that, or two different z fields could be created, one for relative height and one for absolute coordinate.

3) Hexen id's and Doom line tags have slightly different semantics. Tags default to 0, whereas id's default to -1 (lines without an id in Hexen cannot be addressed by a tag-style search at all, since they're never added to the line id lookup table). Should there be a separate "id" field for Hexen? This is how EE handles it internally, and even allows the id to be set explicitly via ExtraData, separately from the line tag.

4) While I'm aware this is just the basic specification, here are the new fields that EE currently supports for sectors:

* damageamount (int, damage every leveltime & damagemask tics)
* damagemask (int, determines frequency of damage)
* damagemod (string or int, method of damage (ice, fire, etc))
* floorterrain (string, name of TerrainType)
* floorangle (float, floor flat angle in degrees)
* floorxoffset (float)
* flooryoffset (float)
* ceilingterrain (string, name of TerrainType)
* ceilingangle (float)
* ceilingxoffset (float)
* ceilingyoffset (float)
* colormap_upper (string, colormap lump name)
* colormap_middle (string, colormap lump name)
* colormap_lower (string, colormap lump name)

I know ZDoom also has sector specials that can be set via scripting and activate on various actions, such as an actor entering the sector or hitting the sector floor. Seems like another natural field to add.

When we get into things like these, determining how and what DB will handle may get tricky. What sort of process will we use to work it out? While the standard that I specified allows for unknown keywords and fields, what recourse will we have if DB does not implement support for a given feature?

Is there perhaps a practical way for DB to support adding arbitrary information to entities? Such as a dialogue box that would pop up and have a text field into which the user could type a keyword/value pair? This would allow for adding fields that DB does not yet or will never support. I understand it adds more work to implementing this format, since things, lines, and sectors will be required to hold a list of string objects that describe their user-added fields. Seems fairly simple on the surface though.

Last edited by Quasar on 01-16-08 at 00:14

01-16-08 00:08 #

Profile || Blog || PM || Email || Homepage || Search || Add Buddy

Report to mod || IP || Edit/Delete || Quote

Moriarti
Newbie

Posts: 7
Registered: 02-04

In regards to flags, why represent them as a bitfield at all? Why not make each flag its own boolean field? This would simplify the map format so a reader wouldn't need to parse a flags string to extract each property out of it. Instead, it just sets a bit in its own internal flag(s) field(s) as it encounters a flag. Presumably any flag that doesn't appear as a field will default to off.

So the example from the opening post could be better written as:

code:
thing 245 { easy = 1; normal = 1; hard = 1; ambush = 1; }

01-16-08 02:05 #

Profile || Blog || PM || Search || Add Buddy

Report to mod || IP || Edit/Delete || Quote

Yagisan
Mini-Member

Posts: 51
Registered: 01-06

I don't believe any Doomsday dev intends to see this fail, but there certainly seems a lot of hate directed at them. The merits of the Doomsday .zip map format and the ZDoom .zip map format could be discussed in another thread, but those haven't taken off so well probably because after Doomsday made theirs, ZDoom made an incompatible one.

It's a shame you won't place it in another format, as assuming this is implemented, we will get users saying these maps say they work in Vanilla/Boom/etc yet can't be run by ports like Chocolate.

It does feel like the entire process is intended to be non-democratic, as thus far, I see 4 port authors (If I missed anyone my apologies), representing 4 projects have posted here. 2 for, 2 against, and the 2 against have been told to STFU even though they have valid concerns.

In summary:
Eternity: For
GZDoom: For
Doomsday: Against
Deng-NG: Against

As my port is on its way to being the least DooM-like, you would think I would support it - as it stands I can't, as it's been presented as a fiat from on high, and my concerns have been dismissed as bias.

At the very least, to reduce technical support requests considering your adamant use of the .wad format, I think the editor should also produce the standard binary lumps when targeting any features that already exist as of now, so if eg a new vanilla map is created, you get your text format, and Chocolate et al can still run it.

I wish you the best of luck with this, but I don't believe you will get the broad cross port support you want Quasar.

01-16-08 02:43 #

Profile || Blog || PM || Homepage || Search || Add Buddy

Report to mod || IP ||