kb1 Posted February 4, 2017 KBDoom will eventually implement UDMF. Has anyone considered a specification for compressing UDMF? Not a general Huffman-based method. I'm talking about tokenizing common keywords, while leaving custom keywords as-is. It seems it could be a relatively simple algorithm, with decent space-saving properties. It might even parse faster. Just a thought. I'm just trying to gauge interest and support for such an idea. 0 Share this post Link to post
Gez Posted February 4, 2017 kb1 said:Has anyone considered a specification for compressing UDMF? Not specifically. UDMF compresses extremely well in zips, so the idea is to use zips. Several ports can load zipped files already so... The UDMF specs were designed to be as simple as possible to implement; that's why it doesn't feature, for example, nested blocks. Putting some sort of lump-specific compression (rather than generic archive compression like with a pk3) would complicate the specs needlessly. 0 Share this post Link to post
Altazimuth Posted February 4, 2017 printz said:But still, why not just use the ZDoom specials in a common namespace? ZDoom (and derivatives) has some specials I can't see Eternity supporting any time soon (or ever), such as: Player_SetTeam, Team_Score, Team_GivePoints, SetGlobalFogParameter, FS_Execute, Sector_SetPlaneReflection, Sector_SetContents, etc. Even some specials that we should support we don't atm. 0 Share this post Link to post
kb1 Posted February 4, 2017 Gez said:Not specifically. UDMF compresses extremely well in zips, so the idea is to use zips. Several ports can load zipped files already so... The UDMF specs were designed to be as simple as possible to implement; that's why it doesn't feature, for example, nested blocks. Putting some sort of lump-specific compression (rather than generic archive compression like with a pk3) would complicate the specs needlessly.Not lump-specific compression, "tokenizing" of keywords, similar to how old dialects of BASIC tokenized keywords to normalize syntax, shrink size, and speed up execution. Something like IE's MHTML web page tokenizing. Again, just getting some feedback on an idea. Thanks. 0 Share this post Link to post
dpJudas Posted February 4, 2017 kb1 said:Not lump-specific compression, "tokenizing" of keywords, similar to how old dialects of BASIC tokenized keywords to normalize syntax, shrink size, and speed up execution. Something like IE's MHTML web page tokenizing. Again, just getting some feedback on an idea. Thanks. Why spend time on implementing an inefficient compression method when standard deflate is much better and easier to add? You can literally add deflate decompression by including one extra source file (miniz) into your project and 10 lines of code to decompress the lump. As for speed gains, if the tokenization part is slow there's something very wrong in how the tokenizer is implemented. There are algorithms there that allow this to be done in a single for loop over all the characters. 0 Share this post Link to post
Graf Zahl Posted February 4, 2017 kb1 said:KBDoom will eventually implement UDMF. Has anyone considered a specification for compressing UDMF? No. It's useless. It only complicates matters and the speed gain is insignificant. Loading the largest available map on my computer (ZDCMP2) takes 0.4 seconds to parse in GZDoom, but then loading all the needed data takes twice as long. Here's the timing I got: https://forum.zdoom.org/viewtopic.php?p=733709#p733709 And parsing the keywords is the least to worry, even then parsing the numeric values into actual binary values is by far the largest chunk of the parse time. dpJudas said:As for speed gains, if the tokenization part is slow there's something very wrong in how the tokenizer is implemented. There are algorithms there that allow this to be done in a single for loop over all the characters. Correct. ZDoom uses its existing Name class to tokenize the UDMF input and that thing is already very efficient at converting a string into a token that can then be used in a 'switch' statement for dispatching. On the other hand, tokenizing the UDMF itself adds a completely new layer of complexity to any tool that needs to write the format and the decision to go pure text was to a large part motivated by making this task as easy as possible, even if it means some minor efficiency loss. Overall, the most costly operations that occur with UDMF are not related to parsing (see my table), the second most expensive part is indeed loading the data from the HD, and zipping helps a lot more here than pre-tokenization. UDMF normally compresses to ca. 1.5x the size of binary maps. Altazimuth said:ZDoom (and derivatives) has some specials I can't see Eternity supporting any time soon (or ever), such as: Player_SetTeam, Team_Score, Team_GivePoints, SetGlobalFogParameter, FS_Execute, Sector_SetPlaneReflection, Sector_SetContents, etc. Even some specials that we should support we don't atm. Some of these are not even in ZDoom but Zandronum exclusives, and Sector_SetContents is deprecated anyway, as its only purpose is to load the few Vavoom maps out there. There's no need to make those part of a unified namespace. 0 Share this post Link to post
scifista42 Posted February 4, 2017 Graf Zahl said:[Compressing UDMF is] useless. It only complicates matters and the speed gain is insignificant. What about the gain of reduced filesize? EDIT: I've just read Gez's post, but still. 0 Share this post Link to post
Graf Zahl Posted February 4, 2017 It would shorten the uncompressed filesize but make very little impact on compressed size, because the keywords appear so frequently in the text that they are normally compressed away easily. 0 Share this post Link to post
RestlessRodent Posted February 4, 2017 If size on the disk is your major concern, you can take a pre-existing compression algorithm and then modify it so that it is optimized for UDMF. For example you could take deflate and use an alternative fixed huffman tree that is optimized for UDMF's text, however you would need to write new utilities just to compress and decompress it. Maintaining these utilities would increase cost and complexity of programs, and in most cases mod authors would not be using it anyway because it just complicates things. 0 Share this post Link to post
Graf Zahl Posted February 4, 2017 RestlessRodent said:Maintaining these utilities would increase cost and complexity of programs, and in most cases mod authors would not be using it anyway because it just complicates things. And that's why I wouldn't even think about considering it. Just look at the history of compressed nodes how such things go. Only after I added an uncompressed version to ZDBSP they gained some traction. 0 Share this post Link to post
Coraline Posted February 4, 2017 3DGE got UDMF recently, but it is still WIP. 0 Share this post Link to post
Altazimuth Posted February 5, 2017 Chu said:3DGE got UDMF recently, but it is still WIP. Ooh that's exciting! I wish you the best of luck in getting it in. On a different note, I have updated the EE UDMF spreadsheet so it now lists all the properties I could find, their type, what they bind to, origin, and which ports support them. Check the "Properties" tab. 0 Share this post Link to post
Graf Zahl Posted February 5, 2017 LOL who cares... :D:D:D Why did you even list Vavoom? It only copied what it got from ZDoom. In any case, if I find some time this evening I'll make a list of ACS functions so that this part can be checked as well. Compiling these 3 lists into a spec shouldn't be too hard then. 0 Share this post Link to post
Gez Posted February 5, 2017 I'm glad to see UDMF finally starting to get some traction. Now that's a motivation to start creating more editor configs. 0 Share this post Link to post
Altazimuth Posted February 5, 2017 Graf Zahl said:LOL who cares... :D:D:D Why did you even list Vavoom? It only copied what it got from ZDoom. In any case, if I find some time this evening I'll make a list of ACS functions so that this part can be checked as well. Compiling these 3 lists into a spec shouldn't be too hard then. I tend to inject some levity into my work, helps me enjoy myself a bit more; this usually manifests itself into jokey comments. You'll find EE's ACS functions listed here. I also recently added compat for a couple line specials that had different versions when called from ACS, and they're here. 0 Share this post Link to post
Graf Zahl Posted February 5, 2017 Thanks. With all this stuff combined I should be able to write something comprehensive. 0 Share this post Link to post
printz Posted February 5, 2017 So, developer side, should I add anything special to the new common namespace? Or simply treat the common namespace the same as Eternity's? While also filtering out undefined stuff, sure, hopefully. 0 Share this post Link to post
Graf Zahl Posted February 5, 2017 Can't say yet. I think all that will be needed is filtering out some of the action specials and namespace-check the UDMF properties and maybe a few other items. ACS cannot be filtered anyway in case you load an add-on that's using other things. 0 Share this post Link to post
Graf Zahl Posted February 5, 2017 I just ran through the ACS functions. I compiled a list of all that appear to be in both engines but this is hardly something conclusive, because Eternity's ACS VM is so radically different that I can't see which feature, aside from the listed functions is supported and which isn't. I'm leaving this part out for now. Where does it translate the ACS bytecode into what the VM runs on? I couldn't find because looking for 'behavior' is rather futile. 0 Share this post Link to post
Coraline Posted February 5, 2017 Graf Zahl said:LOL who cares... :D:D:D Why did you even list Vavoom? It only copied what it got from ZDoom. Hey now, 3DGE is still relevant!! ;-) Should I replace Vavoom with Eternity then? Or another standard? 0 Share this post Link to post
chungy Posted February 6, 2017 If disk space is a concern, you could just use file system compression, which is transparent to applications. NTFS on Windows supports it, HFS+ on Mac OS X supports it, several file systems (eg, btrfs and zfs) on Linux support it. 0 Share this post Link to post
Gez Posted February 6, 2017 Chu said:http://m.quickmeme.com/img/e3/e3346397890b6bf8b3667f4c75039e6b1c270496ac5733c4b4eca12c758c0c27.jpg Hey now, 3DGE is still relevant!! ;-) That wasn't aimed at you. Do go there and do check the properties tab: Altazimuth said:On a different note, I have updated the EE UDMF spreadsheet so it now lists all the properties I could find, their type, what they bind to, origin, and which ports support them. Check the "Properties" tab. 0 Share this post Link to post
Rachael Posted February 6, 2017 Wow - talk about bad timing. >_< But I got the joke now... 0 Share this post Link to post
Chilly Willy Posted February 9, 2017 I'm working on the UDMF support. It's been coming along pretty well. If you use the latest from the repo, you have: Support for uncompressed GL v1/v2/v3 znodes. Sector properties: heightfloor heightceiling xpanningfloor ypanningfloor xpanningceiling ypanningceiling xscalefloor yscalefloor xscaleceiling yscaleceiling rotationfloor rotationceiling gravity texturefloor textureceiling lightlevel lightcolor fadecolor special id desaturation Linedef properties: id v1 v2 special arg0 arg1 arg2 arg3 arg4 sidefront sideback blocking blockmonsters twosided dontpegtop dontpegbottom secret blocksound dontdraw mapped passuse repeatspecial playeruse impact Sidedef properties: offsetx offsety offsetx_bottom offsety_bottom offsetx_mid offsety_mid offsetx_top offsety_top scalex_bottom scaley_bottom scalex_mid scaley_mid scalex_top scaley_top texturetop texturebottom texturemiddle sector Vertex properties: x y zfloor zceiling Thing properties: id x y height angle type skill1 skill2 skill3 skill4 skill5 ambush single dm coop friend standing strifeally translucent invisible dormant class1 class2 class3 Note that not every property is complete. For example, while sector specials are read, only a few are supported just yet. That should expand rapidly. I've been spending the last few days on the important visual properties - scaling, rotation, light color, fog, desaturation, vertex slopes... those all should be working fine now. 0 Share this post Link to post
Gez Posted February 9, 2017 Chilly Willy said:I'm working on the UDMF support. It's been coming along pretty well. If you use the latest from the repo, you have: But which repo, if you don't mind me asking? 0 Share this post Link to post
Coraline Posted February 9, 2017 He's the other lead developer on 3DGE with me :-) 0 Share this post Link to post
Rachael Posted February 9, 2017 Gez said:But which repo, if you don't mind me asking? https://github.com/3dfxdev/hyper3DGE This one. :P 0 Share this post Link to post
Graf Zahl Posted February 9, 2017 What linedef format? Doom or Hexen? Although all ports support the Doom format, editor support is virtually non-existent, if I am not mistaken. 0 Share this post Link to post
Rachael Posted February 9, 2017 As far as I can tell 3DGE uses Hexen special #'s for its UDMF format, but not all have been assigned, yet. Although I may just not be clear where you're coming from in asking that. 0 Share this post Link to post
Chilly Willy Posted February 9, 2017 If you use the doom namespace, you have all the doom specials. If you use hexen/zdoom/zdoomt namespace, you get the hexen/zdoom specials... which are not all supported yet. That's one of the things we'll be working on this weekend - getting all the most commonly used specials going. Note that we don't plan of full zdoom specials or properties, just the common ones needed to make it easier on modders. 0 Share this post Link to post