Cross-platform Doom utilities.

[Maes]

I was thinking that instead of a source port in Java, maybe I should have started a map editor or a WAD management tool that could work cross-platform.

I mean, it's not like there's a lack of cross-platform Doom source ports, but there sure is a lack of cross-platform mapping utilities and WAD management tools: making maps and managing WADs nowadays pretty much means using Windows (sometimes running Win16 applications which will soon be unaccessible) or even DOS-era utilities. On other platforms (e.g. Mac) it can be even worse, with even the most recent utilities being entirely outdated as binaries.

How useful do you Doom folk think a cross-platform version of e.g. Doombuilder or XWE would be (assuming both could be ported to e.g. Java)?

Also, how useful would it be to convert certain of the classic command-line utilities to Java or a similar cross-platform language, assuming that it's no longer convenient to compile them from their (usually C) originals? As a proof of concept, the development version of Mocha Doom contains a fully functional port of QMusToMidi in pure Java, created by _D_.

[Gez]

Maes said:

I was thinking that instead of a source port in Java, maybe I should have started a map editor or a WAD management tool that could work cross-platform.

Why "instead"? Why not add the editor to Mocha itself? You'd have the most faithful "3D preview" mode ever. :)

An integrated port/editor all-in-one is something that has never been done before. Even though the concept is viable I think.

[Maes]

Gez said:

Why "instead"? Why not add the editor to Mocha itself? You'd have the most faithful "3D preview" mode ever. :)

Heh that's true... the benchmarking spinners that I (as of yet) include, sort of do just that: they use a "disembodied" renderer and a standard Doom level loader, without any connection to user input, game logic, etc.

Ofc, it would require certain bullet-proofing measures e.g. handling missing textures, and before that, there should be a fully capable BSP node-building tool in Java...so I think I'll start from that one first, also for the dual-purpose of making Mocha Doom self-reliant in node building.

[Quasar]

Maes said:

Heh that's true... the benchmarking spinners that I (as of yet) include, sort of do just that: they use a "disembodied" renderer and a standard Doom level loader, without any connection to user input, game logic, etc.

Ofc, it would require certain bullet-proofing measures e.g. handling missing textures, and before that, there should be a fully capable BSP node-building tool in Java...so I think I'll start from that one first, also for the dual-purpose of making Mocha Doom self-reliant in node building.

Perhaps you could be the one to pioneer my idea for a parallelized node builder, since you have experience with multithreading in Java.

My idea was, of course, to have a pool of worker threads which decide on a subtree to build at each split. Once a thread finishes building a subtree (reaches a leaf node, or subsector in other words), it would return to the pool to be grabbed by the next thread to generate a split.

You'd probably need some stochastic decision-making on whether it's worth it to assign another worker thread to a subtree or to finish that subtree with the thread currently processing the other side of the split once it finishes its current job, probably based on the number of segs remaining to be classified on that side and/or the depth in the tree. The number of worker threads is also a variable, though in a native solution at least you'd one one per core - I don't know how the JVM handles the scheduling of threads and whether or not it translates to hardware cores the same way as for purely native threads.

I believe that a parallelized node builder could be frighteningly fast, since the process of building subtrees is entirely recursive and therefore completely independent on a thread-by-thread basis.

[Graf Zahl]

As long as you don't need GL nodes, yes, it should be rather simple to parallelize it. But with GL nodes it all gets tricky because you need to synchronize both subtrees so that you can create proper GL segs for the splitter line and that would probably negate all the performance gain.

Concerning speed, of course it can only be faster by a factor of the number of CPU cores you have, meaning 4 with the vast majority of current systems - minus the multithreading overhead of course.

[Maes]

For such problems, I think a "pooled worker" approach combined with a workload threshold would work better e.g. to have both a "cookie jar" of tasks (node splits) to perform and assign to available workers, but also have a threshold beyond which each worker will have to finish its own work alone, without delegating to anywhere else.

Luckily, Java ever since v1.5 has got some pretty decent multithreading tools -not the least because of the work of Thomas W. Christopher and George K. Thiruvathukal, whose pre-v1.5 libraries eventually became part -modernized of course- of Java 5.

Compared to what's available in the standard libraries of other general-purpose languages not specialized for SMP, including even the would-be Java killer C#, they are simply stellar, so it's not too hard to orchestrate even complex start/stop/syncing conditions, if one has done similar work before.

[printz]

They say SLADE3 is cross-platform, and SLADE originated as a map editor. Hopefully SlayeR and Gez will add the map editor to SLADE3 so that we have an up-to-date cross-platform Doom map editor.

[andrewj]

Gez said:

An integrated port/editor all-in-one is something that has never been done before. Even though the concept is viable I think.

The Build and Jedi engines have the massive advantage of not requiring a BSP. The need to constantly rebuild the BSP (even if only part of it) when the user moved or added architecture would make in-game editing in a DOOM software port quite painful I think (for both user and programmer :-)

[natt]

If it's written in portable-ish C, easy to compile on different platforms, then that's cross-platform enough for me honestly. Of course, you'll never really reach that with GUI apps, because GUIs are always complicated. But tools like deutex and nodebuilding (of which there are multiple open source nodebuilders which are simple cross-plaform console apps), I don't think much would be gained by adding a java version.

[printz]

andrewj said:

The Build and Jedi engines have the massive advantage of not requiring a BSP.

How the hell do they manage that?

[Graf Zahl]

They use other methods to sort the geometry. You don't need a BSP for that although it makes things significantly faster.

With Build, many large Doom levels could not be handled because the time needed for sorting increases a lot if there's a lot of data. As an example, I once tested a Build based sorting algorithm in GZDoom. It worked fine for smaller maps like the IWAD's but when things like P:AR or Hellcore 2's MAP09 came into play performance broke down quite significantly. That's why you don't see the code anywhere. It's pointless to release.

[Maes]

natt said:

If it's written in portable-ish C, easy to compile on different platforms, then that's cross-platform enough for me honestly. Of course, you'll never really reach that with GUI apps, because GUIs are always complicated.

And in fact, the only truly cross-platform editor at this very instant is *drum roll* Yadex, which is pretty much stuck at vanilla Doom features and with all the ease of deployment that a "portable-ish C" app that you have to compile yourself has. Consider e.g. a modern, spiffy Power Mac or Ubuntu, Quad Core box: your best option to run a native editor is to run Yadex or, if you still have a PowerPC based Mac with Classic, one of the few editors that existed for it looooong ago. Or even resort to DOSBOX or a VM.

In comparison, the Windows crowd has a much better array of native options (well, that's also a natural consequence of the fact that Doom was primarily a game for IBM PC-compatibles, and much of the actual mod and tool development took part there), although some of the tools used are starting to show their age there as well.

As for the base tools themselves: I agree that, as far as console-level utilities are concerned, porting them to Java would not really add anything new, other than the ease of calling them from other Java applications without having to cross the external program invocation barrier or using JNI (which has to be handled differently per-OS). Or allowing them to be built-in in other Java tools (or engines...) ;-)

[hobomaster22]

http://www.doomworld.com/vb/doom-general/51477-new-doom-wad-editing-utility-in-development/

It's in Qt and I've had it running in both Unix and Windows but I don't have the time for it anymore. It has all the basic stuff though, and a lot of bugs.

[Maes]

From what I understand, the problem with all those separate tools, development environments etc. is that it's really hard to pick up development where one has left over, usually because of external dependencies, libraries, particular environments etc. so for that same reason I would not endorse developing e.g. in C# or Delphi (even though I love both languages and find that they have far more stable development environments than C, or less options, if you wish to see it that way). Just take a look at how difficult it is even for seasoned, hardcore programmers like entryway to set up everything needed to compile even recent and very well documented projects such as Chocolate Doom.

Java might not be perfect in every aspect, but it surely levels the differences between development environments and simplifies development and even deployment, with some careful packaging. Also, as I think I've amply demonstrated, "Java" does not necessarily mean shitty, slow enterprisey app which uses a fuckton of libraries, protocols, databases etc. ;-)

[hobomaster22]

I think the folks at Qt have made this very easy as they have their own IDE (Qt Creator) that loads their own projects, kind of like a cross platform Visual Studio. I've very simply been able to copy the project to my Unix machine and compile it with Qt Creator there. They have more recently supported Mac OSX but I don't own a mac so I can't say anything with respect to that.

I can see the appeal of Java, but I personally can't stand it as I've found GUI development for it to be particularly frustrating (I haven't done so recently, though). Also, JavaDoc is the most asinine format for API documentation I have ever seen. There are things I like about Java, but for most cases I would say no to it. But that's my opinion and I could probably just keep ranting on about it.

[Graf Zahl]

Maes said:

Java might not be perfect in every aspect, but it surely levels the differences between development environments and simplifies development and even deployment, with some careful packaging. Also, as I think I've amply demonstrated, "Java" does not necessarily mean shitty, slow enterprisey app which uses a fuckton of libraries, protocols, databases etc. ;-)

No, but Java means an incomplete and amputated language with far too many omissions to handle certain types of datasets without creating a mess of code. Your Mocha Doom is already a good example of what I don't like about Java. Far too many source files, most of which don't contain anything substantial.

I have been working with Java for the last 7 years and I developed a strong dislike for it during this time. For me C# is Java done right - unfortunately it's not platform independent enough.

That said, Java's platform independence makes Java apps awkward to use at best and plain shit at worst.

[sirjuddington]

SLADE3 runs on windows, linux and mac. The map editor is currently being developed too.

Also, SLADE 2.0 should compile in linux, so there's another map editor option there (until I get the 3 version done). Afaik it's a little buggy on linux, but works sufficiently well.

[Maes]

Graf Zahl said:

Far too many source files, most of which don't contain anything substantial.

That may be more a matter of personal taste. I am -and you are, I suppose- aware that you can have inner classes, interfaces etc. and put almost everything that a package contains in one file, if so you wish but if something doesn't require access to some containing class's fields (e.g. the Menu's items) or it's not used anywhere else but inside a particular class (e.g. PIT iterators or column/span functions), I prefer leaving them in a separate file, even if it's just 2-3 fields or a small amount of info like the definition of vertex_t, for the sake of my sanity and concern separation. Someone else may prefer packing them all together, but it's a matter of taste.

I've worked intensively with C# too, and there are some things I like about it that I wish existed in Java too (partial classes, unsigned types, a cleaner getter/setter system (properties) etc.) but it's not without its flaws, first and foremost the uneven look & feel between different versions of Windows (that caught me by surprise, and even embarassed me in a couple of situations, where I assumed that even though I developed under XP, the look and feel would stay consistent accross 7/Vista. They didn't. Badly. Much worse than any Java/AWT/Swing horror story you've heard).

The other thing is a lack of a free IDE that's on par with Eclipse, in terms of features. Sharpdevelop came reasonably close, but I missed some features like e.g. OTF single-class compilation and dynamic code completion.

That, and the fact that even if it's a managed language M$ didn't give a rat's ass about supporting it on other platforms (Mono? Gimme a break...) make me reluctant to suggest that anyone learns only C#, unless he plans on going full M$ for the rest of his life. Which is a pity, because otherwise the language is really a "Java the way it should be".

[hex11]

Maes said:

Java might not be perfect in every aspect, but it surely levels the differences between development environments and simplifies development and even deployment, with some careful packaging. Also, as I think I've amply demonstrated, "Java" does not necessarily mean shitty, slow enterprisey app which uses a fuckton of libraries, protocols, databases etc. ;-)

Here's an example of a good Java program (great game, btw):
http://sourceforge.net/projects/flapp/

Personally I really like the xwadtools package, due to its shell-based nature, but it could use some work (update deutex, fix some bugs...)

If you can manage to write a portable DoomBuilder-style editor, that would be cool. Then people would at least have the choice between that and DEU-style editor. If you're also going to write tools, my only recommendation is to allow their functionality be accessed from the shell, whenever possible.

[tempun]

hex11 said:

If you're also going to write tools, my only recommendation is to allow their functionality be accessed from the shell, whenever possible.

Like Boom's CLED?

[Bucket]

I think the community is hurting for a 21st century WAD editor. Not only is WIN16 a rotting corpse, but the tools themselves are pretty obtuse. You can brick an entire PWAD just because you imported a .MUS incorrectly.

[esselfortium]

Bucket said:

I think the community is hurting for a 21st century WAD editor. Not only is WIN16 a rotting corpse, but the tools themselves are pretty obtuse. You can brick an entire PWAD just because you imported a .MUS incorrectly.

Have you tried Slade 3.0?

[Jodwin]

Bucket said:

You can brick an entire PWAD just because you imported a .MUS incorrectly.

Ever heard of backups?

[hex11]

tempun said:

Like Boom's CLED?

Yeah, that's the general idea. Might sound dumb in Windows (crusty tools from the bad old DOS days?) but the Unix shell is where a lot of its appeal lies, and this is even more apparent in Plan9 (more powah!)

[Graf Zahl]

Maes said:

The other thing is a lack of a free IDE that's on par with Eclipse, in terms of features. Sharpdevelop came reasonably close, but I missed some features like e.g. OTF single-class compilation and dynamic code completion.

Eclipse is one of the reasons why I hate working with Java. A horrible piece of bloatware. For C# I use Visual C# Express which really suits my working style.

[sirjuddington]

Graf Zahl said:

Eclipse is one of the reasons why I hate working with Java. A horrible piece of bloatware. For C# I use Visual C# Express which really suits my working style.

This is true. Has to be the worst IDE I've ever had the 'pleasure' of dealing with. For Java, Netbeans is much better.

[Maes]

Graf Zahl said:
Eclipse is one of the reasons why I hate working with Java. A horrible piece of bloatware[/i]

So, by that reasoning, I could say that C is a horrible piece of shit by picking on a particular IDE or compiler that has some pet peeves of mine.

SlayeR said:

This is true. Has to be the worst IDE I've ever had the 'pleasure' of dealing with. For Java, Netbeans is much better.

Oh wow...bitter much? When was the last time you guys tried it? 2006?TBQH, I didn't find it "hot stuff" either back then when I was still working with Borland's JBuilder (which was quite bloated itself), but the versions that have been out since about two years like Ganymede and Helios (and I employ in my workplace, where actually using Eclipse was a requirement) are pretty decent. Let alone that Eclipse is pretty much the only way you can do any serious Android development, unless you really enjoy the command line.

I didn't try NetBeans though, which I feel I should.

[hex11]

Life was a lot simpler back in the day of DOS Borland IDEs. Well, I switched to qedit when I found it, but it was still simple, lean & mean. Screw it, I'm gonna wipe my disk and dual boot FreeDOS and Plan9. :P

[Maes]

hex11 said:

Life was a lot simpler back in the day of DOS Borland IDEs.

Heh, those IDEs were the motherfucking ass, back in the day. Somehow Borland just "got it right" and blew M$ or any other competitor out of the water, when it came to textmode "graphical" IDEs, no matter what language they were meant for. C/C++? Sure. Turbo Pascal? You betcha. Turbo Basic? Well maybe not that one, as I felt that M$'s Quick Basic (and even QBasic) was vastly superior. TB looked neglected in comparison (at least the old version I had).

[sirjuddington]

Maes said:

Oh wow...bitter much? When was the last time you guys tried it? 2006?TBQH, I didn't find it "hot stuff" either back then when I was still working with Borland's JBuilder (which was quite bloated itself), but the versions that have been out since about two years like Ganymede and Helios (and I employ in my workplace, where actually using Eclipse was a requirement) are pretty decent. Let alone that Eclipse is pretty much the only way you can do any serious Android development, unless you really enjoy the command line.

I didn't try NetBeans though, which I feel I should.

Yeah the Android thing is unfortunate. I was referring to recent versions of eclipse. It's not a lack of good features I was referring to, moreso an over-abundance of useless ones, making it near impossible to find anything you want in the menus/settings. It reminds me a little of deepsea (though not quite as poorly laid out :P)

Bucket said:

I think the community is hurting for a 21st century WAD editor. Not only is WIN16 a rotting corpse, but the tools themselves are pretty obtuse. You can brick an entire PWAD just because you imported a .MUS incorrectly.

Win16? XWE is getting pretty old now and that's hardly a win16 program. Works fine in windows 7 afaik. Also SLADE3 has already been mentioned, which is a real '21st century wad editor' in that it's currently in active development, and does pretty much everything wintex/xwe can.