Modified prboom+ for producing videos

I was wondering, if there was any interest amongst doomers for some enhanced recording features in a source port. I'm currently diddling around with a prboom-plus modification (so it has the ability to playback any demo prboom-plus can) with the following features:

1) frame-exact non-realtime video recording from -timedemo
2) sound recording (sfx+music) perfectly synced to video recording
3) ability to encode video in place (so no huge uncompressed intermediate file)

So you can video capture from a LMP with absolutely no chance of frame dropping, audio out of sync, or anything bad, regardless of how fast your computer is. As far as i know, no current source port implements this fully.

here is nuts3.wad being played in full 1080p on a 7 year old pentium 4:

The mess of code I've made isn't release quality. The video recording is pretty simple (it's just a minor extension to the -avishot feature already in prboom-plus), and the audio (sfx) recording is pretty simple, but getting properly synched music in there as well is more complicated (and involves dropping sdl_mixer off a bridge). If there's some interest, I'll get it into a releasable form...

natt said:

but getting properly synched music in there as well is more complicated

Even if timidity is used?

Do want. Options for picking quality and resolution? Into avi or multiple formats? Maybe an option for direct to youtube format. Looks to be most awesome.

Definitely want it.

Catoptromancy said:

Do want. Options for picking quality and resolution? Into avi or multiple formats? Maybe an option for direct to youtube format. Looks to be most awesome.

Resolution and original rendering quality is entirely controlled through the game's settings interface. GL vs software mode, filtering, high res texture replacements, etc etc; all works exactly as you would expect. If you mean quality as far as compression quality, you control that through the compressor. Right now I have two video output setups:

1) Video for windows AVI writer. You can use any video codec you have installed and setup for VFW. I must say I don't particularly like this. It's windows only so I have to have a second setup available regardless, and while it's reasonably easy to use, an inexperienced user can easily choose bad settings or formats here (especially if he doesn't have any good VFW compressors installed) and make junk videos.

2) Raw output to file. RGB or YUV format. The file isn't seeked, so you can use mkfifo, pipe it to your compressor, and then there won't actually be any intermediate file on your hard disk. This is a pain on windows as there isn't actually a mkfifo. (I created one, but it's not quite a drop in replacement.)

I'm leaning towards 2) really. I don't want to create a lot of additional dependencies (like libavcodec or anything), but I want something flexible that's not arbitrarily limited. Just need to work on the "easy to use" bit.

entryway said:

Even if timidity is used?

I've been working with fluidsynth actually. (And yes, I have it looping properly). Honestly, it's mostly that I want the code to be clean enough for you to accept as a patch, because I don't have the energy to maintain a fork for very long. And it's kind of a mess right now. Anyway, here's sort of what I've been thinking about:

First of all, SDL_Mixer is gone. The only feature used is the midi/mp3/ogg play. For midi play, we need a soft synth in userspace (so not like bassmididrv, because there's no way to control its output). Bassmidi is incompatible with GPL so I can't bring it in. I'm using chocolate doom's OPL2 emulator right now, which works well enough. I haven't implemented it, but bringing in libfluidsynth as an option as well would be no problem.

For mp3/ogg play, SDL_Mixer does let you capture the data they output if you dump what comes into the Mix_SetPostMix callback, but you're restricted to having it come out at realtime speed (since it's actually being played to your audio system). I haven't done this yet, but libmad and libvorbis seem like the way to go.

There's one last feature that's missing at that point though. For when you're not recording video, you still want to be able to do system MIDI output right? Otherwise Rich Weeds won't be able to pipe output through a 30 year old yamaha synth he found in someone's garbage can. Add on libportmidi for that.

So that's 4 new libraries (fluidsynth, mad, vorbis, portmidi) as well as a dozen source files spliced in from another project (OPL emu). None of it is terribly difficult; the OPL emu is implemented in my current build, and I have test apps which implement both the portmidi and the fluidsynth correctly, but I'm still a bit apprehensive.

natt said:

So that's 4 new libraries (fluidsynth, mad, vorbis, portmidi) as well as a dozen source files spliced in from another project (OPL emu). None of it is terribly difficult;

Question: Are you on Windows or Linux? Over at ZDoom, Randy Heit has reported that compiling Fluidsynth for Windows was a major hassle and as a result provides a precompiled DLL.

natt said:

I'm leaning towards 2) really. I don't want to create a lot of additional dependencies (like libavcodec or anything)

Yes, raw RGB output is enough I think.

Graf Zahl said:

Question: Are you on Windows or Linux? Over at ZDoom, Randy Heit has reported that compiling Fluidsynth for Windows was a major hassle and as a result provides a precompiled DLL.

My adventures in compiling FluidSynth:

1. Download FluidSynth from SVN. Easy.
2. Get CMake.
3. Get pkg-config.
4. Put pkg-config in the cmake/bin directory so that it finds it.
5. Get glib, libsndfile, pulseaudio, midishare and proxy-libintl.
6. Set up environment variables and copy libglib-2.0.dll and intl.dll to cmake/bin directory so that pkg-config accepts to work.
7. Look if pkg-tool now works:

> pkg-config --list-all
gthread-2.0           GThread - Thread support for GLib
sndfile               sndfile - A library for reading and writing audio files
glib-2.0              GLib - C Utility Library
gio-2.0               GIO - glib I/O library
gobject-2.0           GObject - GLib Type, Object, Parameter and Signal Library
gmodule-no-export-2.0 GModule - Dynamic module loader for GLib
gmodule-2.0           GModule - Dynamic module loader for GLib
gio-windows-2.0       GIO Windows specific APIs - Windows specific headers for glib I/O library

It seems it works.
8. Run cmake. See error messages about glib being missing. Go back to 7.
9. Say "screw it" and comment out the lines in CMakeList.txt where the FluidSynth cmake script is checking for whether libs are there that you know are there, so that it stops aborting because it's too stupid to query pkg-config correctly.
10. Load the SLN file in MSVC and set up include/library directories.
11. Compile! It works.
12. Notice that you get a libfluidsynth.dll that's a lot smaller than the one Randy compiled. Try it anyway.
13. Get an error message about the application not working because gthread-2.0.dll cannot be found. Accept it as explanation for 12. Look for how to tell the linker to statically link that in the fluidsynth dll.
14. Discover that you apparently have to compile gthread yourself in "static" mode.
15. Find out that the glib solution refers to source code files that do not exist in the source code package.

This is the step where I am now.

Graf Zahl said:

Question: Are you on Windows or Linux? Over at ZDoom, Randy Heit has reported that compiling Fluidsynth for Windows was a major hassle and as a result provides a precompiled DLL.

Windows. I don't recall it being particularly easy to compile, no. Mingw, Msys, Cmake, precompiled versions of some of dependencies (libglib et al, forget where i got them)... come to think of it, the fluidsynth.dll i compiled probably dynamically links all that other stuff... that's a lot of dlls to distribute.

Is there a better option? I wasn't able to make head or tails of timidity. Bassmidi is extremely easy to use but is not GPL-compatible.

Use the DLL Randy compiled. That one works fine. No need to reinvent the wheel.

Very cool.

First of all, SDL_Mixer is gone.

Unrelated to the actual topic, but isn't SDL_Mixer what causes PrBoom+ to have trouble with MIDIs in Vista/7? Would that mean this version of PrBoom+ would run MIDIs fine in these OSes without having to use Timidity?

Graf Zahl said:

Use the DLL Randy compiled. That one works fine. No need to reinvent the wheel.

Yeah, if I distribute I'll do that.

Phml said:

Unrelated to the actual topic, but isn't SDL_Mixer what causes PrBoom+ to have trouble with MIDIs in Vista/7? Would that mean this version of PrBoom+ would run MIDIs fine in these OSes without having to use Timidity?

Well, I don't have a Vista or 7 test machine, nor do I know exactly what problem SDL_Mixer comes into on them; but fluidsynth in userspace would completely bypass the system-level MIDI APIs, so I doubt there'd be any sort of problem. Same holds true for OPL emulation.

===

I rather like the new setup I made for the capture system. When recording video, the game popens two commands, one for video and one for audio. After the timedemo completes, a third one is popened. For example:

#audio encoding command
ffmpeg -f s16le -ar %s -ac 2 -i pipe:0 -acodec libvorbis -f ogg output.ogg
#video encoding command
ffmpeg -f rawvideo -pix_fmt rgb24 -s %wx%h -r 35 -i pipe:0 -vcodec libx264 -f mp4 -fpre ./libx264-baseline.ffpreset -crf 22 output.mp4
#mux command (after completion of previous two)
ffmpeg -i output.ogg -i output.mp4 -vcodec copy -acodec copy %f

(% codes are replaced at runtime; %w pixel width, %h pixel height, %s sound rate (samples/sec), %f output file name specified on commandline)

This allows the power user the flexibility to use any encoding tools he wants that can accept raw input on stdin. For the casual user, the download could come prepackaged with ffmpeg.exe and default commandlines could be set.

As soon as it's all cleaned up, I'm going to submit what I have now (video dumping but with NO music input) to entryway. I'm hoping it's clean enough and modular enough for him to accept.

natt said:

Well, I don't have a Vista or 7 test machine, nor do I know exactly what problem SDL_Mixer comes into on them

SDL_mixer crashes on multicore systems while playing music, that's because prboom-plus forces single core with SetProcessAffinityMask()

http://bugzilla.libsdl.org/show_bug.cgi?id=629

In Vista/7 you are not able to change midi volume independently from sfx, that's because timidity is preferable on these systems

https://sourceforge.net/tracker/?func=detail&aid=3300070&group_id=148658&atid=772943

=====

I was listening to the eternal doom original midi pack today. Rather impressive. It got me to thinking; midi renderings can sound way different depending on the soundfont used. With the prevalence of soft synths in doom ports these days, it seems like it would be nice for a wad to be able to hint a suggested sf2 name, or even store an sf2 in the wad file (although that sounds like overkill, not to mention the fact that so many soundfonts are of dubious legal status). I was looking through the zdoom SNDINFO spec, and while it includes a way to suggest output type for midi, $mididevice, it doesn't have a way to hint at a suggested soundfont name.

Also, what sorts of non-midi non-mus lumps are commonly used by mapmakers? SDL_mixer can load quite a few different formats, and I'm just trying to get an idea of exactly what I'd need to re-implement...

As far as PrBoom+ goes, I think you'll be fine with just MP3 and Ogg Vorbis.

natt said:

and while it includes a way to suggest output type for midi, $mididevice, it doesn't have a way to hint at a suggested soundfont name.

I wanted to add it but it turned out not to be so easy back then. This was before FluidSynth was available so the only option was Timidity++ - and the only way to change the sound font with Timidity++ is to replace the .cfg file. There is no way to override the config from the command line so I skipped the entire feature. With FluidSynth it might be possible but to be honest, I never could motivate myself because I've yet to find a soundfont that's better than EAWPats.

Also, what sorts of non-midi non-mus lumps are commonly used by mapmakers? SDL_mixer can load quite a few different formats, and I'm just trying to get an idea of exactly what I'd need to re-implement...

ZDoom supports a few more (HMI, HMP, XMI)
If you ever write playback code using FluidSynth it may be worth investigating ZDoom's playback code. Even though the sound stream ultimately uses FMod, everything except the low level interface should be usable, in particular the code playing back the different MIDI formats. (And if you get a sound stream working it may even be worth considering ZDoom'S module player, based on DUMB because at the end it'd end up like the MIDI softsynth feeding a streaming sound channel.)

Graf Zahl said:

I've yet to find a soundfont that's better than EAWPats.

And D_E1M7 sounds like utter garbage™ with it. Which is why this feature is suggested.

Graf Zahl said:

based on DUMB

Is that GPL compatible?

tempun said:

And D_E1M7 sounds like utter garbage™ with it. Which is why this feature is suggested.

Agreed. But for that one I got a high quality Ogg so no big deal. :P

I don't know why but I found that Doom's original music always renders very poorly regardless of soundfont. I've yet to find one with which Doom's music sounds good.

The only versions I liked were the Roland MP2's that are linked somewhere here at Doomworld and the Sycraft versions.

Is that GPL compatible?

In general, yes, although I'm not 100% sure about this clause in the DUMB license:

4. If you are using the Program in someone else's bedroom at any Monday 3:05 PM, you are not allowed to modify the Program for ten minutes.

:D :D :D

Don't worry about that clause, it's just a joke :P

Graf Zahl said:

ZDoom supports a few more (HMI, HMP, XMI)
If you ever write playback code using FluidSynth it may be worth investigating ZDoom's playback code. Even though the sound stream ultimately uses FMod, everything except the low level interface should be usable, in particular the code playing back the different MIDI formats.

I may look at it. I actually have code I wrote for fun a while back that does MIDI file load+parse and MIDI event scheduling relative to a sound sample stream. When combined with fluidsynth it sounds fine as well as looping properly (something fluidsynth can't accomplish with its internal midi file loader).

In addition, chocolate doom's OPL synth has a midi file parser and event scheduler, although I don't know how good they are (I've heard them chew up some MIDI files pretty badly, but that may be just a limitation of the OPL synth emu).

Graf Zahl said:

(And if you get a sound stream working it may even be worth considering ZDoom'S module player, based on DUMB because at the end it'd end up like the MIDI softsynth feeding a streaming sound channel.)

Funny thing, I was actually checking out libmodplug earlier today. The idea of a module player sounds great; although perhaps a bit feature creepy, especially if there aren't existing otherwise prboom-compatible wads that use MOD music.


edit: BTW, I'd like input from anyone familiar with video encoding on default encoding parameters to use. This is what I have at the moment, with %f being an mkv:

"ffmpeg -f s16le -ar %s -ac 2 -i pipe:0 -acodec libvorbis -f ogg -aq 5 output.ogg"
"ffmpeg -f rawvideo -pix_fmt rgb24 -s %wx%h -r 35 -i pipe:0 -vcodec libx264 -f mp4 -fpre ./libx264-baseline.ffpreset -crf 22 output.mp4"
"ffmpeg -i output.ogg -i output.mp4 -vcodec copy -acodec copy %f"

I don't want multiple extra programs to bundle; ffmpeg static link is already 11MB. The probems I'm encountering are:
1) ffmpeg's aac encoder is marked experimental
2) ffmpeg's mkv muxer produces files that stutter severely during playback for me (remuxing them with mkvmerge fixes it)
3) ffmpeg won't put vorbis inside mp4 (not sure if there even is a standard way to do this)
4) ffmpeg won't put avc inside ogg (why?)

natt said:

Funny thing, I was actually checking out libmodplug earlier today.

I have used both. DUMB is definitely the better one.

The idea of a module player sounds great; although perhaps a bit feature creepy, especially if there aren't existing otherwise prboom-compatible wads that use MOD music.

There are none - because at the moment no 'classic' port supports MOD music. But who knows? I think if there was a general music streaming library that can handle all popular formats and stream them to whatever backend a port uses, it'd be a big win for everybody - and maybe even pave the way to someone seriously working on an OpenAL implementation that could be used as an SDL sound replacement.

Gez said:

As far as PrBoom+ goes, I think you'll be fine with just MP3 and Ogg Vorbis.

How about .MOD .S3M .IT .XM? SDL_mixer plays everything I have (midi, mod, mp3, ogg)

Graf Zahl said:

OpenAL

Some ports had OpenAL implementation (at least EDGE, right?), but all of them drop it eventually, right?

Graf Zahl said:

ZDoom supports a few more (HMI, HMP, XMI)

Did not see such files. Where are they from? I think support for midi, mod (just because of Terminal Velocity music :)), ogg, mp3 is enough.

http://pastebin.com/mX4ZmgsV

Video+sound capture, no music capture. Diff is against prboom-plus 2.5.0.8. Compiles fine on mingw as long as https://sourceforge.net/tracker/?func=detail&aid=3300070&group_id=148658&atid=772943 is also fixed. Should be fine on other systems? Makefile.am is changed, so re-whatever the build whatevers. I didn't touch any of the msvc project files.

I'm not so good with this whole diff thing, so if I screwed anything up, I apologize, I'll try to fix it.


As far as video encoding goes, try

prboom-plus -timedemo foo -viddump bar.mkv

The commandlines used will be in prboom-plus.cfg; if you stick with the defaults you need to have ffmpeg in path and have write permissions to current directory. I think you need some sort of x264 presets file, blargh. The video encoding command lines are a pain to debug because you don't get to see the output of the encoder, I will fix that.

If this creation makes it to "good" then I'll proceed with the sound code overhaul.

entryway said:

Did not see such files. Where are they from? I think support for midi, mod (just because of Terminal Velocity music :)), ogg, mp3 is enough.

Descent used HMP, if I am not mistaken. Yes, these types are rare, but if you somehow get ZDoom's MIDI code to work you'll have support for these, too.

Graf Zahl said:

Descent used HMP, if I am not mistaken.

I have all descent music in midi. Have since 19xx. HMP <-> MIDI?

http://prboom-plus.sf.net/track21.mid
http://prboom-plus.sf.net/game21.ogg

natt said:

http://pastebin.com/mX4ZmgsV

I've PMed you

entryway said:

I have all descent music in midi. Have since 19xx. HMP <-> MIDI?

Yes, HMP != MID. The base set of features can be converted but it has some additional looping controllers that cannot be supported in MID. That was the main reason for adding support in ZDoom.

BTW, ZDoom also supports Apogee's MID extensions which some of the songs in Duke Nukem use.

Prboom-plus with video recording (synchronized sound, NO music) is working and will be in the official svn soon. I'm still very much undecided on the actual encoding commands though; I urge anyone with video encoding experience to chime in. ffmpeg just seems to be so BAD at so much (broken x264 presets, broken mkv muxer, substandard aac encoder); but the distribution can't be including a dozen different tools either.

Once that's squared away, I'm going to start cleaning up my music code and preparing it for review. I've jotted down some ideas about this.

A simple music player api. Inspired by the one in chocolate doom, with a few necessary changes:

typedef struct
{
  const char *(*name)(void);
  int (*init)(int samplerate); // load library; 1 on success
  void (*shutdown)(void); // unload library
  void (*setvolume)(int v); // set volume, 0 - ??
  void (*pause)(void);
  void (*resume)(void);
  void *(*register)(const void *data, unsigned len); // return handle or NULL on failure
  void (*unregister)(void *handle);
  void (*play)(void *handle, int looping);
  void (*stop)(void);
  void (*render)(void *dest, unsigned nsamp); // render audio samples to buffer
} music_player_t;

Since the focus of my work is video recording, all of the audio implementations I'm putting in will return their data by render() so it can be either piped to speakers in realtime or piped to file in non-realtime while perfectly synchronized with video capture.

The proposed implementations:
fluidsynth
dumb
opl2 emu (from chocolate doom)
<some mp3 player, probably libmad>
<some vorbis player, probably libvorbis>

Common support files:
mus2midi
<some midi file parser, perhaps chocolatedoom's midifile.c>


The opl2 emu (from chocolate doom) has already been set up by me for this, it just needs to be cleaned up some. I've worked with fluidsynth in other midi playing projects. I've never used dumb before, but I read the reference docs and they looked pretty straightforward. I can't imagine libvorbis or libmad will be all that hard to use; and these implementations will be just very basic wrappers in the end. So the whole project should be over pretty quickly.

As far as dependency creep goes, I'm not really worried; SDL_mixer requires a bunch of libraries to compile, and the modular nature of the thing makes it very easy for a particular build to leave out a particular library. In fact, I think on a whole the setup will be an improvement on the current one, where #ifdef HAVE_MIXER is sprinkled all over i_sound.c.


POP QUIZ: The game engine calls play music with looping = false in three instances. Two are the title screens (doom and doom2), where it's not even noticeable because the screens are fixed 11 second length. What is the third instance?

natt said:

POP QUIZ: The game engine calls play music with looping = false in three instances. Two are the title screens (doom and doom2), where it's not even noticeable because the screens are fixed 11 second length. What is the third instance?

S_StartMusic(mus_bunny) from F_Ticker() ?