Vanilla Custom Sprites? (oddities)

[Doomkid]

So I've looked all around and can't find an answer to this one.

I've heard that if you replace every Doom2 sprite, that you can get custom sprites to run in vanilla Doom, so long as all sprite entries and rotations match up. I just did that, and can't get the damn thing to run! (About 80% of sprites were replaced, but I've ensured the number, order, and rotations all match up. I can provide a link if needed.)

Why the heck won't this thing run in vanilla doom? I keep getting "Bad lump characters in lump 4110". I can't identify lumps, but I dobule and triple checked - Every sprite rotation is there.

It gets a bit weirder, though. There's 1,381 sprite frames in Doom2. However, DWANGO11.wad uses custom sprites, only replacing 302 of them, greatly reducing filesize. What's the story here? Why does it work? I just tested it in vanilla Doom, custom sprites weren't an issue even though they're FAR from fully replaced. What gives?

I copied all the sprites from Dwango11 into an empty wad, so it was only the sprites and the S_START and S_END markers. No luck - crashed, simply cut to black, to error message or anything.

Can anyone please shed light on this?

[Obsidian]

I'm pretty sure you need to rename sprites using DeHackEd if you want to use them in vanilla. Download WhackEd4 and rename the sprites in the Strings table, rename the sprites in SLADE accordingly and you should be fine. If you need an example then look at Strain's DeHackEd patch.

[Gez]

Doomkid said:

Why the heck won't this thing run in vanilla doom? I keep getting "Bad lump characters in lump 4110". I can't identify lumps, but I dobule and triple checked - Every sprite rotation is there.

The message you got is "bad frame character".

    if (frame >= 29 || rotation > 8)
	I_Error("R_InstallSpriteLump: "
		"Bad frame characters in lump %i", lump);

It means it found something in the sprite namespace that didn't fit the requirements to look like a sprite. The 29 accepted frames are A-Z, [, \, ], and the accepted rotation values are 0-8. So if it finds something like a flat named CEIL_A, the "frame" character will '_' and the rotation will be 'A', both of which will fail.

I'm not sure how the dehacked trick works under the hood. From what I know, it looks for sprites between S_END and S_START (Doom engine always searches backward from the last) so either you have an S_END marker and it'll start looking there, eventually getting in the Doom IWAD's flats if you don't have a corresponding S_START first; or you don't have markers and your sprites should be ignored. Yet somehow it works and I don't know why.

[Doomkid]

Thankyou both for the responses!

See, this is what's weird - DWANGO11 doesn't have a .DEH patch - It just has a select few custom sprites and runs in vanilla with no hassle. Supposedly, this is impossible. I've copied the exact sprites from the wad and placed them in another, and it didn't work

Take a look, it's strange: http://soulsphere.org/random/dwango-wads/dwango11.zip (select bfg/plasma)

Only 302 of the ~1,300 sprites are replaced, and it runs under DOS... What the hell did they do to get this to work, without a dehacked patch? Figuring this out could be a big key to making future vanilla-compat graphics wads easier to deal with.

I'm really curious to find out why!

[purist]

I had similar issues replacing sprites in vanilla plutonia. Have a look, it might help you:

http://www.doomworld.com/vb/doom-editing/69236-replacing-sprites-in-vanilla-plutonia/

[scifista42]

The theory is: In vanilla, you either replace exactly ALL sprites, or none. In the latter case though, you can add new sprites with new names into the wad (their number or rotations don't matter at all) and use DEHACKED to change a "stock" sprite name to the new one, then it should work even in vanilla. You need to put the sprites between SS_START and S_END markers - yes, really SS_START and S_END, there shouldn't be S_START.

However, I've encountered instances when it mysteriously didn't work. Sometimes it was due to the subsprites (=letters A,B,C... after the sprite name) not being in order. On the other hand, in other instances this worked, so I don't know. The DWANGO thing is also a mystery to me.

[Da Werecat]

Doomkid said:

Only 302 of the ~1,300 sprites are replaced, and it runs under DOS...

As far as I can see, all sprites that are needed in deathmatch are replaced. I guess it's okay to leave the monsters out, as long as the engine won't come looking for them.

[Doomkid]

Thanks for the further info, fellas.

So, I've done as instructed, and tried to replace the pistol, chaingun and chainsaw sprites using the DEHACKED trick. (CHGG was changed to CHAI, and so on, through strings) In all source ports, it works as intended, but not in Doom2.exe. Can anyone explain why this is happening? I feel like Ive followed the instructions on every thread as close as possible - http://www.mediafire.com/download/a2agx3qdbppwxxb/ugh_poop.zip

If someone can help me out here, I'd be incredibly thankful. I'm also hoping about the mystery of DWANGO11's custom sprites will be solved, it's strange to think there may be something about the vanilla EXE that isn;t yet documented. EDIT: Great find, Da Werecat! I want to replicate this, yet for some reason, doing the same thing causes my WADs to hang in DOS. Hell, even doing the full sprite replace trick causes it to hang. Is there something special aside from renaming S_START to SS_START that's needed?

Boy, Doom2 vanilla sprite editing is even more finnicky than I remember..

[Da Werecat]

Works in Choco. Did you forget to apply the patch to the vanilla Doom executable?

[Doomkid]

Is starting up doom2.exe with the "-file pooper.wad pooper.deh" command enough? If not, I've found my problem. Now what I'd need to do is work out how to patch DOOM2.exe for dehacked stuff, and I'd finally have it figured out.

[scifista42]

Vanilla Doom2.exe doesn't support DEHACKED patches being loaded along with a wad. That's a feature of source ports. You need to make a copy of your "Doom2.exe" and use dehacked.exe to actually hack and rewrite it. That's the correct DOS-compatible way.

[Doomkid]

Thank you so much scifista, Da Werecat, purist, Obsidian and Gez. I now know how to replace vanilla sprites both through DEHACKED, and the "replace every sprite" way.

(I tried replacing all sprites in XWE, but there was an oddity - It renamed all VILE\ sprites to VILE_! obviously Doom didn't recognize the "_" and that's why this method wasn't working for me.)

Thanks a million. I'm now having problems getting DEHACKED itself to run, even after telling it where my Doom2.wad is through the INI. I feel like I've asked so many damn questions, though. Is using the old version the only way of making a patched doom2.exe? Can't be done through WhackEd, I assume?

[scifista42]

Doomkid said:

Can't be done through WhackEd, I assume?

No, WhackEd can merely make and edit DEHACKED patches, not apply them to hack an exe. At least, I'm convinced it is so.

Did you follow the enclosed README.txt of Dehacked? You should specify editname, normalname, wadname, pathname and patchdir in the .ini. Then you manually make the copy of Doom2.exe (calling it as you've specified in editname) and then simply launch "dehacked -load mypatch.deh".

[Gez]

SS_START means nothing to vanilla, though.

Okay, so the frame character check is done in R_InstallSpriteLump(), which is called by R_InitSpriteDefs(). The way R_InitSpriteDefs() works is that it has a list of name, a first sprite lump, and a last sprite lump. It works through its list of name and searches through the lumps for lumps which match the sprite names from its list, so it overlooks those that don't fit. Interestingly, this function searches forward.

Spoiler

//
// R_InitSpriteDefs
// Pass a null terminated list of sprite names
//  (4 chars exactly) to be used.
// Builds the sprite rotation matrixes to account
//  for horizontally flipped sprites.
// Will report an error if the lumps are inconsistant. 
// Only called at startup.
//
// Sprite lump names are 4 characters for the actor,
//  a letter for the frame, and a number for the rotation.
// A sprite that is flippable will have an additional
//  letter/number appended.
// The rotation character can be 0 to signify no rotations.
//
void R_InitSpriteDefs (char** namelist) 
{ 
    char**	check;
    int		i;
    int		l;
    int		intname;
    int		frame;
    int		rotation;
    int		start;
    int		end;
    int		patched;
		
    // count the number of sprite names
    check = namelist;
    while (*check != NULL)
	check++;

    numsprites = check-namelist;
	
    if (!numsprites)
	return;
		
    sprites = Z_Malloc(numsprites *sizeof(*sprites), PU_STATIC, NULL);
	
    start = firstspritelump-1;
    end = lastspritelump+1;
	
    // scan all the lump names for each of the names,
    //  noting the highest frame letter.
    // Just compare 4 characters as ints
    for (i=0 ; i<numsprites ; i++)
    {
	spritename = namelist[ i ];
	memset (sprtemp,-1, sizeof(sprtemp));
		
	maxframe = -1;
	intname = *(int *)namelist[ i ];
	
	// scan the lumps,
	//  filling in the frames for whatever is found
	for (l=start+1 ; l<end ; l++)
	{
	    if (*(int *)lumpinfo[l].name == intname)
	    {
		frame = lumpinfo[l].name[4] - 'A';
		rotation = lumpinfo[l].name[5] - '0';

		if (modifiedgame)
		    patched = W_GetNumForName (lumpinfo[l].name);
		else
		    patched = l;

		R_InstallSpriteLump (patched, frame, rotation, false);

		if (lumpinfo[l].name[6])
		{
		    frame = lumpinfo[l].name[6] - 'A';
		    rotation = lumpinfo[l].name[7] - '0';
		    R_InstallSpriteLump (l, frame, rotation, true);
		}
	    }
	}
	
	// check the frames that were found for completeness
	if (maxframe == -1)
	{
	    sprites[ i ].numframes = 0;
	    continue;
	}
		
	maxframe++;
	
	for (frame = 0 ; frame < maxframe ; frame++)
	{
	    switch ((int)sprtemp[frame].rotate)
	    {
	      case -1:
		// no rotations were found for that frame at all
		I_Error ("R_InitSprites: No patches found "
			 "for %s frame %c", namelist[ i ], frame+'A');
		break;
		
	      case 0:
		// only the first rotation is needed
		break;
			
	      case 1:
		// must have all 8 frames
		for (rotation=0 ; rotation<8 ; rotation++)
		    if (sprtemp[frame].lump[rotation] == -1)
			I_Error ("R_InitSprites: Sprite %s frame %c "
				 "is missing rotations",
				 namelist[ i ], frame+'A');
		break;
	    }
	}
	
	// allocate space for the frames present and copy sprtemp to it
	sprites[ i ].numframes = maxframe;
	sprites[ i ].spriteframes = 
	    Z_Malloc (maxframe * sizeof(spriteframe_t), PU_STATIC, NULL);
	memcpy (sprites[ i ].spriteframes, sprtemp, maxframe*sizeof(spriteframe_t));
    }

}

Edit: Haha, the code accidentally contained valid BBCode and I had to add spaces around any array access on the i index.

This function is responsible for making sure sprites have all their rotations. Note that it has a nested loop construction: it searches for all names, for all lumps.

R_InitSpriteDefs() is called by R_InitSprites(), which is called by P_Init() -- you may have noticed the message "P_Init: Init Playloop state." in vanilla startup. At no point do these function care about S_START or S_END directly. However, R_InitSpriteDefs() use global variables:

    start = firstspritelump-1;
    end = lastspritelump+1;

These variables are set by R_InitSpriteLumps() which is what scans through S_START and S_END.

//
// R_InitSpriteLumps
// Finds the width and hoffset of all sprites in the wad,
//  so the sprite does not need to be cached completely
//  just for having the header info ready during rendering.
//
void R_InitSpriteLumps (void)
{
    int		i;
    patch_t	*patch;
	
    firstspritelump = W_GetNumForName ("S_START") + 1;
    lastspritelump = W_GetNumForName ("S_END") - 1;
    
    numspritelumps = lastspritelump - firstspritelump + 1;
    spritewidth = Z_Malloc (numspritelumps*4, PU_STATIC, 0);
    spriteoffset = Z_Malloc (numspritelumps*4, PU_STATIC, 0);
    spritetopoffset = Z_Malloc (numspritelumps*4, PU_STATIC, 0);
	
    for (i=0 ; i< numspritelumps ; i++)
    {
	if (!(i&63))
	    printf (".");

	patch = W_CacheLumpNum (firstspritelump+i, PU_CACHE);
	spritewidth[ i ] = SHORT(patch->width)<<FRACBITS;
	spriteoffset[ i ] = SHORT(patch->leftoffset)<<FRACBITS;
	spritetopoffset[ i ] = SHORT(patch->topoffset)<<FRACBITS;
    }
}

It is called by R_InitData(), which is called by R_Init(), which is the famous "R_Init: Init DOOM refresh daemon" line. R_Init() is called just before P_Init().


So the real question here is, "how does it get the namelist?". Well it's just a hardcoded global array:

char *sprnames[NUMSPRITES] = {
    "TROO","SHTG","PUNG","PISG","PISF","SHTF","SHT2","CHGG","CHGF","MISG",
    "MISF","SAWG","PLSG","PLSF","BFGG","BFGF","BLUD","PUFF","BAL1","BAL2",
    "PLSS","PLSE","MISL","BFS1","BFE1","BFE2","TFOG","IFOG","PLAY","POSS",
    "SPOS","VILE","FIRE","FATB","FBXP","SKEL","MANF","FATT","CPOS","SARG",
    "HEAD","BAL7","BOSS","BOS2","SKUL","SPID","BSPI","APLS","APBX","CYBR",
    "PAIN","SSWV","KEEN","BBRN","BOSF","ARM1","ARM2","BAR1","BEXP","FCAN",
    "BON1","BON2","BKEY","RKEY","YKEY","BSKU","RSKU","YSKU","STIM","MEDI",
    "SOUL","PINV","PSTR","PINS","MEGA","SUIT","PMAP","PVIS","CLIP","AMMO",
    "ROCK","BROK","CELL","CELP","SHEL","SBOX","BPAK","BFUG","MGUN","CSAW",
    "LAUN","PLAS","SHOT","SGN2","COLU","SMT2","GOR1","POL2","POL5","POL4",
    "POL3","POL1","POL6","GOR2","GOR3","GOR4","GOR5","SMIT","COL1","COL2",
    "COL3","COL4","CAND","CBRA","COL6","TRE1","TRE2","ELEC","CEYE","FSKU",
    "COL5","TBLU","TGRN","TRED","SMBT","SMGT","SMRT","HDB1","HDB2","HDB3",
    "HDB4","HDB5","HDB6","POB1","POB2","BRS1","TLMP","TLP2"
};

That's what you can modify in DEHACKED. From what I can guess, it's safe to change sprites if none of the non-sprite lumps in vanilla after S_END (included) and in your own mod have their first four characters in common with the sprites you are not replacing, which is why replacing all sprites is safe.

In fact I've just made a small PWAD which only contains one lump: the S_END marker. Loading it in vanilla within DOSBox, it didn't cause any error during loading...

Doomkid said:

(I tried replacing all sprites in XWE, but there was an oddity - It renamed all VILE\ sprites to VILE_! obviously Doom didn't recognize the "_" and that's why this method wasn't working for me.)

SLADE can rename lumps to VILE\ without problem. :)

[Doomkid]

Wow, very informative Gez. I always wondered what was the deal with the "DOOM Refresh daemon" line, it's interesting to know after all these years.

So I have just one final question. Is there any other way I can tell vanilla doom2.exe to load a DEH file? What I mean is, so long as I have DEHACKED.exe, Doom2.exe, doom2.wad, something.DEH, and something.wad in the directory, is there and easy way for me to make a simple BAT textfile that loads it properly in dosbox? Or do I have to use the way built into dehacked.exe?

It would be great if whacked was somehow able to create a patch for vanilla D2. I got dehacked booted up through dosbox, but everything kept getting corrupted and garbled on screen, as if it was having trouble dealing with my IWAD. It's just the commercial doom2 1.9 iwad though, not sure why it causes all the garbled text. The strings were just showing up as smiley faces and stuff, unless there's another way, I think my .deh making will be limited to choco-doom and up ;)

All these old TC's like STRAIN come with .BAT files, but they're a bit beyond my understanding. It does work placing it in my doom2 directory and just loading the BAT though - This is what I want to achieve.

[scifista42]

It's perfectly possible to create a DEHACKED patch using WhackEd, then put the patch in Dehacked's directory and just use "dehacked -load mypatch.deh" to generate the proper .exe in a brief moment, as I said. You won't even see Dehacked's interface at all. I do it this way, there's no special effort connected to it.

[Doomkid]

(Sorry for the slight bump)

Alrighty scifista, I think you might be able to help me. I ran dehacked just as you said, "dehacked -load dehthing.deh" and it just said "Line 9:unkown line, Line 10: unknown line. Patch file read, one or more errors detected" and so on.

I then tried to run DOOM2.exe, which had apparently been edited by this. It crashed with the message "fatal error 1313: can't resolve external references".

Any ideas what I can do here? I made sure it was in ultimate doom DEH format and not BEX format when in WhackEd.

[dg93]

scifista42 said:

It's perfectly possible to create a DEHACKED patch using WhackEd, then put the patch in Dehacked's directory and just use "dehacked -load mypatch.deh" to generate the proper .exe in a brief moment, as I said. You won't even see Dehacked's interface at all. I do it this way, there's no special effort connected to it.

This is a bit off topic, but are there any good tutorials on how to use WhackEd? I've tried using it before, but I had absolutely no clue on what the Hell I was doing.

[scifista42]

the_miano said:

This is a bit off topic, but are there any good tutorials on how to use WhackEd? I've tried using it before, but I had absolutely no clue on what the Hell I was doing.

Enjay's tutorials (here) describe DOS Dehacked, but it should be sufficient for you, because the principle is absolutely the same.

@Doomkid: Make sure that you've set up Dehacked's ini file correctly (according to instructions in README.txt), because Dehacked was supposed to edit a copy of your Doom2.exe (which you previously had to make yourself), and not the original file. If that's not the problem, post your dehacked file here.

[dg93]

scifista42 said:

Enjay's tutorials (here) describe DOS Dehacked, but it should be sufficient for you, because the principle is absolutely the same.

Awesome, thanks man

[GreyGhost]

Doomkid said:

I ran dehacked just as you said, "dehacked -load dehthing.deh" and it just said "Line 9:unkown line, Line 10: unknown line. Patch file read, one or more errors detected" and so on.

I think the problem is that WhackEd4 now uses a verbose text format for the Bits field that DeHackEd doesn't understand, it expects to find a numeric value. I don't know if there's any way to force WhackEd to use numeric Bit values for backward compatibility, but exl might add that as an option if asked nicely. In the meantime your only option is to use an earlier version of the editor.

[AD_79]

Ok, so I found this thread really helpful; I was having issues when I replaced the baron sprites, and the baron ended up being invisible for many of its frames in Chocolate Doom. Thanks to whoever said "replace all the sprites".

[scifista42]

AD_79 said:

Ok, so I found this thread really helpful; I was having issues when I replaced the baron sprites, and the baron ended up being invisible for many of its frames in Chocolate Doom. Thanks to whoever said "replace all the sprites".

In Chocolate Doom, there's even better solution: use "-merge" parameter instead of "-file" when launching the wad, then you won't need to replace all sprites. Vanilla itself doesn't support this option, though. You can, however, tell the user to use a program DEUSF to add all non-replaced sprites to the wad on his end. But that's a 90's method. I strongly prefer using the DEHACKED trick nowadays.

[fraggle]

Doomkid said:

Thankyou both for the responses!

See, this is what's weird - DWANGO11 doesn't have a .DEH patch - It just has a select few custom sprites and runs in vanilla with no hassle. Supposedly, this is impossible. I've copied the exact sprites from the wad and placed them in another, and it didn't work

I checked the WAD - it uses the "replace all sprites" method. It's just that as a deathmatch WAD it doesn't need to bother including the monster sprites. If you try loading a single player WAD on top of it, the game crashes as soon as you see a monster.

Here's how sprites actually work in Vanilla:

First off, SS_START and SS_END don't actually do anything. That's just a convention invented by tools like deutex to make sprite replacement more convenient.

When you use the -file parameter, the PWAD is (internally to Doom) appended to the end of the IWAD directory. So for example: doom2.wad has 2919 entries. If you run doom2 -file foo.wad, the first lump of foo.wad is lump #2920, following right on from F_END (the last lump at the end of doom2.wad).

When Doom has to look up a lump by name, it searches backwards from the end of the directory. So suppose that you have a PWAD that includes a TITLEPIC lump. When you load the game and the title screen is displayed, the game searches backwards from the end of the combined directory. Because it searches backwards from the end, it finds the TITLEPIC from the PWAD file first, before TITLEPIC from the IWAD. And your modified TITLEPIC is shown.

This works for almost everything: graphics, sounds, music, levels etc. - because most lumps are looked up by name. The two exceptions are flats and sprites.

Flats and sprites are special because they have to be enclosed in F_START..F_END and S_START..S_END. The game uses these markers to delineate ranges of lumps that can be used as flats or sprites. You can't just put flats or sprites into PWADs because they won't be in these ranges.

Vanilla mappers back in the '90s figured out a trick to include replacement flats in PWADs. They include their replacement sprites in FF_START..F_END. The FF_START is actually unnecessary and just a convention. It's the F_END (with no F_START) that's the important part.

Let's remember that the PWAD directory is joined to the end of the IWAD directory. So it looks like this:

... lumps ...
F_START
FLOOR0_1
...bunch of flats ...
F_END
... more lumps ...
[end of IWAD directory]
[start of PWAD directory]
FF_START
... bunch of replacement /additional flats ...
F_END

When Doom looks up F_START and F_END it gets the F_START from the IWAD and the F_END from the PWAD. So everything within that range is a valid flat, even if it isn't a flat (could be sounds, music, levels etc.). Fortunately the engine doesn't do any kind of checks on the lumps it finds, so it isn't a problem if you have random junk in there that isn't really a flat, and so the trick works.

The sprite code isn't so forgiving - it has some strict checks in place that make sure the directory has the right number of sprites in place for each frame. So if you try to include a replacement sprite in an SS_START..S_END range, you get an error like this:

R_InitSprites: Sprite POSS : A : 1 has two lumps mapped to it

Because you now have POSSA1 twice instead of just once, and that's a no-no.

It seems that Id did try to include some hacky code apparently intended to support sprites in PWAD files. But it doesn't work - I assume it was never tested:

		if (modifiedgame)
		    patched = W_GetNumForName (lumpinfo[l].name);
		else
		    patched = l;

To translate: this means "if we're running with a PWAD, re-lookup the sprite name again to see if it's been replaced in the PWAD". But it doesn't work - there's other code elsewhere that requires all sprites to be within the S_START..S_END range or they don't work. But you can see the effect of this broken code if you try to include a replacement sprite bare in a PWAD: make a PWAD that just contains POSSA1 and nothing else, and when you run it in Vanilla you'll see that that particular frame just disappears.

So PWAD sprite replacements just don't work in Vanilla. The ways to work around it are: (1) replace all sprites in the PWAD - in which case the IWAD sprites are completely ignored, (2) use a tool like deusf to modify the IWAD file and merge in the replacement sprites (lots of mods used this technique) or (3) use dehacked to change the sprite names.

The dehacked technique is kind of clever. As we saw above, replacing sprites by extending the S_END range doesn't work, because now you have two POSSA1 lumps. But if you use dehacked to rename POSS to eg. XOSS, now you only have the XOSS from your PWAD, and the original POSS just gets ignored.

But in the end, hacking up either doom2.exe or doom2.wad is pretty ugly. Personally I think the deusf approach is the best one, just because it's the most common, accepted approach and most source ports support it in some way or other.

[Doomkid]

Okay so, I'm really sorry for the extreme necrobump, but this question is relevant: I'm currently using WhackEd4, and in the "Strings" editing section, I don't see anything where I could change SKEL to SMGN, for example, to alter what sprites it uses. What am I doing wrong, where in WhackEd4 can I edit the sprite strings?

Here's where I'm expecting to be able to find and edit sprite definitions yet can't:

[scifista42]

You can't change sprite strings in Boom-extended-format DEHACKED, only in vanilla-format DEHACKED. You better rename the actual sprites in your wad to match the default names.

[Doomkid]

Oh, damn! That's a major setback, but thanks for the info scifista.

[plums]

Can't you just open the dehacked file in a text editor and change the string anyhow? Or will Boom ports reject that?

(If it does work, no idea if WhackEd will preserve the change if you edit it more.)

Text 4 4
SKELSMGN

[VGA]

It's better to replace the original sprites directly imo. In the source ports, you don't have to replace everything or jump through any hoops.

SLADE is great, it supports mass renaming with asterisks to help you, too.