Jump to content
Search In
  • More options...
Find results that contain...
Find results in...
Sign in to follow this  
wesleyjohnson

Identify a patch at preload textures

Recommended Posts

I have got a legacy demo wad that will cause segfaults. I cannot change the demo wad because it has been around so long that I can expect that every copy encountered will be the original buggy version.

The demo wad has Boom colormap named BLUMAP. It also has a texture that uses a patch named BLUMAP. During texture preload, Doom Legacy will read the texture patches and combine them to form a texture.
It is getting the BLUMAP colormap instead, and trying to interpret it as a patch.
This leads to reading an offset to a post that exceeds the memory footprint.

Checking for textures that are used does not help as the texture is used in the map, in a Boom special effects sector.

So far I am working on checking that offsets to the posts stay within the patch size.
I am trying to come up with a way to get the patch and not the colormap lump. Both come from the PWAD. Most of the Doom Legacy lump functions are the same as in PrBoom.
I cannot think of any way to enhance them to solve this problem better, any ideas ?

PrBoom will preload the same texture and will also get the colormap BLUMAP instead of the patch. But it does not actually interpret the patch at preload so it does not segfault.

Share this post


Link to post

Is the colormap lump located within C_START and C_END marker lumps? Boom set up the "loosely-coupled" namespace thing? I would think it would be safe to assume any lump inbetween the C_'s is not an image, but, who knows? I've seen a lot of funky wads out there too.

You can typically catch 99.9% of non-texture lumps by checking the lump's first 8-bytes for valid values.

It would be nice to see the wad file too - I'd be interested in seeing the problem occur myself.

You say that PrBoom does not segfault. Is it loading textures on demand, and that's why it doesn't crash? Will it crash when that texture is actually found in a visible room?

Share this post


Link to post

If you want to be Boom compatible you have to respect its namespaces fully and properly. I know that the WAD management in Legacy 1.42 did not do this, instead choosing to hack around the issue to avoid a proper implementation.

Share this post


Link to post

Legacy_demo.zip (42.73KB)

Download Link
http://www.sendspace.com/file/wrej1y

This is a demo of legacy features.
I could not find it at idgames. I got it from somewhere but cannot find where. I expect I do not have the only copy.

The problem texture is on one of the sides of a separate small sector on the left side of the level map, used for Boom deep water. It is not visible to the player. I think the texture is left over from some
experimentation.

I would like to avoid making changes to how patches are found in the wad, because I would likely find that some other important wad would no longer work (did not follow the same rules). Thus I am interested in how other ports have dealt with this kind of issue.

The patch handling in Doom Legacy is almost identical as that in PrBoom. In neither do I see a way to limit the lumps considered for patches in textures. In either, reading a lump by name could get any lump in the wad and interpret it as a patch.

The prefetch of textures in DoomLegacy faults sooner because of the conversion to an internal texture format. The patches have to be read and interpreted at prefetch.

Share this post


Link to post

There are three things wrong in Legacy:

1. The BLUMAP colormap is between C_START and C_END markers, so it is in the Boom colormap namespace. It should not be loaded as a patch because of that.
2. The BLUMAP patch is between PP_START and PP_END markers, so it set explicitly in the patch namespace, which means it should be prioritized when looking for a BLUMAP patch.
3. The BLUMAP patch is situated later in the WAD than the BLUMAP colormap, so since the Doom search algorithm is a backward search from the last, it is the first lump that should be found. Since it is found first and it fits the format requirement for a patch, it should be used as the patch.

Share this post


Link to post

As Graf said, there is really only one solution to this problem and that is to augment the lump index with domain specific resource indexes. Trying to tackle this without additional indexes would be extremely awkward and difficult to maintain.

Fact is that the original resource management already uses such a concept for some domains/resource types. However its not standardized and consequently there are some domain/resource specific quirks to contend with.

Whether or not you decide to use a centralized, Boom-like "lump collation" mechanism, or a series of domain specific indexes is entirely up to you, so long as the "namespacing" behavior is emulated.

In the long run, though, it pays to do this the "hard" way. In the process you will uncover and then implement explicit solutions to deal with the quirky behaviors. This means one does not have to be mindful of them every time resource management needs to be extended in some way.

Share this post


Link to post

Thank you, Gez. I could not readily find a tool to reveal that.
Which did you use?

I would be interested in seeing one of these "proper" implementations handle this, but only have prboom-2.5 and ee running. Attempts to compile others (zdoom, vavoom, etc.) have failed so I cannot test this wad on them.

Which do you consider more "proper", prboom-2.5 or Boom202 source ??
I would prefer to only have to reverse engineer one of these.
A cursory examination of them did not reveal the proper handling of the patch space to which has been referred.

I have not found any code that handles markers anywhere !
Does not mean that it is not there, I just can't find it. Anyone got an actual function name ??

I would just start coding up my own unique solution, but as you see that gets comments on not doing it "proper" like Boom. So I would like to find out what Boom did that is considered proper.

DaniJ - I had already concluded that marking the lumps with some categorization was appropriate. I was hoping for more details on the problems I would encounter that are going to make me wish I had never started this.

Share this post


Link to post
wesleyjohnson said:

Which do you consider more "proper", prboom-2.5 or Boom202 source ??
I would prefer to only have to reverse engineer one of these.
A cursory examination of them did not reveal the proper handling of the patch space to which has been referred.



It doesn't really matter in this context. The WAD management hasn't changed much between versions.

Regarding the patch 'namespace', it simply doesn't exist in any port. The markers were only there as a convenience but no engine ever checked them

Share this post


Link to post

For what it's worth, here's what I use, and it's worked on every wad I throw at it:

First of all, I implemented killough's hash, and namespace stuff:

typedef struct
{
	char			name[9];
	FILE			*handle;
	int 			position;
	int 			size;
	
	// killough 1/31/98: hash table fields, used for ultra-fast hash table lookup
	int				index;
	int				next;

	// [kb] index to the wad this lump is in
	int				wadfileindex;

	// killough 4/17/98: namespace tags, to prevent conflicts between resources
	enum
	{
		ns_global = 0,
		ns_sprites,
		ns_flats,
		ns_colormaps,
		ns_any
	} li_namespace; // haleyjd 05/21/02: renamed from "namespace"

} lumpinfo_t;
Next, on wad load, all lumps are marked as namespace 'ns_global':
lump_p->li_namespace = ns_global;
Then, W_InitLumps does the 'coalesce' thing:
...
	// sprites
	s_start = W_CoalesceMarkedResource("S_START", "S_END", ns_sprites);
	
	// flats
	f_start = W_CoalesceMarkedResource("F_START", "F_END", ns_flats);

	// colormaps
	c_start = W_CoalesceMarkedResource("C_START", "C_END", ns_colormaps);
   
	// killough 1/31/98: initialize lump hash table
	W_InitLumpHash();
Finally, my approach differs from other ports. I actually try to respect the namespaces, by writing specialized lump lookup routines:
// W_CheckNumForGlobalLumpName
//  Searches for lumps that are not sprites, flats, or colormaps.
//  Returns the matching lump, or -1 if none found.
//
int W_CheckNumForGlobalLumpName(register const char *name)
{
	register int i = lumpinfo[W_LumpNameHash(name) % (unsigned) numlumps]->index;

	while (i >= 0 && (_strnicmp(lumpinfo[i]->name, name, 8) ||
		lumpinfo[i]->li_namespace != ns_global))
		i = lumpinfo[i]->next;

	return i;
}

// W_CheckNumForSpriteLumpName
//  Searches for sprite lumps only.
//  Returns the matching lump, or -1 if none found.
//
int W_CheckNumForSpriteLumpName(register const char *name)
{
	register int i = lumpinfo[W_LumpNameHash(name) % (unsigned) numlumps]->index;

	while (i >= 0 && (_strnicmp(lumpinfo[i]->name, name, 8) ||
		lumpinfo[i]->li_namespace != ns_sprites))
		i = lumpinfo[i]->next;

	return i;
}

// W_CheckNumForFlatLumpName
//  Searches for flat lumps only
//  Returns the matching lump, or -1 if none found.
//
int W_CheckNumForFlatLumpName(register const char *name)
{
	register int i = lumpinfo[W_LumpNameHash(name) % (unsigned) numlumps]->index;

	while (i >= 0 && (_strnicmp(lumpinfo[i]->name, name, 8) ||
		lumpinfo[i]->li_namespace != ns_flats))
		i = lumpinfo[i]->next;

	return i;
}

// W_CheckNumForColormapLumpName
//  Searches for colormap lumps only
//  Returns the matching lump, or -1 if none found.
//
int W_CheckNumForColormapLumpName(register const char *name)
{
	register int i = lumpinfo[W_LumpNameHash(name) % (unsigned) numlumps]->index;

	while (i >= 0 && (_strnicmp(lumpinfo[i]->name, name, 8) ||
		lumpinfo[i]->li_namespace != ns_colormaps))
		i = lumpinfo[i]->next;

	return i;
}

// W_CheckNumForWallPatchName
//  [kb] Searches for wall patches, first in the non-sprite,
//   non-flat, non-colormap namespaces, and then in the
//   sprite namespace. Returns the matching lump, or -1 if none found.
//
//  [kb] Modified to specifically avoid searching
//   the flat namespace, which was causing BOOMEDIT.WAD to crash
//   rendering ICEFLAT, which exists as a flat AND a patch.
//  
//
//  [kb] Original Boom comment:
// killough 4/17/98:
// Some wads use sprites as wall patches, so repeat check and
// look for sprites this time, but only if there were no wall
// patches found. This is the same as allowing for both, except
// that wall patches always win over sprites, even when they
// appear first in a wad. This is a kludgy solution to the wad
// lump namespace problem.
//
// [kb] Not so kludgy anymore, as only one search is performed. 
//
int W_CheckNumForWallPatchName(register const char *name)
{
	register int i = lumpinfo[W_LumpNameHash(name) % (unsigned) numlumps]->index;
	int sprite = -1;

	while (i >= 0)
	{
		switch (lumpinfo[i]->li_namespace)
		{
			case ns_global:
				if (!_strnicmp(lumpinfo[i]->name, name, 8))
					return i;

				break;

			case ns_sprites:
				if (!_strnicmp(lumpinfo[i]->name, name, 8))
					sprite = i;

				break;
		}

		i = lumpinfo[i]->next;
	}

	return sprite;
}
Now, this last routine would definitely handle your wad file. I am not sure why other ports get away with not having this level of sophistication, but I needed it, for reasons I don't quite remember at this time. I think it works in other ports, because the wall lump is after the colormap lump, so it is found first. (Legacy should have worked this way too, maybe you're searching in the wrong direction? That should be obvious when loading replacement pwads.)

My code basically defaults to original handling, but also does the desired thing (in my opinion) when needed.

When things are arbitrary like this, you have 3 choices: Let the code be arbitrary, or attempt to define exactly what occurs in these situations and stick to it, or make special case clauses. I chose the second one - a definite set of rules that can be applied. In this case, it's obvious that the author at least tried to use the namespaces as intended by Boom. Barring that, as Gez said, the wall lump is located after the colormap lump, so, yes it should get chosen first.

With my implementation, the colormap (with markers) could be moved after the wall lump, and it would still work the same. To me, this is how it's supposed to work. There's no need to handle colormaps arbitrarily. I wish all ports used this logic. Actually, I wish id had followed through with the work started on P_START/P_END, S_START/S_END, and made those actually work properly. Boom didn't even follow through and actually finish their work on namespaces. The code I offer does. And, with a small amount of work, it could be expanded to handle any *_START/*_END namespaces that may be needed. ZDoom has "TX_START" and "TX_END", I think, and maybe some others. I think one of the Doom Alphas put maps within "M_START/END" markers.

There will never be a perfect solution, because:
1. Ports do not use the same logic universally.
2. Some ports try to take the "PP_START" hack solutions, and treat them as legitimate constructs. When I see these in wads, I honestly have no clue how to interpret those.
2. Editors allow people to make all manners of strange nonsensical wads that don't respect these markers.
3. Vanilla Doom (and other ports) will work regardless of a lot of these strange wads.

My implementation handles every strange construct I have encountered so far, but, obviously, someone can craft some wad specifically to break my port, but work fine on someone else's. I am not very upset by that - I'll just edit their wad if I want to play it. If I handle 99.9% of wads out there, I feel pretty good.

So, the question is: "Where do you draw the line? What constructs to you attempt to support?" For me, a wad that actually tries to handle colormaps properly gets my vote as a wad that I should try to support.

Authors of new maps should have mechanisms to be able to remove arbitrary handling of their lumps, so actually supporting C_START/C_END is a positive, sensical step in that direction.

Share this post


Link to post
kb1 said:

my approach differs from other ports. I actually try to respect the namespaces

kb1 said:

Now, this last routine would definitely handle your wad file. I am not sure why other ports get away with not having this level of sophistication

kb1 said:

I wish all ports used this logic


Digging the strong "everyone's doing it wrong but me" vibe. Pretty sure port authors will enjoy being told that they don't try to respect namespaces and aren't sophisticated.

Share this post


Link to post

I would say that most ports are already doing it the "sophisticated" way. Doomsday has a whole subsystem dedicated to specialized resource management and maintains separate indexes for stuff like textures. ZDoom probably does something similar.

In fact, I would guess that the only ports using Boom's "coalesce" approach are those which were based on it originally.

Share this post


Link to post
Gez said:

Digging the strong "everyone's doing it wrong but me" vibe. Pretty sure port authors will enjoy being told that they don't try to respect namespaces and aren't sophisticated.

Good lord, man, that's what you took from what I wrote? Maybe I'm not so good at expressing myself - I'll accept responsibility for that. But I never claimed that anyone was "doing it wrong", or that anyone was "being disrespectful" or "unsophisticated".

My approach is "sophisticated", in that I probably over-engineered it by using multiple lookup routines - if anything, it was a statement suggesting that other port's are probably doing what is required without needing this level of sophistication, and that I'm not exactly sure why it worked out so well for them, and not me. I haven't researched the other ports fully, but I know that a lot of them did not create multiple lump search routines, last time I checked.

When I say that I "respect the namespaces", that means that I based my code logic off of the existence of the namespaces, not that I am using "superior morals with my design." Jeez. And, yes, my approach differs from other ports. Why did you feel the need to quote that?

Finally, Yes, I do wish other ports used this logic, or that they could be depended on to separate resources as defined by Boom. At least, that way, wad lumps would get loaded the same way across all ports, things like the example provided would not have arbitrary results, and resource management could depend on the namespaces being used. Is that a bad thing?

Do you have some personal vendetta against me? I can't for the life of me comprehend what you are trying to accomplish with your post, except to maybe slant others into thinking I was trying to "disrespect" their approaches. What does that gain you?

I spent a considerable amount of time and energy, for the sole purpose of helping Wesley decide what approach to take, to address the issue he was encountering. I do not appreciate your attempt to paint that act in a negative light, with claims that are incongruent to the truth and to the nature of my efforts.

I can appreciate that it is difficult to interpret intent from text, especially when the author has difficulty expressing that intent in so many words, so I suppose you deserve the benefit of the doubt. I could use that logic to interpret your post too.

But, I can say that, yes, Gez, "you're doing it wrong" in understanding my post. Get off it, man, there's no ill will here.

Share this post


Link to post
DaniJ said:

In fact, I would guess that the only ports using Boom's "coalesce" approach are those which were based on it originally.



ZDoom originally used that thing, too, but the only reason for it was that the original sprite/flat initialization code depends on these resources being a contiguous block in the WAD directory.

Ultimately this kind of reorganization will also cause a bootload of problems - especially if things get a bit more complex. In ZDoom it started to cause problems with texture ordering - since the global directory was shuffled around it had become completely impossible to priorizize textures of different type by WAD loading order.


As for kb1's tl;dr post:

Wow, that's bad. Really bad. A perfect example of tunnel vision resulting in a limited, hard to expand and ultimately problematic implementation. And quite a bit of error-friendly redundancy instead of Boom's original namespace parameter.

Justify as much as you want: I'd happily point learning programmers to that post to show how NOT to do stuff. :P

Share this post


Link to post
DaniJ said:

I would say that most ports are already doing it the "sophisticated" way. Doomsday has a whole subsystem dedicated to specialized resource management and maintains separate indexes for stuff like textures. ZDoom probably does something similar.

In fact, I would guess that the only ports using Boom's "coalesce" approach are those which were based on it originally.

DaniJ, you're not getting offended too, are you? I guess my post was not worded too good - oops :)

Interesting - I've known that Doomsday has advanced Zip, and multiple folder support. What approach does Doomsday take to wads with C_START/C_END markers? Does it separate the lumps within, based on the markers, or does it analyze the contents of lumps?

For this specific example, the texture BLUMAP appears after the colormap BLUMAP, so most lump management code would properly pick the texture. But, what would Doomsday do if the markers and the colormap were located after the texture? Would it still be able to choose the texture over the colormap?

Share this post


Link to post
Graf Zahl said:

As for kb1's tl;dr post:

Wow, that's bad. Really bad. A perfect example of tunnel vision resulting in a limited, hard to expand and ultimately problematic implementation. And quite a bit of error-friendly redundancy instead of Boom's original namespace parameter.

Justify as much as you want: I'd happily point learning programmers to that post to show how NOT to do stuff. :P

Graf, I never attempted to justify anything!??? But, please enlighten me:
. What's really bad? (And, how did you know if you didn't read it - heh)
. Tunnel vision - what did I fail to see?
. Error-friendly redundancy? Maybe. I guess 3 lines of code per routine is error-friendly...hasn't failed yet, though. How exactly would you make it less error-friendly? (And keep it fast, and small.)
. You have my blessing, and are encouraged to point anyone you like to gawk at my code.

Seriously, enlighten me - I would love to learn how to do it better. Please, if you have time, explain how to exactly solve the stated problem, in an un-bad, non-tunnel-vision, non-problematic, error-resistant block of code that aspiring programmers can type with pride. Because, when sharing an approach with a fellow programmer, in an area that you've already stubled upon yourself, and have developed a possible solution to, you better damn well make sure it's beautiful production-level code that handles every possible current and future case, is optimized perfectly, and is worthy to show the world over. Otherwise, to Hell with your offer of help - keep your crappy code.

Wes: "Hey, does anyone have $10 I can borrow - I forgot my wallet, and want to get some lunch."

kb1: "No, but I've got half a pizza left you're welcome to."

Graf: "Cold pizza? kb1, what an ass you are. Your pizza sucks. Die in a fire!"

Thanks a lot, man.

Yes, please enlighten me with your wisdom.
And, Graf, I will read your post, no matter how long it is, before I choose to pass judgement on it. I mean, hey, we're all trying to assist Wesley with his original issue, right? Do it for him too, man - take the time out of your schedule to show him how to "code like a man", an exact, beautiful chunk of GrafCode that solves his issue, so that we may all "respect it", and adopt it into our "simple, unsophisticated, error-friendly" ports.

Share this post


Link to post
kb1 said:

DaniJ, you're not getting offended too, are you?

Me? Nah, I was merely disagreeing with your apparent assertion that most ports are "doing it wrong" (Gez's interpretation too).

What approach does Doomsday take to wads with C_START/C_END markers?

Presently we don't use the colormaps at all for anything. If we did support them then we'd implement a separate colormap resource index. The index would be populated explicitly from a one-time scan of the main lump index following any quirky behavior rules specific to this resource type. We'd then load from this index rather than global one and perform data validity checks at first-load time.

For this specific example, the texture BLUMAP appears after the colormap BLUMAP, so most lump management code would properly pick the texture. But, what would Doomsday do if the markers and the colormap were located after the texture? Would it still be able to choose the texture over the colormap?

We only ever pick textures from the texture index (thus namespaced) and not the lump index.

In fact its a lot more complicated than that in Doomsday. Firstly, because textures are abstracted behind a "material" concept (TEXTURE1/2, flats, etc, are never drawn directly, only as a layer of some material or other).

Secondly because we provide mod authors with mechanisms to conditionally enhance materials according to whether they are custom (i.e., did not originate from an IWAD). As you'll know, many old PWADs duplicate the textures, sprites, etc..., from the IWAD. So we also do a cross comparison to determine whether a resource in a PWAD is in fact a copied IWAD resource (which in the case of multipatch textures means checking the definition, the origin of the patch data lumps, etc...).

Share this post


Link to post

Wow, I just re-read my post. I still don't get the "everyone else is doing it wrong" thing, but I guess it could be read that way.

What I really don't get is why some people are so quick to jump. I tend to have a lot of respect for the hard work that everyone puts into their creation, whether it's a pwad, a port, or just a map.

It is always disturbing to me, when people attack other people's works, given for free, for the express purpose of providing enjoyment and entertainment. Let me just state now, that that is never my goal. Doom is supposed to be fun, isn't it?

It sounds like Doomsday is essentially employing the same general idea (treating certain resources as special, and separating resources with some logic above and beyond vanilla's "which resource appears last"). Obviously, Doomday's approach is much more elegant (read: "sophisticated") than my approach :) Do you know of a case that my approach might conflict with Doomsday's?

What does the 'copied IWAD cross comparison' do for you? Does that prevent the resource from being loaded twice, or do you prevent lumps from different wads from contributing to a single texture? Or does it just eliminate the possibility of a foreign wad's resources from getting mismatched with another wad?

I'm not sure I understand: Is Doomsday figuring out what lumps are textures based on what's required by the TEXTURE AND PNAME lumps, vs. using the P_START and P_END markers? If not, that might be an interesting approach, anyway.

This topic is interesting to me - I fretted for a while over how to handle these types of issues, which were present in a number of wads, if I remember correctly. I have never claimed to have the best approach, I just wish everyone used the same approach! These issues only get worse over time, and, especially now with the number of "add-on" wads, such as weapon, gore, enhanced lighting, GL, music, etc. add-ons being made. Loading the proper resources, in the right order, is more important that ever, even if your port does not support all of those features, am I right?

Share this post


Link to post
kb1 said:

Wow, I just re-read my post. I still don't get the "everyone else is doing it wrong" thing, but I guess it could be read that way.

What I really don't get is why some people are so quick to jump...

I won't comment on the other points you made other than to state that I personally wasn't attacking you. I can't speak for Gez but what triggered my response was precisely the use of language in the parts he quoted.

Anyways...

It sounds like Doomsday is essentially employing the same general idea (treating certain resources as special, and separating resources with some logic above and beyond vanilla's "which resource appears last").

For the most part, yes.

Do you know of a case that my approach might conflict with Doomsday's?

Off the top of my head no, it looks like you're respecting the namespaces in a similar manner.

What does the 'copied IWAD cross comparison' do for you?

This is purely to classify texture resources, for the benefit of mod authors who wish to enhance materials with additional effects such as light decorations, reflections, etc... It does not affect which resource is actually loaded (except in the case of highres replacements), or how they are interpreted as textures.

I just wish everyone used the same approach! These issues only get worse over time...

Theres not much any one of us can do about it, though. We are well past the point where a common standard could be imposed. The reason we're all using different solutions in the first place is because the vanilla implementation is so buggy and quirky. This is compounded by iterative attempts to improve it in ports also failing to get it right.

Loading the proper resources, in the right order, is more important that ever, even if your port does not support all of those features, am I right?

Of course.

I have no objection to your offer of help, nor have I commented on your solution. As far as I'm concerned it was pretty cool (and brave) of you to step in with something practical. Just remember that there is a pretty high percentage of coders here, so you're inviting feedback in the process.

Share this post


Link to post
kb1 said:

Good lord, man, that's what you took from what I wrote? Maybe I'm not so good at expressing myself - I'll accept responsibility for that. But I never claimed that anyone was "doing it wrong", or that anyone was "being disrespectful" or "unsophisticated".


I can accept the foot-in-mouth explanation.

For reference, when you say "I'm doing things differently from others; I actually try to do things correctly", you shouldn't be surprised if it gets parsed as "others don't even try to do things correctly".

If you had just talked about your code without adding commentary about other people's work, you wouldn't have seemed so snide.

kb1 said:

When I say that I "respect the namespaces", that means that I based my code logic off of the existence of the namespaces

And other ports do as well. Apparently not Legacy (hence this thread) but at least Doomsday, Eternity, ZDoom, and other ports with ZIP archive support have to have a good namespacing system in place. I've also tried the demo wad with PrBoom+ and while the Legacy effects didn't work (obviously) it loaded correctly and no-clipping into the relevant control sector showed the BLUMAP texture that used the BLUMAP patch was displayed adequately.

And even Legacy at least used to do it right, back when this demo wad was created.

kb1 said:

Finally, Yes, I do wish other ports used this logic, or that they could be depended on to separate resources as defined by Boom.

But my point was that they do. Well, not all ports in existence, especially since most of them are abandoned, but all the big names do (with the obvious exception of ChocoDoom).

kb1 said:

Do you have some personal vendetta against me? I can't for the life of me comprehend what you are trying to accomplish with your post, except to maybe slant others into thinking I was trying to "disrespect" their approaches. What does that gain you?

wow

It was one post pointing out you were being tactless, not the start of a crusade to get you ostracized. Lay off the paranoia a bit.

Share this post


Link to post

I have to give kb1 the benefit here because he was enthused about his solution, noted that Boom did not implement reading their own namespace tags, and noted (like Grah Zahl did) that other ports did not implement it either.

I could take offense at the statement that all the "big names" implement wad namespaces. I did not know of any port that did.
I doubt that we agree on what are the "big names" or on how much we are all supposed to know about their inner workings.

So some say some other ports did implement something that pays attention to the namespace tags. That is news to me. But I have not gleaned any really useful information on how they did it (other than it gets messy).

Legacy, even now in 1.44 alpha5, is still very close to PrBoom.
Even PrBoom mishandles this lump. It continues running only because it does not try to interpret it as patch until the player sees it. If the texture was player visible it could easily segfault too.
Legacy is only now failing due to memory allocation vagaries that make reading the patch at prefetch (bad offsets point way beyond its boundaries) finally (in this version) hit some memory outside the valid address space. It could fail equally well in previous versions and in PrBoom.

As to one BLUMAP lump being before another. I understand the search order applying to PWAD on the command line. The last PWAD gets searched first because it replaces any previous lumps with its own. I do not think it applies within a PWAD as implied, simply because there were no expectations of there ever being two lumps with the same name within a PWAD. Whatever reason the Boom team created the namespaces, it could not be to allow multiple lump with the same name because that required implementing namespace code too (which they did not).

The kb1 solution is close to what I envisioned and is probably what I will have to implement.
I probably will have one search function instead of a separate one for each usage (the only thing I can see in the harsh comment from Zahl).

One addition. I will search within a namespace first, but will accept any lump found in any namespace if necessary.
Most wads only have one lump of any name, but too many wads disregard the namespaces or are disorganized (which would go undetected because most ports do not notice namespace violations).

Thank you all for you responses.

Share this post


Link to post
wesleyjohnson said:

Legacy, even now in 1.44 alpha5, is still very close to PrBoom.
Even PrBoom mishandles this lump.

But PrBoom+ doesn't.

wesleyjohnson said:

I could take offense at the statement that all the "big names" implement wad namespaces. I did not know of any port that did.
I doubt that we agree on what are the "big names" or on how much we are all supposed to know about their inner workings.

As far as I'm concerned:

  • Chocolate Doom
  • Doomsday/Risen3D
  • 3DGE
  • Eternity
  • Odamex
  • PrBoom+
  • Zandronum
  • ZDaemon
  • ZDoom/GZDoom
Those are the ports that are still actively used, maintained, and targeted by mods.

wesleyjohnson said:

As to one BLUMAP lump being before another. I understand the search order applying to PWAD on the command line. The last PWAD gets searched first because it replaces any previous lumps with its own. I do not think it applies within a PWAD as implied, simply because there were no expectations of there ever being two lumps with the same name within a PWAD.

How about level lumps? Plenty of THINGS lumps around... But it's a different issue, since they're identified by position after a lump with a unique name (the marker).

If you want some examples of WADs with duplicated, non-map lumps, I can give you plenty of examples. Starting with doom.wad, which duplicates all the music and sound lumps from the shareware version. Strife also has multiple identical copies of its songs and a few other lumps. Doom2.wad has two copies of SW18_7. Hexen has two SKY2. Granted, they're identical so choosing the first or the second has no effect. You can then try 1 Monster MAP07. See if you get "Shawn's Got the Shotgun" or Descent's Lunar Military Base theme.

And the backward search definitely applies in a PWAD. Doom "glues" all loaded wads together into a single mass in memory (the entire reason why the flat hack, where you put an F_END but no F_START, works).

    // scan backwards so patch lump files take precedence
    lump_p = lumpinfo + numlumps;

    while (lump_p-- != lumpinfo)
    {
	if ( *(int *)lump_p->name == v1
	     && *(int *)&lump_p->name[4] == v2)
	{
	    return lump_p - lumpinfo;
	}
    }
See this check? It's done backward lump by lump. Not backward archive by archive but forward lump by lump.

And again, the demo you've found was made for Legacy, so it used to work in Legacy. It was made to demonstrate Legacy's cool features, not to report a bug. It's something that broke at some point and which should be fixed.

Share this post


Link to post
Gez said:

For reference, when you say "I'm doing things differently from others; I actually try to do things correctly", you shouldn't be surprised if it gets parsed as "others don't even try to do things correctly".

If you had just talked about your code without adding commentary about other people's work, you wouldn't have seemed so snide.

I can remember looking for the exact same thing in the Boom sources, and wondering "Where are the namespaces actually used? Surely I just haven't found it yet.". Hence the statement "I actually try to respect the namespaces" (forgetting to add "unlike Boom", and written way to close to "My approach is different than other ports"). Got you.

Gez said:

wow
It was one post pointing out you were being tactless, not the start of a crusade to get you ostracized. Lay off the paranoia a bit.

My bad - point taken. Peace, brother.

wesleyjohnson said:

I could take offense at the statement that all the "big names" implement wad namespaces...

Please don't. Hey, "Legacy" is a big name, after all!


Gez is dead on about the bottom-to-top in-wad lump search. Implement that and you can avoid adding the namespace code for now, I think.

On the multiple search routines: Searches for lumps come from various distinct places within a given Doom port. Apparently, ports must search for textures first in the proper places, then in the sprite lumps, if I understand killough's comments. The coalesce function, and the rest of the killough code actually reorders the lump in relation to the hash, so that things are searched in the right order. I'm not looking at the code now, so that's the best explanation I have right now. It also helps ports emulate the awful DeuSF iwad hacking, sprite adding tools of yesterday, without needing all of that.

I find the namespace/coalesce implementation to be interesting and useful - I think it's worth the time to, at least, study it, and see if it might work out for you. I recommend taking a look.

I made multiple routines, because it helped me debug some time back, and it lets me place specialized logic into each type of search, at the point where the search occurs, globally. Definitely check out the texture one.

But, again, see if your current code searches backwards. I haven't been able to do any fun coding, much less check out your code, or anyone else's, in a long time. :(

wesleyjohnson said:

Thank you all for you responses.

You are most welcome. Thanks for keeping Legacy alive!

Share this post


Link to post

(sorry about double posting...)

DaniJ said:

I won't comment on the other points you made other than to state that I personally wasn't attacking you.

Didn't really think you were. Thanks for clarifying it!

DaniJ said:

Theres not much any one of us can do about it, though. We are well past the point where a common standard could be imposed. The reason we're all using different solutions in the first place is because the vanilla implementation is so buggy and quirky. This is compounded by iterative attempts to improve it in ports also failing to get it right

I know. If I were brave, and had the time, I would like to collaborate on starting some kind of Doom standards committee, where ports could get "certified" on things like lump management, action and thing numbers, and the like. Not for "brownie points", but to ensure some type of compatibility. Adhering to the standards would help wad authors, by guaranteeing that certain features will be supported in compliant ports. It would also provide port authors with a template to go by, to make sure that they are adding a feature that does not conflict with other ports. But, committees scare me! It would be difficult for everyone to agree. As a first step, I would just want ports to fail gracefully if they encountered a feature that the port does not support. That would be a very positive step. But, everyone would have to support it in some manner. I think it's still possible. Maybe one day...

DaniJ said:

I have no objection to your offer of help, nor have I commented on your solution. As far as I'm concerned it was pretty cool (and brave) of you to step in with something practical. Just remember that there is a pretty high percentage of coders here, so you're inviting feedback in the process.

That's very cool of you to say, and I appreciate it. Feedback is very welcome. I never proclaim to have awesome code - heh. Just wanted to offer some help. No good deed goes unpunished :)


Well, I just took a quick look at DoomLegacy v1.44 Alpha 5 w_wad.c source. You've actually got some "scanforward" stuff going on in there, and it looks like some wad reload capabilities. I'm not sure, but do you allow wads to be loaded via the console? Very cool.

For whatever it's worth, here's my suggestions to support solving the immediate issue:

1. Implement Killough's hash. Basically, the code needs to make a pass across loaded wad's lumps, to populate the hash. And if the loaded wad set changes, you'll have to reinit the hash, but that's quick. Killough comments that this made Doom run 300% faster for him (heh, I doubt it would have that effect today). But, realistically, I believe that you will definitely notice a significant rendering speedup, especially on scenes with a lot of textures, flats, and sprites, and when wads with a lot of lumps are loaded. It always amazed me that the vanilla renderer finds its resources using a linear text lookup, for each wall, flat, and sprite, FOR EACH FRAME!
The hash lookup avoids the linear lookup, so it's very fast. This one is a BIG win, with no real downside (uses another 8 bytes/lump, though). It made quite a noticeable difference for me. And, the hash code presents matches in vanilla order (last lump first), so you get that for free. (texture BLUWALL will be found before colormap BLUWALL, even without assigning namespaces).

2. See if your issue with that wad file goes away, without having to worry about the namespace stuff just yet.

3. Maybe add full bounds checking/sanity checking on your texture compositor, just to prevent further texture issues. maybe using the wrong BLUWALL as a test case.

4. Maybe consider implementing some of Boom's Medusa fix if you haven't already (R_GenerateComposite, and friends). Wow, I just took a look at Legacy's r_data.c - looks like you are doing lots of neat and complex texture checking and fixing stuff - maybe you don't need Boom's approach after all. Take a look at what Boom does though, if you haven't already - it works pretty well for what it does.

Man, I am going to be researching what you've done with your texture engine - there's some interesting things going on there!

Ok, that's all I can think of. Whatever approaches you use, good luck - you'll do the right thing!

Share this post


Link to post

Enough. Patches have been committed. The Legacy_demo works again, including the hidden texture with the BLUMAP patch.

Implemented test to detect bad patch lumps, to protect against segfault.

Implemented namespace tracking, on the fly during lump loading.
Lumps are now marked as to namespace.

Implemented search for patches in patch namespace first, then accept any lump with same name (because of other level maps).

Yes, search within a wad looks to me to be forward, but that is hard to confirm, it goes forward through the indexes.

Done for the time being.

Share this post


Link to post

Sounds like someone did an all-night crunch :)

Share this post


Link to post

Today I have time to comment.

DoomLegacy has cache for textures and other things, but apparently at different places than Boom. This makes it difficult and potentially damaging to try to drop in any solution.

Lump name search is fast because Legacy does not do string searches.
Before this latest change it was done using two integers, but now a single uint64_t comparison compares the 8 byte names. One more instruction at lump load ensures that the trailing bytes are all zero.

Textures are in texture cache as a single picture format or single patch format. Multi-patch textures are combined into a single patch by R_GenerateTexture. Textures are drawn using a texture id, without any lump name lookups at frame rates.

I had looked at PrBoom several times, and could never see the namespace code. With the listing from kb1, I now see the namespace code. I kept looking in the lump loading code and saw no tests for START lumps.


The best way to create standards is to create standard DEMO wads. I use the Legacy_Demo.wad and BOOMEDIT.WAD extensively to do regression testing. There are some other legacy wads that I commonly test with also.

A TEST wad with certain namespace collisions would test implementations (ports). If the wad is handled correctly there will be clearly visible texture displays. Something like a wall saying "Patch name collision test. Should say Blue Patch", and next to it a texture that uses the Blue Patch that says "Blue Patch". A port that gets the Blue music or Blue colormap would display something else (probably messy).

If a majority of ports can display all the tests in the wad correctly, then there would be pressure for other ports to display it correctly too.
This works best with a test wad because feelings and interpretations of the rules are bypassed.
Regression tests can be done often, and very quickly by checking that the test wad still works.

There could be levels of conformance within the TEST WAD, or separate test wads. There cannot be just one set of rules because there already is Vanilla, Boom, and others. Legacy has its own extensions too, which the Legacy_Demo.wad tests.

Some of the things discussed here do not make BOOMEDIT.WAD fail.
They may have been implied, or there may be subtle differences, and some sector self references cause distracting render errors of their own.

Things that could be tested in a TEST WAD.
1. Namespace collisions.
1a. Blue patch, with Blue colormap, with Blue flat.
2. Textures with edge cases.
2a. Patches with offsets.
2b. Texture with single patch, where the patch is smaller than the texture.
2c. Patch much larger than texture. Clipping test.
2d. Multiple patch draw order.
3. Transparent textures.
3a. Fence texture.
3b. Multiple patch grill.
4. Sector self-reference effects.
4a. Invisible bridges.
4b. Floating signs.

Share this post


Link to post

I had to think about it for a while, and, I think the demo idea is excellent - as you mention, yes, having an actual demo file on your hard drive dispenses with all the forum text interpretations, and boils it down to what matters, like nothing else can: Does the port display this construct correctly, or not? Does the port do what's expected here, or not. How do other ports handle X?

Off the top of my head, here's a few more:

3c. Negative offsets
3d. Non-power of 2 width patches. DV5 had some funky width textures that caused some issues.
5. Sky rendering. (Practically every port does something a bit different, it seems).
5a. ...with freelook
5b. ...without freelook.
5c. ...w/widescreen.
5d. ...w/tall patches.

6. Weapon placement (with the same subcategories as skies)
7. HUD elements

Those last two aren't really demo-wad-related, per se, but you could use some non-standard weapon and HUD images, and see what happens. A great idea.


On the lump-seek thing, I strongly suggest trying it out. Even with the fast uint64_t compare, for a 2,000-lump wad, you're average number of elements to search to find a match is 1,000. With the hash, it's theoretically between 1 and 2, regardless of lump count! (Though, in practice, that's overly optimistic - you may have to check a dozen, in average worst case, if that makes sense. And, there's a small setup time). Still, if you're looking to optimize, it's a low-hanging, high-yield fruit, it's totally compatible, and implements easily. Something I'd suggest considering. If you're using texture ids, that certainly bypasses name lookups for textures at game time. In that case, the hash would help during texture generation, at startup. And, there's sprites, sounds, flats, colormaps (heh). Bottom-line: I think you'd be pleasantly surprised.

Share this post


Link to post

The hard part of a test/demo wad would be coming up with test textures that display a clear distinctive indicator of correct rendering.
In many cases an example of how the result should look would have to be next to it.

Example: texture with offsets (some negative).

The patches applied should be clearly visible, like a different color, and with graphic elements, and probably some text that says "patch1".

Next to it needs to be a single patch texture of how it should look.

If there is a common way to fail, then a texture of that failure could be also present with text on it identifying it. Easier to use than having a doc accompany the test wad.

Example: Test elements that crash some engines.
Have to be in separate wads because it is difficult to determine what is crashing a port when two or three possibilities are mixed into one wad. This implies a test suite of test wads.

Share this post


Link to post

You might be able to hide certain classes of crashes behind doors. That would actually be kinda neat - have a hallway with doors, each one concealing a different potential run-time renderer crash. More scary than a archvile behind the door, there's an actual crash instead :)

That could actually be useful in a lot of ways.

Share this post


Link to post

Beware that what crashes an engine only when you look at it, will crash other engines at load time.
An example is this is this crash of DoomLegacy. Legacy precaches all textures used in the wad, even if they are not visible by any path the player could take.

Share this post


Link to post

Create an account or sign in to comment

You need to be a member in order to leave a comment

Create an account

Sign up for a new account in our community. It's easy!

Register a new account

Sign in

Already have an account? Sign in here.

Sign In Now
Sign in to follow this  
×