Doom Marine
Register | User Profile | Member List | F.A.Q | Privacy Policy | New Blog | Search Forums | Forums Home
Doomworld Forums : Powered by vBulletin version 2.2.5 Doomworld Forums > Classic Doom > Source Ports > Extended blockmap format?
Pages (2): « 1 [2]  
Author
All times are GMT. The time now is 04:39. Post New Thread    Post A Reply
natt
Junior Member


Posts: 248
Registered: 05-11



Quasar said:

Not when just building a correct unlimited map from scratch at runtime is both less error prone and also has the same amount of "compatibility".



I'm not sure what you're saying; building a correct limit-free blockmap doesn't fix this.

Old Post 06-29-11 14:37 #
natt is offline Profile || Blog || PM || Search || Add Buddy IP || Edit/Delete || Quote
Maes
I like big butts!


Posts: 8664
Registered: 07-06



Maes said:
rebuild it while having the good sense of NOT using a broken, limited 16-bit format for offsets :-p

Old Post 06-29-11 16:04 #
Maes is offline Profile || Blog || PM || Homepage || Search || Add Buddy IP || Edit/Delete || Quote
natt
Junior Member


Posts: 248
Registered: 05-11



Maes said:




natt said:


I'm not sure what you're saying; building a correct limit-free blockmap does not fix this.

Old Post 06-29-11 16:19 #
natt is offline Profile || Blog || PM || Search || Add Buddy IP || Edit/Delete || Quote
Quasar
Moderator


Posts: 4615
Registered: 08-00



natt said:


I'm not sure what you're saying; building a correct limit-free blockmap doesn't fix this.


Then I guess we're talking about two different things.

This thread so far has been about trying to hack around broken blockmaps by reconstructing the missing upper 16 bits of a nominally 32-bit block offset by assuming that the offsets which are over the limit proceed in a monotonically increasing fashion modulo SHORT_MAX.

If the topic changed at some point, then I missed it.

Either way, a *correct* unlimited blockmap building algorithm which ignores the original broken input entirely should not have any "problems" to be fixed. If you are having problem with this map which involve ports that generate a blockmap to replace broken ones entirely, then there is something else intrinsically wrong with the map in question.

Eternity will build blockmaps for any possible size of level, up til there are more than 2 billion blocks. All other limitations of the game engine would be exceeded long before that ever happened.

I don't consider doing anything less to really be a fix at all if it has to engage in guess work.

Old Post 06-29-11 16:38 #
Quasar is offline Profile || Blog || PM || Email || Homepage || Search || Add Buddy IP || Edit/Delete || Quote
natt
Junior Member


Posts: 248
Registered: 05-11



Quasar said:

Then I guess we're talking about two different things.

This thread so far has been about trying to hack around broken blockmaps by reconstructing the missing upper 16 bits of a nominally 32-bit block offset by assuming that the offsets which are over the limit proceed in a monotonically increasing fashion modulo SHORT_MAX.

If the topic changed at some point, then I missed it.




Considering that I made a post which described the exact specifics of the problem down to source code excerpts, then yes I'd say you missed it. I'll reiterate in condensed form:

Regardless of how the blockmap structure is stored, blockmap functions will cease to operate properly more than 32k units from the blockmap origin (in either x or y direction) because of an overflow in fixed_t arithmetic which is calculating position deltas from the blockmap origin. This gives a hard mapsize limit of 32k by 32k square outside which there will be no functional blockmap even with a correctly done limit-removing blockmap.

And no, I don't advocate trying to fix this limit; just understanding it.

Old Post 06-29-11 16:57 #
natt is offline Profile || Blog || PM || Search || Add Buddy IP || Edit/Delete || Quote
Maes
I like big butts!


Posts: 8664
Registered: 07-06



natt said:
Stuff about overflowing fixed_t



While I got your point, it's moot if we're talking about providing a COMPLETE non-broken solution to the blockmap issue. And if that point in the code is broken as it is, then it obviously needs fixing too, if we are NOT to be talking out of our asses.

OK, so individual fixed_t numbers are indeed limited and subtracting them or doing pretty much anything on them, can't express a difference/result greater than 32K, as you correctly pointed out.

However none is stopping you from extending the fixed_t to a larger type just before doing that operation.
This can be solved very easily by casting the fixed_t numbers (which are really int32) into a larger type e.g. int64 (or ints into longs, if your don't mind the ambiguity).

Et voila', now you have a whooping extra 32 bits to play with (while you really only need one to express 64K). No more overflow ;-)

Last edited by Maes on 06-29-11 at 17:37

Old Post 06-29-11 17:30 #
Maes is offline Profile || Blog || PM || Homepage || Search || Add Buddy IP || Edit/Delete || Quote
Gez
Why don't I have a custom title by now?!


Posts: 7046
Registered: 07-07


Also, do you really need sub-FRACINT precision for blockmaps? You could shift the values to keep only the upper 16 bits and this way get 16 extra bits.

Old Post 06-29-11 17:36 #
Gez is offline Profile || Blog || PM || Search || Add Buddy IP || Edit/Delete || Quote
natt
Junior Member


Posts: 248
Registered: 05-11



Maes said:


While I got your point, it's moot if we're talking about providing a COMPLETE non-broken solution to the blockmap issue. And if that point in the code is broken as it is, then it obviously needs fixing too, if we are NOT to be talking out of our asses.

OK, so individual fixed_t numbers are indeed limited and subtracting them or doing pretty much anything on them, can't express a difference/result greater than 32K, as you correctly pointed out.

However none is stopping you from extending the fixed_t to a larger type just before doing that operation.
This can be solved very easily by casting the fixed_t numbers (which are really int32) into a larger type e.g. int64 (or ints into longs, if your don't mind the ambiguity).

Et voila', now you have a whooping extra 32 bits to play with (while you really only need one to express 64K). No more overflow ;-)




Gez said:
Also, do you really need sub-FRACINT precision for blockmaps? You could shift the values to keep only the upper 16 bits and this way get 16 extra bits.


I think both of these are workable solutions, but I doubt either would hold up to the rigors of demo sync, which would mean another demo compatibility option bolted on.

Old Post 06-29-11 17:39 #
natt is offline Profile || Blog || PM || Search || Add Buddy IP || Edit/Delete || Quote
Maes
I like big butts!


Posts: 8664
Registered: 07-06



Gez said:
Also, do you really need sub-FRACINT precision for blockmaps? You could shift the values to keep only the upper 16 bits and this way get 16 extra bits.


Heh even better:

code:
#define MAPBLOCKSHIFT (FRACBITS+7)




natt said:
I think both of these are workable solutions, but I doubt either would hold up to the rigors of demo sync, which would mean another demo compatibility option bolted on.


Well, in theory those fractional bits COULD alter the result before being shifted, so in that sense extending to 64 bits would actually preserve them. The only way that an extension to 64-bits could break demo compatibility would be if a demo actually relied on the blockmap behaving in a broken way. But, if that's what you want then you don't go fixing it, right?

Cutting away part or all of the fractional part OTOH, would actually introduce outright approximation errors even when no other type of problem would arise.

Last edited by Maes on 06-29-11 at 17:47

Old Post 06-29-11 17:40 #
Maes is offline Profile || Blog || PM || Homepage || Search || Add Buddy IP || Edit/Delete || Quote
natt
Junior Member


Posts: 248
Registered: 05-11



Maes said:

The only way that an extension to 64-bits could break demo compatibility would be if a demo actually relied on the blockmap behaving in a broken way.



That sounds right, but I know better than to trust myself on those sorts of things.

Old Post 06-29-11 17:51 #
natt is offline Profile || Blog || PM || Search || Add Buddy IP || Edit/Delete || Quote
finnw
Junior Member


Posts: 104
Registered: 01-04



Maes said:
While I got your point, it's moot if we're talking about providing a COMPLETE non-broken solution to the blockmap issue. And if that point in the code is broken as it is, then it obviously needs fixing too, if we are NOT to be talking out of our asses.

That only applies if it is necessary to support maps that big. IMHO it is not. Vanilla/Chocolate Doom will never be able to load a map like that, and if you were designing it for a modern port you would probably split it into multiple maps with a hub.

wesleyjohnson's solution (for Legacy) is my favourite - Make a little effort (but not too much) to restore overflowed blockmaps (thus preserving special effects) but accept the 32Kx32K overall limit.

Old Post 06-29-11 18:34 #
finnw is offline Profile || Blog || PM || Search || Add Buddy IP || Edit/Delete || Quote
Quasar
Moderator


Posts: 4615
Registered: 08-00



natt said:

Considering that I made a post which described the exact specifics of the problem down to source code excerpts, then yes I'd say you missed it. I'll reiterate in condensed form:


Yeah, I'm officially thick. Sorry ^_^

But you are correct. DOOM runs into this coordinate space limitation in all of its data structures due to the use of 16.16 fixed point. The problem here is that the blockmap has a different coordinate space origin than the map does, so it's possible for valid space inside the map to be invalid in the blockmap coordinate space. Bad. >_>

Also not easily fixed.

Old Post 06-29-11 22:17 #
Quasar is offline Profile || Blog || PM || Email || Homepage || Search || Add Buddy IP || Edit/Delete || Quote
Maes
I like big butts!


Posts: 8664
Registered: 07-06



Quasar said:
Also not easily fixed.


Well...that one spot where you need to do the subtraction can be definitively fixed by expanding the operands to longs. Signs and absolute numeric value will be preserved, but now you will have more breathing room and an overflow/underflow will be much harder to induce (actually impossible with just a -/+32K coordinate system).

However you need to apply this fix EVERYWHERE where the blockmap is accessed in that way (e.g. in the archvile functions).

And of course it will break whatever expected stuff to be broken.

Old Post 06-29-11 23:55 #
Maes is offline Profile || Blog || PM || Homepage || Search || Add Buddy IP || Edit/Delete || Quote
wesleyjohnson
Member


Posts: 486
Registered: 04-09


That all is interesting. Just to save everyone from downloading DoomLegacy (ha ha), it follows here. Just for reference, or to make commenting on it more accurate. Inserted underlines because this post engine keeps removing the indenting, and it collapses all other multiple spaces to one space.


Notice:
1. The problem is in the blockmap offset table, which gives indexes into the actual blockmap lists.
2. There are protections to keep it disabled for most level maps.
3. The protections arbitrarily uses 0x1000, which could be tightened up to 0x0020 or smaller and still work.
4. How well this succeeds with overly large compressed blockmaps (with offsets that exceed 0xf000), is dependent on whether the shared lists are within 0x1000 of each other and do not occur across the bme increment. I expect compression to reduce the blockmap table to smaller than 0xf000 (keeping it valid), and thus reduce the chance of another Europe.wad problem.

Partial Listing from reading of blockmap, in p_setup.c

firstlist = 4 + (bmapwidth*bmapheight);
lastlist = count - 1;

// read blockmap index array
for (i=4 ; i<firstlist ; i++) // for all entries in wad offset index
{
_ uint32_t bme = LE_SWAP16(wadblockmaplump[i]); // offset
_ // upon overflow, the bme will wrap to low values
_ if ( (bme < firstlist) // too small to be valid
_ && (bme < 0x1000) && (prev_bme > 0xf000)) // wrapped
_ {
_ _ // first or repeated overflow
_ _ overflow_corr += 0x00010000;
_ _ fprintf( stderr,"Correct blockmap offset[%i...] overflow by adding 0x%X\n",
_ _ i, overflow_corr );
_ }
_ prev_bme = bme; // uncorrected
_ // correct for overflow, or else try without correction
_ if ( overflow_corr )
_ {
_ _ uint32_t bmec = bme + overflow_corr;
_ _ // First entry of list is 0, but high odds of hitting one randomly.
_ _ // Check for valid blockmap offset, and offset overflow
_ _ if ( bmec <= lastlist
_ _ && wadblockmaplump[bmec] == 0 // valid start list
_ _ && ((bmec - blockmaphead[i-1]) < 1000)) // reasonably close sequentially
_ _ {
_ _ _ bme = bmec;
_ _ }
_ }

_ if ( bme > lastlist )
_ _ I_Error("Blockmap offset[%i]= %i, exceeds bounds.\n", i, bme);
_ if ( bme < firstlist
_ _ || wadblockmaplump[bme] != 0 ) // not start list
_ _ I_Error("Bad blockmap offset[%i]= %i.\n", i, bme);
_ blockmaphead[i] = bme;
}
// read blockmap lists
for (i=firstlist ; i<count ; i++) // for all list entries in wad blockmap
{
_ // killough 3/1/98
_ // keep -1 (0xffff), but other values are unsigned
_ uint16_t bme = LE_SWAP16(wadblockmaplump[i]);
_ blockmaphead[i] = (bme == 0xffff)? ((uint32_t) -1) : ((uint32_t) bme);
}

Last edited by wesleyjohnson on 06-30-11 at 05:00

Old Post 06-30-11 04:46 #
wesleyjohnson is offline Profile || Blog || PM || Search || Add Buddy IP || Edit/Delete || Quote
All times are GMT. The time now is 04:39. Post New Thread    Post A Reply
Pages (2): « 1 [2]  
Doomworld Forums : Powered by vBulletin version 2.2.5 Doomworld Forums > Classic Doom > Source Ports > Extended blockmap format?

Show Printable Version | Email this Page | Subscribe to this Thread

 

Forum Rules:
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts
HTML code is OFF
vB code is ON
Smilies are OFF
[IMG] code is ON
 

< Contact Us - Doomworld >

Powered by: vBulletin Version 2.2.5
Copyright ©2000, 2001, Jelsoft Enterprises Limited.

Forums Directory