XWE Beta: File was Modified by External Program

[ReX]

In this thread:

Csabo said:

The only problematic one is the "check-if-file-was-modified-by-external-program". I might make this optional or something. Works here, but people have been complaining about it popping up at weird times, and no-one seems to want to track down if there's really an issue with it.

I said:

I have been reading about the "problem". When I have some time I will check the issue out on a test wad.

I had not received the warning message for a while. Then it happened when I had DooM2.wad open, and was doing a search for a specific lump (M_OPTTTL, in this case). The warning message was as follows:

WARNING
The currently opened file was modified by an external program.
To avoid file corruption, you should then close and re-open this file.

I did as suggested by the message, then looked in Windows Explorer to see if the file had changed. Sure enough, the date stamp on the file was today's date, indicating that something had changed. I reopened DooM2.wad and looked through a few entries, but they looked fine. ZDooM and GZDooM both loaded DooM2 and ran Map01 fine, but I am somewhat disturbed that the wad was modified (even if I can't tell what was changed).

I will update this thread if the problem recurs.

[ReX]

Today I got the warning again. This time I was in the Textures viewing/editing screen and had earlier saved my texture changes. I did a search in QuickFind for a patch, and when I clicked on the patch I got the warning message. When I checked the date stamp for the file, it was showing the time of the "modification".

Other programs I had running were MS Office Picture Manager (displaying the contents of a different folder), Wally and MS Windows Explorer (each browsing the contents of a different folder).

Anyhoo, I closed the wad but didn't close XWE. When I tried to load the wad again I got this message: "I/O error 6". I tried to close XWE but the message popped up again. Finally, I used Windows Task Manager to close XWE (at which time I got the usual "This program is not responding ...." message). I don't know if the warning and the error are somehow related.

EDIT: More bad news to report. I looked in the wad that I needed to close after the warning, and my entire TEXTURE1 entries had been wiped out. The lump itself was there; there just weren't any textures in there.

I recovered an earlier, uncorrupted, version of the wad and opened it up. I went into the TEXTURE1 lump, and then wanted to do a search on QuickFind. The Warning message popped up again. I closed the file, then closed XWE. This time there was no I/O error. I re-opened XWE and pulled up the file I had to close, and once again the TEXTURE1 lump was empty. This time, however, there was no backup wad created, so it would appear that I am truly fucked. Thankfully, I have an earlier version of the wad (several days old) with an intact TEXTURE1 lump, which contains all the changes I have made through March 5. I will recover that and insert it into my wad.

The lesson here appears to be that if the TEXTURE1 lump is open, the user should avoid doing a search on QuickFind.

[Lobo]

I don't want to sound like a jerk, but here I go ;)

Why would you repeat the same steps on your uncorrupted wad that you knew caused a bug on the previous attempt *without* first making a copy of it?

[ReX]

Lobo said:

Why would you repeat the same steps on your uncorrupted wad that you knew caused a bug on the previous attempt *without* first making a copy of it?

Good point. The reason, I guess, is that I thought the warning occurs at random (or at least I did not make the connection and realize that the problem would occur when I was doing a search in QuickFind while I had TEXTURE1 open). Thankfully, after March 5 I had only made a few changes to TEXTURE1, which I was able to duplicate without too much extra effort.

At least now I know what not to do the next time. More importantly, I hope this helps Csabo to narrow down and fix the problem. [In the past, users had complained about the warning itself, and had not indicated that there were any problems associated with it. My case indicates that a definite problem could be associated with the warning.]

[Quasar]

Why in the world would a search operation ever even have the potential of modifying the objects being searched over? ...........

[ReX]

Another instance of the problem, this time with a different sort of operation. For some reason my Map01 entries were all the way at the bottom of the lump entries. I cut them and wanted to paste them above the Map02 entries (which were at the top of the list), when I got the warning message. I don't think anything got messed up, but to be sure I deleted the wad and loaded up the most recent saved version. This time the wad did not load up, I got the "I/O Error 6" message, and had to use Windows Task Manager to close and exit XWE.

Next time around I created a dummy entry, and left it UNTITLED. I cut the Map01 entries and clicked on the dummy entry so that the map entries would be copied below that. Again I got the warning message, and went to close XWE, but it crashed with the "I/O Error 6" message.

[ReX]

Same problem, different circumstances. I opened the wad (incidentally, the last few times the problem has occurred it's been on the same wad, but that can't be the cause as the problem also occurred with DooM2.wad) and typed in the first few characters of a patch name in QuickFind. The search engine brought the list of lumps to the appropriate position, but when I clicked on the patch to display it I got the warning. This suggests that Quasar's speculation (of a search operation being the source of the problem) has merit.

When I clicked OK (I needed to do it twice) and the message disappeared, the patch opened up. When I scrolled back up to check the DECORATE lump I got the same warning. The DECORATE lump was now empty. I clicked on TEXTURE1 and I got the same warning. This time clicking on OK closed XWE. I reopened XWE and it loaded the wad. DECORATE was still empty. When I clicked on TEXTURE1 it closed XWE. Tried it again, and XWE closed again.

The first entry in my list of lumps is DECORATE, if that makes any difference. In other words, I had DECORATE open when I did the search (although in the past I had TEXTURE1 open when I did the search and the problems occurred).

[Csabo]

The quickfind box just locates the first matching lump and sets the focus to it. It definitely does not modify the WAD in any way. However, when a lump is clicked on (or selected through the quickfind), that's the first time XWE checks for the file modified date. So chances are, the file is already modified at that point. To better prove this (and to further help file corruption), maybe I should add a timer that would perform this check every one second or so in the background.

I tried to do all of the stuff you described but could not get the file modified warning to come up. I did find a few smaller bugs though that gave I/O Error 6. (That means invalid file handle btw, the most common cause is trying to work on a file that's not open.) For example, if you closed a WAD in XWE, you could still double-click the filter buttons, and it tried to import stuff. I fixed these and uploaded the latest beta.

For the last issue you described: in this case too, it really looks like the file was actually modified. XWE right now just gives you a warning and lets you continue, but maybe it shouldn't. You see, what happens is this: XWE reads the WAD file. The WAD file has a directory that specifies where the lumps are within the file. Let's say your DECORATE lump is at file position $1000. At some point XWE detects that the modified date of this WAD is newer than it should be, so it gives the warning. In the "new" file, DECORATE could be at $2000, or anywhere else. So when you clicked OK twice, and saw empty lumps, that's why it happened: XWE still looked for your lump at $1000, but the file was modified and now something else (or nothing) was at that position.

So thanks for your continued feedback, I really hope we can nail this. For now just grab the new one and see if you get it again.

For everyone else: could you guys please try to reproduce the file-gets-corrupted-on-quickfind-if-textures-lump-is-open issue? It doesn't happen to me, regardless of what I search for, or if I modify the textures lump or not.

[ReX]

Csabo said:

You see, what happens is this: XWE reads the WAD file. The WAD file has a directory that specifies where the lumps are within the file. Let's say your DECORATE lump is at file position $1000. At some point XWE detects that the modified date of this WAD is newer than it should be, so it gives the warning. In the "new" file, DECORATE could be at $2000, or anywhere else. So when you clicked OK twice, and saw empty lumps, that's why it happened: XWE still looked for your lump at $1000, but the file was modified and now something else (or nothing) was at that position.

The thing is that I had not modified the file in any way when I did that search. I will try the new beta to see if it does the same thing.

For everyone else: could you guys please try to reproduce the file-gets-corrupted-on-quickfind-if-textures-lump-is-open issue? It doesn't happen to me, regardless of what I search for, or if I modify the textures lump or not. [/B]

Remember that the problems (i.e., both the warning and the messing up of lumps) occurred not just when I was in the TEXTURE1 lump; it also happened up when I was in the DECORATE lump. Also, it happened even when the first operation I conducted upon opening a wad in XWE was a search. Note that I didn't need to have modified the TEXTURE1 lump - I simply had it open when I did the search.