Single Status Update
tl;dr version: I lost 4 days worth of backups, possibly in a bit of trouble, stress++
Earlier this afternoon I encountered problems with our backup software. Initial investigation indicated the internal database had become corrupted. After trying a few things (over several hours with HP support), I decided to restore the DB from a backup taken this morning. This involved connecting the tape library to another server, importing the 40+ tapes and restoring the DB back to the original server. Not overly complicated but time consuming. It fixed the issue.
A few hours on, I was checking the nights backups (our backups are pretty much running 24/7 and need constant monitoring), I realized the DB was missing session information (records of what was backed up and to what media) from the last 10 days. I soon found out that I hadn't imported all of the tapes to the other server - as a result the software didn't know about the more recent copy and restored an older copy of the DB.
This wouldn't have been an issue, except that I had already started the nights backups, potentially overwriting a backlog of 4 days worth of backups that the DB didn't know about (we backup to hard drive first (fast) then tape(slow)). Effectively, the data in the Internal DB and what was actually on the tapes was out of sync.
I hated backups already - I really hate them now.
Edit: Just realized the tapes have protection applied, which is recorded in the IDB as well as the tape itself - the protection would prevent it being overwritten. The data should still be there. Hmmm. Still hate backups though.