Server Outage.....

For those who want a more detailed explanation of our woes, here goes.

We have a very nice database server that decided it was time for a vacation. This server included a database for current running auctions, and also archived auctions. All bids are stored on another server. When the server with the database of current auctions decided to take a dump, our only option was to restore using a backup taken about 20 hours earlier. Because the rebuilt database was missing about 20 hours of time (new auctions), and records of the bids are stored on another server, any seller that had created an auction after the backup and before the crash, lost that auction.

Due to how our auctions are run, whenever a seller creates an auction, a bid representing the seller's starting price is also created on another table, along with the new item number generated during the item creation process. When we restored from a backup, we essentially reset the order of item numbers back to just before the backup itself was created.

So what you have is a series of item numbers used up for about a 20 hour period while auctions are being created, and then you have the current auctions database being reset back to a time prior to those 20 hours worth of auctions (and item numbers) being created. Then, after the restore, a new batch of auctions gets created, and the database generates that same series of item numbers again, but for a completely different set of auctions.

I'm sure you can see our problem.

We have a 20 hour overlap of auction numbers for a bunch of listed items. Ugh.

So I set off on the herculean task of writing a process that will examine each auction, and try to eliminate bids for the new set of auction that we actually from a prior auction that was created after our last backup, but before the servers crash. I was able to fix the majority of them, but a number of auctions didn't meet the criteria I set up for the screening process, and slipped through.

So that's the saga that kept us sweating for the last couple of days.

If you see anything funky in the bidding history, give us a holler and we'll sort it out on a case-by-case basis. This should iron itself out as old auctions flush out of the database, and new ones are added. We've already seen a dramatic decrease in the amount of oddballness. But I really appreciate your patience, and I know we're just about done with it. Thanks again.
-manny