I was super tired this morning after the trip and just wanted to stay in bed. I really should have had I known what were about to happen today. Those of you know me know that I am a web programmer. I code web pages all day and work on the database. This morning I got an email that there is a page I need to troubleshoot because it's causing an error. I was testing the query in the database to see if it's running correctly. In the older version of the database, there is a warning before the query would commit. However in the newer one, the query just ran without any warning. I basically update over 42,000 records with the exact same information. I tried to undo and rollback but it didn't work.
I ran to the IT guys and ask if they have a backup copy of the database. I was so happy and relieved when they told me they backed up the database every night. All things start falling like a domino effect when the IT guy came with a concerned look on his face. Apparently they backed up all other database but somehow overlooked this one. I went into crisis mode and try to recover data. I have a 2 months old copy of the database on my local computer for testing and went to recover the data from that database. Fortunately I overwrote only some fields and not the main ones nor delete them out. Since my database is 2 months old, there are over 8,000 records that are unaccounted for.
I went to break the news to my boss taking full responsibility and expecting her to reprimand me. However, I was surprised that she was more upset at the Network guys for not backing up the database. To give you a glimpse of the seriousness of this situation, the COO was involved and I wouldn't be surprise if our Executive Director hears about it also. Half of our department will have to go through the files and enter in all these data that I overwrote. Needless to say I felt crappy all day even though everyone tried to sympathize and tell me these things happen.
The politic part of things is that the IT Manager, our Director, and I are all friends. I hang out with both of them after work and know about their family life. The IT manager wasn't in today so I called him and tell him about the crisis. I was surprised he was quite nonchalant about it. I think he figure it's my problem and I have to fix it. I don't think he realize that everyone is putting most of the blame on them for overlooking the backup. I tried to call him after work to give him a heads up of what's going on but he never return my call. Maybe he thinks I am calling to bother him about the database that I messed up on...I am not sure. I also am worry that he thinks our Directory is playing favorites toward me because he gets the blame.
All I can say is that I hope the rest of the week is better.
2 comments:
OMG! What a nightmare for you to have to go through. I'm so sorry. I can totally relate. I've had days where I've walked around feeling miserable over a goof up and it's not a good feeling. Try not to be so hard on yourself and just take it one step at a time. Did everything fall back into place?
Elizabeth - it took 2 weeks of data recovery, 10 temps, 6 staff, and 10 days of late nights for me to go manually go through all the files to match up the corrupted data. We still only managed to recover 6,000 records so 2,000 are just lost.
Post a Comment