It was a quiet Sunday afternoon. I was at home working on some financial stuff on my home computer. I was waiting for my daughter to call from Scotland (she lives there.) The phone rang – oh good Alana is calling – – but wait – – the caller ID say it is the home phone of our operations manager at the County where I work. This can’t be good, we are not social friends, I debate – should I answer it? I figure I should so when he tells me we have about fifty computers infected with a virus and two of his employees are in operations trying to find out what they can do to stop it – beside the fifty desktop PC most of the 80 plus servers are infected. While talking to him, the phone beeped – it was my daughter.
I got to talk to Alana for about fifteen minutes and then headed to the GSC where I work. When I got there Dan and Keith were working on cleaning the servers but it was not working too well. We use Symantec Enterprise for all our protection and it was finding two of the listed bad guys but when they were removed they came right back. We tried some other products and found one that identified Qbot as the real culprit and even that tool removed it but it returned a while later.
Keith worked with engineers at Symantec and after about two hours I realized that I could not do much as until we had a fix from Symantec we were stalled in the water. I discussed shutting off various links to networks we knew were infected but because our server farm had been infected – well it would not make much difference except that emergency services and other important things would be down. The Trojan did not seem to do damage to the systems as it was more for gathering information not havoc.
I went home and tried not to think about Monday morning. Keith had some success with the engineers – they isolated the biggest problem and sent rapid releases to fix it. Like many things we should do and don’t, we had never upgraded all our computers to the newest version of Symantec – that was a big mistake I will take some blame for. Many of our older computers were not capable of running version 10 and we assumed that they would not be able to run version 11 (current version) – the fact is that 11 takes less resources than 10. All the versions we had were pretty much able to stop the spread of the bad guy but it still took time. As anyone can tell you that works around a large network in many locations with different speed connections – time is not on your side. The Trojan moved faster than our up-dates to the software and because of that most of our 1900 computers had the Qbot on them. By noon on Monday we had it pretty much contained and it was not spreading (well when all the machines have it that happens too!)
It is now Thursday and we are just able to tell people in finance that they should be safe to log into on-line accounts, we are 99% sure our out-bound protection kept the Trojan from “calling home” with data from users but we are recommending they change passwords and monitor anything they may have accessed from their desktop PC.
Almost all of the stuff we sent to the engineers at Symantec was new and the first time submitted – lucky us!!!
It looks like a few weeks of vigil to clean any little hot-spots that remain – by today all our computers (that we can access) should be up to date.
I was running Windows 7 (we get it early) and while the Trojan was delivered, it would not run, it did send the Trojan to Microsoft under WER (Windows Error Reporting) because all 14 of the exe files crashed when W7 tried to run them – I guess they were not certified by Microsoft – I hope they don’t fix them so they run!