You are the IT Guy. You come into work early, turn on your system and settle in for what appears to be a nice mellow day…then BAM! The world as you know it ends!
Every workstation in your environment starts to mysteriously hang and freeze. Productivity stops and the user requests start piling in. Tickets, phone calls, walk-by’s. Your day has now been inundated with one task…find out what the HELL is going on?
The first report is that someone opened a virus email and clicked a link, the second is that you have a rogue workstation with a defective NIC backing up the network, None of these add up. If there is a Virus why is there no fallout? No damage, no errors, no missing files? The rogue workstation is quickly ruled out as it is taken off the network.
Still no clues.
You settle in for a day of combing event logs. The first clue pops up. You have 3 Servers on the network that you can ping, browse, and are physically running but you can’t RDP to them. What?!? At this point any processes running above 100k are killed and systems are rebooted. After a bit the network starts to simmer down and people start working. You plan an all out system reboot after hours and hope for the best.
The next morning you are treated to a nice snappy network and everything seems to be running at optimum speed. The first person logs on as you watch with apprehension and everything still looks fine.
Then at 10:22am the world again falls apart. You sit in your cubicle and listen to the wave spread over the entire building. It starts with murmurs that someones email has slowed down, then someone can’t reboot, then people are up and walking around, finally someone stops in and says “Hey my computer stopped working”. OMFG are you kidding?
Now the real fun starts. Why the heck would it start at 10:22am on the button. What the heck is running. You backtrack to the processes you last saw running. Avp.exe. Anti-virus. First thought, isolate the AV Server and start turning off any data mining apps before the entire business collapses.
After shutting down the server and rebooting workstations things start to even out and people are able to work. Only isolated cases of machines having issues are trickling in.
Ok now it’s time to dig in. You unplug the network cable from your AV Server and fire it up. What the heck is this thing doing. Hmmm, update tasks are still trying to run. Applied to 116 computers but not completed? Huh? Time to call in the support.
Deftly punching in the number for Kaspersky Corporate Support you are greeted with an ominous message.
“Please hold for the next available support technician. We are currently experiencing higher then normal coll volumes. You can leave a message or stay on the line to speak with a technician.”
Hmmm, that does not sound good.
After about 20 mins a techician comes on the phone. You explain your current situation and how odd it is. The response: “Yeaaaaa (I am suddenly taken back to Office Space), you’ve just experience a bug caused by a bad update that was pushed out.” Wait…a what? Oh hell no.
The fix:
- Schedule a notification letting all employees know to leave their computers on after leaving.
- Schedule a task to reboot all machines.
- Change the Update workstations task to push out the update once only and directly after the reboot.
- Schedule a second reboot task to follow the update.
Does this work? I will know tomorrow…
It seems kind of silly to me that your Anti-Virus company is sending out updates that are bad and not notifying you. For all the maintenance and support that is paid to keep these things running efficiently and effectively it seems to me that we should have been notified the day it happened.
Instead I have a technician on the phone apologizing in a very unconvincing manner. Looks like I will be putting in a call to my sales rep tomorrow.
The Whiteboard 
i am also facing the same problem since yesterday 18-12-12, what should i do to fix my servers.
This comment has been removed by the author.
What I was told by Kaspersky support is that a patch for 32bit systems will be ready tomorrow (12-18-12).
What you can do is go into the administration kit and stop the update task that is running. Change the setting to Manually update “Once” that way it won't keep trying over and over.
I was also told that the fix was to let the update run through to it's completion. What I am doing is running it off hours to ensure there is no business impact.
To get my servers unlocked I needed to stop the AVP.exe process that was running between 160-170k.
Hope that helps. I will know more tomorrow.
what would u suggest me ? should i exit the KAV ? and wait for an update ?
I would call Kaspersky support and ask about the patch.
i have just exit that KAV. after restarting server. and its working perfect. Please update regarding new S/W update from KAV.
any updates ?
As of yesterday the process of forcing an update overnight and rebooting all machines directly after the reboot has resolved the issue.
Upon checking all versions it looks like Kaspersky pushed out new updates. most likely patches, as of 12/19/12.