Search the Community

I figured it might be entertaining for others to read about the kind of things I run into at work that tend to be rather bizarre yet rather likely that you'll run into when you work in IT. And it gives me a chance to vent a little =p To give a bit of background information, I am a tier 3 sys/network admin at an office with around 200 computers connected to 3 other buildings with 200-250 computers each which each have their own IT infrastructure and staff. (each is its own organization, this might sound weird but for privacy reasons I'm leaving out what kind of office I work at, which would have it make more sense:-) ) Tier 3 means I get to deal with disasters when everybody else is freaking out, considering our small size tier 1 are interns who answer the phone and deal with small (l)user problems, tier 2 are the normal sysadmin's (there's 3 of em) and then there's me. I also deal with smaller problems when there's nothing broken but usually I just growl and let others deal with those, taking a more advisory role instead which gives me time to slack off. I also occasionally pitch in for our neighbors when they have something they cant figure out, being the only networking expert. We also run a rather varied range of OS' on our servers ranging from Win2k3/8 to Enterprise & desktop linux versions, as well as 'true' unix so there's lots of variation in problems. Our main network is not Microsoft so that means we do have a lot less issues then our neighbors on the network. I have also read the BOFH diaries more then once from start to finish =p So today I get an alert that the CPU of our main mailserver(runs on linux) is generating an above average load, while I'm firing up the CPU monitor on a shell the phone rings .. its one of our interns and he's going on about how somebody's mail client is freaking out and after spending about 1 minute to try and figure out what is going on (not the brightest intern) it turns out this user 'magically' had 56000 new emails in their mailbox .. Now I know for a fact that we only have around 100k received e-mails in our system on average, me yelling at people and an system wide auto delete policy having educated our users on a semi decent mailbox management. Right here we obviously had the source of our high cpu load, the server having generated all those 56000 e-mails in about 5 minutes time... After inquiring where these messages were coming from I checked the 'sender' user mailbox for weird rules (rule generated ping-pong messages are frequent, auto reply on auto reply) and what not but that was clean so I decided to take a better look at the stuffed mailbox which they were trying to delete messages from. Deleting messages is a lot slower then creating automatic messages so this was going at an amazing rate of 10 per second. After doing some more inquiring (you REALLY have to learn how to interrogate people and what questions to ask when you work in IT) it turned out that just before they had sent out an appointment to a few (thankfully just a few) other people. This appointment consisted of a repeated appointment for every day for the rest of the year, and they had marked it as that they wanted a notification about who opened it, who accepted or declined it and further status tracking.. So for each user, for each day of the year, upon opening/accepting the appointment, the system generated a pile of messages and sent those back to the originating user .. coming to a grand total of over 56000 messages. Deleted user mailbox, ran a cleanup and created a new mailbox .. problem solved. Protip: over scale your e-mail server hardware EVEN if it runs on linux (dual quadcore laughed at this). This will save you a lot of future problems and user performance complaints as well as allows you to postpone upgrades for an extra year or 2 :-) Next to this, business computers these days come with 22" by default here. When your old standard monitor is 19" and you get a new shipment your company WILL become a statistical anomaly for people with eye problems. Statistically seeing all the people for the whole region who would have trouble with a 19" monitor work in my office. We do not replace all computers at once, there are 4 different models in use that get rotated out. Now you think this would not really be a problem, but people come up with the weirdest reasons for why they need a new computer/monitor. This even leads towards aggressive behavior to IT staff (usually towards the interns), trying to trick interns into making promises that normal staff has already declined, attempts to bypass IT by going straight to the top with angry faces and/or teared up eyes and if that fails they'll actually call in sick. You'll notice people who work close to staff often have high end computers even though all they do is ms office while somebody across the hall is actually having efficiency issues with their brand new software X that runs poorly on a 4 year old computer. IT constantly has to balance things to keep people happy, even 'cheat' by giving somebody who would benefit from a fast computer but already has a 1 year old computer the latest model so you can move last years model with a new monitor to somebody who is causing drama and tell them its a new computer. We do this a lot :-) Gives some extra work for us (us being mostly just the interns) but keeps people happy.