Monitor Everything

by Shawn @ 2:06 am · filed under Server Admin

Okay, is anyone tired of my server admin tips yet? Yes? Too bad.

Monitor everything... Put as much info at your finger tips as easily as possible. Put that info in a place where you will always be looking at it for some reason. For example, I made a vBulletin plug-in that monitors 4 memcached servers (including a latency test it runs) as well as 10 blade servers. This shows every time I'm in the admin of the forum (which is a lot), so I can't help but to not see it.

I wrote a little daemon that runs on my servers that can quickly report back whatever info I want (time, disk RAID status, server load, MySQL replication status, etc.)

The more info you have in one place (especially when you run a bunch of servers), the easier it is to see anything wrong. For example, I had an issue with a web server serving requests slow one day... it ultimately ended up not being a problem with the web server, but the memcached server it was using. The latency test was showing ~2,000 ms latency (2 seconds) vs. the normal 0.5 ms (1/2000 of a second).

And be proactive about monitoring stuff... don't wait until something bad happens to start doing it! Then it's too late.

Can I go to bed now?

Permalink

5 Responses to “Monitor Everything”

Patrick Says:
October 26th, 2006 at 12:00 pm

Very cool. Have you tried Fusion Charts? I’ve been using the standard charts for basic stats…but I have my eye on the real time capability of the instrumentation suite. Not really needed, but I like looking at pretty graphs…

-Patrick

Anthony Cea Says:
October 29th, 2006 at 7:33 pm

Pretty cool if you have enough brains to do it, Nintendo does not, teach him to do this!

Damnz Says:
October 30th, 2006 at 10:44 am

Pretty cool .

Here’s one of my stupid Qs ; Looking at the chart, I saw “Pointer 64″. Based on what I *think* I know, that means the almighty DP servers are running on 64-bit servers?
Feel free to call me an idiot. heck! thats what i call myself .

ViciousSummer Says:
November 5th, 2006 at 2:16 am

Don’t forget about the Sister Monitor that you have running…When you arn’t nerding out in front of your computer, guaranteed that the Sister Backup Monitor is. Granted, the sister backup can’t trouble shoot the problem beyond a phone call to say, “Hey dude, your website done be broke and shit…”. But hey, knowledge is half the battle!

Isds Says:
November 19th, 2006 at 10:22 pm

Os this where to ask?

I share a DSL conection with neighbors and they love to use their “ares” musci downloader. You can imagine that each time they decide to open it i loose any bandwith avalaible and can not get connected to internet. No matter if I limit the bandwith in their software, anyway it practically “sucks” all the bandwith.

The router is a 2wire 1070 in a 1024kb conection.

Was getting into the firewall settings but can’t get how to do it to limit their computer and avoid them to suck all the bandwith…

Any suggestion will be apreciatte.

Thanks.

G.

Leave a Reply