Okay, is anyone tired of my server admin tips yet? Yes? Too bad.
Monitor everything… Put as much info at your finger tips as easily as possible. Put that info in a place where you will always be looking at it for some reason. For example, I made a vBulletin plug-in that […]

Yay, more server admin fun! Here’s a useful *nix command that will let you determine what system calls a program uses… For example, I wanted to double check that libevent calls within memcached were using epoll() and select() or poll() calls (epoll scales better) on my SuSE Linux machines…
server:~ # […]

Purely out of necessity, I’ve become a system administrator/architect for digitalpoint.com servers… and a few people have been asking me for general admin tips to make things stable and scalable, so here’s a good one for you…
Use memcached… no really, use it.
memcached is a distributed memory caching system that allows multiple servers to access the […]

I found some bugs in the web interface for a Linksys WRT54G router I bought to fix some 2Wire “issues”. One of the bugs just would give me a false error when trying to do something and keep the form from being submitted.
Anyway, the short version of this entry is that I found out […]

Did I tell you how much I hate the 2Wire router/gateway you have to use for fiber Internet connections? You can’t get into the admin (which is web based) remotely, you can’t use DDNS so when your IP address changes weekly, you have no idea what it changed to, the firewall assumes you are […]

In theory, Gmail’s spam filtering is supposed to get smarter as you “train” it by tagging spam emails that slipped through their spam filters. I’ve been anal about going through every single email and tagging every spam email as such in the hopes that it would get smarter. In reality, I think it […]

Anyone know of some super awesome secret way to make a 2Wire Internet gateway also act as a dynamic DNS client? Since it’s a FTTP (fiber to the premises) gateway and not a “normal” DSL modem, I’m pretty sure I can’t just swap it out with something else that supports DynDNS.
The little I poked […]

MySQL is working on version 5.0.25 already. 5.0.23 was never released, and 5.0.24 was released on July 27, 2006.
So if 5.0.24 was released, where the hell is it? (it’s not on the download page)
http://dev.mysql.com/doc/refman/5.0/en/news-5-0-x.html
5.0.23 has a couple key bug fixes in it that I’ve been itching to get my hands […]

The new servers and equipment were installed into the data center yesterday (I also had to move the existing servers/equipment to a new rack), so everything is physically at the data center now (it’s not actually in USE yet, but at least it’s at a place where I can start moving stuff over to them).
Kind […]

So I was thinking about maybe doing MySQL fault tolerance and load balancing through hardware load balancers by setting up a virtual cluster for database reads and another for database writes. We could setup 2 master servers in a circular replication, making sure you only actually write to one at a time (define one […]

The new servers will be going into the data center this week (Monday if I can coordinate it), so we are close (finally!).
Just got some stuff fine-tuned with them today… wrote a cluster-copy and cluster-exec app for copying stuff across all blades and executing something on all blades. Made an init.d script that alters […]

So I’ve been fighting this networking crap for about a week now, and finally everything is working like it should. Just took a little editing of the underlying network routing tables.
I win.

“Hate” is a strong word, but in this case I think it’s appropriate…
I’m so f’ing sick of trying to screw around with network/routing problems with the new servers that I’m thinking about selling all my computers and going into construction.
Trying to fix arp routing issues is more than I ever wanted to know about networking… […]

Okay… MySQL Cluster (the storage engine) kind of sucks IMO. It’s terribly annoying that you can’t alter the DB schema of anything running it (even more annoying is that you can’t alter the schema of a database that’s NOT using ndbcluster, but just exists in the same mysqld process). So I think I’m […]

Just a FYI so people stop asking/emailing me about the status of the new equipment…
Just waiting for the 220V circuit to be wired in the data center (which hopefully will be in the next few days).

I’ve had a few people email me asking for a picture of the new blade servers that are going to be taking over as web/database servers for digitalpoint.com soon…
So uhm… here is what they look like right now sitting on the floor in my spare bedroom.
From top to bottom, we have a 48 port gigabit […]

Okay, I think I’m relatively happy with how the first blade is configured for the new servers. So I split the RAID mirror today, and put one of the drives into the second blade and am letting them rebuild their RAID mirrors onto 2 new drives. So soon I’ll have 4 hard drives […]

mpt-status is a command line utility to check the RAID status for LSI 1030 RAID controllers. Now can someone tell me why in the hell SuSE Linux Enterprise bothers to come with a version of mpt-status that doesn’t work with the Linux kernel that SuSE Linux uses???
It’s basically like bundling some application with Windows […]

Finally got the last (physical) piece for the blade servers (120GB RAM [60 x 2GB DIMMs]). I’m a swell counter because I counted them 5 or 6 times, and each time I counted them, I came up with 40, so I thought they shorted me 20 DIMMs for a 10 minutes or so. […]

I spent most of the day getting crap installed on the first blade as well as learning about little quirks with SuSE Linux Enterprise 9.3.
MySQL 5.0.21 was an easy install (a nice little RPM for SuSE Linux comes from MySQL).
Memcached was a pretty easy compile/install… just needed to compile/install eventlib first.
The big bitch was getting […]

Yesterday I spent about 2 hours configuring all sorts of crap in the new network switch… port assignments/names, QoS, SNMP crap, etc.
I forgot that even though changes you make are applied right away, they are lost if you power cycle the equipment. Hahaha… whoops!
Don’t forget this, I know I won’t ever forget it again. […]

The blade chassis arrived on a palette today (and boy, was that an awesome time getting it upstairs), so that means all the new server equipment is here (except for additional RAM).
We got 1 blade chassis, 10 loaded blades (except for RAM right now), 2 load balancers (1 is a hot spare) and a managed […]

The new blade servers were delivered this morning (10 of them). Hopefully should get the chassis tomorrow (it shipped separately), and the new switch and load balancers later in the week.

Wiggly (one of my cats) is terrified of change, and now he won’t leave my office because there are big boxes in the hallway. […]

My primary MySQL server has been VERY overloaded lately (which is the main reason new blades are on the way), but today I decided to see what I can do about it in the meantime (the parameters have already been tuned as much as possible).
First I toyed around with a single node MySQL Cluster… it […]

On one of my ultra-high traffic web servers, I switched from eAccelerator to APC today (an opcode/caching system for PHP). So far it seems pretty nice… Especially the ability to disable stat for each PHP request.
I ended up making a datastore class for vBulletin also so I could use it for the forum, so […]

What in the hell dude???
Now we are back to the $66,030 pricing for the 10 blades. This is $30,000 cheaper than the pricing yesterday.
UpdateRather than wait around until tomorrow when their pricing will probably change again, I went ahead and ordered them (without RAM, which I’ll add myself). So now […]

As I mentioned previously, I’m looking to get a bunch of Dell blade servers, but their pricing system (seemingly random pricing changes every day) is really irritating me. So I think I may have come up with a solution… Just buy stripped down blades and add the RAM, hard drives (and maybe even […]

Can I just tell everyone how annoying Dell’s price fluctuations are? I’m trying to purchase a blade chassis and 10 loaded blades. One day the blades are $66,030, then they are $105,400, then $66,030 again, now they are $88,040 (all pricing for identically configured blades of course). Finally I got pissed and […]

The core of WordPress (this blog software) is pretty much a piece of crap as far as it’s “guts” are concerned (although I knew this already, I just didn’t care because my blog doesn’t get enough visitors to really make that fact matter much).
Anyway, I woke up this morning to my servers being thrashed (database […]

Alternative PHP Cache is a PHP caching mechanism (like Turck mmCache, eAccelerator, etc.) that is being developed directly by PHP developers. In fact, PHP 6.0 is going to include APC Cache in it’s core framework, so it’s certainly something PHP developers/admins should start looking at.
Anyway, if you install APC Cache and Apache fails to […]

I had an issue where Googlebot was spidering parts of my site that were not allowed in the robots.txt file…
My old robots.txt file…

With version 5.1.7, MySQL 5.1 (which is something I’m [not] patiently waiting for) has gone from alpha to beta status.
The stuff I’m really looking forward to is it’s improvements to the NDB Cluster engine…

Disk Data tables (before NDB Cluster required everything to be memory-resident)
Integration of MySQL Cluster and MySQL replication
Variable sized records

Now hurry up through […]

I’m looking at the possibility of moving some stuff I’m doing to blade servers. Anyone have any experience with blade servers in general? And if so, now is the time to gimme some input!
This is what I’ve been looking at lately…
http://www1.us.dell.com/content/products/productdetails.aspx/pedge_1855
Then I could just pop in a new blade (computer) […]

A malformed UDP packet to your DNS server can cause it to respond to an IP address that never made the request (with the response being being more bytes than the request). So someone malicious could use one of your name servers to throw unwanted traffic at a 3rd IP address. Annoying… but […]

If you ever need to figure out what is eating bandwidth on a server, tcpdump comes in handy…
tcpdump -n -i any
That will spew out everything, so you might be able to find anything that looks suspicious in there. Say you find the IP address of 1.2.3.4 doing something suspicious, you can zero in […]

Okay… well *I’m* not big… but my forum is. In the first 21 months of it being around, its received over a half million posts. 500,000 posts puts you into the “big-boards.com” listing of big forums.
http://www.big-boards.com/board/1676/
Should probably break 1,000,000 posts by June.

Wow… there are half a million posts in my forum now. Craziness.

Okay, I *thought* it wasn’t a problem (at the time), but it turns out, it just made it “less” of a problem.
After watching the server closely for a month or two I think I finally figured out what the hell is going on. mysqld is not really efficient with it’s memory (I finally figured […]

If you need to debug memory usage and segments on a BSD based unix machine, vmmap is the ticket.
vmmap [process id]

Google Analytics lets you toggle between pie and bar chart, but it really seems logical to have a 3rd chart option that makes a line graph comparing the top 10 items from any report section. For example for web browsers you could see historically how the percentage of Firefox users goes up and Internet […]

It seems Google has broken up with me. Not only did they not do it in person, they didn’t call or even bother to email me to let me know they have moved on from our relationship. This blog (not all of digitalpoint.com, only this blog) has some sort of weird partial ban […]

I really wish Apple would update the Xserve to use dual core processors (for a total of 4 cores) like their PowerMac towers because I need another Xserve (or 2) and I’m not sure how much longer I can wait for it.

If you own a website or a blog, you need to be using this (really). It’s basically just Urchin 6 (Google bought the company earlier this year), but Google is offering it as a free service now (I think it was like $1,000/month or something before Google bought it).
http://www.google.com/analytics/

I know that Alexa numbers really are pointless, but it’s still fun to watch when you get below 10,000 or so.
The 3 month average dropped below 400 for the first time (389 specifically), which means only 388 domains are more trafficked than digitalpoint.com. According to Alexa’s top 500 list, we just passed MTV.com […]

For the last two months or so, I’ve been having a strange problem with my primary MySQL Server that required that the mysqld process (not the server itself) be restarted. The first image shows the CPU usage of the server, with the red arrows being the points I restarted the mysqld process (queries per […]

I’ve been doing a little tuning with one of my MySQL database servers today, and it’s pretty amazing how fast it is. Just ran a benchmarking thing on it before I applied some optimizations (so this is with no tuning), and this is what it looks like:
Uptime: 747 Threads: 1 Questions: 1409737 […]

The Digital Point forum is now live with vBulletin 3.5 beta 1. I spent some time today and last night recoding all the little hacks I made on vB 3.0.x, and turning them into plug-ins for 3.5 (yay for the new plug-in architecture).
Along the way, I even found some bugs (some of which I […]

I’ve spent a few weeks now with the new Xserves (I’m going to put them physically into the data center tomorrow BTW).
For the most part, Mac OS X Server 10.3 is a very nice server platform (especially when you are using Xserves). But maybe someone at Apple will read my little wish list, so […]

Okay, so I have two new loaded (each has dual 2.3Ghz G5s, 8GB RAM, hardware RAID-5 w/ 1.2TB drive space, dual gigabit ethernet, etc.) Xserves thanks to those that donated, and I’ve been mucking around with them for a few days now.
One is going to be primarily a web server, and the other will be […]

Apple put out a minor speed-bump on their G5 Xserve machines (they went from 2.0 Ghz to 2.3 Ghz and nothing else in the system or architecture changed).
The dual 2.0 Xserve could yield 9 gigaflops of raw processing power (which is really fast to begin with). The dual 2.3 Ghz Xserve yields 35 gigaflops […]