I migrated some databases to ndbcluster (some of the tables were fairly decent sized… 9GB for 1 table spanning 220M records), and was running into a problem where an ALTER TABLE to change the storage engine was spewing out some cryptic error message like so:

ERROR 1205 (HY000): Lock wait timeout exceeded; try restarting transaction

Cryptic mainly because it was coming from MyISAM, which doesn’t have record locking. Long story short is it’s not a setting in your my.cnf file for mysqld, rather a setting in your config.ini for ndbd. The TransactionDeadlockDetectionTimeout setting defaults to 1200 (1.2 seconds), I ended up raising it to 24 hours just for purposes of migrating existing tables to ndbcluster (the setting of 86400000 ms is 24 hours).

Being relatively new to MySQL Cluster, it was also a good opportunity to practice making a config change and doing a rolling restart of the ndbd nodes to have no downtime.

Anyway… long story short is apparently RRDtool pulls the list of fonts from the system, and since it was running as the _www user, it was unable to write a the font cache to disk to make future font access fast.

Logging in as root and running the “fc-list” command to get a list of installed fonts fixed the problem. The first time it ran it took 30 seconds (imagine that! heh), and subsequent times it was instant. And Cacti graphs are back to being instant. Hopefully this helps someone…

I’m assuming I’ll be able to get them relatively soon and have started to think about how I want to lay the servers out architecturally.

As far as hardware goes, gigabit ethernet just really isn’t that fast these days… Real world use has being able to pass about 80MB/sec (theoretical maximum is about 110MB/sec). When you have a cluster of servers passing massive amounts of data amongst themselves, 80MB/sec just doesn’t cut it. Which is why the servers have Infinband QDR (40Gbit interconnects), so they should be able to pass 3.5GB/sec (ish) between themselves.

For software/services, I’m thinking maybe 12 identical servers that are more or less service agnostic… with each server being a web server, MySQL Cluster data node and a MySQL Cluster SQL node. Then each server could probably be setup to handle 10,000 concurrent HTTP connections as well as 10,000 DB connections (1 per HTTP connection). With 12 servers, you could have the capacity of 120k concurrent web connections, 120k DB connections capable of doing millions of SQL queries/sec. If you need more capacity with anything, you could just bring online additional agnostic servers.

This of course is just in theory… who knows how it will test out in actual use, but it does seem like something worth testing at least.

Things that did not work…

• ZTerm let me get to the console just fine, but uploading the firmware with XModem failed for some reason (and you only found out it failed after a 3.5 hour upload time).
• Keyspan device on Windows via Parallels (from all the Googling I did, apparently it’s just an known issue with that piece of hardware and Parallels and the drivers will not recognize it).
• Virtual serial port in Windows/Parallels via a app called SerialClient (it was very difficult to find this app since the site that made it doesn’t exist anymore… people running Parallels said this worked for them, but it was many years ago… so maybe it was an older version of Mac OS X. Either way, I couldn’t get the app to “connect”.).

What DID work (thank God because I was running out of options)…

1. Download/compile/install lrzsz via ./configure;make;make install.
2. From Terminal: screen /dev/tty.Keyserial1 57600
3. Hit CONTROL+A
4. Type: :exec !! lsx -b -X ~/firmware_image.ros

It was infinitely faster than ZTerm (about 15 minutes vs. 3.5 hours), but more importantly it actually WORKED.

Sadly, 12GB in a server just isn’t what it used to be and the database servers are really starting to feel strained as they beg for more memory. Handling ~25GB of databases on servers with 12GB is just no bueno. I thought of upgrading the RAM in the blades, but the maximum RAM the BIOS supports is 16GB, and all the DIMM slots are in-use… so it more or less would mean tossing all the existing RAM and buying 160GB of new RAM. In the end spending that much money to gain 4GB per server isn’t really worth it (especially since 16GB wouldn’t really be enough either).

I started to do a little research on what would be some good options for upgrades… I think I want to stay away from blades simply because we did have the daughter card in the chassis fail once which took down all 10 blades (the daughter card controls the power buttons). Buying 2 complete sets of blades/chassis is overkill just so you have stuff still up if one complete chassis goes down. The “must haves” on my list are hot swappable drives with hardware RAID, hot swappable redundant power supply, some sort of 10Gbit/sec (or higher) connectivity for communicating between servers. On top of it all, the servers need to be generally “dense” (I don’t want to take an entire rack of space).

PowerEdge C6100

The Dell C6100 actually looked really nice at first glace…

Pros

1. Super dense (4 servers in a 2U package)
2. Redundant power supplies (hot swap)
3. 24 (!!) hot swap drives
4. Supports 10GbE or even QDR Infiniband (40Gbit/sec

Cons

1. Age – the server itself came out about 18 months ago without any refresh. That means you have hard drive options and CPU options that are a year and a half old
2. Price – OMG… a single unit loaded up with 4 nodes, drives, RAM, Infiniband, etc. works out to $54,709 (before tax).

The age factor really becomes an issue when it comes to the disk drives… you can get 15k rpm drives in a 2.5″ form factor, but Dell only offers 146GB models (there are 300GB models now). The CPU isn’t really too bad… Dell’s fastest offering is the Xeon X5670… 2.93Ghz, 6 core @95 watts (wattage is important because of so much stuff crammed in there). There is a slightly faster 95 watt, 6 core processor these days… the Xeon X5675… the same thing basically, just 3.06Ghz. 0.13Ghz speed difference isn’t a huge deal… but the hard drive difference is a big deal.

I started to think… well maybe I could just order it stripped down and then I could just replace the CPU/hard drives with better stuff. Doing that, you still end up spending about $60,000 because you end up with 8 Xeon processors (valued at about $1,500 each that you just are going to throw away).

Then I started to think harder… Wait a minute… Dell doesn’t even make their own motherboards (at least I don’t think so)… so maybe I could find the source of these super dense motherboards and build my own systems (or just find the source of something similar… which ended up being the case).

SuperServer 2026TT-H6IBQRF

What do we have here??? The Supermicro SuperServer 2026TT-H6IBQRF is more or less the same thing… 4 servers per 2U, hot swap hard drives, same BIOS/chip/controllers… supports the same Xeon processor family (up to 95 watts)… And as a bonus, Infiniband QDR is built in (it’s a $2,596 add-on for the Dell server) as well as an LSI MegaRAID hardware RAID card (a $2,156 add-on for the Dell server).

So let’s add up the cost to build one of these things with the CPU/hard drives I actually would WANT…

• Chassis (includes 4 motherboards, 2 power supplies, hard drive carriers, etc. – $4,630.98
• Xeon X5675 3.06Ghz CPU – $1,347.84 each, so $10,782.72 for 8
• 8GB ECC/Reg DIMM – $124.26 each, so $5,964.48 for 48
• 600GB Seagate Savvio 10K.5 – $391 each, so $9,384 for 24 (also 410% more capacity, 21% faster and 20% more reliable than the 146GB options from Dell)

Add It All Up…

$30,761.88 would be the total cost (a savings of $24,138.82) and in the end you get slightly faster CPUs and *way* better hard drives. So in a single 2U package, you end up with 48 Xeon cores at 3.06Ghz, 384GB of 1333Mhz memory, 14.4TB of drive space (9.6TB usable after it’s configured as double redundant parity striping with RAID-6… which should be able to do more than 750MB/sec read/write). 8 gigabit ethernet ports and 4 Infiniband QDR (40Gbit) I/O.

Get 2 or 3 of those rigs, and you have some nasty (in a good way) servers that would be gloriously fun to run MySQL Cluster, web servers and whatever else you want.

I have a couple older Xserve G5s in the facility, so I figured why not use them as network boot servers for the Linux machines? By default OS X Server has a service for NetBoot (which is not the same thing and can really only be used to boot other Mac machines). But Mac OS X Server also has all the underlying services already installed to make it able to be a server for PXE booting from normal Intel BIOS.

So what network services do we need exactly (at least for how I did it)? DHCP, NFS, TFTP and optionally Web if you do an auto-install like I did.

Preface

This was written up in about 5 minutes mostly so I wouldn’t forget what I did in case I needed to do it again. Some assumptions are made like you aren’t completely new to Linux/Mac OS X administration. You can also have pxelinux boot to a operating system selection menu and some other things, but for what *I* wanted to do, I didn’t care about being able to boot into multiple operating systems/modes.

Setting Up The Server

Mac OS X Server makes it really simple to get DHCP, NFS and Web servers online since the normal Server Admin has a GUI for each.

Mac OS X has a TFTP server installed by default, but there it’s not running by default and has no GUI. You can of course enable/configure it from the shell, but just to make things simple, there is a free app you can download that will make configuring and starting the TFTP service simple (the app is just a configuration utility, so it does not need to run once you are finished, so adds no overhead). You can grab the app over here.

Files Served By TFTP

I was installing openSUSE 11.4, so that is what my example revolves around… most Linux installations should be similar, if not identical. First, make sure you have syslinux, which you probably already have since most Linux distributions install it by default.

Copy /usr/share/syslinux/pxelinux.0 to your Mac OS X TFTP Server: /private/tftpboot/pxelinux.0

Create a directory on your Mac OS X Server: /private/tftpboot/pxelinux.cfg, and then create a file named default within that folder with the following:

Obviously change the IP address to your Mac OS X Server IP address. The autoyast part is optional and only needed if you have an auto-configuration file for YaST.

Now we want to grab two files from the installation DVD/.iso so we have the “real” kernel.

Copy /boot/x86_64/loader/linux from the installation DVD to your Mac OS X TFTP Server: /private/tftpboot/osuse11-4.krnl (notice the file rename)
Copy /boot/x86_64/loader/initrd from the installation DVD to your Mac OS X TFTP Server: /private/tftpboot/osuse11-4.ird (notice the file rename here too)

Special Options For DHCP Server

There is not a GUI for adding special options to the DHCP server, but it’s easy enough to add them manually. Just edit the /etc/bootpd.plist file, and add two keys/values to it (option 150 is the IP address of your TFTP server, option 67 is the kernel file name the TFTP server is serving):

NFS Sharing Of Installation DVD

Edit your /etc/exports file, and add the following line to it:

/images/opensuse10-4 -ro

Then just copy the contents of the entire installation DVD to /images/opensuse10-4. You can of course just do a symlink or something if you want.

Cross Your Fingers…

Hopefully if all goes well, you should see something along the lines of this when you choose to network boot a machine (for purposes of this video, I did it in a virtualized environment of Parallels so I didn’t need to boot any running servers):

• 3:26 am – Reading and posting seems to work… I suppose that’s good enough for now (still working on the other stuff, but don’t need to force everyone off to do it).
• 3:24 am – Skimming over various areas of the forum to see if things (mostly) work. I could let you guys in before the templates are fully updated possibly.
• 3:22 am – New moderator permissions set
• 3:17 am – Static CSS files located on single server in web cluster and spread around properly (I hate this about vBulletin BTW… gimme a hook location to do it please!)
• 3:09 am – Core upgrade done.
• 3:07 am – Recoded some of the upgrade scripts so they aren’t making that change to the reputation table. Will deal with issue this later instead.
• 3:00 am – Stupid reputation table was altered in 3.8.0 to not allow negative userids so it can accommodate 4 billion users instead of “only” 2 billion. Stupid. I used negative userids internally for some stuff. /thinking what to do about this…
• 2:55 am – First problem… reputation table alterations not going well. Going digging in raw database…
• 2:50 am – Running “ALTER TABLE pmtext” on DB servers. That’s a big one… going to get another beer.
• 2:49 am – Up to version 3.8.0 alpha 1
• 2:43 am – It’s almost 3am. If you are a cute girl reading this, please post your picture in the comments, k thanks.
• 2:41 am – I forgot we have to go through all versions to get to the newest… lol… going through 3.7.0 beta 4 at the moment.
• 2:40 am – Userlist rebuilding
• 2:36 am – Watching DB servers alter thread table for tagging support… (this is really boring)
• 2:34 am – Watching DB servers alter thread table for prefix support… /bored
• 2:30 am – New PHP files in place and synced across web server cluster
• 2:29 am – DB backup done
• 2:27 am – Thinking I might not have enough beer for this…
• 2:25 am – DB backup still running (it’s huge… many, many, many gigs)
• 2:18 am – Got beer and craisens
• 2:17 am – Backing up DB
• 2:13 am – Making this post

On a better note, I have 2 years of life to write about now… stay tuned…

I decided to try and do something about it so I don’t have to weed through them manually (the time consumption on this task is one of the biggest reasons I don’t post as often as I used to).

So let’s see how it works… if you see me posting more often, then you know it worked.

Monitor everything… Put as much info at your finger tips as easily as possible. Put that info in a place where you will always be looking at it for some reason. For example, I made a vBulletin plug-in that monitors 4 memcached servers (including a latency test it runs) as well as 10 blade servers. This shows every time I’m in the admin of the forum (which is a lot), so I can’t help but to not see it.

I wrote a little daemon that runs on my servers that can quickly report back whatever info I want (time, disk RAID status, server load, MySQL replication status, etc.)

The more info you have in one place (especially when you run a bunch of servers), the easier it is to see anything wrong. For example, I had an issue with a web server serving requests slow one day… it ultimately ended up not being a problem with the web server, but the memcached server it was using. The latency test was showing ~2,000 ms latency (2 seconds) vs. the normal 0.5 ms (1/2000 of a second).

And be proactive about monitoring stuff… don’t wait until something bad happens to start doing it! Then it’s too late.

Can I go to bed now?

Sure enough!

Use memcached… no really, use it.

memcached is a distributed memory caching system that allows multiple servers to access the same shared memory (you can use it just for single local server too of course). You can cache pretty much anything… SQL query results (especially ones that can’t hit indexes), dynamically generated web pages, dynamic RSS feeds, etc.

If you compile the memcache() functions/class directly into PHP, you have a an easy (yet powerful) way to incorporate it.

For example, let’s say I have some existing code that makes a call to some sort of API (I use memcached for the keyword suggestion tool in this way)… you can add a couple lines of code to cache the results.

[code=php]$data = @file_get_contents ("http://some_api_call_url?keywords=$keywords");[/code]

could become this:

[code=php]$memcache = new Memcache;
$memcache->addServer('192.168.1.1');
$memcache->addServer('192.168.1.2');
if (!$data = $memcache->get($keywords)) {
$data = @file_get_contents ("http://some_api_call_url?keywords=$keywords");
$memcache->set($keywords, $data, false, 86400); // 24 hour cache
}
$memcache->close();[/code]

Now we are only hitting the API for the same keywords once every 24 hours… that setup utilizes 2 memcached servers working as a single entity for pooling and failover.

I use memcache for all sorts of things (this blog’s pages are cached for 60 seconds as another example because WordPress sucks, and protects against a massive influx of traffic [front-page digg for example]).

Here’s some things I personally use it for…

• Caching this blog’s pages (for 60 seconds)
• Caching Keyword Suggestion Tool queries (for 7 days)
• Caching AdSense Sandbox queries (for 24 hours)
• Temporary counters in Keyword Tracker (using decrement() for displaying remaining keywords on a “Check All”)
• Recent forum topics that show on all the webmaster tools
• vBulletin forum datastore

Any high-traffic site that is dynamically generated (especially if it includes SQL queries) could benefit from it… you could manage which memcache items need to be updated when you insert/update to the database, or an easy way to do it is just check the cache for the pages that potentially have high traffic… if they aren’t in the cache, put them in the cache with a certain expiration date. Then the page is dynamically generated no more often than the cache expiration time (in the case of this blog, the pages are generated no more than once every 60 seconds).

P.S. - Make sure you block outside access to your memcached port since there’s no authentication used!

Anyway, the short version of this entry is that I found out that you can flash your Linksys with an entirely new firmware. Ultimately, I just wanted to fix the bug so I could do what the Linksys is supposed to do to begin with. But after installing it, it looks like you can do all sorts of other interesting things… like run crontab processes (since it’s Linux).

Anyway, if you have a Linksys WRT54G (they are pretty popular), and you are a dork, you may want to give it a go… but don’t blame me if you screw yours up (mine was fine).

http://www.dd-wrt.com/dd-wrtv2/ddwrt.php

So the IP address changed over the weekend, and the only way to get the new IP is to drive out there and look to see what it is. On the way out there, I had an idea… why not buy a Linksys WRT54G broadband router and just route all in-bound traffic to it. Then you can use it’s firewall and DDNS functions. So that’s what I did… and it works *so* much better… remote admin access and everything else. I even tried to get tricky and route traffic to the Linksys and then right back to the 2Wire to see if I could get to it’s web admin remotely, but no such luck… It ends up redirecting it to a different URL (I’m assuming based on the client not being on the local network)… whatever that part isn’t that important anyway now.