Anyway… long story short is apparently RRDtool pulls the list of fonts from the system, and since it was running as the _www user, it was unable to write a the font cache to disk to make future font access fast.

Logging in as root and running the “fc-list” command to get a list of installed fonts fixed the problem. The first time it ran it took 30 seconds (imagine that! heh), and subsequent times it was instant. And Cacti graphs are back to being instant. Hopefully this helps someone…

I’m assuming I’ll be able to get them relatively soon and have started to think about how I want to lay the servers out architecturally.

As far as hardware goes, gigabit ethernet just really isn’t that fast these days… Real world use has being able to pass about 80MB/sec (theoretical maximum is about 110MB/sec). When you have a cluster of servers passing massive amounts of data amongst themselves, 80MB/sec just doesn’t cut it. Which is why the servers have Infinband QDR (40Gbit interconnects), so they should be able to pass 3.5GB/sec (ish) between themselves.

For software/services, I’m thinking maybe 12 identical servers that are more or less service agnostic… with each server being a web server, MySQL Cluster data node and a MySQL Cluster SQL node. Then each server could probably be setup to handle 10,000 concurrent HTTP connections as well as 10,000 DB connections (1 per HTTP connection). With 12 servers, you could have the capacity of 120k concurrent web connections, 120k DB connections capable of doing millions of SQL queries/sec. If you need more capacity with anything, you could just bring online additional agnostic servers.

This of course is just in theory… who knows how it will test out in actual use, but it does seem like something worth testing at least.

And to be totally honest, I would rather have a burrito from Rubio’s even if an aircraft carrier was the same price.

I’m a saving machine… I just saved $4,500,000,000 and I didn’t even have to switch to GEICO!

Watching the whole deficit stuff unravel is like watching kindergardeners argue about something they don’t understand.

So let’s forget for a second that the US has around $115 TRILLION in unfunded liabilities, and just focus on a much “smaller” amount… the US debt is currently around $14.5 TRILLION.

Now the latest plan for debt reduction goes something like this… “We will only borrow and extra $800,000,000,000 per year instead of $1,200,000,000,000.” And it’s being billed as “saving $3 trillion over the next 10 years”. Last time I saved money, it was money I DIDN’T spend at all.

Let’s say I have $10,000 to my name, and I wanted to buy a new phone for $176,400 (the price of a Princess Plus iPhone)…

It’s $166,400 more than I can afford, so I decide to take out a loan for it. I decide that I’m going to instead get the less expensive version for $66,150 (less quality diamonds), so I only need a loan for $56,150.

I don’t know much about math, but I DO know that I’m amazing… I only had $10,000 to my name, yet I was able to save $110,250! If I do this every year for 10 years, I can save $1,102,500.

I’m a FUCKING GENIUS… I just solved my lifelong debt problem! The amount of money I can save is limitless.

Things that did not work…

• ZTerm let me get to the console just fine, but uploading the firmware with XModem failed for some reason (and you only found out it failed after a 3.5 hour upload time).
• Keyspan device on Windows via Parallels (from all the Googling I did, apparently it’s just an known issue with that piece of hardware and Parallels and the drivers will not recognize it).
• Virtual serial port in Windows/Parallels via a app called SerialClient (it was very difficult to find this app since the site that made it doesn’t exist anymore… people running Parallels said this worked for them, but it was many years ago… so maybe it was an older version of Mac OS X. Either way, I couldn’t get the app to “connect”.).

What DID work (thank God because I was running out of options)…

1. Download/compile/install lrzsz via ./configure;make;make install.
2. From Terminal: screen /dev/tty.Keyserial1 57600
3. Hit CONTROL+A
4. Type: :exec !! lsx -b -X ~/firmware_image.ros

It was infinitely faster than ZTerm (about 15 minutes vs. 3.5 hours), but more importantly it actually WORKED.

Sadly, 12GB in a server just isn’t what it used to be and the database servers are really starting to feel strained as they beg for more memory. Handling ~25GB of databases on servers with 12GB is just no bueno. I thought of upgrading the RAM in the blades, but the maximum RAM the BIOS supports is 16GB, and all the DIMM slots are in-use… so it more or less would mean tossing all the existing RAM and buying 160GB of new RAM. In the end spending that much money to gain 4GB per server isn’t really worth it (especially since 16GB wouldn’t really be enough either).

I started to do a little research on what would be some good options for upgrades… I think I want to stay away from blades simply because we did have the daughter card in the chassis fail once which took down all 10 blades (the daughter card controls the power buttons). Buying 2 complete sets of blades/chassis is overkill just so you have stuff still up if one complete chassis goes down. The “must haves” on my list are hot swappable drives with hardware RAID, hot swappable redundant power supply, some sort of 10Gbit/sec (or higher) connectivity for communicating between servers. On top of it all, the servers need to be generally “dense” (I don’t want to take an entire rack of space).

PowerEdge C6100

The Dell C6100 actually looked really nice at first glace…

Pros

1. Super dense (4 servers in a 2U package)
2. Redundant power supplies (hot swap)
3. 24 (!!) hot swap drives
4. Supports 10GbE or even QDR Infiniband (40Gbit/sec

Cons

1. Age – the server itself came out about 18 months ago without any refresh. That means you have hard drive options and CPU options that are a year and a half old
2. Price – OMG… a single unit loaded up with 4 nodes, drives, RAM, Infiniband, etc. works out to $54,709 (before tax).

The age factor really becomes an issue when it comes to the disk drives… you can get 15k rpm drives in a 2.5″ form factor, but Dell only offers 146GB models (there are 300GB models now). The CPU isn’t really too bad… Dell’s fastest offering is the Xeon X5670… 2.93Ghz, 6 core @95 watts (wattage is important because of so much stuff crammed in there). There is a slightly faster 95 watt, 6 core processor these days… the Xeon X5675… the same thing basically, just 3.06Ghz. 0.13Ghz speed difference isn’t a huge deal… but the hard drive difference is a big deal.

I started to think… well maybe I could just order it stripped down and then I could just replace the CPU/hard drives with better stuff. Doing that, you still end up spending about $60,000 because you end up with 8 Xeon processors (valued at about $1,500 each that you just are going to throw away).

Then I started to think harder… Wait a minute… Dell doesn’t even make their own motherboards (at least I don’t think so)… so maybe I could find the source of these super dense motherboards and build my own systems (or just find the source of something similar… which ended up being the case).

SuperServer 2026TT-H6IBQRF

What do we have here??? The Supermicro SuperServer 2026TT-H6IBQRF is more or less the same thing… 4 servers per 2U, hot swap hard drives, same BIOS/chip/controllers… supports the same Xeon processor family (up to 95 watts)… And as a bonus, Infiniband QDR is built in (it’s a $2,596 add-on for the Dell server) as well as an LSI MegaRAID hardware RAID card (a $2,156 add-on for the Dell server).

So let’s add up the cost to build one of these things with the CPU/hard drives I actually would WANT…

• Chassis (includes 4 motherboards, 2 power supplies, hard drive carriers, etc. – $4,630.98
• Xeon X5675 3.06Ghz CPU – $1,347.84 each, so $10,782.72 for 8
• 8GB ECC/Reg DIMM – $124.26 each, so $5,964.48 for 48
• 600GB Seagate Savvio 10K.5 – $391 each, so $9,384 for 24 (also 410% more capacity, 21% faster and 20% more reliable than the 146GB options from Dell)

Add It All Up…

$30,761.88 would be the total cost (a savings of $24,138.82) and in the end you get slightly faster CPUs and *way* better hard drives. So in a single 2U package, you end up with 48 Xeon cores at 3.06Ghz, 384GB of 1333Mhz memory, 14.4TB of drive space (9.6TB usable after it’s configured as double redundant parity striping with RAID-6… which should be able to do more than 750MB/sec read/write). 8 gigabit ethernet ports and 4 Infiniband QDR (40Gbit) I/O.

Get 2 or 3 of those rigs, and you have some nasty (in a good way) servers that would be gloriously fun to run MySQL Cluster, web servers and whatever else you want.

I have a couple older Xserve G5s in the facility, so I figured why not use them as network boot servers for the Linux machines? By default OS X Server has a service for NetBoot (which is not the same thing and can really only be used to boot other Mac machines). But Mac OS X Server also has all the underlying services already installed to make it able to be a server for PXE booting from normal Intel BIOS.

So what network services do we need exactly (at least for how I did it)? DHCP, NFS, TFTP and optionally Web if you do an auto-install like I did.

Preface

This was written up in about 5 minutes mostly so I wouldn’t forget what I did in case I needed to do it again. Some assumptions are made like you aren’t completely new to Linux/Mac OS X administration. You can also have pxelinux boot to a operating system selection menu and some other things, but for what *I* wanted to do, I didn’t care about being able to boot into multiple operating systems/modes.

Setting Up The Server

Mac OS X Server makes it really simple to get DHCP, NFS and Web servers online since the normal Server Admin has a GUI for each.

Mac OS X has a TFTP server installed by default, but there it’s not running by default and has no GUI. You can of course enable/configure it from the shell, but just to make things simple, there is a free app you can download that will make configuring and starting the TFTP service simple (the app is just a configuration utility, so it does not need to run once you are finished, so adds no overhead). You can grab the app over here.

Files Served By TFTP

I was installing openSUSE 11.4, so that is what my example revolves around… most Linux installations should be similar, if not identical. First, make sure you have syslinux, which you probably already have since most Linux distributions install it by default.

Copy /usr/share/syslinux/pxelinux.0 to your Mac OS X TFTP Server: /private/tftpboot/pxelinux.0

Create a directory on your Mac OS X Server: /private/tftpboot/pxelinux.cfg, and then create a file named default within that folder with the following:

Obviously change the IP address to your Mac OS X Server IP address. The autoyast part is optional and only needed if you have an auto-configuration file for YaST.

Now we want to grab two files from the installation DVD/.iso so we have the “real” kernel.

Copy /boot/x86_64/loader/linux from the installation DVD to your Mac OS X TFTP Server: /private/tftpboot/osuse11-4.krnl (notice the file rename)
Copy /boot/x86_64/loader/initrd from the installation DVD to your Mac OS X TFTP Server: /private/tftpboot/osuse11-4.ird (notice the file rename here too)

Special Options For DHCP Server

There is not a GUI for adding special options to the DHCP server, but it’s easy enough to add them manually. Just edit the /etc/bootpd.plist file, and add two keys/values to it (option 150 is the IP address of your TFTP server, option 67 is the kernel file name the TFTP server is serving):

NFS Sharing Of Installation DVD

Edit your /etc/exports file, and add the following line to it:

/images/opensuse10-4 -ro

Then just copy the contents of the entire installation DVD to /images/opensuse10-4. You can of course just do a symlink or something if you want.

Cross Your Fingers…

Hopefully if all goes well, you should see something along the lines of this when you choose to network boot a machine (for purposes of this video, I did it in a virtualized environment of Parallels so I didn’t need to boot any running servers):

I have AT&T fiber to my house (actual fiber to the premise), so I use U-verse for TV and Internet in my house, so it’s break this down (I’m not including any on-time costs or fees/taxes)…

Television
I pay $138/month for TV (includes set top box rentals). I only watch MAYBE 8 hours of TV per month, but it’s nice to have it when you actually want to watch it. So I pay AT&T $17.25/hour to watch TV.

Cell Phone
I pay $159.99/month for 2 cell phones. Last month I used 17 minutes (which is actually more than normal). So I pay AT&T $9.41/minute for cell phone service. Oh yeah, I also paid them $125 for a MicroCell because their service is so crappy.

Internet
I pay $55/month for Internet because I can only get 18Mbit to my house (remember… I have AT&T fiber straight into my house). I’d like to pay for the $65/month plan to give me 24Mbit, but you know… it’s hard to get more bandwidth out of fiber I guess. I really wish Verizon would buy AT&T’s fiber to my house so I could get FiOS (they offer 150Mbit down/35Mbit up already and I heard they are testing 1Gbit up/down). Meanwhile I’ll be stuck at DSL speeds with my fiber. Sweet.

Mobile Internet
Now if I want a 3G connected iPad, that would be another $25/month (and I would rarely use it… so probably would be about $25/hour).

If I want tethering enabled on one of my phones, that’s another $20… even though I already pay for an unlimited data plan on that phone. lol

It would be nice if I could just pay $10/hour to use whatever I want… TV, cell phone, cell phone tethering, iPad connectivity, etc.

Now let’s swap a couple words here…

The problem is that I’m an anal perfectionist and the Flash object was eating 80-90% of a computer’s CPU when being viewed… and me being me, I couldn’t just be okay with that.

…so I rebuilt it with CSS3/HTML5/jQuery. It turned out much nicer in my opinion, only uses about 1% of the resources that Flash needed and as an unintentional bonus, it works on things like iPad and phones/computers without Flash support.

I’m not anti-Flash (as I said, I *wanted* to do the site with Flash), but I AM anti-inefficient. There are still some things you can’t do with HTML5 that you can do with Flash, but those things are becoming fewer and fewer it seems.

It’s basically a boat that you can hit a button, and it will go underwater (up to 100 feet). It would be pretty awesome to go scuba diving and instead of leaving your boat at the surface, just take it under water with you when you get to the dive site. It’s not pressurized, so it’s not a submarine (you would still need scuba gear of course).

It also has an onboard computer that controls your ascent and descent so you don’t go to quickly and give yourself the bends.

I love you, Santa!

All of a sudden it was changes to 8 pages (still 8 results per page), so now you can look at the first 64 results. This is quite handy.

Thanks Google, you’re swell!

I Googled the issue, and came up with hundreds of people having the same problem, but no one seemed to figure out a solution.

After a lot of digging and poking around, I think I found the solution. She imported large folders into iPhoto from her old photo management application, and she had some movies in her photo folders. So this is how I fixed it…

• Quit iPhoto first.

• Then get into the guts of your iPhoto Library (right-click the iPhoto Library file in your Pictures folder and do “Show Package Contents”).
• Go to the Originals folder in there and get your movies out of there.
• Delete the face.db and face_blob.db files.
• Load iPhoto up and it will start scanning all your faces and with any luck actually get through them all and be kosher going forward.

Warning… this will delete all your face tags you had previously (so don’t do it if you tagged a bunch of people manually and don’t want to lose that).

]]> http://www.shawnhogan.com/2009/10/iphoto-09-faces-spinning-forever.html/feed 6