I migrated some databases to ndbcluster (some of the tables were fairly decent sized… 9GB for 1 table spanning 220M records), and was running into a problem where an ALTER TABLE to change the storage engine was spewing out some cryptic error message like so:

ERROR 1205 (HY000): Lock wait timeout exceeded; try restarting transaction

Cryptic mainly because it was coming from MyISAM, which doesn’t have record locking. Long story short is it’s not a setting in your my.cnf file for mysqld, rather a setting in your config.ini for ndbd. The TransactionDeadlockDetectionTimeout setting defaults to 1200 (1.2 seconds), I ended up raising it to 24 hours just for purposes of migrating existing tables to ndbcluster (the setting of 86400000 ms is 24 hours).

Being relatively new to MySQL Cluster, it was also a good opportunity to practice making a config change and doing a rolling restart of the ndbd nodes to have no downtime.

Anyway… long story short is apparently RRDtool pulls the list of fonts from the system, and since it was running as the _www user, it was unable to write a the font cache to disk to make future font access fast.

Logging in as root and running the “fc-list” command to get a list of installed fonts fixed the problem. The first time it ran it took 30 seconds (imagine that! heh), and subsequent times it was instant. And Cacti graphs are back to being instant. Hopefully this helps someone…

I’m assuming I’ll be able to get them relatively soon and have started to think about how I want to lay the servers out architecturally.

As far as hardware goes, gigabit ethernet just really isn’t that fast these days… Real world use has being able to pass about 80MB/sec (theoretical maximum is about 110MB/sec). When you have a cluster of servers passing massive amounts of data amongst themselves, 80MB/sec just doesn’t cut it. Which is why the servers have Infinband QDR (40Gbit interconnects), so they should be able to pass 3.5GB/sec (ish) between themselves.

For software/services, I’m thinking maybe 12 identical servers that are more or less service agnostic… with each server being a web server, MySQL Cluster data node and a MySQL Cluster SQL node. Then each server could probably be setup to handle 10,000 concurrent HTTP connections as well as 10,000 DB connections (1 per HTTP connection). With 12 servers, you could have the capacity of 120k concurrent web connections, 120k DB connections capable of doing millions of SQL queries/sec. If you need more capacity with anything, you could just bring online additional agnostic servers.

This of course is just in theory… who knows how it will test out in actual use, but it does seem like something worth testing at least.

Things that did not work…

• ZTerm let me get to the console just fine, but uploading the firmware with XModem failed for some reason (and you only found out it failed after a 3.5 hour upload time).
• Keyspan device on Windows via Parallels (from all the Googling I did, apparently it’s just an known issue with that piece of hardware and Parallels and the drivers will not recognize it).
• Virtual serial port in Windows/Parallels via a app called SerialClient (it was very difficult to find this app since the site that made it doesn’t exist anymore… people running Parallels said this worked for them, but it was many years ago… so maybe it was an older version of Mac OS X. Either way, I couldn’t get the app to “connect”.).

What DID work (thank God because I was running out of options)…

1. Download/compile/install lrzsz via ./configure;make;make install.
2. From Terminal: screen /dev/tty.Keyserial1 57600
3. Hit CONTROL+A
4. Type: :exec !! lsx -b -X ~/firmware_image.ros

It was infinitely faster than ZTerm (about 15 minutes vs. 3.5 hours), but more importantly it actually WORKED.

Sadly, 12GB in a server just isn’t what it used to be and the database servers are really starting to feel strained as they beg for more memory. Handling ~25GB of databases on servers with 12GB is just no bueno. I thought of upgrading the RAM in the blades, but the maximum RAM the BIOS supports is 16GB, and all the DIMM slots are in-use… so it more or less would mean tossing all the existing RAM and buying 160GB of new RAM. In the end spending that much money to gain 4GB per server isn’t really worth it (especially since 16GB wouldn’t really be enough either).

I started to do a little research on what would be some good options for upgrades… I think I want to stay away from blades simply because we did have the daughter card in the chassis fail once which took down all 10 blades (the daughter card controls the power buttons). Buying 2 complete sets of blades/chassis is overkill just so you have stuff still up if one complete chassis goes down. The “must haves” on my list are hot swappable drives with hardware RAID, hot swappable redundant power supply, some sort of 10Gbit/sec (or higher) connectivity for communicating between servers. On top of it all, the servers need to be generally “dense” (I don’t want to take an entire rack of space).

PowerEdge C6100

The Dell C6100 actually looked really nice at first glace…

Pros

1. Super dense (4 servers in a 2U package)
2. Redundant power supplies (hot swap)
3. 24 (!!) hot swap drives
4. Supports 10GbE or even QDR Infiniband (40Gbit/sec

Cons

1. Age – the server itself came out about 18 months ago without any refresh. That means you have hard drive options and CPU options that are a year and a half old
2. Price – OMG… a single unit loaded up with 4 nodes, drives, RAM, Infiniband, etc. works out to $54,709 (before tax).

The age factor really becomes an issue when it comes to the disk drives… you can get 15k rpm drives in a 2.5″ form factor, but Dell only offers 146GB models (there are 300GB models now). The CPU isn’t really too bad… Dell’s fastest offering is the Xeon X5670… 2.93Ghz, 6 core @95 watts (wattage is important because of so much stuff crammed in there). There is a slightly faster 95 watt, 6 core processor these days… the Xeon X5675… the same thing basically, just 3.06Ghz. 0.13Ghz speed difference isn’t a huge deal… but the hard drive difference is a big deal.

I started to think… well maybe I could just order it stripped down and then I could just replace the CPU/hard drives with better stuff. Doing that, you still end up spending about $60,000 because you end up with 8 Xeon processors (valued at about $1,500 each that you just are going to throw away).

Then I started to think harder… Wait a minute… Dell doesn’t even make their own motherboards (at least I don’t think so)… so maybe I could find the source of these super dense motherboards and build my own systems (or just find the source of something similar… which ended up being the case).

SuperServer 2026TT-H6IBQRF

What do we have here??? The Supermicro SuperServer 2026TT-H6IBQRF is more or less the same thing… 4 servers per 2U, hot swap hard drives, same BIOS/chip/controllers… supports the same Xeon processor family (up to 95 watts)… And as a bonus, Infiniband QDR is built in (it’s a $2,596 add-on for the Dell server) as well as an LSI MegaRAID hardware RAID card (a $2,156 add-on for the Dell server).

So let’s add up the cost to build one of these things with the CPU/hard drives I actually would WANT…

• Chassis (includes 4 motherboards, 2 power supplies, hard drive carriers, etc. – $4,630.98
• Xeon X5675 3.06Ghz CPU – $1,347.84 each, so $10,782.72 for 8
• 8GB ECC/Reg DIMM – $124.26 each, so $5,964.48 for 48
• 600GB Seagate Savvio 10K.5 – $391 each, so $9,384 for 24 (also 410% more capacity, 21% faster and 20% more reliable than the 146GB options from Dell)

Add It All Up…

$30,761.88 would be the total cost (a savings of $24,138.82) and in the end you get slightly faster CPUs and *way* better hard drives. So in a single 2U package, you end up with 48 Xeon cores at 3.06Ghz, 384GB of 1333Mhz memory, 14.4TB of drive space (9.6TB usable after it’s configured as double redundant parity striping with RAID-6… which should be able to do more than 750MB/sec read/write). 8 gigabit ethernet ports and 4 Infiniband QDR (40Gbit) I/O.

Get 2 or 3 of those rigs, and you have some nasty (in a good way) servers that would be gloriously fun to run MySQL Cluster, web servers and whatever else you want.

I have a couple older Xserve G5s in the facility, so I figured why not use them as network boot servers for the Linux machines? By default OS X Server has a service for NetBoot (which is not the same thing and can really only be used to boot other Mac machines). But Mac OS X Server also has all the underlying services already installed to make it able to be a server for PXE booting from normal Intel BIOS.

So what network services do we need exactly (at least for how I did it)? DHCP, NFS, TFTP and optionally Web if you do an auto-install like I did.

Preface

This was written up in about 5 minutes mostly so I wouldn’t forget what I did in case I needed to do it again. Some assumptions are made like you aren’t completely new to Linux/Mac OS X administration. You can also have pxelinux boot to a operating system selection menu and some other things, but for what *I* wanted to do, I didn’t care about being able to boot into multiple operating systems/modes.

Setting Up The Server

Mac OS X Server makes it really simple to get DHCP, NFS and Web servers online since the normal Server Admin has a GUI for each.

Mac OS X has a TFTP server installed by default, but there it’s not running by default and has no GUI. You can of course enable/configure it from the shell, but just to make things simple, there is a free app you can download that will make configuring and starting the TFTP service simple (the app is just a configuration utility, so it does not need to run once you are finished, so adds no overhead). You can grab the app over here.

Files Served By TFTP

I was installing openSUSE 11.4, so that is what my example revolves around… most Linux installations should be similar, if not identical. First, make sure you have syslinux, which you probably already have since most Linux distributions install it by default.

Copy /usr/share/syslinux/pxelinux.0 to your Mac OS X TFTP Server: /private/tftpboot/pxelinux.0

Create a directory on your Mac OS X Server: /private/tftpboot/pxelinux.cfg, and then create a file named default within that folder with the following:

Obviously change the IP address to your Mac OS X Server IP address. The autoyast part is optional and only needed if you have an auto-configuration file for YaST.

Now we want to grab two files from the installation DVD/.iso so we have the “real” kernel.

Copy /boot/x86_64/loader/linux from the installation DVD to your Mac OS X TFTP Server: /private/tftpboot/osuse11-4.krnl (notice the file rename)
Copy /boot/x86_64/loader/initrd from the installation DVD to your Mac OS X TFTP Server: /private/tftpboot/osuse11-4.ird (notice the file rename here too)

Special Options For DHCP Server

There is not a GUI for adding special options to the DHCP server, but it’s easy enough to add them manually. Just edit the /etc/bootpd.plist file, and add two keys/values to it (option 150 is the IP address of your TFTP server, option 67 is the kernel file name the TFTP server is serving):

NFS Sharing Of Installation DVD

Edit your /etc/exports file, and add the following line to it:

/images/opensuse10-4 -ro

Then just copy the contents of the entire installation DVD to /images/opensuse10-4. You can of course just do a symlink or something if you want.

Cross Your Fingers…

Hopefully if all goes well, you should see something along the lines of this when you choose to network boot a machine (for purposes of this video, I did it in a virtualized environment of Parallels so I didn’t need to boot any running servers):

I have AT&T fiber to my house (actual fiber to the premise), so I use U-verse for TV and Internet in my house, so it’s break this down (I’m not including any on-time costs or fees/taxes)…

Television
I pay $138/month for TV (includes set top box rentals). I only watch MAYBE 8 hours of TV per month, but it’s nice to have it when you actually want to watch it. So I pay AT&T $17.25/hour to watch TV.

Cell Phone
I pay $159.99/month for 2 cell phones. Last month I used 17 minutes (which is actually more than normal). So I pay AT&T $9.41/minute for cell phone service. Oh yeah, I also paid them $125 for a MicroCell because their service is so crappy.

Internet
I pay $55/month for Internet because I can only get 18Mbit to my house (remember… I have AT&T fiber straight into my house). I’d like to pay for the $65/month plan to give me 24Mbit, but you know… it’s hard to get more bandwidth out of fiber I guess. I really wish Verizon would buy AT&T’s fiber to my house so I could get FiOS (they offer 150Mbit down/35Mbit up already and I heard they are testing 1Gbit up/down). Meanwhile I’ll be stuck at DSL speeds with my fiber. Sweet.

Mobile Internet
Now if I want a 3G connected iPad, that would be another $25/month (and I would rarely use it… so probably would be about $25/hour).

If I want tethering enabled on one of my phones, that’s another $20… even though I already pay for an unlimited data plan on that phone. lol

It would be nice if I could just pay $10/hour to use whatever I want… TV, cell phone, cell phone tethering, iPad connectivity, etc.

The problem is that I’m an anal perfectionist and the Flash object was eating 80-90% of a computer’s CPU when being viewed… and me being me, I couldn’t just be okay with that.

…so I rebuilt it with CSS3/HTML5/jQuery. It turned out much nicer in my opinion, only uses about 1% of the resources that Flash needed and as an unintentional bonus, it works on things like iPad and phones/computers without Flash support.

I’m not anti-Flash (as I said, I *wanted* to do the site with Flash), but I AM anti-inefficient. There are still some things you can’t do with HTML5 that you can do with Flash, but those things are becoming fewer and fewer it seems.

All of a sudden it was changes to 8 pages (still 8 results per page), so now you can look at the first 64 results. This is quite handy.

Thanks Google, you’re swell!

I Googled the issue, and came up with hundreds of people having the same problem, but no one seemed to figure out a solution.

After a lot of digging and poking around, I think I found the solution. She imported large folders into iPhoto from her old photo management application, and she had some movies in her photo folders. So this is how I fixed it…

• Quit iPhoto first.

• Then get into the guts of your iPhoto Library (right-click the iPhoto Library file in your Pictures folder and do “Show Package Contents”).
• Go to the Originals folder in there and get your movies out of there.
• Delete the face.db and face_blob.db files.
• Load iPhoto up and it will start scanning all your faces and with any luck actually get through them all and be kosher going forward.

Warning… this will delete all your face tags you had previously (so don’t do it if you tagged a bunch of people manually and don’t want to lose that).

]]> http://www.shawnhogan.com/2009/10/iphoto-09-faces-spinning-forever.html/feed 6 http://www.shawnhogan.com/2009/07/how-to-get-twitter-followers.html http://www.shawnhogan.com/2009/07/how-to-get-twitter-followers.html#comments Wed, 29 Jul 2009 09:18:26 +0000 Shawn http://www.shawnhogan.com/2009/07/how-to-get-twitter-followers.html Well actually, I’m not exactly sure how because I have Twitter followers since I don’t have a Twitter account, but apparently I have have a couple Twitter accounts…

http://twitter.com/digitalpoint/ – 1,127 followers
http://twitter.com/shawnhogan/ – 63 followers

Stupid, but funny. lol

]]> http://www.shawnhogan.com/2009/07/how-to-get-twitter-followers.html/feed 10 http://www.shawnhogan.com/2009/07/vbulletin-383-upgrade.html http://www.shawnhogan.com/2009/07/vbulletin-383-upgrade.html#comments Fri, 24 Jul 2009 09:13:01 +0000 Shawn http://www.shawnhogan.com/2009/07/vbulletin-383-upgrade.html Okay, gonna go ahead and upgrade the Digital Point Forums to vBulletin 3.8.3… Since so many of you are freaks and are on it 24/7, I made this post to keep you up to date on what’s going on (I’ll update it periodically throughout the process). You can also use this blog post as the new temporary forum for all your discussion needs while I do the upgrade. hah

• 3:26 am – Reading and posting seems to work… I suppose that’s good enough for now (still working on the other stuff, but don’t need to force everyone off to do it).
• 3:24 am – Skimming over various areas of the forum to see if things (mostly) work. I could let you guys in before the templates are fully updated possibly.
• 3:22 am – New moderator permissions set
• 3:17 am – Static CSS files located on single server in web cluster and spread around properly (I hate this about vBulletin BTW… gimme a hook location to do it please!)
• 3:09 am – Core upgrade done.
• 3:07 am – Recoded some of the upgrade scripts so they aren’t making that change to the reputation table. Will deal with issue this later instead.
• 3:00 am – Stupid reputation table was altered in 3.8.0 to not allow negative userids so it can accommodate 4 billion users instead of “only” 2 billion. Stupid. I used negative userids internally for some stuff. /thinking what to do about this…
• 2:55 am – First problem… reputation table alterations not going well. Going digging in raw database…
• 2:50 am – Running “ALTER TABLE pmtext” on DB servers. That’s a big one… going to get another beer.
• 2:49 am – Up to version 3.8.0 alpha 1
• 2:43 am – It’s almost 3am. If you are a cute girl reading this, please post your picture in the comments, k thanks.
• 2:41 am – I forgot we have to go through all versions to get to the newest… lol… going through 3.7.0 beta 4 at the moment.
• 2:40 am – Userlist rebuilding
• 2:36 am – Watching DB servers alter thread table for tagging support… (this is really boring)
• 2:34 am – Watching DB servers alter thread table for prefix support… /bored
• 2:30 am – New PHP files in place and synced across web server cluster
• 2:29 am – DB backup done
• 2:27 am – Thinking I might not have enough beer for this…
• 2:25 am – DB backup still running (it’s huge… many, many, many gigs)
• 2:18 am – Got beer and craisens
• 2:17 am – Backing up DB
• 2:13 am – Making this post
]]> http://www.shawnhogan.com/2009/07/vbulletin-383-upgrade.html/feed 236 http://www.shawnhogan.com/2009/07/sony-blu-ray-bdp-cx7000es-400-disc-changer.html http://www.shawnhogan.com/2009/07/sony-blu-ray-bdp-cx7000es-400-disc-changer.html#comments Thu, 23 Jul 2009 23:20:03 +0000 Shawn http://www.shawnhogan.com/2009/07/sony-blu-ray-bdp-cx7000es-400-disc-changer.html So I’ve been waiting for this thing for awhile now, and it looks like Sony is finally going to be releasing it (even $400 cheaper than previously rumored). Yum, yum…

400 discs, serial control, Ethernet port for pulling meta data via Internet, etc… looks nice to me.


Press Release

SONY ORGANIZES YOUR ENTERTAINMENT COLLECTION

WITH NEW 400-DISC BLU-RAY DISC/DVD MEGACHANGER

Company Expands Blu-ray Disc Player Options with Three New Models
NEW YORK, July 23, 2009 – Looking to help consumers simplify and manage their living room entertainment experience, Sony today announced the new BDP-CX960 and the “Elevated Standard” (ES) BDP-CX7000ES 400-disc Blu-ray Discâ„¢ MegaChangers.

The company also announced the BDP-S1000ES single disc ES player expanding the Sony’s Blu-ray Disc line to 13 models offering a broad set of performance and features.

The Blu-ray Disc MegaChanger models store and play 400 Blu-ray Discs, DVDs, and CDs, allowing consumers to relocate their disc library to a convenient, easy-to-access location.

They also feature the ability to download information from Gracenote’s MusicID® and VideoIDâ„¢ products to organize movies and music. When a disc is inserted into the device, Gracenote technology automatically downloads information across an existing Internet broadband service and saves title, director, some cast information, release year, and genre information for most titles, allowing users to easily navigate the library intuitively through the player’s xross media barâ„¢ menu system.

“Think about all those great movie and music discs in your collection,” said Chris Fawcett, vice president for Sony Electronics’ home audio and video business. “Wouldn’t it be great to instantly play those movies or CDs — definitely gives a new meaning to the notion of a movie marathon!”

BDP-CX7000ES and BDP-CX960

Leading the MegaChanger line up is the BDP-CX7000ES which outputs full HD 1080/60p and 24p True Cinemaâ„¢. Designed to integrate with third-party automation control systems, the model features an RS232 connection and IR-in and is built to high-quality ES standards.

It offers exceptional audio quality with 7.1 channel analog output and superior video incorporating Sony’s HD Reality Enhancer and Super Bit Mapping technologies. The HD Reality Enhancer technology continually analyzes the original source bit by bit, sharpening edges and reproducing detail, while reducing the effects of film grain. Super Bit Mapping delivers smoother color gradation realizing true 14-bit equivalent color tone from 8-bit sources.

The model also upscales standard definition DVDs to 1080p through Sony’s Precision Cinema HD Upscale technology when connected to a 1080p display via HDMI.

Precision Cinema HD Upscale uses advanced conversion and processing to detect image changes at the pixel level, rather than the level of whole scan lines. Additionally, separate algorithms are used to process the moving and still parts of an image, resulting in sharp backgrounds with moving objects that are free from motion artifacts.

Complementing the BDP-CX7000ES, is the new BDP-CX960. This 400-disc MegaChanger also delivers full HD 1080/60p and 24p True Cinema output and Precision Cinema HD Upscale technology.

Both the BDP-CX7000ES and BDP-CX960 feature an Ethernet port for connectivity to a local home broadband network to download metadata from Gracenote, software updates and to access BD-Liveâ„¢ content. There is also an external port for local storage so users can add their USB flash storage device to support BD-Live content.

The models support 7.1 channel Dolby® TrueHD and Dolby® Digital Plus, DTS®-HD High Resolution Audio, and DTS-HD Master Audio decoding as well as bit-stream output via HDMI. It is compatible with an array of video formats, including BD-ROM/R/RE (BDMV and BDAV modes), DVD-+ Video/R/RW, CD/R/RW, and MP3/JPEG on BD/DVD/CD recordable media and supports x.v.Color™ (xvYCC) technology. The BDP-CX7000ES also supports Deep Color via HDMI (v1.3).

The BDP-CX7000ES is available this August for about $1,900, while the BDP-CX960 will be available this fall for about $800.
BDP-S1000ES

The single-disc BDP-S1000ES Blu-ray Disc player bolsters the ES line adding built-in Wi-Fi® (802.11N/G/B/A) capability for easy BD-Live access. The model offers full HD 1080/60p and 24p True Cinemaâ„¢ and upscales standard definition DVDs to 1080p through Sony’s Precision Cinema HD Upscale technology.

The model can easily connect to the Internet through an existing wireless home network to download and stream BD-Live content from select titles including additional scenes, short subjects, trailers, interactive games, and more. It also enables easy firmware updates to assist in keeping your player up-to-date with the latest Blu-ray Disc media and features.

While compatible with most wireless routers, the BDP-S1000ES also supports Wi-Fi Protected Setupâ„¢, for a quick and easy connection to enabled wireless routers. Additionally, the player is Digital Living Network Alliance (DLNA) ready, allowing it to connect to other DLNA compliant devices to watch digital photos on your TV.

The model also incorporates Sony’s HD Reality Enhancer, Super Bit Mapping, and Precision Drive HD technology.

It supports Deep Color video output and x.v.Color technology. The player is compatible with an array of video formats, including BD-ROM/R/RE (BDMV and BDAV modes), DVD-+ Video/R/RW, CD/R/RW, and MP3/JPEG on BD/DVD/CD recordable media.

The model also is compatible with advanced audio codecs including 7.1 channel Dolby TrueHD, Dolby Digital Plus, DTS -HD Master Audio, and DTS-HD High Resolution Audio decoding and bit-stream output, as well as analog 7.1 channel output.

The BDP-S1000ES is available this August for about $700.

]]> http://www.shawnhogan.com/2009/07/sony-blu-ray-bdp-cx7000es-400-disc-changer.html/feed 5 http://www.shawnhogan.com/2009/07/models-et-al.html http://www.shawnhogan.com/2009/07/models-et-al.html#comments Wed, 22 Jul 2009 07:35:44 +0000 Shawn http://www.shawnhogan.com/?p=849 I gotta say, iMovie HD is really nice for hacking together quick videos with pretty decent quality. I threw together this video for a friend real quick for NVR Strings. Oh darn, I get to look at girls in bikinis while editing video. Sad day for me.

The original video I output was full 1920×1080 high def, but here’s the YouTube version…

]]> http://www.shawnhogan.com/2009/07/models-et-al.html/feed 11