Solved! ndbcluster ALTER TABLE yields “ERROR 1205 (HY000): Lock wait timeout exceeded; try restarting transaction”
0This is mostly a reminder to myself… but maybe someone else will find it useful as well.
I migrated some databases to ndbcluster (some of the tables were fairly decent sized… 9GB for 1 table spanning 220M records), and was running into a problem where an ALTER TABLE to change the storage engine was spewing out some cryptic error message like so:
ERROR 1205 (HY000): Lock wait timeout exceeded; try restarting transaction
Cryptic mainly because it was coming from MyISAM, which doesn’t have record locking. Long story short is it’s not a setting in your my.cnf file for mysqld, rather a setting in your config.ini for ndbd. The TransactionDeadlockDetectionTimeout setting defaults to 1200 (1.2 seconds), I ended up raising it to 24 hours just for purposes of migrating existing tables to ndbcluster (the setting of 86400000 ms is 24 hours).
Being relatively new to MySQL Cluster, it was also a good opportunity to practice making a config change and doing a rolling restart of the ndbd nodes to have no downtime.
Random Email #38
1I don’t really use email… but once in awhile I skim through them. I forgot how funny email is sometimes.
Email I got in it’s entirety:
Not sure but I think I was violated I may be drugged or hypnotized from home. Maybe check cause I can not. 213 0451(Samsung tv
Errr… is that some dude’s phone number without the area code? Who are you, and why are you emailing me about maybe being hypnotized from home? lol
Cacti Slow On Mac OS X (specifically RRDtool)
2So this has been driving me mad since I installed Mac OS X 10.7… Any time Cacti tried to output a graph, it would output, but it was painfully slow (like 30 seconds)… prior to installing 10.7, it was a fraction of a second.
Anyway… long story short is apparently RRDtool pulls the list of fonts from the system, and since it was running as the _www user, it was unable to write a the font cache to disk to make future font access fast.
Logging in as root and running the “fc-list” command to get a list of installed fonts fixed the problem. The first time it ran it took 30 seconds (imagine that! heh), and subsequent times it was instant. And Cacti graphs are back to being instant. Hopefully this helps someone…
Agnostic Servers
9Seagate is still not producing the hard drives I want for new servers in sufficient quantity, but they do seem to be actually producing them now. I’ve seen places where I can get 30 of them lately… but I need 72.
I’m assuming I’ll be able to get them relatively soon and have started to think about how I want to lay the servers out architecturally.
As far as hardware goes, gigabit ethernet just really isn’t that fast these days… Real world use has being able to pass about 80MB/sec (theoretical maximum is about 110MB/sec). When you have a cluster of servers passing massive amounts of data amongst themselves, 80MB/sec just doesn’t cut it. Which is why the servers have Infinband QDR (40Gbit interconnects), so they should be able to pass 3.5GB/sec (ish) between themselves.
For software/services, I’m thinking maybe 12 identical servers that are more or less service agnostic… with each server being a web server, MySQL Cluster data node and a MySQL Cluster SQL node. Then each server could probably be setup to handle 10,000 concurrent HTTP connections as well as 10,000 DB connections (1 per HTTP connection). With 12 servers, you could have the capacity of 120k concurrent web connections, 120k DB connections capable of doing millions of SQL queries/sec. If you need more capacity with anything, you could just bring online additional agnostic servers.
This of course is just in theory… who knows how it will test out in actual use, but it does seem like something worth testing at least.
I Saved $4.5 Billion
2
I was going to buy a Nimitz class aircraft carrier today, but instead I bought lunch at Rubio’s.
And to be totally honest, I would rather have a burrito from Rubio’s even if an aircraft carrier was the same price.
I’m a saving machine… I just saved $4,500,000,000 and I didn’t even have to switch to GEICO!
Math Has No Relevance
2Is there some sort of Congressional entrance exam where you need to NOT be able to pass 1st grade math?
Watching the whole deficit stuff unravel is like watching kindergardeners argue about something they don’t understand.
So let’s forget for a second that the US has around $115 TRILLION in unfunded liabilities, and just focus on a much “smaller” amount… the US debt is currently around $14.5 TRILLION.
Now the latest plan for debt reduction goes something like this… “We will only borrow and extra $800,000,000,000 per year instead of $1,200,000,000,000.” And it’s being billed as “saving $3 trillion over the next 10 years”. Last time I saved money, it was money I DIDN’T spend at all.
Let’s say I have $10,000 to my name, and I wanted to buy a new phone for $176,400 (the price of a Princess Plus iPhone)…

It’s $166,400 more than I can afford, so I decide to take out a loan for it. I decide that I’m going to instead get the less expensive version for $66,150 (less quality diamonds), so I only need a loan for $56,150.
I don’t know much about math, but I DO know that I’m amazing… I only had $10,000 to my name, yet I was able to save $110,250! If I do this every year for 10 years, I can save $1,102,500.
I’m a FUCKING GENIUS… I just solved my lifelong debt problem! The amount of money I can save is limitless.
Router Firmware Upload From Mac Via Xmodem
5I had a switch that “lost” it’s operating system and was stuck in a reboot loop. Long story short is I needed to upload the OS via a direct serial connection to fix it. I didn’t have a Windows machine available, so I used my Mac and a Keyspan USA-19 (USB -> DB9) thingie (heh). This is mostly a note for myself in case I need to ever do it again, and I forget.
Things that did not work…
- ZTerm let me get to the console just fine, but uploading the firmware with XModem failed for some reason (and you only found out it failed after a 3.5 hour upload time).
- Keyspan device on Windows via Parallels (from all the Googling I did, apparently it’s just an known issue with that piece of hardware and Parallels and the drivers will not recognize it).
- Virtual serial port in Windows/Parallels via a app called SerialClient (it was very difficult to find this app since the site that made it doesn’t exist anymore… people running Parallels said this worked for them, but it was many years ago… so maybe it was an older version of Mac OS X. Either way, I couldn’t get the app to “connect”.).
What DID work (thank God because I was running out of options)…
- Download/compile/install lrzsz via ./configure;make;make install.
- From Terminal: screen /dev/tty.Keyserial1 57600
- Hit CONTROL+A
- Type: :exec !! lsx -b -X ~/firmware_image.ros
It was infinitely faster than ZTerm (about 15 minutes vs. 3.5 hours), but more importantly it actually WORKED.
Starting To Think About Server Upgrades
11Amazing that it’s been half a decade now since we got the new blade servers that we are running on (got them in June of 2006).
Sadly, 12GB in a server just isn’t what it used to be and the database servers are really starting to feel strained as they beg for more memory. Handling ~25GB of databases on servers with 12GB is just no bueno. I thought of upgrading the RAM in the blades, but the maximum RAM the BIOS supports is 16GB, and all the DIMM slots are in-use… so it more or less would mean tossing all the existing RAM and buying 160GB of new RAM. In the end spending that much money to gain 4GB per server isn’t really worth it (especially since 16GB wouldn’t really be enough either).
I started to do a little research on what would be some good options for upgrades… I think I want to stay away from blades simply because we did have the daughter card in the chassis fail once which took down all 10 blades (the daughter card controls the power buttons). Buying 2 complete sets of blades/chassis is overkill just so you have stuff still up if one complete chassis goes down. The “must haves” on my list are hot swappable drives with hardware RAID, hot swappable redundant power supply, some sort of 10Gbit/sec (or higher) connectivity for communicating between servers. On top of it all, the servers need to be generally “dense” (I don’t want to take an entire rack of space).
PowerEdge C6100
The Dell C6100 actually looked really nice at first glace…
Pros
- Super dense (4 servers in a 2U package)
- Redundant power supplies (hot swap)
- 24 (!!) hot swap drives
- Supports 10GbE or even QDR Infiniband (40Gbit/sec
Cons
- Age – the server itself came out about 18 months ago without any refresh. That means you have hard drive options and CPU options that are a year and a half old
- Price – OMG… a single unit loaded up with 4 nodes, drives, RAM, Infiniband, etc. works out to $54,709 (before tax).
The age factor really becomes an issue when it comes to the disk drives… you can get 15k rpm drives in a 2.5″ form factor, but Dell only offers 146GB models (there are 300GB models now). The CPU isn’t really too bad… Dell’s fastest offering is the Xeon X5670… 2.93Ghz, 6 core @95 watts (wattage is important because of so much stuff crammed in there). There is a slightly faster 95 watt, 6 core processor these days… the Xeon X5675… the same thing basically, just 3.06Ghz. 0.13Ghz speed difference isn’t a huge deal… but the hard drive difference is a big deal.
I started to think… well maybe I could just order it stripped down and then I could just replace the CPU/hard drives with better stuff. Doing that, you still end up spending about $60,000 because you end up with 8 Xeon processors (valued at about $1,500 each that you just are going to throw away).
Then I started to think harder… Wait a minute… Dell doesn’t even make their own motherboards (at least I don’t think so)… so maybe I could find the source of these super dense motherboards and build my own systems (or just find the source of something similar… which ended up being the case).

SuperServer 2026TT-H6IBQRF
What do we have here??? The Supermicro SuperServer 2026TT-H6IBQRF is more or less the same thing… 4 servers per 2U, hot swap hard drives, same BIOS/chip/controllers… supports the same Xeon processor family (up to 95 watts)… And as a bonus, Infiniband QDR is built in (it’s a $2,596 add-on for the Dell server) as well as an LSI MegaRAID hardware RAID card (a $2,156 add-on for the Dell server).
So let’s add up the cost to build one of these things with the CPU/hard drives I actually would WANT…
- Chassis (includes 4 motherboards, 2 power supplies, hard drive carriers, etc. – $4,630.98
- Xeon X5675 3.06Ghz CPU – $1,347.84 each, so $10,782.72 for 8
- 8GB ECC/Reg DIMM – $124.26 each, so $5,964.48 for 48
- 600GB Seagate Savvio 10K.5 – $391 each, so $9,384 for 24 (also 410% more capacity, 21% faster and 20% more reliable than the 146GB options from Dell)
Add It All Up…
$30,761.88 would be the total cost (a savings of $24,138.82) and in the end you get slightly faster CPUs and *way* better hard drives. So in a single 2U package, you end up with 48 Xeon cores at 3.06Ghz, 384GB of 1333Mhz memory, 14.4TB of drive space (9.6TB usable after it’s configured as double redundant parity striping with RAID-6… which should be able to do more than 750MB/sec read/write). 8 gigabit ethernet ports and 4 Infiniband QDR (40Gbit) I/O.
Get 2 or 3 of those rigs, and you have some nasty (in a good way) servers that would be gloriously fun to run MySQL Cluster, web servers and whatever else you want.
PXE/Network Boot Linux With Mac OS X Server
1So I was in a situation where I want to upgrade the operating system on 10 blade servers that are in a data center. The problem is I really didn’t want to sit there and install the new operating systems on each one by one. The other issue is the servers don’t have CD/DVD drives in them since are blades.
I have a couple older Xserve G5s in the facility, so I figured why not use them as network boot servers for the Linux machines? By default OS X Server has a service for NetBoot (which is not the same thing and can really only be used to boot other Mac machines). But Mac OS X Server also has all the underlying services already installed to make it able to be a server for PXE booting from normal Intel BIOS.
So what network services do we need exactly (at least for how I did it)? DHCP, NFS, TFTP and optionally Web if you do an auto-install like I did.
Preface
This was written up in about 5 minutes mostly so I wouldn’t forget what I did in case I needed to do it again. Some assumptions are made like you aren’t completely new to Linux/Mac OS X administration. You can also have pxelinux boot to a operating system selection menu and some other things, but for what *I* wanted to do, I didn’t care about being able to boot into multiple operating systems/modes.
Setting Up The Server
Mac OS X Server makes it really simple to get DHCP, NFS and Web servers online since the normal Server Admin has a GUI for each.

Mac OS X has a TFTP server installed by default, but there it’s not running by default and has no GUI. You can of course enable/configure it from the shell, but just to make things simple, there is a free app you can download that will make configuring and starting the TFTP service simple (the app is just a configuration utility, so it does not need to run once you are finished, so adds no overhead). You can grab the app over here.

Files Served By TFTP
I was installing openSUSE 11.4, so that is what my example revolves around… most Linux installations should be similar, if not identical. First, make sure you have syslinux, which you probably already have since most Linux distributions install it by default.
Copy /usr/share/syslinux/pxelinux.0 to your Mac OS X TFTP Server: /private/tftpboot/pxelinux.0
Create a directory on your Mac OS X Server: /private/tftpboot/pxelinux.cfg, and then create a file named default within that folder with the following:
Obviously change the IP address to your Mac OS X Server IP address. The autoyast part is optional and only needed if you have an auto-configuration file for YaST.
Now we want to grab two files from the installation DVD/.iso so we have the “real” kernel.
Copy /boot/x86_64/loader/linux from the installation DVD to your Mac OS X TFTP Server: /private/tftpboot/osuse11-4.krnl (notice the file rename)
Copy /boot/x86_64/loader/initrd from the installation DVD to your Mac OS X TFTP Server: /private/tftpboot/osuse11-4.ird (notice the file rename here too)
Special Options For DHCP Server
There is not a GUI for adding special options to the DHCP server, but it’s easy enough to add them manually. Just edit the /etc/bootpd.plist file, and add two keys/values to it (option 150 is the IP address of your TFTP server, option 67 is the kernel file name the TFTP server is serving):

NFS Sharing Of Installation DVD
Edit your /etc/exports file, and add the following line to it:
/images/opensuse10-4 -ro
Then just copy the contents of the entire installation DVD to /images/opensuse10-4. You can of course just do a symlink or something if you want.
Cross Your Fingers…
Hopefully if all goes well, you should see something along the lines of this when you choose to network boot a machine (for purposes of this video, I did it in a virtualized environment of Parallels so I didn’t need to boot any running servers):
How Much Does AT&T/U-verse Cost?
6So I heard that AT&T is going to start putting monthly usage caps on it’s Internet users (which is me), so it got me thinking about how much I actually pay for AT&T services…
I have AT&T fiber to my house (actual fiber to the premise), so I use U-verse for TV and Internet in my house, so it’s break this down (I’m not including any on-time costs or fees/taxes)…
Television
I pay $138/month for TV (includes set top box rentals). I only watch MAYBE 8 hours of TV per month, but it’s nice to have it when you actually want to watch it. So I pay AT&T $17.25/hour to watch TV.
Cell Phone
I pay $159.99/month for 2 cell phones. Last month I used 17 minutes (which is actually more than normal). So I pay AT&T $9.41/minute for cell phone service. Oh yeah, I also paid them $125 for a MicroCell because their service is so crappy.
Internet
I pay $55/month for Internet because I can only get 18Mbit to my house (remember… I have AT&T fiber straight into my house). I’d like to pay for the $65/month plan to give me 24Mbit, but you know… it’s hard to get more bandwidth out of fiber I guess. I really wish Verizon would buy AT&T’s fiber to my house so I could get FiOS (they offer 150Mbit down/35Mbit up already and I heard they are testing 1Gbit up/down). Meanwhile I’ll be stuck at DSL speeds with my fiber. Sweet.
Mobile Internet
Now if I want a 3G connected iPad, that would be another $25/month (and I would rarely use it… so probably would be about $25/hour).
If I want tethering enabled on one of my phones, that’s another $20… even though I already pay for an unlimited data plan on that phone. lol
It would be nice if I could just pay $10/hour to use whatever I want… TV, cell phone, cell phone tethering, iPad connectivity, etc.
Government Economics or Ponzi Scheme?
1Here’s an interesting exercise… Let’s take the first two paragraphs from Wikipedia about a Ponzi scheme…
Now let’s swap a couple words here…
Maybe Steve Jobs Was Right About Flash?
4I was building a website for Digital Point Ads, and for what I wanted it to do, I figured Flash would be the way to go. So it was built, and worked fine in Flash.
The problem is that I’m an anal perfectionist and the Flash object was eating 80-90% of a computer’s CPU when being viewed… and me being me, I couldn’t just be okay with that.
…so I rebuilt it with CSS3/HTML5/jQuery. It turned out much nicer in my opinion, only uses about 1% of the resources that Flash needed and as an unintentional bonus, it works on things like iPad and phones/computers without Flash support.
I’m not anti-Flash (as I said, I *wanted* to do the site with Flash), but I AM anti-inefficient. There are still some things you can’t do with HTML5 that you can do with Flash, but those things are becoming fewer and fewer it seems.
WordPress 3.x
8So I figured it was about time to update WordPress. Went from WordPress 1.5 (haha!) to the latest (WordPress 3.0.1). I gotta say it’s quite a bit nicer now.
What Does Carmen Electra, Cyber-Terrorism and Meg Whitman Have In Common? eBay!
6Note: This was moved from blogs.digitalpoint.com to here, because well… blogs.digitalpoint.com is no longer a sub-domain we use (user blogs were wiped when we migrated to XenForo).
I haven’t said much about it to date about the dealings between myself and eBay, because well… I didn’t see a point. But now with people’s imaginations running wild about what did (and didn’t) happen, I suppose I’ll talk about it. The story itself is far more interesting than you would think possible within an affiliate program.
The Beginning
I started doing things with the eBay affiliate program in the fall of 2004. On October 20, 2004, I decided I would see if I could rank well for one of the “holy grail” SEO keywords… “eBay“. On November 9, 2004, I was in the top 10 in Google for “ebay”… specifically I was #9 (at the time, the top 50 results were just the various official eBay sites for various countries). On December 10, 2004, I held the #4 *and* #5 position in Google for the keyword “eBay” and this was a position I held in Google until April, 2006 (when Google updated their algorithm, and I no longer cared about the ranking). I also held the #1 spot for other things like, “eBay Registration” (even higher than ebay.com’s registration page). The rankings were partially done with the Co-op Ad Network. People saw me outranking eBay’s own domain for the keyword “eBay” and in December of 2004 the Ad Network exploded in popularity. All of a sudden we had hundreds of millions of pages on the Internet serving billions of Ad Network ads every day.
I got the attention of eBay because my affiliate income was going crazy and they assigned someone to be my “go to” person for anything I needed within the eBay program. This person was assigned to me in early 2005.
Personally I was more than happy with the income I was getting as an eBay affiliate, but eBay was not and helped me come up with new/innovative ideas for driving more traffic to eBay.
Spring 2005
Due to the overwhelming popularity and reach of the Ad Network, eBay came up with the idea in the spring of 2005 that we should use our ad network for more than just helping people rank well in search engines. The logic was that we are serving billions of ads every day, so why not use it as a traditional advertising delivery system? It made sense, so we tried it out by using a small percentage of the Ad Network ad space to serve up tens of millions of eBay ads every day that ultimately were affiliate links. Affiliate income jumped another 300% around that time (as expected).
Summer 2005
eBay was pleased… they were getting massive amounts of traffic and it sure made their affiliate program look good. In the summer of 2005, eBay decided it needed more traffic from me. I told eBay I couldn’t drive any more traffic. They responded that I should “experiment” with what they deemed “grey area” things (this is what eBay called anything that violated their terms of service).
At this point, things started to seem strange to me. eBay was paying affiliates millions per month, when they had no competition… for the most part it was traffic they would receive anyway. And why was eBay *happy* (and they were) that I was outranking them for their own company name and paying me for it? Someone searched Google for “eBay”, came to my site and I would send them to ebay.com to get affiliate revenue. Finally I confronted eBay about it all. I told them numerous times that I didn’t understand even why they HAD an affiliate program, and that I would gladly do what I do for them for 1% of what they were paying me. The response I was met with with ultimately was (and I quote), “Well don’t tell anyone that. Why do you not like ‘Free money’?”
When I asked them why they would knowingly allow affiliates to violate their terms of service, they were very good at avoiding answering my actual question. Finally after pestering them with the same question for weeks, they broke down and informed me that their terms of service (and even the entire affiliate program to some degree) was a bit of a facade. It allowed eBay to do things they wanted to do (like spam search engines, deploy in countries where they had no actual presence, etc.), while also giving them a way to wash their hands of any wrong-doing when any of their large partners (like Google) would question them about it (like why there are so many spam sites directing people to eBay). They could simply say, “It’s our affiliates, and they are violating the terms of service we set forth.” To me, I suppose it sort of made sense and I stopped questioning them about it. BTW, one of the times this was explained to me was at PubCon in Las Vegas, *while* an eBay employee was going around to each public access computer with a USB dongle he developed that would automatically install something that would redirect any user to eBay when they tried to access Yahoo Auctions.
I was informed by eBay that they understood that in order to keep the interest of their large affiliates and keep them creative/innovative they allowed them to experiment with doing pretty much anything as long as the affiliate let them know if they put something into large-scale deployment and it violated their terms of service. So in the summer of 2005 I played around with all sorts of things (most things did not violate their terms of service, and most things were ultimately bad ideas for driving a decent amount of traffic). One of the things that was toyed with was a mechanism to force the end user to click through to a site that they didn’t actually click on.
I first heard the name Ben Edelman towards the end of the summer. Apparently eBay contracts with Ben to do random compliance checking on their affiliates and issues a monthly compliance report to them. I showed up on his compliance report because this was the time they gave me the go ahead to play with non-compliant things. eBay then proceeded to amend what they told me prior. I was free to do experiment with whatever I wanted, as long as I didn’t show up on any outside compliance reports. They said outside compliance was something they had to do as a publicly traded company, but wasn’t something they paid much attention to internally. When I first showed up on Ben’s compliance report eBay told me it would be best if I just blanket filter (via geo-targeting) the area were Ben worked as well as locations Ben Edelman might be. This included the bay area, Santa Barbara (CJ was located there), Boston, Washington DC as well as an area in upstate New York. eBay even sent me a copy of the “secret monthly compliance report that no one was supposed to know about”. The fact that I showed up on his compliance report was a bit irrelevant anyway since I wasn’t experimenting with any grey area stuff any longer by the time the report was given so in the end there wasn’t anything I needed to change.
Geo Visitors
eBay knew how widely used our Geo Visitors tool was (installed on millions of web pages, MySpace profiles, etc.). So they asked me to direct traffic to eBay when someone clicked on the Geo Visitors button that was widely installed, which looked like this:
Instead of sending the user to the map they were expecting, they would end up on eBay (this was done sometimes, not all the time). I actually brought up the point to eBay that this violates their own terms of service… specifically the part of their terms of service that state you can’t mislead the end user or trick them into clicking something. After further discussion my eBay rep came back, saying they talked to their legal department and I was in fact correct. Their solution wasn’t to stop the traffic though, but instead make the Geo Visitors button display as a small eBay ad instead. Specifically, the Geo Visitors button would sometimes show as:

Personally, I was not okay with this, as it really seemed like a bait and switch… The whole reason I brought up the original implementation as a violation of their terms of service was because I didn’t want to do it anymore and was hoping they would stop applying pressure for me to do it. In the end, the pressure eBay put to leave it as a “sometimes” eBay ad won out. And affiliate revenue again nearly doubled.
Spring 2006
My life started to get very strange. People were finding out how much money I was making by being an eBay affiliate, which made people go a little crazy I think. My servers were getting around 100,000 hack attempts per day, people were showing up on my doorstep from Europe (literally) threatening me with crap if I didn’t tell them what I was doing for eBay. My car got stolen from a “secure” parking structure. More than one person found where I was building a house, broke into the development to take pictures and post on their blog, etc.
I didn’t like the attention and this was not the life I wanted.
I proceeded to inform eBay that I no longer wanted to participate in the eBay affiliate program… but every time I brought this up with them (which was every time I talked to them at this point), I was guilted into staying, with them asking me to stay “one more month” or “can you just wait until the end of the quarter so it doesn’t wreck our numbers?”. Finally I REALLY wanted to be done and not strung along anymore and again was guilted into staying with them telling me how many hours they spent getting me a “special rate”, etc, etc. and now it would all be for nothing if I was going to quit.
eBay Live! – June 2006
The top affiliates get invited to an eBay affiliate conference that takes place during eBay Live! each year. The conference was in Las Vegas this year and eBay affiliate program managers (along with some other eBay executives) invited me to a “secret meeting/dinner” where I was the only non-eBay employee invited. We had dinner at Mix, a swanky restaurant at THE Hotel.
Who cares, about a private dinner, right? Normally yes, until you get into what the topic of discussion at this dinner was. This dinner meeting had two main topics… 1. How can I drive more traffic to eBay (always the topic with eBay) as well as something far more interesting.
The Black Budget
One topic that was discussed was eBay secret “black budget”. This was described as a large allotment of money that eBay was free to do what they wanted with, without it being reported on accounting sheets (and in turn shareholders). eBay wanted me to REALLY ramp up spamming the web with eBay ads. I told them I wasn’t interested at ALL and in fact still wanted to quit the program completely, not “ramp it up”. I explained to them that Google was pretty good to me as far as sending me traffic and that I had no interest in spamming Google search results. Then they offered to buy any hardware I wanted with their black budget and get it co-located offshore if I wanted so that no one could trace the spamming back to me or digitalpoint.com. I still told them I wasn’t interested.
Then they made it very clear that they have no love for Google at all and would actually pay me whatever I wanted from their black budget to “hurt Google in any way I can”. I didn’t really understood what they were asking or even why they would want to “hurt Google”. I pointed out that Google is not their competitor and hurting Google ultimately would only hurt them in the end since they get traffic FROM Google. Finally they came out and said they were angry at Google because eBay was one of the largest AdWords advertisers and Google recently changed their AdWords pricing to take into account “ad quality” (more on that here) and it was costing eBay exponentially more in advertising dollars. I still wasn’t clear exactly what they meant by “hurt Google”, and pressed them to be a little more specific. Their answer was, “You are creative… think of anything you can think of and just name your price. Maybe you could figure out a way to take down Google datacenters somehow?” eBay even flew down an executive from their pay per click advertising division to talk about this.
Yeah, no. eBay asking me to engage in cyber-terrorism against Google… thanks, but no thanks. I’m not going down that road.
After I made it blatantly clear that I was not interested in “hurting Google” for any price the topic turned back to how I could drive more traffic to the affiliate program. I told them that I really didn’t think there was a way I could drive more traffic. They questioned me about any of the “grey area” stuff I experimented with in the previous summer and if there anything in there that could drive traffic. I told them that it was *possible* to add additional traffic, but the only traffic they could get from that would be non-compliant. Their response was, “As long as you don’t show up on compliance reports, it’s compliant as far as we are concerned.”
Carmen Electra & The Super Bowl
In 2006, I had a new idea for driving traffic to the eBay affiliate program that started as a joke. I had the idea to “Win A Date With Carmen Electra” (or some other person along those lines that would agree to it). The idea entailed running a Super Bowl ad that directed you to a site that passed traffic through to eBay via an affiliate link. My joke became serious when eBay actually wanted me to run the Super Bowl ad. They even went so far as to get special approval from their legal department after I expressed my concern about spending the money to run the ad when they could decide they didn’t want to do it (or pay the commissions on it) after I already foot the bill. There wasn’t enough time to get it ready for the Super Bowl that winter (and eBay was rather annoyed I couldn’t do it that year).
Fall 2006
I don’t recall the exact time, but I believe it was in the fall of 2006. I showed up on Ben Edelman’s monthly compliance report again, and this time he was absolutely furious with eBay employees about it. What *I* heard is that during the conference call with the eBay affiliate managers, he did a lot of yelling and screaming about why I was still participating in the eBay affiliate program when I should have been terminated last time I showed up on his report.
eBay’s response to him? “We’ll take care of it.”
eBay’s response to me? “Ben can only cross reference affiliates by PID. Please change your PID in case you show up on Ben’s report in the future.” Yep, that’s right… eBay only wanted me to change my tracking ID so Ben Edelman couldn’t see it was an affilate that was on his report in the past.
Private Jets
eBay knew I was not driven by the money (clearly since I told them constantly I didn’t want to participate in the program any longer and that I would do what I did for eBay for 1% of what they were paying). They ended up getting creative to keep me interested in the program for the last year. Going so far as to trying to get clearance so I earned hours on a private jet instead of commissions (even if those commissions were worth more, it was more interesting to me).
Rover
In the fall of 2006 eBay was switching over to their in-house rover links and I was very slow to switch my links to rover. I didn’t see a point in it really… the old links were working just fine. They were oddly insistant that I move my links to rover, but would never actually tell me why. Their refusal to tell me why made me not want to do it, so we went around and around for months with this. I told them I would switch my links to rover as soon as they told me why it’s so important. Finally they said it was important to them because traffic going through rover links had no compliance checking.
The End
In June of 2007, my affiliation with eBay ended. And truthfully, I was very happy about it (in the previous year, nearly every communication I had with eBay, I would bring up the fact I didn’t want to participate in the eBay affiliate program any longer), so it was an easy way out without them begging/pleading that I stay in the program “just a little longer”.
eBay’s Numbers
eBay was clearly driven somehow by the overall commissions paid out to affiliates. There were months where their numbers would be “low”, so they would give retroactive bonuses out to all (or at least the top) affiliates for the previous month, and were constantly upping the payout rate to affiliates (which is odd only because they had no real competition).
I suspect eBay’s management staff within the affiliate program were probably getting quarterly bonuses based on how much commissions were paid out to affiliates.
Non-Compliant Traffic
eBay’s “favorite” traffic source back then was also technically violating their own terms of service. Cloaking search engines via server-side redirects was the thing they loved the most. But again… they stated their terms of service acted more like a scapegoat they could point to when their partners (notably Google) would question them about it.
So What Happened?
This part is purely speculation, but the feeling I get is that someone higher up in eBay got wind of what their affiliate program managers were doing and encouraging affiliates to do and “cleaned house”. I heard a rumor that the majority of the top 100 eBay affiliates were axed at the same time I was. I also heard a rumor that the eBay affiliate program managers inside the company were also “let go” (or at the very least relocated to different departments) at the same time. Did it have something to do with Meg Whitman’s (eBay’s CEO at the time) departure from eBay? Who knows, but I can only assume eBay knew of Meg Whitman’s departure at least a few months before it was official and started to bring in the new management team. It would all coincide with a timeline of someone from the new management team looking deep into how things were run.
So if someone higher up wanted to “clean up the department”, what do you do after you get rid of most of your top affiliates and replace the internal employees running the department? You take your previous top affiliate (me) and attempt to make an example out of them. In the course of this lawsuit, eBay has even said it’s not about the money.
Okay, So Why The District Attorney Then?
Off the record, one FBI agent told me that they (personally) thought this whole thing was a waste of their resources, and was a civil matter between eBay and myself. Politics is politics, eBay is eBay and coincidentally one of eBay’s civil lawyers used to work in the district attorney’s office.
A 3rd Party CAN Get You Banned From AdSense
0Note: This was moved from blogs.digitalpoint.com to here, because well… blogs.digitalpoint.com is no longer a sub-domain we use (user blogs were wiped when we migrated to XenForo).
I’ve heard people claim they were banned from AdSense unfairly for this, that and whatever other reason though the years… and to be honest, I just chalked it up to them doing something they shouldn’t have been and just not admitting it.
Low and behold, it *can* happen to even larger publishers (I believe we were approaching over 1,000,000,000 [yes, billion] AdSense impressions over the years). Note: I can’t confirm the exact number because, well… my AdSense account was disabled.
We get advertising inquires daily and we even go so far as directing people to AdWords and explain to them how to use Site/Placement targeting if they wish to advertise on digitalpoint.com. It’s less money for us, but in the end it’s easier and less to manage. I would guess Google has gained at least 200 NEW AdWords advertisers because of this.
Even more ironic is we actually run the largest AdSense support site to help publishers. Google’s own AdSense support site has 75,783 discussions vs. 760,145 for our AdSense help forum. We’ve even had Google AdSense representatives posting in our forum.
The Warnings
In the last month, we received 30 warnings for running AdSense ads on non-compliant websites (gambling related). These are sites that I don’t own, have no affiliation with, nor do I know who the owner is. I have no idea why someone would want to use my AdSense publisher ID on their site, but I guess that’s beside the point really. AdSense allows you to set a whitelist of your sites just so this doesn’t cause problems. We have used this whitelist for a long time (since we first heard about it), and none of these gambling related sites were on our whitelist.
Hell, I even got an email from Google *because* I use a whitelist (and STILL didn’t turn off the whitelist function)…
Your Allowed Sites settings blocked $220 in earnings last week
We noticed that you’ve been receiving ad activity on sites which aren’t included in your Allowed Sites list. If a URL displaying your AdSense ad code is not on your Allowed Sites list, ads will still be displayed, but you won’t receive any earnings for that URL.
For your reference, sites that display your ad code, but aren’t included in your Allowed Sites list generated roughly $220 from May 2 through May 8.
My Response
After seeing a zillion of these notices coming in, I responded and let them know that these are not my sites and that I use their whitelisting feature (and that none of these sites are on my whitelist).
Sarah H. from Google’s AdSense Team responded back a couple weeks later letting me know that, “If that site or URL is not in the Allowed Sites List within your account, no further action is needed and this issue won’t negatively affect your account in any way.“. Alright… no further action is needed.
Account Disabled
Fast forward a couple days and I get this email from Google letting me know my account is now disabled because of violations of program policies… specifically, AdSense publishers are not permitted to place Google ads on sites with content related to gambling or casinos. I *still* don’t own a gambling/casino related site (nor have I ever), so I’m assuming it’s related to the 30 warnings I got in the last month for someone else trying to run my publisher ID on their sites.

While I still think the majority of people who claimed to have their AdSense account unfairly terminated are probably just whiners that got caught doing something they shouldn’t be, I can say for 100% certainty now that it can (and clearly does) happen sometimes.
I guess it’s time to finally start managing advertising in-house… Just one more thing to add to the “to-do” list. /sigh
Genius Tasers Himself
24This kid might not be the brightest human to ever walk the face of the planet, but his unintentional comic relief if priceless.
Scubacraft
18So I think I decided what I want for Christmas… A Scubacraft (I saw it in Wired magazine).
It’s basically a boat that you can hit a button, and it will go underwater (up to 100 feet). It would be pretty awesome to go scuba diving and instead of leaving your boat at the surface, just take it under water with you when you get to the dive site.
It’s not pressurized, so it’s not a submarine (you would still need scuba gear of course).
It also has an onboard computer that controls your ascent and descent so you don’t go to quickly and give yourself the bends.
I love you, Santa!


Google Ups AJAX Search API To 8 Pages
6It seems Google has quietly raised it’s limitations on how deep you can look into search results for the AJAX Search API. It’s always been 4 pages (8 results per page), so you could only see the first 32 results.
All of a sudden it was changes to 8 pages (still 8 results per page), so now you can look at the first 64 results. This is quite handy.
Thanks Google, you’re swell!
iPhoto ’09 Faces Spinning Forever
6Someone I know uses iPhoto, but the Faces part of it wasn’t working. New pictures wouldn’t get scanned for faces and the spinning icon next to the Faces label on the navigation bar would just spin forever (and apparently it’s always been stuck like that).
I Googled the issue, and came up with hundreds of people having the same problem, but no one seemed to figure out a solution.
After a lot of digging and poking around, I think I found the solution. She imported large folders into iPhoto from her old photo management application, and she had some movies in her photo folders. So this is how I fixed it…
- Quit iPhoto first.
- Then get into the guts of your iPhoto Library (right-click the iPhoto Library file in your Pictures folder and do “Show Package Contents”).
- Go to the Originals folder in there and get your movies out of there.
- Delete the face.db and face_blob.db files.
- Load iPhoto up and it will start scanning all your faces and with any luck actually get through them all and be kosher going forward.
Warning… this will delete all your face tags you had previously (so don’t do it if you tagged a bunch of people manually and don’t want to lose that).
Random Email #37
9I haven’t posted a random email in awhile, so here’s a new one…
I am Mr Mark Jo—-, I will like to make reservation for 6 people that will be coming for vacation in your area on the 20th November to 30th November 2009.
If you have vacancy for the specified period, give me the total cost for the whole period of 10 nights, with either 6 single rooms or 3 double rooms for 6 guests.
Number of Persons: 6
Mr & Mrs Alb—-{37 and 33yrs}
Mr & Mrs Ben—-{40 and 36yrs}
Mr & Mrs Ari—-{44 and 38yrs}
Arrival date: 20th November 2009
Departure date : 30th November 2009.
Number of days: 10 days
Number of Guest: 6,
Confirm availability and get back to me with your rates and total cost and if you will accept major credit card for your payment and convert the total price to British Pounds or Euro.
Thank you and looking forward to hear from you soon.
Regards,
Mark
I’m open to suggestions on how much I should charge these guys to stay at my home… let me know.
The Best News Show Ever
8So I’d like to put together a newscast… I want this guy to be my weatherman, and the guy from the previous blog entry needs to be my field reporter.
Random Quotes
6So I found a draft post for this blog from about 3 years ago that I forgot to post (oops). Better late than never, right?
San Diego Chargers vs. Oakland Raiders
2
The first Monday Night Football of the season, and it’s the Chargers vs. the Raiders. lol
The Chargers play the Raiders twice a year (being in the same division and all), and the Raiders haven’t won since 2002. So uh… yeah… If you are a Raiders fan, good luck on Monday.
Philip Garrido’s Blog
8So the fruitloop that kidnapped an 11 year old girl, held her captive in his backyard shed for 18 years while she bore 2 of his children has a blog.
http://voicesrevealed.blogspot.com/
So here’s the deal… Google is really good at determining relevancy of a webpage and what a webpage is about. Maybe they should modify their algorithms a bit to detect crazy people automatically.
Facebook Is Hard To Use
26Proof that you should have to take some sort of IQ test before having a Facebook account.


MySpace Traffic
6I know Alexa data isn’t terribly accurate, but for sites under an Alexa rating of 500 *and* within the same market, it’s pretty accurate as far as relative traffic to sites you are comparing. That being said, how long do you think before they sell it or shut it down?
Their traffic is almost cut in half from a year ago (and still dropping)…

Red Bull Cola
9
Here’s some wise words for you based on an experience I had last night. If it’s 2am and you are getting ready to go to bed, maybe don’t drink two Red Bull cola drinks.
Someone was telling me about them the other day and I was like, “gee, okay… I guess I’ll try it to see what it’s like.”
Also, for the record, it’s not very good. Tastes like they just mixed Red Bull and Coke 50/50.
How To Get Twitter Followers
10
Well actually, I’m not exactly sure how because I have Twitter followers since I don’t have a Twitter account, but apparently I have have a couple Twitter accounts…
http://twitter.com/digitalpoint/ – 1,127 followers
http://twitter.com/shawnhogan/ – 63 followers
Stupid, but funny. lol
vBulletin 3.8.3 Upgrade…
236Okay, gonna go ahead and upgrade the Digital Point Forums to vBulletin 3.8.3… Since so many of you are freaks and are on it 24/7, I made this post to keep you up to date on what’s going on (I’ll update it periodically throughout the process). You can also use this blog post as the new temporary forum for all your discussion needs while I do the upgrade. hah

- 3:26 am – Reading and posting seems to work… I suppose that’s good enough for now (still working on the other stuff, but don’t need to force everyone off to do it).
- 3:24 am – Skimming over various areas of the forum to see if things (mostly) work. I could let you guys in before the templates are fully updated possibly.
- 3:22 am – New moderator permissions set
- 3:17 am – Static CSS files located on single server in web cluster and spread around properly (I hate this about vBulletin BTW… gimme a hook location to do it please!)
- 3:09 am – Core upgrade done.
- 3:07 am – Recoded some of the upgrade scripts so they aren’t making that change to the reputation table. Will deal with issue this later instead.
- 3:00 am – Stupid reputation table was altered in 3.8.0 to not allow negative userids so it can accommodate 4 billion users instead of “only” 2 billion. Stupid. I used negative userids internally for some stuff. /thinking what to do about this…
- 2:55 am – First problem… reputation table alterations not going well. Going digging in raw database…
- 2:50 am – Running “ALTER TABLE pmtext” on DB servers. That’s a big one… going to get another beer.
- 2:49 am – Up to version 3.8.0 alpha 1
- 2:43 am – It’s almost 3am. If you are a cute girl reading this, please post your picture in the comments, k thanks.
- 2:41 am – I forgot we have to go through all versions to get to the newest… lol… going through 3.7.0 beta 4 at the moment.
- 2:40 am – Userlist rebuilding
- 2:36 am – Watching DB servers alter thread table for tagging support… (this is really boring)
- 2:34 am – Watching DB servers alter thread table for prefix support… /bored
- 2:30 am – New PHP files in place and synced across web server cluster
- 2:29 am – DB backup done
- 2:27 am – Thinking I might not have enough beer for this…
- 2:25 am – DB backup still running (it’s huge… many, many, many gigs)
- 2:18 am – Got beer and craisens
- 2:17 am – Backing up DB
- 2:13 am – Making this post
Sony Blu-Ray BDP-CX7000ES 400 Disc Changer
5So I’ve been waiting for this thing for awhile now, and it looks like Sony is finally going to be releasing it (even $400 cheaper than previously rumored). Yum, yum…
400 discs, serial control, Ethernet port for pulling meta data via Internet, etc… looks nice to me.

Models et al
11I gotta say, iMovie HD is really nice for hacking together quick videos with pretty decent quality. I threw together this video for a friend real quick for NVR Strings. Oh darn, I get to look at girls in bikinis while editing video. Sad day for me.
The original video I output was full 1920×1080 high def, but here’s the YouTube version…
Override CSS
6So I found myself wanting to override a few items in a sitewide CSS file, but only for a single page… Rather than make entire new CSS elements, I figured there had to be a way to override the CSS file with a <style> clause within the page itself…
Sure enough, I finally figured it out… !important…
[code=css]
[/code]
!important can go at the end of any style attribute and it will override anything that isn’t “important”.
I’m Getting Married!
14…well at least I assume so. I can’t imagine he would turn me down.
Dave, you have a Shawn Hogan category on your blog… marry me? <3 We can frolic in a sea of code and whatnot.
http://www.dellanave.com/blog/
If I have a pool party in San Diego, will you come?
