Wordpress before 3.5 to Wordpress 3.5. 1 button click. 1. Count em. 1. Uno. No dos. I am going to take that lesson to heart. One button.

Streaming reads and writes. Far beyond memory/cache/… all spinning disk. Remember, this is our “slow” storage. [root@dv4-1 ~]# df -h /data Filesystem Size Used Avail Use% Mounted on /dev/md2 55T 65G 55T 1% /data Run status group 0 (all jobs): WRITE: io=65505MB, aggrb=1467.7MB/s, minb=1467.7MB/s, maxb=1467.7MB/s, mint=44633msec, maxt=44633msec Run status group 0 (all jobs): READ: io=65412MB, aggrb=1814.5MB/s, minb=1814.5MB/s, maxb=1814.5MB/s, mint=36050msec, maxt=36050msec

I wrote this some time ago. This is even more true this year than last. So when people call me up and try to tell me of the glamour of working for another company, they need to take this into consideration. But they don’t. So they call all the extensions on our phone. And leave messages for everyone. Um … yeah. We are growing, much faster than I had anticipated. We are actually on a real live hockey stick revenue curve.

So I just dealt with a hack on the @sijoe twitter account. And I went through a process of re-locking everything down. What occurs to me, is that this is our cloudy future. Where resources could be effectively stolen from us, say CPU cycles and storage, not merely hacking useless social media sites, by fairly determined hacking groups. Think about this for a moment. You have a large allocation on EC2 for some reason, and your account gets hacked.

Somehow/somewhere, the @sijoe twitter account was compromised, and a bad tweet generated. I deleted the tweet. Then revoked all access to twitter from all accounts. Then made sure I’ve got two factor authentication up everywhere possible. Then changed all passwords on all accounts. Are we having fun yet? Somehow, I have a sense that this is our computational future. I’ll elaborate on this shortly. Let me finish hooking up the newly re-secured bits to each other (well, a more limited version of this …) And no, I wasn’t able to identify the culprit vector.

… I still can’t talk about it publicly, until everything is live, and I get the OK. But it is awesome, and its a pleasure to work with the large extended team we are working with. And yes, this is killing me. I love to talk about cool stuff.

This post. That was written in the late evening of the 29th of November 2011. Today is the 1 year anniversary of that visit. Chris Samuel (@chris_bloke) and his wife went through a similar event somewhat before we did. And he pointed out this XKCD to everyone on his twitter feed. We got our surgical slot quickly, I believe we were given priority. Not sure why, but the post-operative analysis indicated that the cancer had broken out of the duct, and was growing rapidly.

Our new JackRabbit tightly coupled storage and computing unit is on the test track, and about to go out the door to a customer. Need a few minutes with it, after quick tuning to generate some performance data. This is a single 4U server unit with 1/4 PB within it. Streaming 1TB from disk. This is using our tuned software RAID6. Our hardware accelerated RAID results will be generated later in the next batch of tests with new units we are building.

Its only when you stress a distribution infrastructure that you truly see its limits. And as often as not, the fail winds up being widespread. Our new 60 bay JackRabbit unit with CentOS 6.3 on it … and this is not a bash at CentOS, they do a great job rebuilding the Red Hat distribution without the copyrighted bits … has a number of software RAID elements on it. 9 in the current test.

Ok, this took me forever to get this done. But, I’ve had inquiries from a large number of people/companies, so here it goes: Have a looksy at the repo here This is the older code, with a single host at a time (plumbing is for many many hosts at once), with no triggers. That code is about a week away (I don’t like committing broken code). For what its worth, this is going to be used at scale in one of our projects.