Congrats to the SliceHost team on their sale! I’ve had a chance to do some hosting there and I was really impressed with what they did. It’s pretty incredible what a few very smart people who clearly love their field can do in just two years.

That said, I’m not sure that this is the best news. Who knows what is going to happen to customer service and that sort of thing once the company is gobbled up by RackSpace. It looks like SliceHost is remaining somewhat independent, at least initially, but we are going to see a lot of technology crossing over in both directions, which is good in a lot of ways, but when it comes down to it SliceHost customers might not be able to expect the same level of support and interaction with Slice’s main team as before. I don’t have any specific issues with RackSpace, but it’s a big organzation and working with them won’t be like working with Slice, even if so much of the process is automated.

My server at The Planet (which we still employ for storage) is actually online at the moment so not everyone was affected.

Despite some bad press from time to time, I actually had a great experience with The Planet. I had a few strong servers with excellent uptime and the support was OK. I ended up using a third party security/management team, but The Planet always fulfilled their end of the bargain. An outage like this looks incredibly bad for them, but given the fire departments stance on activating the backups, it’s not really The Planet’s fault. The good news is that they said they will be honoring their SLA.

You can follow the matter on their forums. It’s nice to see how transparent they’ve gotten, this was not the case a few years ago (but then again few companies were).

What’s really bothersome about this is the despite all the redundancy pr0n and security that these hosts talk about when are you researching who to use, a single incident normally takes them out. All those huge diesel generators and retinal-scanning security doors and earthquake-proof building foundations are great, but it seems like something else manages to take these places down. I guess you can only prepare so much and we don’t hear about it when the redundancy systems work and a emergency goes as planned (well, as close to as planned as possible) but it still seems like the systems are more fragile than we’d like to believe.

We really need data stored across multiple data centers in different parts of the country (or world) but that leads to all sorts of problems with syncing and balancing and can get quite expensive quickly, especially if for someone who just needs a server or two.

If you haven’t been following the Bingodisk/Strongspage, you should because it is an interesting one. Basically after updating its Sun server they started to experience some issues and took the services offline. At this point it’s been about three days, and it’s been noticed by a lot of people. The people at Joyent apparently encountered a known issues with Sun’s OpenSolaris OS and its file system, which is known as ZFS. Not only has there been serious downtime, but the backups are corrupted as well.

The blog post points out that both systems were based on a single Sun X4500 server. Even now that the problem has been identified it is taking a significant about of time to get the server’s (up to) 24TB of data sorted out and restored. Once this is done all the data should be restored.

A little clip from Bingodisk’s site:

Protected against disk failure
BingoDisk uses hardware and disk architecture that is an order of magnitude more reliable than that hard drive you may be tempted to buy down at the local CompuMall. Rather than writing your data to a single drive, BingoDisk spreads the data across up to 45 disks, which significantly improves reliability while never sacrificing access speed.

Not to kick Joyent while they’re down but this does raise some interesting questions about online storage and the entire cloud mentality. The reason people store things in the cloud is for security and access. We trade privacy and certain aspects of control for access from anywhere and the comfort of knowing that professionals are taking care of our data and that it is 100% secure. This is why people, even technically proficient people, use webmail and online storage and it is the fundamental case for cloud computing.

As it turns out, all this data was stored on a single X4500, which amounts to a $35,000 piece of hardware.

…

I forgot to work on this draft for a few days, but it actually worked out well because now the two services are back online. No data was lost, which is a testament to skill of the Joyent team and ZFS. Another of semi-good news, is that this happened with their consumer products, not Accelerator, which is their main commercial product (though there have been complaints about this- mainly from Twitter’s people I think). Still there were ten days of downtime. This is being addressed by replacing the Strongspace service and giving 9 months of free service to current Strongspace users. Users will also get two months free for the new service and Strongspace will be opened sourced, which is really cool. Bingodisk users get two months free service and people can opt out of extended contracts with a refund if they want.

Overall Joyent is doing a great job of cleaning up their mess. Yes, they are just throwing money at people and hoping the stick around, but these things happen and they have handled everything extremely well. Not being a customer of theirs I am feeling really forgiving, but if I was relying on that data and could not get to it for 10 days, that might be a different story.

As as side note- I am still considering using Joyent for hosting in future projects. This is not the best thing for their reputation, but the Accelerator (the important service) was untouched and Joyent handled the issue with transparency. Plus they are really taking care of users now, which is nice, as a 10-day refund would not have been nearly enough.

This blog is over a year old, and for the entirety of that time it has been hosted on Media Temple’s Grid. The Grid (from the best I can tell, opened two days before, on Oct 17 2006, so I have been on it since about Day 3. As for me- I am not a professional server admin but have experience with more than a few different hosting services and I have use a number of dedicated servers as well.

I originally started the blog not only to get some thoughts out there, but also to use as a sandbox for web-based ideas and technologies. The Grid seemed perfect for this- cheap, scalable, stable, and easy to use. It is effectively a form of shared hosting so there are some sacrifices that have to be made, but overall the trade-off seemed worth it. For $20 a month with promised stability and the ability to withstand large spikes in traffic it seemed too good not to try. I am guessing a number of people chose it for the same reasons, though I know a number of developers who have used it as a starter hosting account for clients.

The Grid is a complex ecosystem which Media Temple has put together in order for people to share the resources of a cloud of servers while keeping prices down to a minimum. It’s a interesting idea and one that is theoretically much better than relying on a single server to do the job. The problem is that the scale of the Grid and getting each of the parts to work together has caused a number of problems along the way. Along the way the Grid has had serious problems with databases and email, as well as more than a little downtime and some latency issues. Things have gotten better along the way, but it’s been a bumpy ride for a lot of people. [edit- some of these problems occurred on the Grid's Cluster.1 today]

Personally I have not experienced any considerable downtime (that I have noticed) since Oct/Nov/Dec 06, but I have talked to people who have. Part of the confusion with talking with other people about the Grid accounts is due to the complexity of the system and the intricacies of the infrastructure, only some of which is ever seen by the user. (For example, each account has it’s own subdomain [http://sXXXX.gridserver.com/] and there are different Grid Clusters [for example Cluster.2]). From what I have heard, Cluster.2 has had more problems than the first cluster and customers cannot have their account switched from one to the other. As a side point, I don’t use Media Temple’s email, so I can’t comment on that.

I have found the Grid to excel in two areas- stability and ease of use. The cloud design of the Grid means that even under extreme loads (Slashdot, Digg, etc) the site will remain stable and responsive, with no drop in performance. What Media Temple does to account for this is charge you by the GPU (a unit of processor time) and if your account sustains a high load you will be charged for overages. That noted- it is well worth it because what is important to most people is uptime and the more traffic the better, so long as it does not take down the site. As far as ease of use is concerned, the Grid is extremely simple to operate. The backend is slow and somewhat tedious, but does the job without problems or confusion. MT did a great job here, and anyone who is not a server admin should be able to appreciate the simplicity and power of tools available with just a click or two.

One problem I have to this day with the Grid is latency. Regardless of what MT’s support team says, which is not a periodic problem, the average latency times on the Grid are high and they have been since the very start. The image above, courtesy of Pingdom, demonstrates what I am talking about and it has been very disappointing. I am not sure exactly the cause for this, but my guess it that it is a result of two things- the complexity of the Grid’s construction and capacity. The Grid has been a wildly popular service and I am sure that capacity and scaling is an issue.

Another major problem with the Grid is backing up. The backup tool that is built into the admin panel is slow and cumbersome, both of which I am fine with- but lately it has been disabled. For some time there was no automated backup solution and after that there was one, but it did not work reliably. Today (and for some time now) the backup tool doesn’t work. Their recommendation? Try FTP.

Along the way though, the Grid has seen some considerable improvements- enough that even detractors have not given up hope. The SQL containers are a great add-on (even if they do more than double the price of hosting) as are minor, but crucial tweaks, like allowing multiple DB users, an iPhone compatible admin panel, MySQL SmartPool improvements, and better webmail. MT has also been very open about problems and their blog is a great source of information for news and system incidents.

So based on my own experience, I would rate Media Temple’s Grid highly and I would have no problem recommending it to a colleague, so long as they were choosing it for the right reasons. It does have some incredible strengths, and it can scale to handle high traffic loads (if you don’t mind the GPU fees) but it is not a replacement for a dedicated server or multiple dedicated servers. If you have a project that you would not put on a shared server or trust to a $20-a-month hosting solution, you probably should not put it on the Grid. That noted, some very high profile and high traffic sites have been on the Grid (include ReadWrite/Web up until July).

Based on what I have heard- mainly the complaints of people I have worked with- I do have some doubts about the Grid. The problems seem to skew more towards Cluster.2 and use under professional circumstances, but these people had problems that should not have happened, especially some downtimes issues. Lots of problems have been documents in angry blog posts.

As for my own issues, I’m looking forward to the backup tool to be fixed and I have pretty much given up on the hope of decreased latency. As long as my uptime stays at or very close to 100% I can’t really complain too much, especially because the service allow me to very easily host and control a number of sites for just $20 a month. Response times from MT’s tech team have been slow (sometimes over 12hrs) but I have not had any mission-critical problems (at least ones that were not happening system-wide) so I can’t say for sure how much of a problem this is. The knowledge base is excellent so most of the time problems can be worked around with a little research anyway.

Overall, Media Temple’s Grid is still the best low cost hosting solution I have ever used. It has problems, but my account does not get pulled when there is a big traffic burst, it’s very easy to use, and it’s getting better. From the complaints I have read and heard, it seems like my experience may not be representative of everyones, or even a majority of people, but it’s hard to say because the most vocal people are generally those who are the most angry and/or disappointed.

Building a cluster
http://qualityhumans.com/images/cluster_build.mov

courtesy of Engine Yard

According Read/Write Web MT will be offering another solution in the near future, Nitro. This is a dedicated physical server (aka a dedicated server, just like those available anywhere) but you get access to Media Temple’s virtualization toolkit.

What does this mean? Well, from what we can see so far, it is a step up from the dv system- basically they get that virtual system on a shared box and move it to a dedicated system in a completely seamless upgrade. The virtualization is what makes this possible, so it’s a major attraction to sites that are on the rise as an upgrade will be easier than ever. Plus it will then be possible now to do hardware upgrades or to load balance across multiple Nitros. If your site has outgrown the dv Extreme package ($150 a month) this is the only feasible option.

The Nitro server is a very capable system. Here are the specs:

• HP Proliant Server
• Quad Core Woodcrest Xeon 2.33 Ghz
• 4 GB Fully Buffered DDR-2 RAM
• 2x 300GB 15,000 RPM hard drives
• 350 io/sec
• Fully Redundant RAID-1
• Battery-backed write-cache enabler
• 1GB Network switch
• 2 TB Monthly transfer

Those are 15K RPM SAS drives so I/O performance should be excellent. Despite the solid hardware specs a main attraction will be getting all that power while working through Media Temple, a company that does a very good job at making hosting as painless as possibe.

What’s the rub? Well, things start at $750 a month, so the Nitro system is not cheap. You could get one or two managed dedicated systems for that much at another reputable host, but you would miss out on some of the Media Temple frills- something that might not be as important to a professional server administrator. This deal is going to appeal mainly to sites that are on the dv system and are looking to upgrade without leaving Media Temple. They might pull in some new business we well, as they have been doing a very good job lately, but in this price/performance range they are competing with companies like Rackspace and Datapipe and are trying to attract sites who have full-time admins on board, so it is a big step for them (despite their previous experience in this segment).

One important issue will be that of support. From what I have seen, Media Temple’s support ticket system is quite slow, so this will not be acceptable for site’s paying this much money and who are lose out on hundreds or thousands of dollars for every hour down. My only experience with their support is through the Grid, so maybe the dv/Nitro (or just the Nitro) customers have their own support staff who can respond more quickly. I don’t know this to be true but it would make sense to keep these customers happy as they are high profile sites that bring in a lot of new customers (just look for that Media Temple logo on sites like TechCrunch or Read/Write Web) and they can easily afford to move to a fully-manage hosting service.

• This should come as no surprise, but- if you’re not a server admin/Linux pro then EC2 is not for you. Even if you don’t know a lot about hosting you can probably set up a web site using certain services, but EC2 is not one of them. Here you have to set up the server itself and then worry about setting up the applications you need. This is compounded by the fact that EC2 does not have persistent storage so you need to work with Amazon’s S3.
• Another obvious one- EC2 does not have a GUI, so you have to do everything on the command line. Developers have been writing tools, like Firefox plugins to do the basics, like initialize or reboot AMI instances, but these will not get you very far. That said, it is pretty easy to setup an instance and to start to play around with it.
• EC2 has no guarantees and limited support, so aside from some documentation and a few tutorials, you are on your own. The documentation is abundant, but none of it is amazing and almost all of it presupposes knowledge in certain areas.
• EC2 offers a trade-off of low computing costs (1.7GHz Xeon server with 1.75GB of RAM, 160GB of local disk, and 250Mb/s of network bandwidth [bursting to 1Gb] for about $75 a month) for relatively high bandwidth costs.
  
• There are alternatives to EC2. This service is very interesting but it’s not unique.
• A lot of us web guys think of EC2 only as a hosting solution, but this is not it’s only, or even it’s ideal, application. What EC2 offers is almost unlimited computing power with no setup fees, no need to buy tens of thousands of dollars worth of servers and SANs or anything like that. For example, here is an explanation of video conversion using EC2.
• The main point I have taken so far is that EC2 is not for the uninformed. If you are relying on tutorials to do the setup and you don’t know some of the basics, you will probably be able to get an AMI running and get it to do a few tasks, but you can easily get in over you head. You need to know about setting up redundancy and what will happen in case an instance crashes, because your data can easily be lost.
  

In case you are curious- I would rate my server administration skills at “just enough to be dangerous”. As in, I know the basics but am not at all a professional or even an expert. According to my own observations it seems clear that I am not up to the task of working with EC2, but I can afford to blow 10 cents/instance/hour on learning about it. I worked with EC2 on both a Windows system and a Linux one (Ubuntu 7.04) and would recommend Linux as it gave me easy access to developer tools.

For most people who are interested in hosting traditional web sites, S3 is really the more useful tool. It is only for static storage, but with some manipulation a lot can be done with it. I am going to keep in tinkering with each, but ultimately expect to be spending more time with S3.

Two picks from the blog…

What is utility computing?

Is there a difference between Grid and Utility Computing?

Traditionally, choosing your hosting solution was pretty basic- you could go with a shared hosting account (a lot of sites on one server) or a dedicated one (one site on one or more servers). This is a gross oversimplification and a lot of thought could be put into who does the hosting, what type of account you have, what kind of server, the operating system, how much control you have, and things like that. If you had a small site and did not need a lot of control you would go shared but if you needed power and/or root level access you would go dedicated. There is a lot more to it, like management and load-balancing, etc, but for most smaller sites that covers it.

Now though, things are getting much more interesting. People are still offering the traditional shared/dedicated accounts, but there are other options. Personally I hate to deal with hosting, but I am fascinated with what I see becoming available, many of which solve the problems I have had with a dedicated solution, eliminate the need for shared hosting, and challenge the traditional ways we have understood scalability. Personally I am most interested in grid/cloud computing and virtualization options. While they are not the best for all applications they not only reflect great advances in technology but also the future of web hosting.

Some of the companies I have been investigating-

• Media Temple’s Grid- a super cheap, really powerful computing grid. It has had a lot of problems, but works well and it super easy to use. I host this blog on it and originally opened the account to use as a sandbox…
• Amazon EC2- AWS’ elastic computing cloud. You pay for the computing power that you use. Works with Amazon’s S3 storage.
• Mosso- $100 a month grid backed by Rackspace. Aimed at resellers. It looks incredibly cool, but I have not heard good things about it…
• Joyent Accelerator- scalable computing, somewhat like EC2. This is the one I am most excited about, but I have yet to play around with it. I will be opening an account soon… (they used to be called Textdrive)
• The Gridlayer- from the people who run Layered Technologies. It’s a grid in which you can get a virtual private server or a virtual “data center”. Starts at $99 a month and looks interesting, but Layered is hit and miss (from what I hear). Seemed like an alternative to Media Temple’s Dedicated Virtual…
• Engine Hosting- from the people that run Express Engine. I’m not sure that this qualifies to be with the rest, but it looks cool. They have load-balanced, high-performance shared hosting, but it’s expensive.

These are definitely not your typical shared/dedicated solutions, but they can serve as replacements in many cases. For example MT’s Grid, once they work out the kinks, could be the best shared-style hosting on the net for just $20 a month and it theoretically has the power to handle just about any site load so long as you want to pay the extra GPU charges. For higher demand sites, where you need root access and lots of computing power, there are options like Joyent and EC2, but these are for more advanced users and may require a lot of setting up. One key characteristic of all of these is the loss of user control when compared to a more traditional solution, like colocation. You are sacrificing total control of the hardware for the scalability of a grid. This also means that, because you are on a grid there there are aspects of shared hosting in that what happens to other sites can affect your account (this is generally true, regardless of what the sales people tell you).

This is a really interesting subject and one I could probably go on talking about for some time. It’s amazing how the landscape of web hosting has changed yet some many sites are still going the traditional route (myself partly among them). I did not mention the impact of free hosting on sites like typepad and blogger, but those all play a major role in this. That is something to discuss in another post, but for the time being it’s definitely worth it to check out some of these sites and at least understand what they have to offer.