Scaling 101 - We are Failing the Next Generation

Tue, Jun 26, 2012 12:01 AM
The other day Twitter was down and I had no place to comment on Twitter being down. This got us to talking about scaling at work. I was reminded of the recent slides posted from Instagram about their scaling journey. They are great slides. There is only one problem I have with them. They are just the same slides that you would find from 2000 about scaling.

I have to say, I like Instagram. My daughter has something like 1,000 followers on Instagram. And good for them for being bought by Facebook for a bajillion dollars. This is not a dig on them really. This is a dig on our industry. Why did Instagram have to learn the hard way how to scale their app? I want to point out some of their issues and discuss why its silly they had to learn this the hard way.

Single machine somewhere in LA

Why would anyone deploy an app to the app store when the backend is all on one server in this day and age? I am not big poponent of the cloud, but that has to be better than a single server in a rack somewhere. And L.A.? Go for Dallas or somewhere geographically neutral.

favicon.ico

So, this is one of the biggest mistakes I see in all of web application developement. People use Apache or Nginx or whatever and have a mod_rewrite command that sends ALL requsets into their appliction stack. They do this because they are lazy. They want to write whatever code they want later and just have the request picked up without any work. At dealnews, we don't do that. We have controllers. But, we specify in our Apache config what paths are routed to those controllers. The most general we have is:

    RewriteCond %{REQUEST_URI} (\/|.html|.php)$
    RewriteRule ^.+$ /torpedo.php/$1 [L]

So, if the request ends with / or .html or .php, send it to the controller. This is the controller our proxy servers use. Any other requests like robots.txt, images, etc. are all served off disk by Apache. No sense having a PHP process handle that. It's crazy. This is not Instagram's fault. They likely followed some examples they found that other developers before them put on the internet. Why are we doing this to people? Is it some rite of passage? I suspect that the problem is actually that 90% of the web does not require any scaling. Only 10% of the sites and services out there have to actually worry about load. So, this never comes up.

So, the next part of their slides basically glorify the scaling process. This is another problem with people in our field. We thrive on the chaos. These are our glory days. Let's face it, most geeks never won the high school football championship. The days when we are faced with a huge scaling challenge are our glory days. I know I have had that problem. And I played sports as a youth. But, nothing is better than my 2006 war story about a Yahoo front page link. Man, I rocked that shit. But, you know what. The fact that I had to struggle through that means I did not do my job. We should have never been facing that issue. It should have just all worked. That is what it does now. It just works. We don't even know when we get a spike now. It just works. The last thing I want in my life now is an unexpected outage that I have to RAGE on to get the site up again. That leaves me feeling like a failure.

Then they realize they are out of the depths. They need to do things they never thought they would need to do. Why not? Why did they not know they would need to do these things? Unexpected growth? Maybe. Why is this not common knowledge though? With all the talk of the cloud being awesome for scaling, why was this not a button on the AWS dashboard that said [SCALE] that you just push? That is what the cloud does right?

In the end, Instagram learned, the hard way again, that you have to build your own architecture to solve your problem. I learned it the hard way. LiveJournal learned it the hard way. Facebook, Twitter, etc. etc. They have all learned the hard way that there is no single solution for massive scale. You have to be prepared to build the architecture that solves your problem in a way that you can manage. But, there are basic building blocks of all scaling that need to be in place. You should never, ever, ever start with an application on a single server that reads and writes directly to the database with no cache in place. Couch, Mongo, blah blah blah. Whatever. They all need cache in front of them to scale. Just build it in from the start.

Instagram was storing images. Why were they surprised when they ran out of room for the images? I just can't fathom that. Its images. They don't compress. You have to put them somewhere. This has to be an education issue. LiveJournal solved this in 2003 with MogileFS and Gearman. Why did they not build their arch on top of that to start with? Poor education, that is why.

One thing they bring up is one that has me bugged is monitoring. There is no good solution for this. Everyone ends up rolling their own solutions using a few different tools. The tools are often the same, but the metrics, and how they are reported and monitored are all different. I think there is a clear need in the industry for some standards in this area.

if you’re tempted to reinvent the wheel ... don't

I did find this slide funny. Reading these slides for me is like seeing them reinvent the whole wheel that is scaling. This has all been done before. Why are they having to learn it the hard way?

don’t over-optimize or expect to know ahead of time how site will scale

I take exception to this slide however. You should have some idea how to scale your app before you deploy it. It is irresponsible and cowboy to deploy and think "oh, we will fix it later". That is true no matter what you are doing. Don't give me the lines about being a start up and all that. It is just irresponsible to deploy something you know won't hold up. For what it is worth, I think these guys really had no clue. And that is because we, as an industry, did not make it known to them what they were up against.

How do we fix this?
4 comments
Gravatar for Sean hull

Sean hull Says:





Great post Brian, and many important points. In a way we might say that scalability solutions are commoditized. Nevertheless the art of scalability or even just technical know how in general remains in scarce supply.

I would also argue that firms that are growing quickly are grappling with big people & process problems & rarely have the luxury to build a Taj Mahal.

Gravatar for Luke

Luke Says:

> if you’re tempted to reinvent the wheel ... don't
> I did find this slide funny. Reading these slides for me is like seeing > them reinvent the whole wheel that is scaling. This has all been done > before. Why are they having to learn it the hard way?

Didn't you say on /dev/hell that you wrote a custom proxy for dealnews in... PHP? Really? Was Varnish, HAPRoxy, or 100 other industry proven proxies too hard to learn? Seems like you're reinventing things yourself.

Gravatar for Brian Moon

Brian Moon Says:

@Luke, yes I did say that. I also said that Varnish did not exist, that I wanted to use memcached for the cache data which none do and that I found Squid to be less than usable. HAProxy is a tcp/ip proxy and not a web proxy. It can't do the kind of caching and inspection of the request that you can do with a real proxy. If I was starting from scratch today, I would use Varnish. In fact, I am looking into pushing some things in Varnish currently.

Gravatar for

Says:

Jamie Dornan 'Fifty Shades' Meets 'House of Cards' in New Pic! Talk about two good things in one breath, and we have photographic evidence to prove that it is possible. In a recent picture that has surfaced Jamie Dornan has been spotted showing his favouritism for the Netflix series 'House of Cards' starring Kevin Spacey and Robin Wright. The series, which is in its second season, is already a money spinner and the recipient of Primetime Emmy Awards and Golden Globe Awards. In reference to this political thriller, Jamie Dornan proved that he is a fan by wearing a 'Freddy's BBQ Joint' tshirt and posing with "Fifty Shades..." producer Dana Brunetti. Brunetti is also, incidentally the executive producer of the show "House of Cards". We bet, you didn't know this! The setting of the picture is the trailer of the handsome Irish actor Jamie Dornan, who plays the role of Christian Grey in the big screen adaptation of the novel "Fifty Shades of Grey", on the set of this very movie. One of the producers of this movie, who also happens to be the executive producer of the series "House of Cards", which is in its second season and has been renewed for a third reason, shared this photograph on the social networking site, Facebook. The picture has been taken in Vancouver, Canada, where the cast and the crew and even the producers have come together to wrap up the last leg of the shooting of this cinematic adaptation of the racy novel written by British author E.L. James. When it comes to television shows, the cast of "Fifty Shades..." is no stranger to them. First off, let us consider the curious case of Charlie Hunnam. Charlie Hunnam is probably now better known as the actor who turned down the role of Christian Grey in the film adaptation of the novel of the same name, for his television show. But before he was even considered for the show, he became a household name thanks to his role in the television series, "Sons of Anarchy", and it was his commitment to this series that he suddenly left such a lucrative role. And another piece of trivia is that the next actor who stepped in to play the role of the enigmatic businessman, Christian Grey, has also carved a niche for himself by appearing in successful shows such as "The Fall" and "Once Upon a Time". And now that the shoot of the film is close to wrapping up, Jamie Dornan, the 31 year old actor who plays the role of Christian Grey, will be back in Belfast to shoot the second season of "The Fall" opposite the very beautiful Gillian Anderson. But why are we talking about only the leading men of this venture, when we can say the same about the leading lady of the film, Dakota Johnson, who plays the role of Anastasia Steele. This movie will be her first major role as a leading actress and before she auditioned and was chosen as Ana, she was only seen in blink and you miss roles in films. But if there was something that gave her recognition, it was her role in "Ben and Kate". When it comes to television shows, the cast of "Fifty Shades Movie " is no stranger to them. First off, let us consider the curious case of Charlie Hunnam. Charlie Hunnam is probably now better known as the actor who turned down the role of Christian Grey in the film adaptation of the novel of the same name, for his television show

Add A Comment

Your Name:


Your Email:


Your URL:


Your Comment: