Ramblings of a web guy

Being a Better Manager: Communication

brianlmoon — Mon, 06 May 2013 08:17:43 -0500

As I have worked on being a better manager, I have been trying to determine what our strengths and weakness are as a development team. Communication is once place I think we can do a better job.

The problem we have had was how other departments communicated with the development team. The company had been small and now it is large. When the company was small, anyone could talk to anyone. If Bob knows that Tom works on his revenue report, he could just ask him about it. The problem comes when there are 10 Bob’s and 5 Tom’s and people are switching roles in the development team and in other teams. Bob is working on something else and is still coming to Tom with his problems. Tom has no idea how to help him. But, he is a good employee so he tries. He gets distracted from his project and probably does not help Bob all that much in the end. No one is at fault here really. Tom just wants to get his job done. Bob just wants to help a coworker.

When teams start to grow, communication needs to be directed. It’s not some hard and fast rule and people are not punished for talking outside of the chain. However, it helps that when Tom in marketing has a problem, he knows who to talk to every time. By default that is me. It is then my job to know what development resource to tap to solve it. On the other side, the developers have to feel comfortable with telling people “I don’t know. Will you file a ticket or talk to Brian about that?” People are generally helpful. They want to help. I have tried to let people know it is OK to not help if they can’t help or perhaps they are worried about the scope of the problem. I have found that some people click and natural connections will happen. I have no problem with that. There are some people that I expect to reach out to the person they are ultimately writing code for to get feedback. I also stress to them that if the scope of what the person needs changes, we need to talk about it.

From my position, I have to be ready to listen. I want the communication going through me. If I am a jerk, don’t reply to email, or tell people no all day long, that will probably not help me achieve my goals or help the people that need development resources. I have learned to keep a more open mind. I have learned to not say no, but instead say “I don’t think this is a good idea because of these reasons.” And if I think I can modify the idea to something that is workable, I will offer that as an alternative. I feel like this is working well for me and I have been told by others that it is well received.

One place I can not seem to get right is quarterly managers’ meetings. Everyone takes a turn talking about what is going on in their department. When marketing talks, everyone is really interested. Same with financial and sales. When it comes time for me to talk, everyone seems to glass over. I firmly believe it is the content that is the problem. I am going to try a new tactic as recommended by our CEO. Rather than talk about what we did, talk about why we did it. For example, instead of saying we made a change to our code to do X I should talk about the business reasons we spent time on making that change. Instead of “We rolled a new release of the app.” I tell them “Our new app is catching up with the features of the web site.” I am still working on it. Even that still sounds boring to me. The only good news is that technical operations follows me. It is really hard to talk about the fact that we didn’t go down and that we installed new servers sound interesting to non-geeks.

Test Driven Development Conversion

brianlmoon — Mon, 22 Apr 2013 08:00:00 -0500

The concept of unit testing is not new, however it has only gained popularity in the last 10 years or so. When I was asked about it for the first time, I was unsure where it would fit in to my development life. I dismissed it for a long time. I have begun to come around however. I wanted to share how and why.

I have been a professional software developer since 1996. Neither in school nor in practice in the early days of my career was I exposed to the concept of writing tests to confirm portions of my code worked as expected. Testing was always done via user testing. You or someone else would use the program to ensure it produced the expected result. That is how I learned to do my job. And I am good at my job.

My initial reaction to unit testing was negative. I know professional developers that sit in cubicles all day long and write code without any knowledge of the overall application. They are asked to write a function that takes a certain input and returns a certain output. They neither have nor desire additional knowledge about the use of their code. I find those people to be very unattractive as coworkers. They are often out of touch with technology. When I told someone in this line of work I used a Mac, he asked "Do they still make software for those?" He was dead serious. This was in 2005. My initial feelings about unit testing reminded me of these people. Just sitting in cubes, all day, writing functions that took X and returned Y. They required no knowledge about where X came from or where Y was going. Furthermore, I felt like people that must have to have that kind of structure in place to do their jobs must not be capable of understanding the big picture. If they did understand the big picture, they would not need to have these things. They should somehow KNOW how the application worked.

The language I use most (and have for the last 17 years) is PHP. PHP development has always had a haphazard culture around it. I don’t see that as a bad thing. Lots of people all doing things their way. That is often a criticism. People in other language communities where things are a bit more dictated and regimented find it disturbing. I find it refreshing. It’s more real world than the rest to me. But, that is a different blog post.

PHP didn’t come with any sort of testing framework out of the box. And none of the core developers put out a way to test your PHP code. The first tests of any kind that I was aware of for PHP were those built by the PHP QA team. It was targeted at testing the PHP engine by writing PHP code that used the engine. That made sense to me since there are so many moving parts in there and you have lots of people all making changes. It makes sense to have some sanity checks.

The first time I heard about testing your own PHP code was PHPUnit. My first impression was that it was an imitation of JUnit which is the de facto standard way to test Java. I don’t really care for Java. I don’t care for the language. I don’t care for the culture. I don’t care for how a java daemon behaves on my server. In general, I don’t care for the platform. It is my personal preference. It is not right or wrong. It is just how I feel. So, when I see something being copied from Java into PHP, I retreat. That probably makes me a bad person. It is a personality flaw. The end result is that I ignored PHPUnit no matter who said they were using it or why. And the only thing I ever heard about when people talked about testing PHP applications was using PHPUnit.

So, how does someone like me start advocating test driven development as a useful tool? It snuck up on me. A few things had to happen.

The first thing that happened was that we started growing our team. In 2006, we were back down to a two person development team. We had been as large as 5 during the dotcom boom. But, we slowly dwindled back down to two people doing development and systems administration every day as their full time jobs. When there are only two of you, you kind of have to know how everything works. If you don’t, you can’t keep running. As we started to add developers, I realized it was increasingly harder for other people to grok the whole application they way I did. And at first, I thought that was just a matter of time. After a year or so they would get it all. So, we hired more people. We are now at six developers, plus myself and one new one starting in two weeks. On top of that we are actively looking to hire two more as soon as possible. Along the way there have been some others come and go as well. Now, to people in the VC bubbles, this may sound boring. I know places where the team grows 20x in 6 months. I can’t imagine. Our application and code base is huge though. Experienced engineers that are now on the team have told me they have never worked with a code base this large. In the last couple of rounds of hiring, I was asked about testing. My standard answer was that if you wanted to write tests for your code and could develop a sane strategy for deploying those tests, I had no problem with it. As a team however, there was no official policy or systems in place for testing. One of the new hires pointed me to a something that peaked my interest. It was doctest for Python. Essentially, you put a bit of Python code in the doc block for the code you are writing. doctest will scan files for these tests and run them. Partly out of curiosity and partly to hopefully satisfy the guys wanting some way to test their code, I wrote a version of doctest for PHP. To prove it worked, I added a test block to all the methods of an array manipulation class I had worked on recently. I was still not convinced that this was a good use of my time however.

The second thing that happened was I met up with Chris Hartjes (@grmpyprogrammer). Chris is an avid supporter of test driven development. I had heard the term test driven development before meeting Chris, but my testing bias had shielded me from really understanding the concept. He and I talked and later I was interviewed by Chris and Ed Finkler (@funkatron) on the /dev/hell podcast. On that podcast we talked about testing as well as other things. Looking back on those two conversations, I now know what Chris did that most changed how I looked at people who are adamant believers in dynamic unit testing. Chris didn’t make it a personal thing. He acknowledged my perspective. We do have a code base and business that has been working for 10+ years. We do have extensive monitoring and graphing in place that tells us when things are going wrong. He told me that if that was working for me, more power to me. Wait, wasn’t he supposed to tell me how I was doing it wrong? Wasn’t he supposed to tell me I was a cowboy and how unprofessional I am? He did none of those things. So, you know what happened? I kept listening to him.

The third thing that happened was also because of Chris (just so you know now, most of this blog post is indirect or direct praising of Chris). Once I started listening, there was something else I was not hearing from him. I was not hearing how I should be using PHPUnit or any particular product for that matter. He only said that code should be written to satisfy a test. He didn’t seem to care how it was tested. He only cared that it was tested. This was again a breathe of fresh air. In this business where every group likes to tell the other group how wrong they are doing things, having someone that didn’t seem to care how you did it, as long as you did it was awesome. So, again, I kept listening to him.

The fourth thing that happened was when Chris made me realize that I already was a test driven developer. I don't think he realized that he did it. He likely does not remember the conversation. And I probably remember it poorly. For all I know, there were beers and/or whiskey involved. It went something like “So, how do you test your code?” said Chris. I replied “Well, if the web page loads, it worked.” To which he asked “So, you don’t test changes as you make them?” I said “Well, yeah, I write some test code and run it till it works while I am making changes. I just don’t save that code. Once its working I don’t need it.” I don’t think it hit me right then. I think it took days or weeks to sink in. Mother F--ker, I already write code in a test driven manner a lot of the time. I have a test.php script in my home directory on my development server. I reuse that thing all the time to test the code I am working on. Instead of assertions, lots of the time I will just print_r() the output and assert with my eyes. But, the end result is the same. So, why not take that extra step? I had the doctest stuff in place. I could just add my test code there and run the test over and over until it worked like I wanted it to. And what do you know, I kind of liked it.

And what I think was the last straw was when I was merging code onto staging and a test failed. Oh my gosh! I made a change to our code base and my all knowing, all seeing eye did not realize that my changes were going to break something. How is this possible? How did I not see this coming? I know everything in the application don’t I? I architected the whole thing. The reality is, this likely (and by likely I mean, it did) happened before. However, sometime very soon, either a monitor or a human would have noticed and a bug report would have been filed. I would have then fixed it. I would have likely justified the bug as the cost of progress. Now I have a way to help prevent these small bugs from rolling out in the first place. And now that I have this tool, there is no excuse. The only reason this should happen now is that we don’t have enough tests.

We are working on improving our test coverage. I am not to the point yet where I require tests for all code. Perhaps we will get to that point. I don’t know. My team is aware of the tool now. And they are aware that I use it and think it is wise to use it. I told them recently that I am not telling them to use it. But, if they roll a bug, and there was not a test, the fix should probably be written using test driven development.

I have intentionally not talked about what we use other than the a fore mentioned doctest. I hope the message that you take away from what I have learned is that you should be using SOMETHING to test code. That is more important than how you test your code. I also have not talked about how you write code that can be tested. That is another hurdle I have had to (still am) get over. I thought a lot of our code was not testable. Because you know, we are solving problems that no one ever in the history of the world has had to solve, like retrieving a web page, hard stuff. There are better resources to learn how to write code to be more testable, especially for PHP. I highly recommend everything Chris Hartjes has ever said or written about the subject. You can find his thoughts on the topic at http://grumpy-learning.com/, http://grumpy-phpunit.com/, and https://leanpub.com/grumpy-testing.

We will be improving doctest.php. We have lots of ideas. It just fits better for us. If PHPUnit works for you, great, use it. Just search for `unit testing PHP`and find something that works for you. In addition, we are starting to work with Selenium for interactive testing of our site and some commercial products that are really good at testing APIs. I am getting a hard time from some guys at work. I have been critical of testing in the past. I just needed to understand that I already worked this way or thought this way.

College Basketball in 2013

brianlmoon — Tue, 09 Apr 2013 11:51:04 -0500

I am a huge sports fan. I particularly love college sports. We don't have any pro sports teams in Alabama. So, we take our college sports very seriously. Like many sports fans I have watched the NCAA tournament this year. I have to say, if the current trend continues, I don't think I will in the future. I don't like the product.

In particular, I pretty much hate the way Louisville plays basketball. Yes, the won. Kudos to them. I don't blame them really. No, the real problem is the lack of offensive foul calls. Players push and shove each other. Guys going up for a lay up from under the back board are free to bully their way up to the rim including jumping backwards into a well placed defender. The only thing that matters to the NCAA is scoring, at any cost.

Luke Hancock stealing the game from Witchita State by fouling although it was called a jump ball. / Jaime Green/MCT via freep.com

So, getting back to Louisville, Rick Petino and his players have figured out just how far they can push these limits. They are almost playing hockey. But, in basketball, there is no penalty box. You get 5 fouls. If you have a deep bench like Petino, you just plug in another guy and they keep fouling. On the other side you have Luke Hancock, named MVP of the Final Four (geez, talk about bad role models) throwing a pump fake and then leaning into a player in a direction that is not toward the basket or any kind of natural shooting motion, just to draw the foul. Those should be no calls or offensive fouls, IMO.

I coach youth basketball. I teach my kids to not foul. I teach my kids to not touch each other when at all possible. On defense, they need to be in position and ready to move their feet. On offense, don't try and run over people. If you have to foul to stop someone, then we need a better game plan. Or maybe they are just better than we are. I have on occasion praised a kid for being aggressive which led to a foul. These are eight year olds. And some of them are still in a shell about being athletic. But, I never say "foul him". And I never have my kid try and draw a foul the way Luke Hancock does. That is just dirty basketball. I will tell my kids to drive the lane and go for the goal. And "if" you get fouled it's OK. But, never go looking for the foul.

Contrastingly, I watched some of the New York Knicks vs. Oklahoma City Thunder game Sunday. These guys hardly ever touch each other. They play within, what I believe, are the real rules of basketball. The game is a little fast and there is a lot of one on one play that can be tedious. But, it was more fun to watch than NCAA basketball. Maybe all the good basketball players go to the NBA and we are left with scrubs in the NCAA. They are too small (size wise) or to slow to play corner back in football so they end up playing basketball as an "athlete". I hope something is done about this. The game is just getting trashy.

The Web We Lost

brianlmoon — Tue, 19 Mar 2013 19:34:36 -0500

I was reading Chris Shiflett's blog and he mentioned a blog post about the web the way it used to be before Facebook and Twitter. I almost tweeted it. Then I thought, nah, a good old fashion blog post that linked to it was way better.

The Web We Lost - Anil Dash

The tech industry and its press have treated the rise of billion-scale social networks and ubiquitous smartphone apps as an unadulterated win for regular people, a triumph of usability and empowerment. They seldom talk about what we've lost along the way in this transition, and I find that younger folks may not even know how the web used to be.

It is a good read for all us "old timers". He ironically uses Facebook for comments. One really good one that I can't actually link to afaik because... Facebook said:

No sarcasm here.. I legitimately miss the "web-rings" of old. With niche interests, it was a great way to find like minded sites, and I found many of my still bookmarked favorites with those old crappy left and right arrows :)

Yeah, you know, Web Rings weren't that bad. I had sites on a couple of those things. I think they are still around. And I think you could still pull them off and not worry about Google juice and all that. 302 redirects don't pass that crap. Maybe I should write up some quick javascript that indexs the Planet PHP blogs and adds a Planet PHP web ring to your page. Hmmmm.

Reliable Delivery

brianlmoon — Mon, 11 Mar 2013 08:00:00 -0500

There is plenty to read about continuous delivery in terms of rolling out code. In my journey to be a better leader and manager, I have realized there is something we are doing badly. While we are continuously integrating, testing, and deploying code to production, we are not reliably delivering a product for our client. I use client loosely here. The client for the development team at dealnews is the company itself. With every project however, there are people interested in its progress. We were (still are some) underserving those people. Those interested often have no clue of when something will be done. Sometimes things would are done and they do not know it. So, how do we solve this problem?

There was a time when we were a team of two people. If some server issue popped up, it totally derailed whatever project was being worked on. So, we grew accustomed to missing deadlines out of necessity. Now that we are a real team, that excuse is no longer valid.

Part of the problem also lies in the team’s (i.e. me) OSS roots. I really got started doing web development by writing Phorum. In fact, my first job when I was hired full time for, then, dealmac.com was to update the Phorum software to scale better. Banner ads were at an all time high in the late 90s. We made a lot of money off those page views. In OSS, the answer to “When will X be done?” is often “it’s done when it’s done”. Blizzard Entertainment, creators of World of Warcraft, have been quoted as saying that about their products.

Telling someone it will be done when it is done sounds really cool. I feel like a bad ass. It’s art! It’s not about a timeline. I can’t be bothered with your pesky expectations. Except, that is bullshit. In reality, there are people depending on me and my team to get work done. So, that was one of the first things I wanted to change. We have gotten better. Here are some things I have learned.

Think before speaking. If I am in a meeting and talking about a new task or project, I try not to throw out a time frame for completion. I tell them I will get back to them. I try and tell them when I will get back with them. I then gather anyone on the team that needs to have input and evaluate the changes. Once I have a solid answer, I report back to the other department. Very often I worry people will be mad when I say “two weeks” so I say “one week” and hope for a miracle. But if it is going to be two weeks, I need to tell them that. That may be too late. Or it may be not worth it to them to take that much of our time. Of course not all tasks need this kind of time commitment to deciding a time frame. Deciding which do and which don’t is tough sometimes.

I wanted to start communicating deadlines to our developers. We had never had a ticketing system that supported due dates. We had the “done when done” philosophy. I was really worried about adding them. I didn’t want people to feel like they were being micro-managed. After a couple of months, everyone is much happier. Turns out developers really like knowing when things are expected to be done. It also helps to prioritize different tasks. If a developer has 5 things assigned to them, they can look at the due dates to decide which is more important. Because, you know, they are all marked “highest” priority. Developers have the freedom to speak up and say “there is no way I can finish this by that date”. It’s possible I completely misjudged the scale of the change. It is also possible I wrote a horrible ticket and the developer is confused by my 2AM stream of consciousness.

There is another hurdle for me. I have gotten better at managing expectations of other departments and helping developers know what is expected of them. It is better. It is not perfect. It may never be. I am still struggling with doing the same with my own development tasks. I catch myself thinking “Well, that is just how it is when I have to manage and develop.” But that is a total excuse and a cop out. I have to learn to do that better. Managing my own time may be the toughest of all. If and when I figure something out, I will write about it.

Becoming a Better Manager

brianlmoon — Sat, 09 Mar 2013 20:02:47 -0600

I have typically blogged on this site about things I have learned in the web
application world that may help others. In the last year or so, I have been
learning a lot of new things. Most of them are not technical in nature however.
You see, I have moved into a role of being a manager. I am a developing manager.
I still write code. And a lot of my time is dedicated to management as well. I
think this has caused me to stop blogging as much. My mind didn't see these
topics as interesting to what I perceive to be the audience of my blog as things
I have blogged about in the past. The problem is, I miss blogging.

So, going forward, there may be some non-technical things on this blog. My hope
is that someone out there finds them as useful as some of my more technical
blogs posts have been.

Developers and Entropy

brianlmoon — Fri, 13 Jul 2012 12:07:00 -0500

This is a selfish blog post. I read a great blog post titled "Why You Need To Hire Great Developers" but I could not find it in my browser history or chat history. It talks about entropy creators versus entropy reducers and how bad we are at knowing which one someone is during the hiring process. I wanted to mention it here so my followers could read it and so I could find it again when I was looking for it.

Lock Wait Timeout Errors or Leave Your Data on the Server

brianlmoon — Wed, 27 Jun 2012 01:12:34 -0500

If you use MySQL with InnoDB (most everyone) then you will likely see this error at some point. There is some confusion sometimes about what this means. Let me try and explain it.

Let's say we have a connection called A to the database. Connection A tries to update a row. But, it receives a lock wait timeout error. That does not mean that connection A did anything wrong. It means that another connection, call it B, is also updating a row that connection A wants to update. But, connection B has an open transaction that has not been committed yet. So, MySQL won't let you update that row from connection A. Make sense?

The first mistake people may make is looking at the code that throws the error to find a solution. It is hardly ever the code that throws the error that is the problem. In our case, it was code that was doing a simple insert into a table. I had a look at our processing logs around the time that the errors were thrown and I found a job that was running during that time. I then looked for code in that job that updates the table that was locked. This was where the problem lied.

So, why does this happen? Well, there can be very legitimate reasons. There can also be very careless reasons. The genesis of this blog post was some code that appeared to be legitimate at first, but upon further inspection was careless. This is basically what the code did.

Start Transaction on database1
Clear out some old data from the table
Select a bunch of data from database2.table
Loop in PHP, updating each row in its own query to update one column
Select a bunch of data from database2.other_table
Loop in PHP, updating each row in its own query to update another column
Commit database1

This code ran in about 20 minutes on the data set we had. It kept a transaction open the whole time. It appeared legit at first because you can't join the data as there are sums and counts going on that have a one to many relationship which would cause some duplication of the sums and counts. It also looks legit because you are having to pull data from one database into another. However, there is a solution for this. We need to stop pulling all this data into PHP land and let it stay on the server where it lives. So, I changed it to this.

Create temp table on database2 to hold mydata
Select data from database2.table into my temp table
Select data from database2.other_table into my temp table
Move my temp table using extended inserts via PHP from database2 to database1
Start Transaction on database1
Clear out some old data from the table
Do a multi-table bulk update of my real table using the temp table
Commit database1

This runs in 3 minutes and only requires a 90 second transaction lock. Our lock wait timeout on this server is 50 seconds though. However, we have a 3 time retry rule for any lock wait timeout in our DB code. So, this should allow for our current workload to be processed without any data loss.

So, why did this help so much? We are not moving data from MySQL to PHP over and over. This applies to any language, not just PHP. The extended inserts for moving the temp table from one db to another really help. That is the fastest part of the whole thing. It moves about 2 million records from one to the other in about 1.5 seconds.

So, if you see a lock wait timeout, don't think you should sleep longer between retries. And don't dissect the code that is throwing the error. You have to dig in and find what else is running when it happens. Good luck.

Bonus: If you have memory issues in your application code, these techniques can help with those too.

Scaling 101 - We are Failing the Next Generation

brianlmoon — Tue, 26 Jun 2012 00:01:36 -0500

The other day Twitter was down and I had no place to comment on Twitter being down. This got us to talking about scaling at work. I was reminded of the recent slides posted from Instagram about their scaling journey. They are great slides. There is only one problem I have with them. They are just the same slides that you would find from 2000 about scaling.

I have to say, I like Instagram. My daughter has something like 1,000 followers on Instagram. And good for them for being bought by Facebook for a bajillion dollars. This is not a dig on them really. This is a dig on our industry. Why did Instagram have to learn the hard way how to scale their app? I want to point out some of their issues and discuss why its silly they had to learn this the hard way.

Single machine somewhere in LA

Why would anyone deploy an app to the app store when the backend is all on one server in this day and age? I am not big poponent of the cloud, but that has to be better than a single server in a rack somewhere. And L.A.? Go for Dallas or somewhere geographically neutral.

favicon.ico

So, this is one of the biggest mistakes I see in all of web application developement. People use Apache or Nginx or whatever and have a mod_rewrite command that sends ALL requsets into their appliction stack. They do this because they are lazy. They want to write whatever code they want later and just have the request picked up without any work. At dealnews, we don't do that. We have controllers. But, we specify in our Apache config what paths are routed to those controllers. The most general we have is:

RewriteCond %{REQUEST_URI} (\/|.html|.php)$
RewriteRule ^.+$ /torpedo.php/$1 [L]

So, if the request ends with / or .html or .php, send it to the controller. This is the controller our proxy servers use. Any other requests like robots.txt, images, etc. are all served off disk by Apache. No sense having a PHP process handle that. It's crazy. This is not Instagram's fault. They likely followed some examples they found that other developers before them put on the internet. Why are we doing this to people? Is it some rite of passage? I suspect that the problem is actually that 90% of the web does not require any scaling. Only 10% of the sites and services out there have to actually worry about load. So, this never comes up.

So, the next part of their slides basically glorify the scaling process. This is another problem with people in our field. We thrive on the chaos. These are our glory days. Let's face it, most geeks never won the high school football championship. The days when we are faced with a huge scaling challenge are our glory days. I know I have had that problem. And I played sports as a youth. But, nothing is better than my 2006 war story about a Yahoo front page link. Man, I rocked that shit. But, you know what. The fact that I had to struggle through that means I did not do my job. We should have never been facing that issue. It should have just all worked. That is what it does now. It just works. We don't even know when we get a spike now. It just works. The last thing I want in my life now is an unexpected outage that I have to RAGE on to get the site up again. That leaves me feeling like a failure.

Then they realize they are out of the depths. They need to do things they never thought they would need to do. Why not? Why did they not know they would need to do these things? Unexpected growth? Maybe. Why is this not common knowledge though? With all the talk of the cloud being awesome for scaling, why was this not a button on the AWS dashboard that said [SCALE] that you just push? That is what the cloud does right?

In the end, Instagram learned, the hard way again, that you have to build your own architecture to solve your problem. I learned it the hard way. LiveJournal learned it the hard way. Facebook, Twitter, etc. etc. They have all learned the hard way that there is no single solution for massive scale. You have to be prepared to build the architecture that solves your problem in a way that you can manage. But, there are basic building blocks of all scaling that need to be in place. You should never, ever, ever start with an application on a single server that reads and writes directly to the database with no cache in place. Couch, Mongo, blah blah blah. Whatever. They all need cache in front of them to scale. Just build it in from the start.

Instagram was storing images. Why were they surprised when they ran out of room for the images? I just can't fathom that. Its images. They don't compress. You have to put them somewhere. This has to be an education issue. LiveJournal solved this in 2003 with MogileFS and Gearman. Why did they not build their arch on top of that to start with? Poor education, that is why.

One thing they bring up is one that has me bugged is monitoring. There is no good solution for this. Everyone ends up rolling their own solutions using a few different tools. The tools are often the same, but the metrics, and how they are reported and monitored are all different. I think there is a clear need in the industry for some standards in this area.

if you’re tempted to reinvent the wheel ... don't

I did find this slide funny. Reading these slides for me is like seeing them reinvent the whole wheel that is scaling. This has all been done before. Why are they having to learn it the hard way?

don’t over-optimize or expect to know ahead of time how site will scale

I take exception to this slide however. You should have some idea how to scale your app before you deploy it. It is irresponsible and cowboy to deploy and think "oh, we will fix it later". That is true no matter what you are doing. Don't give me the lines about being a start up and all that. It is just irresponsible to deploy something you know won't hold up. For what it is worth, I think these guys really had no clue. And that is because we, as an industry, did not make it known to them what they were up against.

How do we fix this?

Stop comparing stuff you don't understand

brianlmoon — Mon, 25 Jun 2012 23:09:28 -0500

I normally don't do this. When I see someone write a blog post I don't agree with, I often just dismiss it and go on. But, this particular one caught my attention. It was titled PHP vs Node.js: Yet Another Versus. The summary was:

Node.js = PHP + Apache + Memcached + Gearman - overhead

What the f**k? Are you kidding me? Clearly this person has NEVER used memcached or Gearman in a production environment that had any actual load.

Back in the day, when URLs and filesystems had a 1:1 mapping, it made perfect sense to have a web server separate from the language it is running. But, nowadays, any PHP app with attractive URLs running behind the Apache web server is going to need a .htaccess file, which tells the server a regular expression to check before serving up a file. Sound complex and awkward with unnecessary overhead? That’s because it is.

Node has a web server built in. Some people call this a bad thing, I call those people crazy. Having the server built in means that you don’t have the awkward .htaccess config thing going on. Every request is understood to go through the same process, without having to hunt through the filesystem and figure out which script to run.

He believes that PHP inside Apache requires a .htaccess file. Welcome to 1999. I have not used a .htaccess file since then. Anyone that cares at all about scaling Apache would disable .htaccess files. And as for running regexs, how does he propose you decide your code path in the controller of his Node.js code? Something somewhere has to decide what code is going to answer a given request. mod_rewrite is wire speed fast and compiled in C. Javascript nor PHP code could ever beat that.

The official website is quite ugly and outdated.

Really? You choose your tools based on that? I don't know what to say.

Since a PHP process starts, does some boilerplate work, performs the taks the user actually wants, and then dies, data is not persistent in memory. You can keep this data persistent using third party tools like Memcache or traditional database, but then there is the overhead of communicating with those external processes.

He clearly has no understanding of how memcached is supposed to be used. You don't put things in Memcached so you can use it on the next request on this server. You put things in memcached so you can use it in any request on any server in your server pool. If you just have one web server, you can write Perl CGI scripts. Performance and up time is not important to you. If you want to share things across requests in PHP, APC and XCache fill the need very well.

The number one bottleneck with web apps is not the time it takes to calculate CPU hungry operations, but rather network I/O. If you need to respond to a client request after making a database call and sending an email, you can perform the two actions and respond when both are complete.

This sums up the mythical magic of Node.js. People think just because your code is not "running" that somehow the server is not doing anything. The process does not get to go do other shit. No, that would be multi-threaded. And Node.js is not multi-threaded. It is single threaded. That means the process can only be doing one thing at a time. If you are waiting on a DB call, you are waiting. I don't care what world you think you live in. You are waiting on that DB call. How your code is structured is irrelevant to how computers actually work. The event driven nature that is Node.js is much like OOP. You are abstracting yourself from how computers really work. The further you get from that, the less you will be able to control the computer.

Node.js is a very new, unstable, untested platform. If you are going to be building a large corporate scale app with a long lifetime, Node.js is not a good solution. The Node API’s are changing almost daily, and large/longterm apps will need to be rewritten often.

So, if I plan on making a living, don't use Node.js. Got it. We finally agree on something.

Being so new, it doesn’t have a lot of baggage leftover from days of old. Having a server built in, the stack is a lot simpler, there are less points of failure, and there is more control over what you can do with HTTP responses (ever try overwriting the name of the web server using PHP?).

No, why would you? It is not the web server. Apache or nginx would be your web server.

Are you building some sort of daemon? Use Node.
Are you making a content website? Use PHP.
Do you want to share data between visitors? Use Node.
Are you a beginner looking to make a quick website? Use PHP.
Are you going to run a bunch of code in parallel? Use Node.
Are you writing software for clients to run on shared hosts? Use PHP.
Do you want push events from the server to the client using websockets? Use Node.
Does your team already know PHP? Use PHP.
Does your team already know frontend JavaScript? Node would be easier to learn.
Are you building a command line script? Both work.

Yes, of course you would not build a daemon in PHP. Do you plan to share that same data across servers? He already told us Node is not multi-threaded, so how can it run code in parallel? Websockets have a ton of their own pain to deal with that is not even related to Node vs. PHP. Things like proxies. If I was building a command line tool and wanted to use Javascript, I would just use V8.

Listen, I write code in PHP and JavaScript all day. I also use some Ruby, Lua and even dabble in C. I am not a language snob. Use what works for you. I do however take exception when people write about things they clearly have no idea about. He claims to have written a lot of PHP. He clearly has never deployed a lot of PHP in environments that matter. If you are building small sites that don't have a lot of traffic, you can use anything. If you are building massive sites that have to scale, any technology is going to require a full understanding of what it takes to scale it out. I leave you with this wisdom that I am reminded of by his blog post.

PHP Coding Standards

brianlmoon — Fri, 25 May 2012 19:27:56 -0500

Update: Matthew Weier O'Phinney, one of the core members of the group, has cleared up the naming history in the comments.

During the /dev/hell podcast at Tek12, someone asked the guys their opinion about PSR. I did not know what PSR was by that name. A quick search lead me to the Google Group named PHP Standards Working Group. I had vaguely remembered a consortium of frameworks, libraries and applications that were organizing to attempt to make their projects cooperate better. But, this did not sound like the same project. Another search and I found the PHP Framework Interoperability Group on Github. A bit more searching led me to a post where apparently the PHP FIG changed their name at some point citing people not knowing what FIG meant. But, this is not a history post. The group had done some work on setting a standard for auto loaders in PHP. This is a very good thing and much needed. That is a real thing that impacts real developers.

The person asking the question had asked about PSR1 and PSR2. These are the first two standards proposals in the group and they deal with coding standards. There were mixed feelings in the room about the proposals. I asked (being me, probably with very little tact) why in 2012 were a group of really smart people still discussing coding standards such as tabs vs. spaces. Because this is what immediately came to mind for me.

Source: http://xkcd.com/927/

There are already coding standards for PHP and any other language out there. Why does anyone need to make a new one? For Phorum we chose the PEAR standard (ok, with 2 minor modifications). On top of that, every one of the projects in this group already have coding standards. Why not just pick one of those? Are 10 projects that currently have their own standards going to actually all change to something else? I highly doubt it. My guess is that, at best, they will all end up with a modified version of the groups standards.

This reminds me a lot of Open Source licenses. There are tons of these things. And in the end, most (GPL has its issues I know) of the open source licenses represent the same idea. I suppose you could say that most of all of them fall into GPL like or BSD like. Anyhow, I quit worrying about having my own license years ago. I now just use a BSD style license that you can generate with several online BSD license generators.

When I voiced my concern about what is, in my opinion, a waste of very smart people's time, my good friend Cal Evans (He has bled in my car. So, I think he is my friend. And I hope he feels the same.) said that I was misunderstanding the point of the group. It was a group of projects that were collaborating to try and use similar standards and practices to make the PHP OSS community better. And that is exactly what I thought PHP FIG was. However, the group name is now "PHP Standards Working Group". That reminds me of the W3C HTML Working Group. And in my mind that means a group that is deciding the future of a technology. In addition the proposal being discussed is titled "PSR-1, a standard coding convention for PHP". If you pair that with the name of the group, it sounds very authoritative. And I don't think that is by accident. If I was heading up such an effort, I would hope that every PHP developer on the planet would follow it too. If you saw Terry Chay's keynote at the PHP Community Conference last year, he talked about frameworks and platforms. He pointed out that the reason people like Facebook were sharing their data center technology was in hopes that people would start using it and it would become common. Thus meaning the equipment they are custom building would be cheaper and people they hire would already be familiar with it. But, if the point of the group is *only* cooperation between lage OSS PHP projects, I wish they would pick a name that is more indicative of that. As it stands, when I landed on the page, my immediate assumption was that this groups intention was to dictate to the rest of the PHP world how to write their PHP code.

In the end, cooperation is good. And if these guys want to cooperate I say more power to them. I just hope they get into really good things soon. Like, can we talk about a maximum number of files, functions or classes used for any one single page execution? *That* would be valuable to the PHP community. I can deal with funny formatting. I can't deal with poorly performing code that his dragged down in abstracton and extension. Or how about things like *never* running queries inside loops that are reading results from another query. That would be a great thing to make examples of and show people the best practices. Tabs vs. spaces? That should have been solved 10 years ago. When in doubt, PHP code should do what the PHP core does. This is PHP we are talking about. Would it not make sense to have the people who write PHP writing code that is somewhat similar in style to those that make PHP? C and PHP syntax are very, very similar. So, why don't we all just refer to the PHP CODING_STANDARDS file when in doubt and not even worry with the little stuff that does not affect performance?

So, as of a few minutes ago, I have joined the group. If for no other reason, just to see what is discussed. Perhaps I should follow the advice I give people when they ask for features in my projects and do something about my issues and worries.

Living in the Prove It Culture

brianlmoon — Tue, 06 Mar 2012 22:45:18 -0600

Engineering cultures differ from shop to shop. I have been in the same culture for 13 years so I am not an expert on what all the different types are. Before that I was living in Dilbert world. The culture there was really weird. The ideas were never yours. It was always some need some way off person had. A DBA, a UI "expert" and some product manager would dictate what code you wrote. Creativity was stifled and met with resistance.

I then moved to the early (1998) days of the web. It was a start up environment. In the beginning there were just two of us writing code. So, we thought everything we did was awesome. Then we added some more guys. Lucky for us we mostly hired well. The good hires where type A personalities that had skills we didn't have. They challenged us and we challenged them. On top of that, we had a CEO who had been a computer hacker in his teens. So, he had just enough knowledge to challenge us as well. Over the years we kept hiring more and more people. We always asked in the interview if the person could take criticism and if they felt comfortable defending their ideas. We decided to always have a white board session. We would ask them questions and have them work it out on a white board or talk it out with us in a group setting. The point of this was not to see if they always knew the answer. The point was to see how they worked in that setting. Looking back, the hires that did not work out also did not excel in that phase of the interview. The ones that have worked out always questioned our methods in the interview. They did not belittle our methods or dismiss them. They just asked questions. They would ask if we had tried this or that. Even if we could quickly explain why our method was right for us, they still questioned it. They challenged us.

When dealing with people outside the engineering team, we subconsciously applied these same tactics. The philosophy came to be that if you came to us with an idea, you had to throw it up on the proverbial wall. We would then try to knock it down. If it stuck, it was probably a good idea. Some people could handle this and some could not. The ones that could not handle that did not always get their ideas pushed through. It may not mean they were bad ideas. And that is maybe the down side of this culture. But, it has worked pretty well for us.

We apply this to technology too. My first experience on Linux was with RedHat. The mail agent I cut my teeth on was qmail. I used djbdns. When Daniel Beckham, our now director of operations, came on, he had used sendmail and bind. He immediately challenged qmail. I went through some of the reasons I prefered it. He took more shots. In the end, he agreed that qmail was better than sendmail. However, his first DNS setup for us was bind. It took a few more years of bind hell for him to come around to djbdns.

When RedHat splintered into RedHat Enterprise and Fedora, we tried out Fedora on one server. We found it to be horribly unstable. It got the axe. We looked around for other distros. We found a not very well known distro that was known as the ricer distro of the Linux world called Gentoo. We installed it on one server to see what it was all about. I don't remember now whose idea it was. Probably not mine. We eventually found it to be the perfect distro for us. It let us compile our core tools like Apache, PHP and MySQL while at the same time using a package system. We never trusted RPMs for those things on RedHat. Sure, bringing a server online took longer but it was so worth it. Eventually we bought in and it is now the only distro in use here.

We have done this over and over and over. From the fact that we all use Macs now thanks to Daniel and his willingness to try it out at our CEO's prodding to things like memcached, Gearman, etc. We even keep evaluating the tools we already have. When we decided to write our own proxy we discounted everything we knew and evaluated all the options. In the end, Apache was known and good at handling web requests and PHP could do all we needed in a timely, sane manner. But, we looked at and tested everything we could think of. Apache/PHP had to prove itself again.

Now, you might think that a culture of skepticism like this would lead to new employees having a hard time getting any traction. Quite the opposite. Because we hire people that fit the culture, they can have a near immediate impact. We have a problem I want solved and a developer that has been here less than a year suggested that Hadoop may be a solution, but was not sure we would use it. I recently sent this in an email to the whole team in response to that.

The only thing that is *never* on the table is using a Windows server. If you can get me unique visitors for an arbitrary date range in milliseconds and it require Hadoop, go for it.

You see, we don't currently use Hadoop here. But, if that is what it takes to solve my problem and you can prove it and it will work, we will use it.

Recently we had a newish team member suggest we use a SAN for our development servers to use as a data store. Specifically he suggested we could use it to house our MySQL data for our development servers. We told him he was insane. SANs are magical boxes of pain. He kept pushing. He got Dell to come in and give us a test unit. Turns out it is amazing. We can have a hot copy of our production database on our dev slices in about 3 minutes. A full, complete copy of our production database in 3 minutes. Do you know how amazing that is? Had we not had the culture we do and had not hired the right person that was smart enough to pull it off and confident enough to fight for the solution, we would not have that. He has been here less than a year and has had a huge impact to our productivity. There is talk of using this in production too. I am still in the "prove it" mode on this. We will see.

I know you will ask how our dev db works, here you go:

1. Replicate production over VPN to home office

2. Write MySQL data on SAN

3. Stop replication, flush tables, snapshot FS

4. Copy snapshot to a new location

5. On second dev server, umount SAN, mount new snapshot

6. Restart MySQL all around

7. Talk in dev chat how bad ass that is

We had a similar thing happen with our phone system. We had hired a web developer that previously worked for a company that created custom Asterisk solutions. When our propietary PBX died, he stepped up and proved that Asterisk would work for us. Not a job for a web developer. But he was confident he could make it work. It now supports 3 offices and several home bound users world wide. He also had only been here a short time when that happened.

Perhaps it sounds like a contradiction. It may sound like we just hop on any bandwagon technology out there. But no. We still use MySQL. We are still on 5.0 in fact. It works. We are evaluating Percona 5.5 now. We tried MySQL 5.1. We found no advantage and the Gentoo package maintainer found it to be buggy. So, we did not switch. We still use Apache. It works. Damn well. We do use Apache with the worker MPM with PHP which is supposedly bad. But, it works great for us. But, we had to prove it would work. We ran a single node with worker for months before trusting it. Gearman was begrudgingly accepted. The idea of daemonized PHP code was not a comforting one. But once you write a worker and use it, you feel like a god. And then you see the power. Next thing you know, it is a core, mission critical part of your infrastructure. That is how it is with us now. In fact, Gearman has went from untrusted to the go to tech. When someone proposes a solution that does not involve Gearman, someone will ask if part of the problem can be solved using Gearman and not whatever idea they have. There is then a discussion about why it is or is not a good fit. Likewise, if you want to a build a daemon to listen on a port and answer requests, the question is "Why can't you just use Apache and a web service?" And it is a valid question. If you can solve your problem with a web service on already proven tech, why build something new?

This culture is not new. We are not unique. But, in a world of "brogramming" where "engineers" rage on code that is awesome before it is even working and people are said to be "killing it" all the time, I am glad I live in a world where I have to prove myself everyday. I am the most senior engineer on the team. And even still I get shot down. I often pitch an idea in dev chat and someone will shoot it down or point out an obvious flaw. Anyone, and I mean anyone, on the team can question my code, ideas or decisions and I will listen to them and consider their opinion. Heck, people outside the team can question me too. And regularly do. And that is fine. I don't mind the questions. I once wrote here that I like to be made to feel dumb. It is how I get smarter. I have met people that thought they were smarter than everyone else. They were annoying. I have interviewed them. It is hard to even get through those interviews.

Is it for everyone? Probably not. It works for us. And it has gotten us this far. You can't get comfortable though. If you do foster this type of culture, there is a risk of getting comfortable. If you start thinking you have solved all the hard problems, you will look up one day and realize that you are suffering. Keep pushing forward and questioning your past decisions. But before you adopt the latest and greatest new idea, prove that the decisions your team makes are the right ones at every step. Sometimes that will take a five minute discussion and sometimes it will take a month of testing. And other times, everyone in the room will look at something and think "Wow that is so obvious how did we not see it?" When it works, it is an awesome world to live in.

/dev/hell Podcast Episode #5

brianlmoon — Fri, 03 Feb 2012 12:12:10 -0600

I was privileged to be invited to be a part of the /dev/hell podcast this week. Thanks to Chris and Ed for having me on. Check it out. And subscribe to their podcast.

Errors when adding/subtracing dates using seconds

brianlmoon — Mon, 16 Jan 2012 15:55:49 -0600

This just came up today again for me. I have said it before, but even I get lazy and forget. When doing math with dates such as adding days it is really quick to think this works:




$date = "2011-11-01";



// add 15 days



$new_date = date("Y-m-d", strtotime($date) + (86400 * 15));



// $new_date should be 2011-11-16 right?



echo $new_date;



?>

This yields `2011-11-15`. The problem with this is that it assume that there are only 86400 seconds in every day. There are in fact not. On days when the clocks change for daylight savings time, there are either 1 hour more than that or 1 hour less than that. In addition, there are also leap seconds put into our time system to keep us in line with the sun. There is one this year, 2012, on June 30th in fact. Since they don't happen with the regularity that daylight savings time does, it may be easy to forget those. Luckily, for this problem, the solution is the same. You have two choices. And the solution you choose depends on the particular problem you have. For the simple problem above, you can simply let strtotime take care of it for you.

$new_date = date("Y-m-d", strtotime($date." +15 days"));

strtotime() is the most awesome date/time related function in all of computer programming. I have written about it before. It handles all those nasty weird seconds issues. But, if you are not solving a problem this simple or you are reading this and need help with another language, there is another solution. You do all your date math at noon. Simply only run code that does date math during lunch time. No, just joking. That would be silly. What I mean is to adjust all your date/time variables to represent the time at 12:00 hours.

$new_date = date("Y-m-d", strtotime($date." 12:00:00") + 86400 * 15);

Now that you are doing date math at noon, you will be safe for most any date range you are doing math on. Daylight savings time always gives and hour and takes an hour every year, so those cancel each other out. It would take a whole lot of leap seconds to cause the offset from the start date to the end date to shift enough to make this technique no longer work, but it is technically feasible. So, if you are adding hundreds of years to dates, this won't work for you. There, disclaimer added.

UPDATE: I should point out that the examples in this post only apply to US time zones that recognized Daylight Savings Time and the rules for DST as they apply after the changes in 2007.

Google is breaking the Web

brianlmoon — Tue, 18 Oct 2011 17:09:05 -0500

So, Google announced today that they would start doing a couple of new things. First, they are going to start sending all logged in users to their SSL enabled search page. Secondly, they claimed they are going to stop sending search terms to the sites in the referring URL. They claim all of this for your security. The first one I buy. SSL is better for those people on public wifi, no doubt. The second, I don't think so.

Let's back up a step. For those that don't know how the web works, this is a quick lesson. When you click on a link on a site, your browser connects to that new site to get the page. Part of that communication is to tell the site you are going to what site the user was on when they clicked on your URL. This is a good thing. There is never any information passed between Google and your site. Its all between your browser on your computer and the site you are asking your compter to load. It helps site owners know who is linking to them. In the case of search engines, the referring URLs often contain the search terms someone typed in to find their site. This is also helpful for lots of reasons. None of them involve a user's security.

Ok, so, Google claims they are going to remove your search terms. But, my tests show they are removing the whole referring URL. Yes, you will not know what users are coming from Google. Let me show you. This is what I did.

I typed http://www.google.com/ into my browser
I searched for dealnews
I clicked on the first link, which is the dealnews.com front page.

Using a tool called HTTPFox I am able to see what information is being passed between my computer and the web sites. This is what I see:

http://www.google.com/ with no referring URL
http://www.google.com/#hl=en&sugexp=kjrmc&cp=5&gs_id=j&xhr=t&q=dealnews&qe=ZGVhbG4&qesig=YtB_HodN2qCOIiqwx_wetA&pkc=AFgZ2tlle01GJ99f38Ol-HvrY0sbiq4vzJfAPDSXGQ2js5QqyHGJ9-5HIgoFXbUujrU81pfyhEVO8jpmFouC09MG1fRbqd0GVA&pf=p&sclient=psy-ab&site=&source=hp&pbx=1&oq=dealn&aq=0&aqi=g4&aql=f&gs_sm=&gs_upl=&bav=on.2,or.r_gc.r_pw.,cf.osb&fp=7b65204da701ddb7&biw=1295&bih=1406 with no referring URL because Google use javascript to load the search results.
http://www.google.com/url?sa=t&source=web&cd=1&sqi=2&ved=0CCwQFjAA&url=http%3A%2F%2Fdealnews.com%2F&rct=j&q=dealnews&ei=EPOdTtaUN4XOiAKZlIntCQ&usg=AFQjCNEN2YJ8XgSAJm6FOUqK2PuBUOkfxA&sig2=N2jBSsJb8sgPsrTkGgFCfw&cad=rja with a referrring URL of http://www.google.com/
http://dealnews.com/ with a referring URL of http://www.google.com/url?sa=t&source=web&cd=1&sqi=2&ved=0CCwQFjAA&url=http%3A%2F%2Fdealnews.com%2F&rct=j&q=dealnews&ei=EPOdTtaUN4XOiAKZlIntCQ&usg=AFQjCNEN2YJ8XgSAJm6FOUqK2PuBUOkfxA&sig2=N2jBSsJb8sgPsrTkGgFCfw

As you can see, the request to http://dealnews.com/ was sent a URL by my browser telling that site that Google was linking to dealnews. In that URL you will see q=dealnews. That is the search term I typed into Google. Now, lets see what happens when I do the same thing on SSL.

https://www.google.com/ with no referring URL
Redirected to https://encrypted.google.com/ with no referring URL
https://encrypted.google.com/#hl=en&sugexp=kjrmc&cp=8&gs_id=f&xhr=t&q=dealnews&tok=wzChADhZTTjwPuXR1iOwSA&pf=p&sclient=psy-ab&site=&source=hp&pbx=1&oq=dealnews&aq=0&aqi=g4&aql=f&gs_sm=&gs_upl=&bav=on.2,or.r_gc.r_pw.,cf.osb&fp=47f2f62d0e6da959&biw=1295&bih=1406 with no referring URL because Google use javascript to load the search results.
https://encrypted.google.com/url?sa=t&source=web&cd=1&sqi=2&ved=0CCsQFjAA&url=http%3A%2F%2Fdealnews.com%2F&rct=j&q=dealnews&ei=x_edTvjlGeKviQKzmdHqCQ&usg=AFQjCNEN2YJ8XgSAJm6FOUqK2PuBUOkfxA&sig2=OEhW8Z_BhHcCboIzu_Z2zQ with a referring URL of https://encrypted.google.com/
http://dealnews.com/ with no referring URL.

So, if dealnews.com does not get a referring URL, how do we know this came from Google. This is the quote from the Google blog post:

When you search from https://www.google.com, websites you visit from our organic search listings will still know that you came from Google, but won't receive information about each individual query.

I ask you how the site will know that if there is no referring URL? Referring URLs are a fundamental part of the web. If Google wants to strip data off the URL, that is one thing. It is not great IMO, but whatever. But, not sending referrers at all is just wrong and should be changed.

If you care, please share this post. Tweet it, +1 it, whatever. This is just bad news for the web.

Edit: I wanted to make sure everyone knew, I observed the same behavior in both Firefox 7 and latest Google Chrome

Edit 2: I have also confirmed with the Apache access logs that no referring URL was sent.

Lightbox using YUI

brianlmoon — Sun, 04 Sep 2011 20:53:55 -0500

I made this a while back but have not blogged about it. I needed a lightbox for a project. So, I looked around. Like usual, none of them were right for me. In addition to that, I am not on the jQuery band wagon. No particular reason. I just never latched on to the weird syntax (They made a function named $ so it could kind of sort of read like PHP, come on). Anyhow, not a jQuery hate post. I did however take a liking to YUI. It is more my style. Less magic stuff and more of a real library. I feel like I can use it where I need to and not use it when I don't. Again, not a YUI love fest post. Long story short, I chose to make my lightbox with the help of YUI. I have just uploaded the code to GitHub. Maybe someone else will find it useful. I have an example page as well.

Features

Lightbox and contents can be created via:
- Javascript
- Existing markup
- Asynchronus (AJAX) request (contents only)
Lightbox will close when ESC is pressed
Rounded corners and box shadow using CSS3
Exposes an onClose event that can be subscribed to

Compatibility

Firefox 3.5+
Chrome 4+
Safari 5+
Internet Explorer 9 (Animation is not working due to YUI's lack of opacity support in IE9)
Internet Explorer 8 (Via IE9 in IE8 compatibility mode. No rounded corners, no shadows)
Internet Explorer 7 (Via IE9 in IE7 compatibility mode. No rounded corners, no shadows)
Opera 10.53

YUI Requirements

Dom
Event
Connection Core
JSON
Selector
Animation (if animate set to true)

Check for a TTY or interactive terminal in PHP

brianlmoon — Thu, 01 Sep 2011 23:42:54 -0500

Many UNIX tools do different things if they are connected to an interactive terminal, also called a TTY. This can be handy for lots of reasons. I had a use case today that prompted me to find out how to do it in PHP.

Here is the situation. We log errors to the PHP error log. We then have processes that monitor that error log and alert us about any uncaught exceptions or fatal errors very quickly so we can address issues. We also monitor non-fatal errors and alert on those on a less frequent schedule. However, this can be annoying if a user is running some code on a terminal that is generating errors. Let's say I am trying to find out why some file import did not happen. Running the job that is supposed to do it may yield an error. Maybe it was a file permission issue or something. There are other people watching the alerts. What they don't know is that I am running the code and looking at these errors in real time. So, they may start digging into the issue when I am the one causing it and can see it happening already. So, I thought it would be nice, if in my error handler, I could not send errors to the error log that are being sent to an interactive terminal. A few quick searches for "php check for tty" did not find anything. In the end, a coworker cracked open a book to see how it was done in C. That got me on the right path to finding two PHP functions: posix_isatty and posix_ttyname. These seem to do the trick. They take a file descriptor like STDOUT and will tell you if that is an interactive terminal and what the tty name is if it is one.

It took me a few tries to get the full effect I wanted in PHP. The thing I always forget is that fatal errors don't use my error handler. I understand why. The engine is in an unknown state when that happens, so it can't keep running code. In the end I added this code to my auto prepend file which is where my error handler, auto loader and other start up stuff is defined for all PHP code we run, both CLI and Apache. Since PHP will send errors to STDERR if it is defined, that is what we want to check for. If it is defined and it is a TTY, we just disable error logging. I should note that I already check the log_errors ini setting in my error handler before I even call the error_log function.

if(defined("STDERR") && posix_isatty(STDERR)){

    ini_set("log_errors", false);

    ini_set("error_log", null);

}

Hopefully anyone else searching for "php check for tty" or "php interactive terminal" will find this blog post and it will help them out.

Talking about Gearman at Etsy Labs

brianlmoon — Fri, 05 Aug 2011 08:11:16 -0500

I find myself flying to New York on Monday for some dealnews related business. Anytime I travel I try and find something fun to do at night. (Watching a movie by myself in Provo, Utah was kinda not that fun.) So, this week I asked on Twitter if anything was happening while I would be in town. Anything would do. A meetup of PHP/MySQL users or some design/css/js related stuff for example. Pretty much anything interesting. Well, later that day I received an IM from the brilliant John Allspaw, Senior VP of Technical Operations at Etsy. He wanted me to swing by the Etsy offices and say hi. Turns out it is only a block away from where I would be. Awesome! He also mentioned that he would like to have me come and speak at their offices some time. That would be neat too. I will have to plan better next time I am traveling up there.

Fast forward another day. I get an email from Kellan Elliott-McCrea, CTO of Etsy wanting to know if I would come to the Etsy offices and talk about Gearman. At first I thought "That is short notice, man. I don't know that I can pull that off." Then I remembered the last time I was asked to speak at an event on short notice based off a recommendation from John Allspaw.

It was in 2008 for some new conference called Velocity. That only turned out to be the best conference I have ever attended. I have been to Velocity every year since and this year took our whole team. In addition, I spoke again in 2009 at Velocity, wrote a chapter for John's book Web Operations that was released at Velocity in 2010 and was invited to take part in the Velocity Summit this year (2011) which helps kick off the planning for the actual conference. The moral of that story for me is: when John Allspaw wants you to take part in something, you do it.

In reality, it was not that tough a decision. Even without John's involvement, I love the chance to talk about geeky stuff. The Etsy and dealnews engineering teams are like two twins separated at birth. Every time we compare notes, we are doing the same stuff. For example, we have been trading Open Source code lately. They are using my GearmanManager and we just started using their statistics collection daemon, statsd. So, speaking to their people about what we do seem like a great opportunity to share and get input.

The event is open to the public. So, if you use Gearman, want to use Gearman, or just want to hear how we use Gearman at dealnews, come here me ramble on about how awesome it is Tuesday night in Dumbo at Etsy Labs. You can RSVP on the event page.

Best Practices for Gearman by Brian Moon
Etsy Labs
55 Washington St. Ste 712
NY 11222

Tuesday, August 09, 2011 from 7:00 PM - 10:00 PM (ET)

Taking a personal day

brianlmoon — Thu, 26 May 2011 12:57:23 -0500

I mostly talk about geeky stuff on this blog, but sometimes I like to get a little more personal. Today is one of those days.

Like this post, I took my public speaking to a personal place this month. At the Ignite Birmingham event this month I decided to talk about what life is like with a large family. This was a different experience for me. It was very well received and my wife was there to watch it. She had not seen the slides so it was a surprise to her. Here is the video.

My daughter's 8th grade class had a "graduation" ceremony on Tuesday. Now, I kind of think that is a bit silly. I approached as an awards ceremony. They give out awards for best GPA per core class. They also award student athletes of each sport. the SGA and drama department are honored. I think those are all great things. Well, Macy, my 13 year old received seveal honors. If I told you I did not expect them I would be lying. She was a head cheerleader, class historian/photographer, played the lead role in the school play and is an all around good student. She received awards for all of those things. Those were expected. After the students all walked across the stage, the vice principal and principal had one more award to give. Each of them chooses one boy and one girl based on their own criteria to receive the Vice Principal's Award and Principal's Award. We sat listening to the principal talk about who she chose the students to receive the award. She spoke of character, kindness, school spirit, work ethic and all the qualities that one hopes their child has. Then, much to our surprise, she says Macy's name. I was floored. I admit, while she was saying all those things, I was thinking "Macy fits all those to a T, she should get that, but probably won't". I figured it would go to some goody two shoes kid I had never met because their mom was always helping out at the school and they would always chat it up with the principal in the hallways. In reality, my wife and I have not always seen eye to eye with the principal and I feared that relationship would also be a hurdle. But, no, kudos to the principal. She evaluated Macy for being Macy. And I won't lie, that will probably help the relationship between myself and the principal as my other kids go into that school.

PHP Frameworks

brianlmoon — Mon, 25 Apr 2011 10:00:00 -0500

Last week I spoke at and attended the first ever PHP Community Conference. It was very good. It was also very different from my normal conference. I usually go for very technical stuff. I don't often stop and smell the roses of the community or history of my chosen (by me or it I am not sure sometimes) profession. There was a lot of community at this one.

One thing that seemed to be a hot topic at the conference was frameworks. CakeDC, the money behind CakePHP was the platinum sponsor. I chatted with Larry Masters the owner of CakeDC for a bit while walking one night. Great guy. Joël Perras gave a tutorial about Lithum. I attended most of this one. He did very well. Joël was frank and honest about the benefits and problems with using frameworks, including having to deal with more than one at a time. There was also a tutorial about Zend Framework patterns by Matthew Weier O'Phinney. I missed this one. On the second day, things were different. Rasmus Lerdorf warned about the bloat in most of the PHP frameworks and expressed hope that they would do better in the newer versions. I received several questions about frameworks in my session. I also spoke out about them a bit. Terry Chay wrapped up the day with his closing keynote and touched on them again. More on that later. I want to kind of summarize what I said (or meant to say).

PHP is a framework

In my session, I talked about the history of Phorum. One of the things I covered was the early days of PHP. Back in the 90s, before PHP, most dynamic web work was done in C or Perl. At that time, in those worlds, you had to do all the HTTP work yourself. If you wanted a content type of text/html, you had to set it, in code, on every single response. Errors in CGI scripts would often result in Apache internal error pages and made debugging very hard. All HTML work had to be done by writing to output. There was no embedding code with HTML even as a templating language. PHP changed all that. You had a default content type of text/html. You had automatic handling of request variables. Cookies were easily ingested and output. You could template your HTML with script instead of having to write everything out via print or stdout. It was amazing. Who could ask for more?

Frameworks as a tool

Well, apparently a world that I honestly don't work in could ask for more. There are three major segments of web developers these days. There are developers that work for a company that has a web site, but its business is not the web site. Maybe it is a network hardware company or some other industry where their business merits having a staff to run their site, but it is not their core business. Then there are developers like myself that work for a company where the web site is the business. Everything about the business goes through the web. We have the public web site and the internal web site our team uses. It is everything. The last type are those developers that are constantly building new sites or updating existing sites for clients. I will be honest, this is not a segment I have considered much in the past when writing code or blog posts. But, I met more of those people at this conference than any of the other two types. They seem to be the ones that are motivated and interested. Or at least, because PHP and the web are their business, they sent their people to the conference.

You see, I have spoken out about frameworks. Not very publicly, but those that know me have heard me comment about them. I have never really seen the point. Why start with something generic that will most likely not fit your ultimate need when you need to scale or expand beyond its abilities? Well, for thousands of web sites, that are likely being built by agencies, that time never occurs. Most likely, before that happens, the site will be redesigned and completely replaced. So, if you spend every day building a new site, why do all that groundwork every time?

In addition, why have to deal with every different client's needs? I often say that Apache is my controller. I don't like to use PHP as my controller. But, if I was deploying a site every week to a different stack, I can't rely on Apache with mod_rewrite or whatever things I rely on in my job today. So, you need to have full control in the application. What database will the client this week use? I don't care, the framework abstracts that for me. These are all very good reasons to use a framework.

Framework Trade-Off

There are some trade-offs though. The biggest one I see is the myriad of choices. Several of the pro-framework people even mentioned that there are a lot out there. And it seems that someone is making a new one everyday. With all these choices, it is likely that some of the benefit you get from a framework could be lost. If a client already has a site based on CakePHP and your agency uses Lithium what do you do? Say no to the work or have to deal with the differences? Some of them are big enough to be a real issue. Some are so small, you may not notice them until it's too late. That is a tough place to be.

The other issue is performance. Frameworks are notoriously inefficient. It has just been their nature. The more you abstract away from the core, the less efficient you are. This is even true with PHP. Terry Chay pointed out that PHP is less efficient than Java or C in his keynote. But, you gain power with PHP in way of quicker development cycles. Frameworks have that same benefit. But, have not solved this issue any better than PHP has over C. They abstract away the low level (for PHP at least) stuff that is going on. And that means loss of efficiency. This can be solved or at least worked on, however, and I hope it is.

Frameworks as a Commodity

So, this gets me back to something Terry Chay said. He talked about the motivation of companies to open source their technology. He used Facebook's Open Compute Project as an example. He pointed out that a major reason Facebook would open up this information would be in hopes that others would do the same in their data centers. If that happened, it would be easier for Facebook to move to a new data center because it was already mostly setup the way they like it.

Transitioning this same thought frameworks, the commoditization here, that I see, is in the interest of developers. If the framework you support becomes the de facto standard, then all those developers working in agencies using it are now ready to come to work for you. Plus, if you are the company behind it, there are opportunities for books, conferences, training, support, and all the other peripherals that come from the commercial/open source interaction. Need proof of that? Look no further than the "PHP Company", Zend. They could have committed developers to PEAR, but instead created Zend Framework. I see job listings very often for Zend Framework experience. Originally Zend tried to monetize the server with their optimizers and Zend Server. They had moderate success. The community came up with APC and XCache that sort of stole their thunder. I feel they have had much better success with Zend Framework in terms of market penetration. The money is with the people that write the code, not run the servers.

Frameworks are EVERYWHERE

I will close with something else that Terry Chay said. This was kind of an aha! moment for me. Terry pointed out that frameworks are everywhere. Wordpress, Drupal and even my project, Phorum, are frameworks. You can build a successful site using just those applications. It is not just the new breed code libraries that can be viewed as frameworks. In fact, Phorum's very own Maurice Makaay is building his new web site using only Phorum 5.3 (current development branch). Phorum offers easier database interaction, output handling, templating, a pluggable module system and even authentication and group based permissions. Wow, I have always kind of shunned this idea. In fact, when Maurice first showed me his site, I kind of grimaced. Why would you want to do that? You know why? Because the main thing that drives his site is Phorum. His users come to the site for Phorum. So, why would he want to install Phorum, invest in making it all it can be and then have to start from scratch for all the other parts of the site that are not part of the message board. Duh, I kind of feel stupid for never looking at things from this perspective before. Feeling dumb is ok. I get smarter when I feel dumb. New ideas make me a better developer. And I hope that is what comes out of this experience for me. You never know, I may throw my name in this hat and see how Phorum's groundwork could be useful outside of the application itself.

HTML5 Experiment

brianlmoon — Mon, 28 Mar 2011 08:00:00 -0500

I was looking through dealnews.com browser stats the other day to see how many of our visitors had browsers that could use CSS3. This was how it broke down.

58% of dealnews visitors have browsers that support some CSS3 elements like border-radius and box-shadow. This includes recent Webkit, Firefox 3.5+, Opera 10+ and IE9. Awesome! Another 37% fully support CSS2. This includes IE7, IE8 and Firefox 2 - 3.0.x.

We have started to sneak in some CSS3 elements into the CSS on dealnews.com. In places where rounded corners are optional (like the lightbox I created), we use border-radius instead of a lot of images. The same for shadows. We have started using box-shadow instead of, well, nothing. We just don't have shadows if the browser does not support box-shadow. In our recent redesign of our mobile site, we used all CSS3 for shadows, corners and gradients. But these were all places where things were optional.

The header of dealnews.com, on the other hand, requires a certain consistency. It is the first thing you see. If it looks largely different on your work computer running IE7 than it does on your Mac at home using Safari, that may be confusing. So, we have stuck with images for rounded corners, shadows and gradients. The images have problems though. On the iPad for instance, the page starts zoomed out a bit. The elements holding the rounded corner images don't always line up well at different zoom levels. Its a math thing. When zooming, you end up with an odd number of pixels at times which causes pixels to shift. So, we get gaps in our tabs or buttons. Not pretty. This has been bugging me, but its really just the iPad. Its not mission critical to fix a pixel on the iPad. Armed with the numbers above, I decided to try and reproduce the dealnews.com header using the most modern techniques possible and see how well I could degrade the older browsers.

Here is a screen shot in Firefox 4 of the current dealnews header. (You can click any of these images to see them full size.)

Now, here is the HTML5/CSS3 header in all the browsers I tried it in. I developed in Firefox 4 and tested/tweaked in others.

Firefox 4 (Mac)

Firefox 3.6 (Windows)

Chrome 10

Opera 11

Internet Explorer 9

Internet Explorer 8 (via IE9 Dev Tools)

Internet Explorer 7 (via IE9 Dev Tools)

Internet Explorer 6 (ZOMG!!1!)

Wow! This turned out way better than I expected. Even IE6 renders nicely. I did have to degrade in some older versions of Internet Explorer. This gets me 97% coverage for all the dealnews.com visitors. And, IMO, degrading this way is not all that bad. I am seeing it more and more around the internet. CNET is using border-radius and CSS3 gradients in their header. In Internet Explorer you see square corners and no gradient. Let's look a little deeper into what I used here.

Rounded corners

For all the rounded corners, I used a border-radius of 5px. I used CSS that would be most compatible. For border-radius that means a few lines just to get the one effect across browsers. The CSS for the tabs looks like this.


-moz-border-radius-topleft: 5px;
-moz-border-radius-topright: 5px;
border-top-left-radius: 5px;
border-top-right-radius: 5px;

The -moz is to support Firefox 3.5+. Everything else that supports border-radius recognizes the non-prefixed style.

For the older browsers, they just use square corners. I think they still look nice. The one thing we lose with this solution is the styled gradient border on the brown tabs. It fades to a silver near the bottom of the tabs. There is no solution for that in CSS at this time. That is a small price to pay to skip loading all those images IMO.

Shadows

For the shadows on the left and right side of the page (sorry, not real visible in the screen shots), I used box-shadow. This requires CSS such as:


-webkit-box-shadow: 0 0 0 transparent, 0 2px 2px #B2B1A6;
-moz-box-shadow: 0 0 0 transparent, 0 2px 2px #B2B1A6;
box-shadow: 0 0 0 transparent, 0 2px 2px #B2B1A6;

Again, -moz for Mozilla browsers and -webkit for WebKit based browsers. Now, there are some gotchas with box-shadow. The first is that Firefox requires a color. The other supporting browsers will simply use an alpha blended darker gradient. This makes it a little more work to get it all right in Firefox. The other tricky part was getting the vertical shadow to work. The shadow you make actual curves around the entire element. It has the same z-index as the element itself. So, a shadow on an element will appear on top of other elements around it with a lower z-index. I had issues keeping the shadow from the lower area (the vertical shadow) from appearing above the element and on top of the tabs. If you look really, really, really close where the vertical and horizontal shadows meet, you will see a tiny gap in the color. Luckily for me, it works due to all the blue around it. That helps mask that small flaw.

For the older IE browsers, I used conditional IE HTML blocks and added a light gray border to the elements. It is a bit more degraded than I like, but as time passes, those browsers will stop being used.

Gradient Backgrounds

Completing the holy trinity of CSS3 features that make life easier is gradient backgrounds. This is the least unified and most complex of the three features I have used in this experiement. For starters, no two browser use the same syntax for gradient backgrounds. Firefox does use the W3C recommended syntax. Webkit uses something they came up with and Internet Explorer uses its super proprietary filter CSS property and not the background property. The biggest problem with the filter property is that it makes IE9 not work with border-radius. The tabs could not use a gradient in IE9 because it applied a rectangular gradient to the element that exceeded the bounds of the rounded top corners. Bummer. Once the standard is set this should all clear itself up. As things stand today, I had to use the following syntax for the shadows.


background: #4b4ba8; /* old browsers */

background: -moz-linear-gradient(top, #4b4ba8 0%, #3F3F9A 50%, #303089 93%); /* firefox */

background: -webkit-gradient(linear, left top, left bottom, color-stop(0%,#4b4ba8), color-stop(50%,#3F3F9A), color-stop(93%,#303089)); /* webkit */

filter: progid:DXImageTransform.Microsoft.gradient( startColorstr='#4b4ba8', endColorstr='#303089',GradientType=0 ); /* ie */

As you can see, this is pretty complex. See the next section for a quick way to configure that block of CSS.

Another thing that makes working with these gradients more complex than images is a lack of rendering control. When you are making images, you can control the exact RGB values of the colors in the images. When you leave it up to the browser to render the gradient, sometimes they don't agree. I had to do a lot of fiddling with the RGB values to get all the browsers to render the gradient just right. Some of this had to do with the short vertical area I am using in the elements. That limits the number of colors that can be used to make the transition. So, just be careful when you are lining up two elements with gradients that transition from one element to the other like I have here with the blue tab and the blue navigation bar.

CSS3 Online tools

Remembering all the CSS3 syntax is a little daunting. Luckily, there are some cool online tools to generate some of this stuff. Here are few I have used.

CSS3 Generator - This tool can generate lots of different CSS3 effects.
CSS3 Border Radius Generator - This tool only generates border radius, but lets you set each corner individually.
Ultimate CSS Gradient Generator - This is a great tool that lets you generate gradient backgrounds and even supports Internet Explorer compatible background gradients.

HTML5 Elements

To make this truly an HTML5 page, I wanted to use the new doctype and some of the new elements. I make use of the

and

tags in this page. The

tag is just what you would think it is. It surrounds your header. This is all part of the new semantic logic behind HTML 5. The

tag surrounds major site navigation. Not just any navigaion, major navigation. The HTML5 Doctor has more on that.

To make IE support them, I used a bit of javascript I found on the internet along with some CSS to set them to display as block level elements. The CSS is actually used by older Firefox versions as well.

A couple of non-CSS3 techniques

I did a couple of things that are not CSS3 or HTML5 in this page. One is that I put the CSS into the page and not in its own file. With modern broadband, the biggest issue in delivering pages fast is the number of HTTP requests, not (always) the total size of the data. The more HTTP requests required to start rendering your page, the longer it will take. Your optimization goals will determine if this technique is right for you. I currently include all CSS needed to render the header in the page so that the rendering can start without any additional HTTP requests. The CSS for the rest of the page is included via a link tag.

The other technique I used in this page that is not new, but is not widely used. Again, depending on your needs, it may be a possible win when trying to reduce HTTP requests. I used embedded image data URIs in the CSS for the three images I still needed for this page. Basically, you base64 encode the actual image file and put it into the CSS. The benefit is that this page is fully rendered with just one HTTP request. The downside is that the base size for the page (or CSS if it is external) that has to be downloaded every time is much larger. Probably a good compromise would be to put the CSS into an external file. This would mean just two HTTP requests would be needed and the CSS could be cached. For IE6, I just used conditional HTML to include an actual URL to the background images.

Th data URI technique is a little bit of a mysterious technique in that it does make for a larger page, but can help with render time on first load. This really comes down to what you are optimizing for. If you are optimizing for repeat visitors, it may be that the images are better off being separate requests. If you are optimizing for new visitors, this technique will yield a faster rendering page. In Chrome and IE9, the onLoad event fired much sooner (as much as half the time) using this technique than having the images as a separate reqeust. In Firefox, something else is going on. I am not sure what. The onLoad event still fires sooner, but not a whole lot sooner. The DOMContentLoaded event in Firefox however fires later with this technique than with the images in a separate request. Firefox was the only browser that showed this pattern.

OOCSS

It is not really HTML5 or CSS3 related, but I do want to give some credit to OOCSS. I used it for much of the layout. It makes laying out elements in different browsers very easy. I am using it in the current site as well as the HTML5 experiment. You should use it. It is awesome.

Conclusion

HTML5 and CSS3 have a lot to offer. And if you have a user base that is fairly modern, you can start using things now. While I may not redo the current dealnews web site using HTML5 and CSS3, our next redesign and any upcoming new designs will definitely include aspects of HTML5 and CSS3 where we can. It can save time and resources when you use these new techniques.

Here is a link to the HTML5 source. I also created a stripped down version of the HTML4 in use on the site now.

Lies, Damned Lies and Google Analytics

brianlmoon — Thu, 10 Mar 2011 10:21:36 -0600

Google Analytics (GA) has changed the world of web analytics. It used to be that you only had applications that analyzed logs from your web server. Those were OK for the first few years. But, with bots (and especially ones that lied about being a bot) and more complex web architectures, those logs became less useful for understanding how your users used your web site.

Once such product was Urchin. Years ago, we were users of Urchin, the product and company that Google purchased to create GA. It was the last really good log analyzer out there. We were kind of excited when Google bought them as we were looking at going to their hosted solution which eventually became GA. At the time however they were young and we decided to go with Omniture, the 800 pound gorilla in the space. However, Omniture prices were such that increased traffic meant increased cost with no additional importance in their numbers and no new features from their product. So, we left them. We turned back to GA. In addition we started using Yahoo's recently acquired product, Yahoo Analytics (formerly IndexTools). They were both free so we figured why not use them both. All this was over 2 years ago.

The one big lacking thing for us with all these tools was being able to tie actual user activity in the analytics to revenue. You see, dealnews does some affiliate based business. This means that we get a percentage of sales after the transactions are all closed. So, we may not see revenue for a user action for days or even weeks. All the analytics packages are made for shopping carts. In a shopping cart, you know at the checkout how much the user spent so you can inject that into the system along with their page view. We don't have that luxury. To try and fill this gap, we started keeping our own javascript based session data in logs for various purposes in addition to the two analytics systems.

Recently we decided to try and use the GA API to get page views, visits and unique visitor data about parts of our site so we could, at least at a high level, tie the numbers all together in one report. This gets me to the meat of this post. What we found was quite disturbing. This is a query for a single day for a particular segment of our site. We used the Data Feed Query Explorer to craft our queries for the data we wanted. This was the result.

As you can see, some pages received visitors but no visits. Pages consistently received more visitors than visits which I find quite odd. Sometimes we found that our internal logging would show activity on pages and Google would simply not show any activity for that page for an entire month. Now, I know they have a cutoff. I believe it is still 10,000, the old Urchin number. (C'mon Google, you are Google. You still have this?) They will only store up to 10,000 unique items (page URLs in this case) for a given segment of their reporting data. So, 10k referring domains, 10k pages, etc. etc. Perhaps that is what happened? We dug in and found that to not be the case. And besides, it shows visitors but not visits. What is up with that? Maybe its a date thing. We should look at more days. This is an entire month for the same filters.

Whoa, still have visitors and no visits. In addition, there are lots of multiples of 17 there. 17, 34 both appear a lot. We found that littered throughout our results when filtering the page path. This has to be made up data or something. I don't see how it could be based in any reality.

The odd data was not limited to this apparent missing data. There is apparently extra data in there too. In another discussion about the impact of recent social marketing efforts we went to the analytics to see what kind of traffic social networking sites were sending our way. When comparing GA to YA and our internal numbers, we were left baffled. Sorry, no numbers allowed here - super corporate secrets and all that. But, I can tell you that Google Analytics claims that several large, well known referring sites send 5x to 10x the traffic to us that our other tracking reports. I even wrote custom code to try and follow a user through our logs to see if maybe they were attributing referrers to a visitor on a second visit in the day as belonging to the earlier referrer but could never find any pattern that matched their data. There simply is no way this is right.

There is good news. If you stick with the high level numbers like overall page views, visitors, visits and things like new vs. old visitors, the numbers appear to line up well. In all cases, the differences of all 3 of our resources are within acceptable ranges. But, apparently, drilling down too deep in the data with GA will yield some very unsavory answers.

Questions for Git users

brianlmoon — Thu, 24 Feb 2011 12:36:09 -0600

So, we have been using Subversion for a long time. We are very comfortable with it. We know how it works. Even when things go sideways, we have people here that understand it and can set things straight again. But, we do find some things lacking. I had an SVN conflict today where the diff showed the problem to be that I was replacing nothing with something. That was a conflict. Sigh. Also, it is getting slower and slower as time passes. Committing or updating my code base can take long enough that I start checking email and forget I was committing something. So, we are thinking about a switch. I use GitHub for some of my OSS software I have released. I like GitHub a lot. I find the Git command line tool a little confusing (commit -a seems silly for example). But, I figure I can get over that. I am going to end up wrapping it up in our merge and deploy tools anyway. But, I have some questions for people that use Git on a daily basis. I have searched around and there is either more than one answer or no answers to these things.

How big is your repository?

We have 1,610 directories and 10,215 files in our code library. How does Git deal with repositories that size?

What are you using for a Git server?

From what I can tell, there is no server component to Git. I think I understand that you can use a shared Git use and allow ssh keys to access it that way? Or there are some other 3rd party projects that act as a Git server. We currently use Apache+SVN which lets us A) Use our LDAP server and B) control access to parts of the repository using Apache configuration. It looks like we would lose the LDAP for sure but maybe something like Gitolite would allow us to do some auth/acl stuff. Maybe we could dump our LDAP data on a schedule to something it can read.

What is your commit, push, deploy procedure?

This is really aimed at the guys doing continuous deployment in a web application with Git. Our current methodology is that each developer has a branch. They commit to their branch. When a set of commits is ready for release, they push it to trunk. In a staging environment, the changeset in trunk is merged with the production branch. That branch is then rolled to the servers. We like having the 3 layers. It lets us review changes in one place (trunk) for large commits. Meaning there can be 10 commits in a developer's branch, but when all the changes are merged into trunk (using one SVN merge of course) you get one nice neat commit in the trunk branch that can easily be evaluated. It also lets users collaborate easily by committing changes to trunk that others may need. Other users can just merge back from trunk to get the changes. From what I have read or heard people talking about, this seems to fly in the face of how Git works. Or, at least what Git is good at. Also, I think the way Git works, 10 commits in a branch would merge as 10 commits in trunk. So, we would lose the unification of a lot of little changes getting merged into one change. Some of our developers will use their branch to commit things in progress and not at a finished point. So, by the time they are done, we will have 10+ commits for one task. Any input here?

What are you using for visualization?

We use Trac. We love and hate Trac. Its good enough. I know Redmine supports Git natively and I know there is a Trac Hack for making Trac support it. Anything else? Any comments on Redmine?

Any tips for importing from Subversion?

I tried importing our Subversion repos into Git using svn2git. It got hung up on some circular branch reference or something. The error message was confusing. So, I guess if we do migrate, we will be starting with little to no history. Perhaps we just import our current production branch and start from there. Any tips?

What is next for message board software?

brianlmoon — Thu, 24 Feb 2011 07:00:00 -0600

When I was hired at dealnews.com in 1998, my primary focus was to get our message board (Phorum) up to speed. I had written the first version as a side project for the site. Message boards were a lot simpler back then. Matt's WWWBoard was the gold standard of the time. And really, the functionality has been only evolutionary since. We added attachments to Phorum in 2003 or something. That was a major new feature. In Phorum 5 we added a module system that was awesome. But, that was just about the admin and not the user. From the user's perspective, message boards have not changed much since 1997. I saw this tweet from Amy Hoy and it got me to thinking about how message boards work. Here is the typical user experience:

Go to the message board
See a list of categories
Drill down to the category they want to read
Scroll through a list of messages that are in reverse cronological order by original post date or most recent post date
Click a message and read it.
Go to #3, repeat

Every message board software package pretty much works like that and has for over 10 years. And it kind of sucks. What a user would probably rather experience is:

Go to the message board
The most interesting things (to this user) are listed right there on the page. No drill down needed.
Click one and read it.
Goto #2, repeat.

Sounds easy? That #2 is easy to type but very hard to accomplish. I think it is conceivably doable if you are running a site that has all the data. Stackoverflow comes close. When you land on the site, they default the page to the "interesting" posts. However, they are not always interesting to me. They are making general assumptions about their audience. For example, right now, the first one is tagged "delphi". I could care less about that language and any posts about it. Its a good try, but misses by oh so far. This is not a Stackoverflow hate post. They are doing a good job. So, what do I do when I land there? I ignore the front page and click Tags (#2 in the first list), then pick a tag I want to read about (#3 in the first list). Low and behold the page I get is "newest". So, I end up doing exactly what is in the first list I mentioned. They do offer other sort options. But, they chose newest as the default. And from years of watching user behavior, 80% - 90% of people go with the good ol' default. This kind of brings me to another point though about the types of message boards there are.

Stackoverflow is a classic example of a help message board. People come there and ask a question. Other people come along and answer the question. Then more people come along and vote on whether the answers (and questions) are any good. This is one really nice feature that I think will have to become a core feature in any message board of the future. The signal to noise ratio can get so out of whack, you need human input to help decide what is good and what is noise. I think the core of the application has to rely on that if we are ever going to achieve the desired experience.

The second type of message board is a conversational system. It is almost like a delayed chat room. People come to a message board and post about their cat or asking who watched a TV show, that kind of thing. This has a completely different dynamic to it than the help message board. You can't really vote if a post is good or bad. The obvious exception being spam would of course want to be recognized and dealt with.

So, how do you know what content is desirable for the user that is entering the site right now? This concept has already been laid out for us: the social graph. You have to give users a way to associate with other users. If Bob really likes Tom's posts, he is probably more interested to read Tom's post from 30 minutes ago than some new guy that just joined the site and posted 1 minute ago. The challenge here is getting people to interconnect...but not too much. Everyone has that aunt on Facebook that follows you, your roommate and anybody else she can. She would follow your dog if he had a Facebook account. So, those people would still get a crappy experience if the whole system relied on the social graph. The other side is the people that will never "follow", "like" or whatever you call it another person. Their experience would lack as well. One key ingredient here is that you need to own this data. You can't just throw like buttons and Facebook connect on your message board and think you can leverage that data. That data is for Facebook, not you. I think the help message boards could benefit from the social graph as well.

Another aspect of what is most important to a user is discussions they are involved in. That could mean ones they started, ones they have replied to or simply ones they have read. Which of those aspects are more important than the others? Clearly if you started a discussion and someone has replied, that is going to interest you. If you posted a reply, you may be done with the topic or you may be waiting on a response. It would take some serious natural language algorithms to decide which is the case. For things you have read, I think you have to consider how many times the user has read the discussion. If every time it is updated they read it, they probably will want to read it again the next time it is updated. If they have only read it once, maybe they are not as interested.

The last aspect of message boards is grouping things. This is the part I actually struggle with the most. The easy first answer is tagging. Don't force the user down a single trail, let them tag posts instead of only posting them in one neat contained area. That gets you half way there. Let's use Stackoverflow (I really do like the site) as an example again. The first thing I do is go to Tags and click on PHP. I like helping people with PHP problems. So, is that really any different from categorization? Sure, there could be someone out there that really likes helping with Javascript. And if the same post was tagged with both tags then their coverage of potential help is larger. But, some of the time those tags are wrong when they tag it with more than one tag. The problem they need help with is either PHP or Javascript, most likely not both. They just don't know what they are doing. For example, there is this post on Stackoverflow. The user tagged it PHP and database-design. There is no PHP in the question. I am guessing he is using PHP for the app. But, it really never comes up and he is only talking about database design. So, who did the PHP tag help there? I don't think it helped him. And it only wasted my time. Having written all that, a free-for-all approach where there is no filtering sucks too. ARGH! It just all sucks. That brings us back to what Amy said in a way. Perhaps moderated tagging is an answer. I have not seen a way on Stackoverflow to untag a post. That would let people correct others. I am gonna write that down. If you work at Stackoverflow and are reading this, you can use that idea. Just put a comment in the code about how brilliant I am or something that aliens will find one day.

So, I am done. I know exactly what to do right? I just have to make code that does everything I put in the previous paragraphs. Man I wish it were that easy. When you want to write a distributed application to do it, the task is even more daunting. If I controlled the data and the servers and the code, I could do crazy things that would make great conference talks. But, it kind of falls apart when I want to give this code to a 60 year old retired guy that is starting a hobby site for watching humming birds on a crappy GoDaddy account. Yeah, he is not installing Sphinx or HandlerSocket or Gearman. Those are all things I would want to use to solve this problem in a scalable fashion. At that point you have two choices. Aim for the small time or the big time. If you aim for the small time, you may get lots of installs, but, you will be hamstrung. If you aim for the big time, you may be the only guy that ever uses the code. That is a tough decision.

What have I missed? I know I missed something. Are there other types of message boards? I can definitely see some sub-types. Perhaps a board where ideas instead of help messages are posted. Or maybe the conversations are more show off based as in a user posting pictures or videos for comment. Is there already something out there doing this and I have just missed it? Let me know what I have missed please.

How does a geek end up doing public speaking?

brianlmoon — Thu, 20 Jan 2011 08:00:00 -0600

I just finished reading Confessions of a Public Speaker (O'Reilly) by Scott Berkun. It is the first book I have read cover to cover in 16 years. I even read the colophon (which I had never seen in a book before). I found the entire book to be very engaging. While reading it, I kept reviewing every mistake I had ever made in every talk I have given. Even if you are not a regular public speaker, there is great stuff in the book that can help you with talking to your coworkers, boss, etc. There is a chapter that is more about teaching than speaking that I found really good. I can really apply a lot of that knowledge to working with my team on new technologies.

Public speaking was never something I thought I would do when I was younger. I was not on the debate team in high school. I was never the leader of any clubs. The most I ever did was run for class president my senior year. I did not win. The eventual valedictorian who had been the president of our class every year of high school won. That was probably for the best. In fact, I was considered a "rebellious", quiet outsider I later discovered. Ha, a rebel? Me. That seems quite far fetched.

I first felt like I wanted to share my knowledge with my peers was when I went to Apachecon in 1999. We were a 4 or 5 person company at the time. But, our CTO had been to regular conferences in his past jobs and found value in the experience. So, he and I took off for Orlando, FL. At the time, Apachecon was really the only well rounded web conference that existed (I find it is not as much this way now). I was excited by the environment I found at the conference. The speakers were not talking heads for faceless corporate giants. They were guys that, like us, were trying to make the web work. Also like us, a lot of them did not have massive investment cash, but were operating on shoestring budgets. Performance and availability mattered. Coming up with new ideas really mattered. We found that we had ideas that others had concurrently. And in some cases, we had ideas that some had not thought about. I would wait another two years to have my turn on stage.

The web dev/ops team had grown to 5 people. I received an email for the CFP (Call for Proposals) for Apachecon 2001. I thought I had a good idea for a session. The reaction we received from people when we talked about our caching scheme was interesting. Most people had one of two reactions. They either thought the idea was very interesting or they thought we were just dumb. But, the key was that almost everyone had a reaction. So, I thought it would make a good talk. I made every rookie mistake there was. I made my slides the night before the talk. I did not go to the room I was speaking in until it was time to talk. You name it, I did it for the actual presentation. The talk felt like a train wreck. The only good thing about it was, at that time, Apachecon required the presenters to submit a full written version of the session for printing in a book that attendees received. So, we (I was doing the work along with a coworker) had become very familiar with the content. The response we received from the people that attended was incredible. They probably had no idea that the slides were written the night before. But, I have to say, from that point on, I was kind of hooked.

It would be a few years before I spoke on that type of stage again (MySQL Conference, 2008). I seem to have found a home at O'Reilly conferences. Like Apachecon, they embrace people that are doing the work in the field. Sure, you see some talks that feel like sales pitches, but the community quickly exposes those sessions. In the last 5 years, I have spoken at several O'Reilly conferences, another Apachecon and several regional conferences.

All of this public speaking helped me in a way I never expected. In late 2009, my grandfather passed away. My grandmother, his wife, passed away in 1995. I remember at then having the urge to get and and talk about her. She was a fascinating woman. And there were so many things in my head. But, I lacked the confidence to just get up and say them. So, when my grandfather passed away, I had a completely different mindset. I made it known that I would like to say some things about him. When trying to decide what to say, I found myself using techniques I use for preparing a presentation. I started with several points to make, worked on the right order for the points. Then I went back through and filled in the details of each point. Practicing that "talk" was the hardest practice I have ever had to do for public speaking. No practice or preparation has seemed hard sense. It was an odd clash of my worlds. But, I am so happy I did it. I still have the notes on my iPhone. When I miss him, I get them out and read them.

Lately, I have branched out from talking to geeks. O'Reilly's Ignite series of events leap out of the geek culture and into your community. We are lucky to have a very good group of people organizing Ignite Birmingham in my home town. And while my talks thus far with Ignite Birmingham have had a tech slant, they were not for geeks. I was speaking to a room full of smart, but not nesicarily technical people. It's fun to get outside of my comfort zone. Also, they video all the talks. So, I get to see myself and judge my performance. It's great. Speaking to a wider audience is a fun journey that I am finding very exciting and look forward to exploring more. My next challenge for myself is to find a completely non-technical topic for Ignite Birmingham.

Release vlualogger 1.1

brianlmoon — Wed, 12 Jan 2011 13:38:28 -0600

I have made a small change to vlualogger to flush the file handle after every write. I discovered that lua was not flushing the writes when I was tailing a the log file to do some debugging. This fixes that issue.

Also see my original post about vlualogger.

Sharing gotchas on Facebook and Twitter

brianlmoon — Thu, 18 Nov 2010 07:00:00 -0600

I have been working on adding some sharing features to dealnews.com. Dealing with Facebook and Twitter has been nothing if not frustrating. Neither one seems to understand how to properly deal with escaping a URL. At best they do it one way, but not all ways. At worst, they flat out don't do it right. I thought I would share what we found out so that someone else my be helped by our research.

Facebook

Facebook has two main ways to encourage sharing of your site on Facebook. The older way is to "Share" a page. The second, newer, cooler way to promote your page/site on Facebook is with Facebook's Like Button. Both have the same bug. I will focus on Share as it is easier to show examples of sharing. To do this, you make a link and send it to a special landing page on Facebook's site. But, lets say my URL has a comma in it. If it does, Facebook just blows up in horrible fashion. The users of Phorum have run into this problem too. In Phorum, we dealt with register_globals in a unique way long ago. We just don't use traditional query strings on our URLs. Instead of the traditional var1=1&var2=2 format, we decided to use a comma delimited query string. 1,2,3,var4=4 is a valid Phorum URL query string.

According to RFC 3986, a query string is made up of:

query = *( pchar / "/" / "?" )

where pchar is defined as:

pchar = unreserved / pct-encoded / sub-delims / ":" / "@"

and finally, sub-delims is defined as:

sub-delims = "!" / "$" / "&" / "'" / "(" / ")" / 
 "*" / "+" / "," / ";" / "="

That is RFC talk for "A query string can have an un-encoded comma in it as a delimiter." So, in Phorum we have URLs like http://www.phorum.org/phorum5/read.php?61,145041,145045. That is the post in Phorum talking about Facebook's problem. It is a valid URL. The commas do not need to be escaped. They are delimiters much like an & would be in a traditional URL. So, what happens when you share this URL on Facebook? Well, a share link would look like http://www.facebook.com/share.php?u=http%3A%2F%2Fwww.phorum.org%2Fphorum5%2Fread.php%3F61%2C146887%2C146887. If I go to that share page and then look in my Apache logs I see this:

66.220.149.247 - - [18/Nov/2010:00:47:51 -0600] "GET /phorum5/read.php?61%2C146887%2C146887 HTTP/1.1" 302 26 "-" "facebookexternalhit/1.1 (+http://www.facebook.com/externalhit_uatext.php)"

Facebook sent %2C instead of comma? It decoded the other stuff in the URL. The slashes, the question mark, all of it. So, what is their deal with commas? Well, maybe I can hack Facebook and not send an encoded URL to the share page. Nope, same thing. So, they are proactively encoding commas in URL's query strings.

This has two effects. The first is that the share app attempts to pull in the title, description, etc. from the page. In this case, we redirect the request as the query string is invalid for a Phorum message page. So, they end up getting the main Phorum page. In the case of dealnews, we usually throw a 400 HTTP error when we get invalid query strings. Neither of these get the user what he wanted. The second problem is that the URL that is clickable when the user has shared the URL is not valid. So, the whole thing was just a huge waste of time.

I have submitted this to the Facebook Bugzilla. The only work around is to use a URL shortener or don't use commas in your URLs. Just make sure the shortener does not use commas. I guess you could use special URLs for Facebook that used something besides comma that are then redirected to the real URL with commas. I don't know what that character is, I am just guessing.

Twitter

Twitter's issues deal with their transition from their old interface to their new interface. Twitter is in the process of (or is done with) rolling a new UI on their site. The link in the old site to share something on Twitter was something like: http://twitter.com/home?status=[URL encoded text here]. This worked pretty darn well. You could put any valid URL encoded text in there and it worked. However, that now redirects you to their new interface's way of updating your status and they don't encode things right.

If I want to tweet "I love to eat pork & beans" I would make the URL http://twitter.com/home?status=I+love+to+eat+pork+%26+beans. Twitter then takes that, decodes the query string and redirects me to http://twitter.com/?status=I%20love%20to%20eat%20pork%20&%20beans. The problem is that they did not re-encode the &. It is in the bare URL. So, when I land on my twitter page, my status box just says "I love to eat pork ". Which while true, is not what I mean to tweet. This bug has been submitted to Twitter, but has yet to be fixed.

The second problem is with the new site and how they deal with validly encoded spaces. Spaces can be escaped two ways in a URL. The first, older way (which the PHP function urlencode uses) is to encode spaces as a plus (+) sign. This comes from the standard for how forms submit (or used to submit) data. It is understood by all browsers. The second way comes from the later RFC's written about URLs. They state that spaces in a URL should be escape like other characters by replacing a space with %20. The old Twitter UI would accept either one just fine. And, if you send that to the old status update URL it will redirect you (see above) with %20 in the URL instead of +. However, if you send + to the new Twitter UI, as above, you get "I+love+to+eat+pork+&+beans" in your status box. The only solution is to not send + has an encoding for space to Twitter. In PHP you can use the function rawurlencode to do this. It conforms to the RFC(s) on URL encoding. Doing so, with thew new linking pattern generates the URL http://twitter.com/?status=I%20love%20to%20eat%20pork%20%26%20beans which works great. This was also reported to Twitter as a bug by our team.

So, maybe that will help someone out that is having issues with sharing your site on the two largest social networks. Good luck with your social media development.

Monitoring PHP Errors

brianlmoon — Mon, 08 Nov 2010 08:00:00 -0600

PHP errors are just part of the language. Some internal functions throw warnings or notices and seem unavoidable. A good case is parse_url. The point of parse_url is to take apart a URL and tell me the parts. Until recently, the only way to validate a URL was a regex. You can now use filter_var with the FILTER_VALIDATE_URL filter. But, in the past, I would use parse_url to validate the URL. It worked as the function returns false if the value is not a URL. But, if you give parse_url something that is not a URL, it throws a PHP Warning error message. The result is I would use the evil @ to suppress errors from parse_url. Long story short, you get errors on PHP systems. And you don't need to ignore them.

In the past we just logged them and then had the log emailed daily. On some unregular schedule we would look through them and fix stuff to be more solid. But, when you start having lots of traffic, one notice error on one line could cause 1,000 error messages in a short time. So, we had to find a better way to deal with them.

Step 1: Managing the aggregation

The first thing we did was modify our error handler to log both human readable logs to the defined php error log and to write a serialzed (json_encode actually) version of the error to a second file. This second file makes parsing of the error data super easy. Now we could aggregate the logs on a schedule, parse them and send reports as needed. We have fatal errors monitored more aggressively than non-fatal errors. We also get daily summaries. The one gotcha is that PHP fatal errors do not get sent to the error handler. So, we still have to parse the human readable file for fatal errors. A next step may be to have the logs aggregated to a central server via syslog or something. Not sure where I want to go with that yet.

Step 2: Monitoring

Even with the logging, I had no visibility into how often we were logging errors. I mean, I could grep logs, do some scripting to calculate dates, etc. Bleh, lots of work. We recently started using Circonus for all sorts of metric tracking. So, why not PHP errors? Circonus has a data format that allows us to send any random metrics we want to track to them. They will then graph them for us. So, in my error handler, I just increment a counter in memached every time I log an error. When Circonus polls me for data, I just grab that value out of memcached and give it to them. I can then have really cool graphs that show me my error rate. In addition I can set alerts up. If we start seeing more than 1 error per second, we get an email. If we start seeing more than 5 per second, we get paged. You get the idea.

The Twitterverse:

I polled my friends on Twitter and two different services were mentioned. I don't know that I would actually send my errors to a service, but it sounds like an interesting idea. One was HopToad. It appears to be specifically for application error handling. The other was loggly which appears to be a more generic log service. Both very interesing concepts. The control freak in me would have issues with it. But, that is a personal problem.

Most others either manually reviewed things or had some similar notification system in place. Anyone else have a brilliant solution to monitoring PHP errors? Or any application level errors for that matter?

P.S. For what it is worth, our PHP error rate is about .004 per second. This includes some notices. Way too high for me. I want to approach absolute zero. Lets see if all these new tools can help us get there.

Boycott NuSphere. They are spammy spammers

brianlmoon — Thu, 28 Oct 2010 00:41:12 -0500

It took a lot for me to finally write this post. I tweeted about it a while back. It seems NuSphere needs more business. They have resorted to spamming people to promote their PhpED product. I have gotten emails to email addresses that:

I know are not on any mailing list
Are on web pages as plain mailto: anchor tags for good reasons.

In one case, it was the security@phorum.org address we have on the site to make it easy for people to report any security related issues. We get all kind of spam because of this, but it is worth it to have an easy access address for security issues.

In the other case, the email addresses used were on the dealnews.com jobs page. It was the addresses that are used to accept resumes.

Now, today, I started getting them to my personal inboxes.

Apparently, they sent it out the first time with a misspelling, so they had to send it out again!?!?

Now, the NuSphere people posted a tweet that claims the emails are not coming from them. That it's an independent marketer that "have permission to sell our products". First, the links in the email to go the NuSphere site, not a 3rd party. I would think an independent would want to take me to their site. Second, if that is the case, revoke the permission you have given them to sell your products. Don't use a marketing/sales agreement as a shield to allow people in the PHP Community to be spammed in your name.

PHP Community, please do no support NuSphere. They are spammers, directly or indirectly. If they do this to promote their product, how will they find a way to hose their user base later on?

PHP 5.3 and mysqlnd - Unexpected results

brianlmoon — Tue, 03 Aug 2010 08:00:00 -0500

I have started seriously using PHP 5.3 recently due to it finally making it into Portage. (Gentoo really isn't full of bleeding edge packages people.) I have used mysqlnd a little here and there in the past, but until it was really coming to my servers I did not put too much time into it.

What is mysqlnd?

mysqlnd is short for MySQL Native Driver. In short, it is a driver for MySQL for PHP that uses internal functions of the PHP engine rather than using the externally linked libmysqlclient that has been used in the past. There are two reasons for this. The first reason is licensing. MySQL is a GPL project. The GPL and the PHP License don't play well together. The second is better memory management and hopefully more performance. Being a performance junky, this is what peaked my interests. Enabling mysqlnd means it is used by the older MySQL extension, the newer MySQLi extension and the MySQL PDO driver.

New Key Feature - fetch_all

One new feature of mysqlnd was the fetch_all method on MySQLi Result objects. At both dealnews.com and in Phorum I have written a function to simply run a query and fetch all the results into an array and return it. It is a common operation when writing API or ORM layers. mysqlnd introduces a native fetch_all method that does this all in the extension. No PHP code needed. PDO already offers a fetchAll method, but PDO comes with a little more overhead than the native extensions and I have been using mysql functions for 14 years. I am very happy using them.

Store Result vs. Use Result

I have spoken in the past (see my slides and interview: MySQL Tips and Tricks) about using mysql_unbuffered_query or using mysqli_query with the MYSQLI_USE_RESULT flag. Without going into a whole post about that topic, it basically allows you to stream the results from MySQL back into your PHP code rather than having them buffered in memory. In the case of libmysqlclient, they could be buffered twice. So, my natural thought was that using MYSQLI_USE_RESULT with fetch_all would yield the most awesome performance ever. The data would not be buffered and it would get put into a PHP array in C instead of native code. The code I had hoped to use would look like:

$res = $db->query($sql, MYSQLI_USE_RESULT);
$rows = $res->fetch_all(MYSQLI_ASSOC);

But, I quickly found out that this does not work. For some reason, this is not supported. fetch_all only works with the default which is MYSQLI_STORE_RESULT. I filed a bug which was marked bogus. Which I put back to new because I really don't see a reason this should not work other than a complete oversight by the mysqlnd developers. So, I started doing some tests in hopes I could show the developers how much faster using MYSQLI_USE_RESULT could be. What happened next was not expected. I ended up benchmarking several different options for fetching all the rows of a result into an array.

Test Data

I tested using PHP 5.3.3 and MySQL 5.1.44 using InnoDB tables. For test data I made a table that has one varchar(255) column. I filled that table with 30k rows of random lengths between 10 and 255 characters. I then selected all rows and fetched them using 4 different methods.

In addition, I ran this test with mysqlnd enabled and disabled. For mysqli_result::fetch_all, only mysqlnd was tested as it is only available with mysqlnd. I ran each test 6 times and threw out the worst and best result for each test. FWIW, the best and worst did not show any major deviation for any of the tests. For measuring memory usage, I read the VmRSS value from Linux's /proc data. memory_get_usage() does not show the hidden memory used by libmysqlclient and does not seem to show all the memory used by mysqlnd either.

So, that is what I found. The memory usage graphs are all what I thought they would be. PDO has more overhead by its nature. Storing the result always uses more memory than using it. mysqli_result::fetch_all uses less memory than the loop, but more than directly using the results.

There are some very surprising things in the timing graphs however. First, the tried and true method of using the result followed by a loop is clearly still the right choice in libmysqlclient. However, it is a horrible choice for mysqlnd. I don't really see why this is so. It is nearly twice as slow. There is something really, really wrong with MYSQLI_USE_RESULT in mysqlnd. There is no reason it should ever be slower than storing the result and then reading it again. This is also evidenced in the poor performance of PDO (since even PDO uses mysqlnd when enabled). PDO uses an unbuffered query for its fetchAll method and it too got slower. It is noticably slower than libmysqlclient. The good news I guess is that if you are using mysqlnd, the fetch_all method is the best option for getting all the data back.

Next Steps

My next steps from here will be to find some real workloads that I can test this on. Phorum has several places where I can apply real world pages loads to these different methods and see how they perform. Perhaps the test data is too small. Perhaps the number of columns would have a different effect. I am not sure.

If you are reading this and have worked on or looked at the mysqlnd code and can explain any of it, please feel free to comment.