Living in the Prove It Culture

Tue, Mar 6, 2012 10:45 PM
Engineering cultures differ from shop to shop. I have been in the same culture for 13 years so I am not an expert on what all the different types are. Before that I was living in Dilbert world. The culture there was really weird. The ideas were never yours. It was always some need some way off person had. A DBA, a UI "expert" and some product manager would dictate what code you wrote. Creativity was stifled and met with resistance.

I then moved to the early (1998) days of the web. It was a start up environment. In the beginning there were just two of us writing code. So, we thought everything we did was awesome. Then we added some more guys. Lucky for us we mostly hired well. The good hires where type A personalities that had skills we didn't have. They challenged us and we challenged them. On top of that, we had a CEO who had been a computer hacker in his teens. So, he had just enough knowledge to challenge us as well. Over the years we kept hiring more and more people. We always asked in the interview if the person could take criticism and if they felt comfortable defending their ideas. We decided to always have a white board session. We would ask them questions and have them work it out on a white board or talk it out with us in a group setting. The point of this was not to see if they always knew the answer. The point was to see how they worked in that setting. Looking back, the hires that did not work out also did not excel in that phase of the interview. The ones that have worked out always questioned our methods in the interview. They did not belittle our methods or dismiss them. They just asked questions. They would ask if we had tried this or that. Even if we could quickly explain why our method was right for us, they still questioned it. They challenged us.

When dealing with people outside the engineering team, we subconsciously applied these same tactics. The philosophy came to be that if you came to us with an idea, you had to throw it up on the proverbial wall. We would then try to knock it down. If it stuck, it was probably a good idea. Some people could handle this and some could not. The ones that could not handle that did not always get their ideas pushed through. It may not mean they were bad ideas. And that is maybe the down side of this culture. But, it has worked pretty well for us.

We apply this to technology too. My first experience on Linux was with RedHat. The mail agent I cut my teeth on was qmail. I used djbdns. When Daniel Beckham, our now director of operations, came on, he had used sendmail and bind. He immediately challenged qmail. I went through some of the reasons I prefered it. He took more shots. In the end, he agreed that qmail was better than sendmail. However, his first DNS setup for us was bind. It took a few more years of bind hell for him to come around to djbdns.

When RedHat splintered into RedHat Enterprise and Fedora, we tried out Fedora on one server. We found it to be horribly unstable. It got the axe. We looked around for other distros. We found a not very well known distro that was known as the ricer distro of the Linux world called Gentoo. We installed it on one server to see what it was all about. I don't remember now whose idea it was. Probably not mine. We eventually found it to be the perfect distro for us. It let us compile our core tools like Apache, PHP and MySQL while at the same time using a package system. We never trusted RPMs for those things on RedHat. Sure, bringing a server online took longer but it was so worth it. Eventually we bought in and it is now the only distro in use here.

We have done this over and over and over. From the fact that we all use Macs now thanks to Daniel and his willingness to try it out at our CEO's prodding to things like memcached, Gearman, etc. We even keep evaluating the tools we already have. When we decided to write our own proxy we discounted everything we knew and evaluated all the options. In the end, Apache was known and good at handling web requests and PHP could do all we needed in a timely, sane manner. But, we looked at and tested everything we could think of. Apache/PHP had to prove itself again.

Now, you might think that a culture of skepticism like this would lead to new employees having a hard time getting any traction. Quite the opposite. Because we hire people that fit the culture, they can have a near immediate impact. We have a problem I want solved and a developer that has been here less than a year suggested that Hadoop may be a solution, but was not sure we would use it. I recently sent this in an email to the whole team in response to that.
The only thing that is *never* on the table is using a Windows server. If you can get me unique visitors for an arbitrary date range in milliseconds and it require Hadoop, go for it.
You see, we don't currently use Hadoop here. But, if that is what it takes to solve my problem and you can prove it and it will work, we will use it.

Recently we had a newish team member suggest we use a SAN for our development servers to use as a data store. Specifically he suggested we could use it to house our MySQL data for our development servers. We told him he was insane. SANs are magical boxes of pain. He kept pushing. He got Dell to come in and give us a test unit. Turns out it is amazing. We can have a hot copy of our production database on our dev slices in about 3 minutes. A full, complete copy of our production database in 3 minutes. Do you know how amazing that is? Had we not had the culture we do and had not hired the right person that was smart enough to pull it off and confident enough to fight for the solution, we would not have that. He has been here less than a year and has had a huge impact to our productivity. There is talk of using this in production too. I am still in the "prove it" mode on this. We will see.
I know you will ask how our dev db works, here you go:
1. Replicate production over VPN to home office
2. Write MySQL data on SAN
3. Stop replication, flush tables, snapshot FS
4. Copy snapshot to a new location
5. On second dev server, umount SAN, mount new snapshot
6. Restart MySQL all around
7. Talk in dev chat how bad ass that is

We had a similar thing happen with our phone system. We had hired a web developer that previously worked for a company that created custom Asterisk solutions. When our propietary PBX died, he stepped up and proved that Asterisk would work for us. Not a job for a web developer. But he was confident he could make it work. It now supports 3 offices and several home bound users world wide. He also had only been here a short time when that happened.

Perhaps it sounds like a contradiction. It may sound like we just hop on any bandwagon technology out there. But no. We still use MySQL. We are still on 5.0 in fact. It works. We are evaluating Percona 5.5 now. We tried MySQL 5.1. We found no advantage and the Gentoo package maintainer found it to be buggy. So, we did not switch. We still use Apache. It works. Damn well. We do use Apache with the worker MPM with PHP which is supposedly bad. But, it works great for us. But, we had to prove it would work. We ran a single node with worker for months before trusting it. Gearman was begrudgingly accepted. The idea of daemonized PHP code was not a comforting one. But once you write a worker and use it, you feel like a god. And then you see the power. Next thing you know, it is a core, mission critical part of your infrastructure. That is how it is with us now. In fact, Gearman has went from untrusted to the go to tech. When someone proposes a solution that does not involve Gearman, someone will ask if part of the problem can be solved using Gearman and not whatever idea they have. There is then a discussion about why it is or is not a good fit. Likewise, if you want to a build a daemon to listen on a port and answer requests, the question is "Why can't you just use Apache and a web service?" And it is a valid question. If you can solve your problem with a web service on already proven tech, why build something new?

This culture is not new. We are not unique. But, in a world of "brogramming" where "engineers" rage on code that is awesome before it is even working and people are said to be "killing it" all the time, I am glad I live in a world where I have to prove myself everyday. I am the most senior engineer on the team. And even still I get shot down. I often pitch an idea in dev chat and someone will shoot it down or point out an obvious flaw. Anyone, and I mean anyone, on the team can question my code, ideas or decisions and I will listen to them and consider their opinion. Heck, people outside the team can question me too. And regularly do. And that is fine. I don't mind the questions. I once wrote here that I like to be made to feel dumb. It is how I get smarter. I have met people that thought they were smarter than everyone else. They were annoying. I have interviewed them. It is hard to even get through those interviews.

Is it for everyone? Probably not. It works for us. And it has gotten us this far. You can't get comfortable though. If you do foster this type of culture, there is a risk of getting comfortable. If you start thinking you have solved all the hard problems, you will look up one day and realize that you are suffering. Keep pushing forward and questioning your past decisions. But before you adopt the latest and greatest new idea, prove that the decisions your team makes are the right ones at every step. Sometimes that will take a five minute discussion and sometimes it will take a month of testing. And other times, everyone in the room will look at something and think "Wow that is so obvious how did we not see it?" When it works, it is an awesome world to live in.
9 comments
Gravatar for Edmar

Edmar Says:

Great read, healthy culture. Thanks!

Gravatar for Dan

Dan Says:

Hi Brian,

Interesting stuff. But I was wondering if you had found any flaws in this 'prove it' culture. Do some people feel miffed? Are you slowed down by endless discussions? Do you pass up tech that later turned out to would have been a good choice? Do you have to regularly water the culture of the place? Is it dependent on an understanding CEO/management team?

Not trying to attack your culture at all, just wanted to know what the tradeoffs you think you have made (with the understanding that your previous culture was Dilbertesque and so not a good comparison).

Gravatar for Tony Marston

Tony Marston Says:

I agree with your "prove it!" philosophy 100%. Sometimes the only way to make progress is to challenge the established view and try new things. Someone who cannot stand to have their viewpoint challenged, or be prepared to justify or defend it in the face of legitimate criticism, would be the weak point in any team. I regularly come across people whose only answer to the question “why do you do it that way?” is either “because we have always done it this way” or “because this is the way it should be done”. That shows that they can only follow the herd and are incapable of evaluating new or alternative ideas.

I think of myself as results-oriented instead of rules-oriented, but sometimes this attitude does cause friction with others who are tied dogmatically to a single viewpoint. I have occasionally been called “arrogant”, “a heretic” and “not a team player” for daring to voice a view which is contrary to theirs, but as my methods regularly outperform those of my critics I dismiss their winging and whining as sour grapes. They tell me that my methods are “wrong” and “impure”, but how can they be if they work? They tell me that their methods are “right”, but how can they be if they are slower to develop and harder to maintain?

Gravatar for Tony Marston

Tony Marston Says:

This blog has a bug! The reason that I posted the same comment twice was because on the first attempt I received the error message "Page invalid, go back and try again"

Gravatar for Tony Marston

Tony Marston Says:

It happend again! The full error message is "The form information you submitted is invalid. Please go back and reload the page before submitting the form."

Is this because I'm using IE9?

Gravatar for Brian Moon

Brian Moon Says:

Tony, that would be the CSRF token failing. Odd that you are seeing that but the comment is making it through. I can't replicate that error in any browser.

Gravatar for Brian Moon

Brian Moon Says:

@Dan, all good and valid questions.

"Do some people feel miffed?"

Yes, some people outside the team have felt miffed before. I don't think anyone on the team has really felt miffed. Maybe disappointed their idea was not embraced, but everyone has success and everyone has rejections. So, it all evens out.

"Are you slowed down by endless discussions?"

No, hardly ever. It is a show me culture, not a tell me culture. We will give anyone the time to try things. And if their results show merit, it is obvious.

"Do you pass up tech that later turned out to would have been a good choice?"

Perhaps. But doing nothing is always an option. Too many people don't understand that. Not changing thing things is a valid choice in some circumstances. Re: the SAN. We evaluated iSCSI SAN tech several years ago and it was not ready. We are now starting to use it.

"Do you have to regularly water the culture of the place?"

Hmm, the elders of the team regularly instill the the idea in everyone if that is what you mean. It is not an effort. It is not on my calendar to remind people about it. It is very organic.

"Is it dependent on an understanding CEO/management team?"

Yes, if you have a CTO or CEO that actually cares what database you use for business reasons or demands things are written some language because they read about it on TechCrunch you can't have this culture. That is Dilbert land. Maybe it is the fact that the senior team members here have been here since the company was small. I don't know. We are trusted to make the correct decisions for the good of the company. Which is a whole other blog post. Playing with new stuff is fun, but it is not what we are paid to do. We are paid to help the company succeed. And sometimes that means playing with new tech to help the company.

Add A Comment

Your Name:


Your Email:


Your URL:


Your Comment: