Sun, Jan 14, 2007 10:25 PM
I should throw up a fanboy alert right here. You have been warned. =)
I was reading a heated discussion about security (no link, MARC is read only right now) on the PHP internals list this past week. In the middle of it, Zeev Suraski writes: "No remotely accessible software has a perfect track record, perhaps other than qmail." For those that don't know, qmail is the second most used MTA (Mail Transfer Agent) on the internet. It was written by Dan J. Bernstein (DJB). DJB, as I like to refer to him around the office, is a professor at University of Illinois at Chicago. You can read all about him at his web site.
The basis for Zeev's comments is DJB's qmail security guarantee. As Dan writes, he was fed up with security holes in sendmail. So, he decided to do something about them. He just avoided the whole app and wrote his own. Besides being rock solid, the application takes a very intuitive (to me) approach to internet mail. DJB believes in separating jobs into separate daemons that run with separate users and permissions. One daemon accepts incoming mail and puts it in a queue. Another reads that queue and then decides if it is an internal or external delivery. I then hands that to an local or remote daemon responsible for those jobs. Everything has its job. Nice and neat.
DJB did not stop there. He also wrote (IMO) the best darn DNS server ever in djbdns. Like qmail, it has a security guarantee. It uses the same logical design as qmail. Honestly, DNS propagation is a bit of mystery to me. Bind zone files confused the hell out of me. But, djbdns is easy as pie to use.
I have been lucky enough to use qmail for my entire career. The first host I ever signed up with used qmail and it was all I ever wanted to use. When our current systems administrator, a life long sendmail and bind user, came to work for us, I showed him qmail and djbdns. It took a little while, but now he will never go back. Even with the occasional annoyance, its better than the alternative to him.
You do have to adjust to the DJB style. His applications don't have the normal configure, make, make install setup. He is a FreeBSD user. At times there are errors on non FreeBSD systems that are in his opinion flaws of those systems and not qmail. He is usually right. At the least, you can't say he is wrong. djbdns for example does propagate data between hosts "automatically" like bind does. You have to rsync the data somehow yourself. That is a turn off at first for some. Then they realize how much more control that will give them.
He is very diligent when it comes to sticking strictly to whatever RFC exist for each daemon he writes. One guy I know complains that qmail is the only MTA that requires the \r\n at the end of emails. qmail will reject them straight away. As you soon discover, there is a huge community of "patches" to make qmail do all sorts of things. There is a patch for that "feature" as well.
For more on qmail, see qmail.org, a collection of patches, documents and add-ons. The most popular of those documents is likely Life with qmail. It is sort of a noobs guide to qmail.
For more on djbdns, see DJB's page about it.
Fri, Dec 22, 2006 12:03 AM
Everyone has heard of being slashdotted or maybe dugg. But have you ever been Yahoo!ed?
Phones started beeping, mayhem ensued. The first thing we looked at was the database. Is some MyISAM table locked? Is there a hung log processor running? The database was busy, but it looked odd. The web servers were going nuts.
As we soon discoverd, we (dealnews.com) were mentioned in an article on Yahoo!. At 5Pm Eastern, that article made it to be the featured article on the Yahoo! front page. It was there for an hour. We went from our already high Christmas traffic of about 80 req/s for pages and 200 req/s for images to a 130 req/s for pages and 500 req/s for images. We survived with a little tinkering. We have been working on a proxy system and this sounded like as good a time as any to try it out. Thanks to the F5 BIG-IP load balancers, we could send all the traffic from Yahoo! to the proxy system. That allowed us to handle the traffic. Just after 6PM, Yahoo! changed the featured article and things returned to normal.
Until 9PM. It seems the earlier posting by Yahoo! must not have went out to all their users. Because at 9PM the connections came back with a vegance. We started hitting bottleneck after bottleneck. We would up one limit and another would bottleneck would appear. The site was doing ok during this time. Some things like images were loading slow. That was a simple underestimation of having our two image servers set to only 250 MaxClients. Their load was nothing. We upped that and images flowed freely once again. Next we realized that all our memcached daemons were maxed out on connections. So, again, we up that and restart them. That's fixed now. Oh, now that we are not waiting on memcached, the Apache/PHP servers are hitting their MaxClients. We check the load and the servers are not stressed. So, up those limits go. The proxy servers were not doing well using a pool of memcached servers. So, we set them to use just one server each. This means several copies of the same cache, but better access to the data for each server. After all that, we were handling the Yahoo! load.
In the end, it was 300 req/s for pages and 3000 req/s for images. It lasted for over 2 hours. The funny thing is, we have been talking all week about how to increase our capacity before next Christmas. Given our content, this is our busy time. Our traffic has doubled each December for the last 3 years. At one point, during the Yahoo! rush, the incoming traffic was 10MB/s. A year and a half ago, that was the size of our whole pipe with our provider. Luckily we increased that a while back.
The silver lining is that I got to see this traffic first hand for over 2 solid hours. This will help us to design our systems to handle this load and then some all the time in the future. In some ways it was a blessing.
Digg? Slashdot? They can bring traffic for sure. We have been on both several times. But wow, just getting in the third paragraph of an article that is one page deep from the Yahoo! front page can bring you to your knees if you are not ready. But, in this business, I will do it again tomorrow. Bring it on.
Update: Yahoo! put the article on their front page again on the 26th. Both our head sys admin and I were off. No phones went off. We handled 400 req/s for the front pages and 1500 req/s for images. This lasted for 3 hours. Granted, some things were not working. You could not change your default settings for the front page for example. But, all in all, the site performed quite well.
Fri, Aug 11, 2006 01:00 PM
From our jobs page:
As a dealnews web developer, you will help maintain our current stable of deal and price-tracking web sites, and build new features and new web sites as we continue to grow. You'll be part of a small, fast-moving team of developers that are involved at every stage of product development, from concept to rollout.
We use Gentoo, Apache, PHP and MySQL. While that does not need to be your expertise, it is a plus. We have a little Perl and Python thrown in as well. You will need to code on a non-Windows system as we run our development environment on our local machines. Currently, we all use Macs.
A big plus, however, is disc golf. We play weekly as an um, team building excercise. Yeah, team building.
Tue, Aug 1, 2006 01:30 PM
Well, I am back from OSCON 2006. Portland is a really nice city. I hope they keep this conference there forever. I hosted a BoF on memcached. That was fun. There were a lot of people there. I did not expect that. They had something this year that was either new or I had not seen before. It was called OSCAMP. They fall below a BoF in terms of plans and organization. A neat idea. You just go in the room, post on the wall what you are interested in and what time you will be back to talk about it. Other folks show up and you kick it. No approval, no rules. I tried to set one up, but I got in there too late to get my topic noticed.
Rasmus Lerdorf gave a good talk about debugging PHP and making your life easy when using PHP. His title was different, but that was what I took from it. I was glad to see someone near the top of the PHP food chain say that speed matters to him. Having him work at Yahoo! helps too.
As usual, Robert "r0ml" Lefkowitz was great. His keynote compared Open Source to vegetables. He is a great speaker.
I did not have a epiphany at this conference. It's still good to go. There will not always be something to blow you away every time you go to one of these things. It was good to just get away and hob nob with other open source developers.
