Ramblings of a web guy (Tag: linux)

The rise of the GLAMMP stack

brianlmoon — Fri, 22 May 2009 11:34:39 -0500

First there was LAMP. But are you using GLAMMP? You have probably not heard of it because we just coined the term while chatting at work. You know LAMP (Linux, Apache, MySQL and PHP or Perl and sometimes Python). So, what are the extra letters for?

The G is for Gearman - Gearman is a system to farm out work to other machines, dispatching function calls to machines that are better suited to do work, to do work in parallel, to load balance lots of function calls, or to call functions between languages.

The extra M is for Memcached - memcached is a high-performance, distributed memory object caching system, generic in nature, but intended for use in speeding up dynamic web applications by alleviating database load.

More and more these days, you can't run a web site on just LAMP. You need these extra tools (or ones like them) to do all the cool things you want to do. What other tools do we need to work into the acronym? PostgreSQL replaces MySQL in lots of stacks to form LAPP. I guess Drizzle may replace MySQL in some stacks soon. For us, it will likely be added to the stack. Will that make it GLAMMPD? We need more vowels! If you are starting the next must use tool for running web sites on open source software, please use a vowel for the first letter.

dealnews is hiring a Systems Administrator

brianlmoon — Wed, 20 Aug 2008 18:02:15 -0500

We are hiring! dealnews is looking for a full time systems administrator. The developers have been sharing the sys admin load for over 10 years now. But, we really need a dedicated person now. If you are interested, see our jobs page.

Replication is much better than cold backups

brianlmoon — Sun, 17 Aug 2008 01:58:47 -0500

So, I wrote about the begining of our wild database issues. Since then, I have been fighting a cold, coaching little league football and trying to help out in getting our backup solutions working in top shape. That does not leave much time for blogging.

Never again will we have ONLY a cold backup of anything. We were moving nightly full database dumps and hourly backups of critical tables over to that box all day long. Well, when the filesystem fails on both the primary database server and your cold backup server, you question everything. A day after my marathon drive to fix the backup server and get it up and running, the backup mysql server died again with RAID errors. I guess that was the problem all along. In the end, we had to have a whole new RAID subsystem in our backup database server. So, my coworker headed over to the data center to pull the all nighter to get the original, main database server up and running. The filesystem was completely shot. ReiserFS failed us miserably. It is no longer to be used at dealnews.

Well, today at 6:12PM, the main database server stops responding again. ARGH!! Input/Ouput errors. That means RAID based on last weeks experience. We reboot it. It reports memory or battery errors on the RAID card. So, I call Dell. Our warranty on these servers includes 4 hour, onsite service. They are important. While on the phone with Dell, I run the Dell diagnostic tool on the box. During the diagnostic test, the box shuts down. Luckily, the Dell service tech had heard enough. He orders a whole new RAID subsystem for this one as well.

There is one cool thing about the PERC4 (aka, LSI Megaraid) RAID cards in these boxes. They write the RAID configuration to the drives as well as on the card. So, when a new blank RAID card is installed, it finds the RAID config on the drives and boots the box up. Neato. I am sure all the latest cards do it. It was just nice to see it work.

So, box came up, but this time we had Innodb corruption. XFS did a fine job in keeping the filesystem in tact. So, we had to go from backups. But, this time we had a live replicated database that we could just dump and restore. We should have had it all along, but in the past (i.e. before widespread Innodb) we were gun shy about replication. We had large MyISAM tables that would constantly get corrupted on the master or slave and would halt replication on a weekly basis. It was just not worth the hassle. But, we have used it for over a year now in our front end database servers with an all Innodb data set. As of now, only two tables in our main database are not Innodb. And I am trying to drop the need for a Full-Text index on those right now.

So, here is to hoping our database problems are behind us. We have replaced almost everything in one except the chassis. The other has had all internal parts but a motherboard. Kudos to Dell's service. The tech was done with the repair in under 4 hours. Glad to have that service. I recommend it to anyone that needs it.

An Introduction to MySQL - Birmingham, AL

brianlmoon — Mon, 16 Jun 2008 19:00:25 -0500

I am giving a talk titled "An Introduction to MySQL" here in Birmingham, AL on June 21, 2008 at 3PM.

I love living in Alabama. I was born and raised in Huntsville. However, Birmingham has always seemed a bit behind in technology compared to what I do for a living. There is good reason. The industry here is medical, banking, industrial and utilities. I don't really want my doctors keeping my medical records in an alpha release of anything. Same goes for my banking and utilities. But, as this page shows, the companies here are catching up. So, I am happy to present MySQL to as many people as I can in this town. Hopefully I will help some folks that have not been exposed to MySQL or any open source for that matter.

The event is part of our local Linux user group's (BALU) planned events.

Example my.cnf files

brianlmoon — Tue, 06 May 2008 10:17:00 -0500

UPDATE: There are some examples being added at the MySQL Forge now.

When I first started installing MySQL for myself, it was quite handy to have the example my.cnf files in the source package. I was a noob to the MySQL configuration. Even after I became more experienced, I would use them as a starting point. However, I now find that they are so behind the times they are not as useful. Here are some of the comments from the files.

my-small.cnf

# This is for a system with little memory (<= 64M) where MySQL is only used
# from time to time and it's important that the mysqld daemon
# doesn't use much resources.

my-medium.cnf

# This is for a system with little memory (32M - 64M) where MySQL plays
# an important part, or systems up to 128M where MySQL is used together with
# other programs (such as a web server)

my-large.cnf

# This is for a large system with memory = 512M where the system runs mainly
# MySQL.

my-huge.cnf

# This is for a large system with memory of 1G-2G where the system runs mainly
# MySQL.

I end up using the large or huge files as a starting point for every server I set up by hand. The small and medium should be renamed underpowered and teeny-tiny. Who has less than 64MB of RAM on a server now? Can you even buy sticks of memory that small in any modern system? Most come with 256MB sticks minimum. And they never come with just one stick.

I will use the large example as a starting point for a server that has 2GB of RAM and will be running an entire site on one server. I use huge for any server that runs only MySQL. And even then, most of them have 4GB of RAM or more.

I don't know if anyone at MySQL has plans on tweaking these files or not. Perhaps those good guys at the MySQL Performance Blog or Percona could create some example my.cnf files. I could put some out there, but I fear their sole purpose would be for someone to point out what I am doing wrong. =P Hey, they work for me. Hmm, maybe this would make a good MySQL Forge section. A whole area of user contributed my.cnf files. They could be architecture specific and everything. What runs best on Solaris? Linux? BSD? Windows? 32-bit? 64-bit?

One thing I would for sure like to see is example files for InnoDB dominant servers. Most of our servers all run primariy InnoDB tables. None of these above examples covers InnoDB. They have comments, but no preconfigured values. I have seen more than one server using InnoDB tables without any custom configuration in their my.cnf. In the end that is the fault of the server admin/owner no doubt.

What do you say? Anyone up for a MySQL Forge section for my.cnf files?

Local: Best practices for SQL backed web applications

brianlmoon — Thu, 06 Mar 2008 13:08:35 -0600

When
Tuesday, March 11, 2008 at 12:00 PM

Where
BizTech
515 Sparkman Drive
Huntsville , AL 35816

Details
Brian Moon of dealnews.com will be discussing best practices for writing database backed web based applications. Many users teach themselves SQL and programming on the web. Other developers may have experience in enterprise desktop applications. No matter what your background, there are common mistakes made when deploying web based applications that use a database.

Also, at this event, we will be giving away two copies of NuSphere's PhpED. Plus, everyone who attends can purchase any NuSphere product at 50% off.

Lunch will be served at this event.

Speaking at MySQL Conference 2008

brianlmoon — Wed, 20 Feb 2008 15:12:01 -0600

I had mentioned a while back that I submitted three proposals for the 2008 MySQL Conference. Well, two were accepted.

From one server to a cluster

In the last 10 years, dealnews.com has grown from a single shared hosting account to an entire rack of equipment. Luckily, we started using PHP and MySQL very early in the company's history.

From the early days of growing a forum to surviving Slashdotting, Digging and even a Yahoo! front page mention, we have had to adapt both our hardware and software many times to keep up with the growth.

I will discuss the traps, bottlenecks, and even some big wins we have encountered along the way using PHP and MySQL. From the small scale to using replication and even some MySQL Cluster. We have done many interesting things to give our readers (and our content team) a good experience when using our web site.

MySQL hacks and tricks to make Phorum fast

Phorum is the message board software used by MySQL. One reason they chose Phorum was because of its speed. We have to use some tricks and fancy SQL to make this happen. Things we will talk about in this session include:

Using temporary tables for good uses.

Why PHP and MySQL can be a bad mix with large data sets.

What mysqlnd will bring to the table with the future of PHP and MYSQL.

How Phorum uses full text indexing and some fancy SQL to make our search engine fast.

Forcing MySQL to use indexes to ensure proper query performance.

You can find my conference page here. (as Terry would say, me, me, me!)

Apache Worker and PHP

brianlmoon — Wed, 13 Feb 2008 01:09:32 -0600

The PHP manual basically tells you not to use Apache 2 with a threaded MPM and PHP as an Apache module. In general, it may be good advice. But, at dealnews.com, we have found it very valuable.

Apache threaded MPMs

Well, first, what is an MPM? It stands for Multi-Processing Module. It is the process model that Apache uses for its children process. Each request that comes in is handed to a child. Apache 1 used only one model for this, the prefork model. That uses one process per Apache child. The most commonly used threaded MPM is the Worker MPM. In this MPM, you have several processes that run multiple threads within it. This is the one I will be talking about. You can read more on Apache MPMs at the Apache web site.

Huge memory savings

With the Apache prefork or even FastCGI, each apache/php process allocates its own memory. Most healthy sites I have worked on use about 15MB of memory per apache process. Code that has problems will use even more than this. I have seen some use as much as 50MB of RAM. But, lets stick with healthy. So, a server with 1GB of RAM will only realistically be able to run 50 Apache processes or 50 PHP children for FastCGI if each uses 15MB or RAM. That is 750MB total. That leaves just 256MB for the OS and other applications. Now, if you are Yahoo! or someone else with lots of money and lots of equipment, you can just keep adding hardware. But, most of us can't do that.

As I wrote above, the worker MPM apache uses children (processes) and threads. If you configure it to use 10 child processes, each with 10 threads you would have 100 total threads or clients to answer requests. The good news is, because 10 threads are in one process, they can reuse memory that is allocated by other threads in the same process. At dealnews, our application servers use 25 threads per child. In our experience, each child process uses about 35MB of RAM. So, that works out to about 1.4MB per thread. That is 10% the usage for a prefork server per client.

Some say that you will run out of CPU way before RAM. That was not what we experienced before switching to worker. Machines with 2GB of RAM were running out of memory before we hit CPU as a bottleneck due to having just 100 Apache clients running. Now, with worker, I am happy to say that we don't have that problem.

Building PHP for best success with Worker

This is an important part. You can't use radical extensions in PHP when you are using worker. I don't have a list of extensions that will and won't work. We stick with the ones we need to do our core job. Mainly, most pages use the mysql and memcached extension. I would not do any fancy stuff in a worker based server. Keep a prefork server around for that. Or better yet, do funky memory sucking stuff in a cron job and push that data somewhere your web servers can get to it.

Other benefits like static content

Another big issue you hear about with Apache and PHP is running some other server for serving static content to save resources. Worker allows you to do this without running two servers. Having a prefork Apache/PHP process that has 15MB of RAM allocated serve a 10k jpeg image or some CSS file is a waste of resources. With worker, like I wrote above, the memory savings negate this issue. And, from my benchmarks (someone prove me wrong) Apache 2 can keep up with the lighttpds and litespeeds of the world in terms of requests per second for this type of content. This was actually the first place we used the worker mpm. It may still be a good idea to have dedicated apache daemons running just for that content if you have lots of requests for it. That will keep your static content requests from over running your dynamic content requests.

Some issues we have seen

Ok, it is not without problems (but, neither was prefork). There are some unknown (meaning undiagnosed by us) things that will occasionally cause CPU spikes on the servers running worker. For example, we took two memcached nodes offline and the servers that were connected to them spiked their CPU. We restarted Apache and all was fine. It was odd. We had another issue where a bug in my PHP code that was calling fsockopen() without a valid host name and a long timeout would cause a CPU spike and would not seem to let go. So, it does seem that bad PHP code makes the server more sensitive. So, your mileage may vary.

As with any new technology, you need to test a lot before you jump in with both feet. Anyone else have experience with worker and want to share?

One last tip

We have adopted a technique that Rasmus Lerdorf had mentioned. We decide how many MaxClients a server can run and we configure that number to always run. We set the min and max settings of the Apache configuration the same. Of course, we are running service specific servers. If you only have one or two servers and they run Apache and MySQL and mail and dns and... etc. you probably don't want to do that. But, then again, you need to make sure MaxClients will not kill your RAM/CPU as well. I see lots of servers that if MaxClients was actually reached, they would be using 20GB of RAM. And, these servers only have 2GB of RAM. So, check those settings. If you can, configure it to start up more (all if you can) Apache process rather than a few and make sure you won't blow out your RAM.

Managing two data centers

brianlmoon — Thu, 07 Feb 2008 13:53:38 -0600

Call it paranoia. Call it being prepared. Whatever your stance, we are considering using more than one data center for dealnews.com. It is not a capacity issue. We can keep growing our current data center without a problem. But, stories of power outages and power outages we have experience have us wanting to explore the idea.

Here is the problem. No one in our company has experience with this. And, there does not seem to be any resources on the internet talking about this. Our problems are not so much with managing the data between the two. The problem is failover and how to deal with one data center being out. Here are some of the ideas that have been thrown on to the wall.

Round Robin DNS

This was the first idea. It seems simple enough. We have two data centers. We publish different DNS for each data center and traffic goes to each one. The problem here is that it is, well, random.

Global Traffic Management

There are devices that "balance" traffic across multiple different locations. But, I am unsure how those deal with outages at one of the locations. It seems like there is still one point of failure.

BGP Routing

This is the biggest mystery to me. I know what it is. I know what it means. I have no idea how to deploy this type of solution. I understand that you can "move" your IP addresses with routing changes. But, that means running routers. Where are these routers? Does this happen at some provider? Is there a provider that handles this? Does that mean that all of our data centers are with one provider? I think one more peace of mind feature of this is that we would not be tied to just one vendor. So, if one vendor had major issues or there was some legal troubles (we lived through the dot come boom and bust) we would have security in knowing we had other equipment that was not affected.

Is there something else? Are we being way paranoid? Maybe it is not cost effective in the end. I/we have no idea really. Anyone out there that has knowledge on this subject?

GoDaddy support is awful

brianlmoon — Thu, 07 Feb 2008 11:43:31 -0600

Luckily, I don't have personal experience with them. But, based on the 2 to 3 users per week that come to the Phorum support forums and IRC, they have the worst support of any host on the internet.

Example 1

A user comes to the forums having trouble with his Phorum install. In the user's words, GoDaddy tells him "they couldnt help me costumize my scripts because it wasnt their job". In this case "customize" meant filling in the MySQL permissions into the Phorum config files. In the end, GoDaddy had to move him to a Linux hosting account. They claimed that the Windows hosting accounts do not support PHP. However, they are clearly wrong about their own hosting as this all started because the user received a PHP error about not connecting to MySQL.

Example 2

This user found that GoDaddy is using MySQL 4 on their servers. Their web site does not mention a version anywhere. So, users are locked in to a hosting plan before knowing this.

Example 3

This happened today in IRC. It was much like the first example. In this case, GoDaddy support told him "permissions are set via ftp". Um, MySQL permissions are set via FTP? The user had a MySQL server name. For some reason it did not exist. So, either he typed it wrong or they gave him the wrong server name. Either way, their support should recognize this and be able to help their clients.

Example 4

This is less a support issue and just plain crappy of them IMO. GoDaddy does not allow the creation of temporary tables. The Phorum search engine makes use of them to save lots of CPU and memory on the PHP side. Luckily for their users, Thomas felt sorry enough for them to make a module that used good old fashioned slow LIKE queries. So, that will work until their account is shut off because they have search queries clogging up the database servers.

So, if you are on GoDaddy, I feel for you. From where we sit, it really seems like they do not provide very good support. We end up having to support their users for things that GoDaddy should be able to answer.

How NOT to get support and how to turn the other cheek.

brianlmoon — Mon, 04 Feb 2008 10:23:30 -0600

So, I checked my email this morning and found this jewel:

I might use Phorum if you brain deads knew how to upload or download your files via FTP. Your documentation has no order to it, its all a mess. I even dropped a release level to see if it was just that release. Ill give you a clue, DONT TRANSFER YOUR FILES VIA AUTO, EXPECIALY YOUR TXT FILES. TRANSFER THEM IN ASCII MODE ONLY, THIS INCLUDES YOUR PHP FILES. Then you just f---ing* MIGHT get readable files. Now you might say hey wait a min, we have full documentation on our web site, but you forget, someone has to open the sample.config.php file and read the crap that resides there.

* edited for content

Should I respond? If so, how? I decided to respond in as nice a way as I could.

I normally don't answer direct support emails. Neither do I normally answer very angry emails. However, I view this as an educational experience.

Judging by your email, I would say you are using Notepad on Windows to edit and read files. That is mistake number one. Notepad only reads one file format: Windows text files. Windows natively uses a CRLF for it's line endings. It is the only operating system that does so. Notepad is the only application on the Windows platform that only reads that format. If you would use Wordpad instead, this would not have been a problem for you. For some reading on the subject, you may want to read:

http://en.wikipedia.org/wiki/Newline
http://www.cs.toronto.edu/~krueger/csc209h/tut/line-endings.html

Because PHP scripts are most commonly deployed on a Linux platform, the Unix line feed (LF or \n) is best for PHP applications. Here are some suggestions for some great text editors for Windows.

TextPad - http://www.textpad.com/
Metapad - http://www.liquidninja.com/metapad/
PSPad - http://www.pspad.com/en/

I hope this has helped educate you on the world of new lines and how real programming works. In the future, a kind word in the forums would be much more appreciated than an email like this. Not all people would be as kind as I am being and want to help you grow.

What do you think? Should have just let this guy go? Should have been as ugly to him as he was to me?

This is so geeky and cool!

brianlmoon — Fri, 01 Feb 2008 16:57:26 -0600

My coworker Rob is building a little project at home. After some bad experience with cheap black box servers, he got some used Dell's. Now, he has done this. Heh, is pretty cool.

MacBreak missing a demographic

brianlmoon — Fri, 02 Nov 2007 23:32:25 -0500

I listen to the MacBreak Weekly podcast every week. I have liked Leo Laporte ever since The ScreenSaver days. He has several good regulars on the show and mixes in topical guests as well. However, I think there is a demographic of Mac user that the show is missing.

There is a growing audience of new Mac users in the tech sector. Just to the O'Reilly Open Source Conference and take a count. Mac OS X and the switch to the Intel platform has brought about the most stable, easy to use *nix based desktops and laptops the world has ever seen. I was a long time Windows user. I made fun of Mac users. I even ran Linux on a Dell laptop for a while. Boy, that was fun. Nothing like waking up and having to edit X configurations so you can work. Apple just got it right. I can run my AMP stack on my MacBook Pro with no problems. And the Mac UI is wonderful. I am becoming a fan boy.

So, on this weeks MBW, Leo and the panel were talking about Leopard. The subject came up about the best new feature for home users, power users and mac software developers. There was neither anyone on the show that fit into my demographic of Mac user nor did anyone mention us. No mention of Apache 2.2 or PHP 5.2. No mention of a much improved Terminal.app. No mention of a built in SSH Agent that works with your keychain. If you work with Linux/BSD server, you use Terminal almost as much as any other application.

So, Leo, please include this growing Mac demographic into your discussions. There has to be someone out there in our space that is as knowledgable as Andy Ihnatko and Scott Bourne are about their topics. Merlin comes close when he is there, but I think he is still and old school Mac user that happens to have gotten into the geekier parts of Mac OS X.

Still, love the show. Keep up the good work.

O'Reilly Open Source Conference Day Two

brianlmoon — Fri, 27 Jul 2007 03:03:09 -0500

So, day two was the cool keynote day. Day one keynotes were from Tim O'Reilly (not that he is not cool) and the vendors sponsors. The Intel building blocks stuff was neat, but most of it was vendor stuff IMO.

Today we had the "cool thing to here and see, but I proabably won't use it" keynote. It was The Processing Development Environment. It was really cool. You can read more about it at processing.org.

The next keynote was hard for me to follow. There were no slides he stood behind the podium the whole time. Gnat seemed to love it as he all told us in IRC. You can read the guys blog at overcomingbias.com. It was basically about overcoming the biases you have.... I think.

Interestingly, (speaking of bias) the next keynote was from Microsoft. Coincidence? According to the speaker, MS (or at least this guy) is really trying to make some Open Source stuff. Time will tell. Also, they are "working" with the OSI to get their licensing approved as Open Source licenses. As somone in IRC said, its a win/win from them. If they don't get approved, they can just blame the OSI for being inflexible. Nate kind of put him on the spot about patents after his talk. He handled it well and kind of rode the fence.

The last keynote was, for me, the pay off keynote. Its the one I will remember from this year the most. It was about branding. The poor guy did not have his slides due to technical issues and still did a great job. You can read Steve's blog at steve-yegge.blogspot.com. Maybe he will post the slides.

I attended a couple of good sessions today. One was about caching, mostly with APC. But, if you stripped down the APC stuff and just took some of his concepts, you could apply some of it to lots of caching methods. The talk was given by Gopal Vijayaraghavan of Yahoo! I don't have a URL for the site where his slides may be. If I find it, I will post it.

Another one was about legacy PHP code. I didn't agree with 100% of what he was saying, but if you are in the boat he described, anything is better than where you are. The guys site is clintonrnixon.net. Hopefully he will put of the slides and maybe a blog post about it.

The last talk that I want to tell you about was from Amy Hoy. She gave the "When Interface Design Attacks!" again this year. Just like last year, it was brilliant. There were new topics like web 2.0. I was happy to see that the Phorum 5.2 template I have been working on (emerald) already included many of her recommendations. I guess she rubbed off on me last year. Amy has started her own consulting company. If we need a usability and/or interface design help again (bleh, the last one was less than exciting) I will push for using her for sure. Check out her site (linked above) for more stuff from her.

The day (and conference really) ended with parties. We went to the Sourceforge Open Source Awards party. phpBB won best tool for communication. Gag me with a chicken bone. I guess it has a large install base. But, MySpace has lots of users too. That does not mean its not a black eye on the internet. Ok, MySpace is worse than phpBB for sure. But, c'mon, I write Phorum. I am biased (see above keynote =). It was a popularity contest and I guess there are more kiddies to vote for them than say Pidgin which is what I voted for. With all the trouble they have had with their name, I wonder if "Gaim" would have gotten more votes. (see other keynote on branding =). The phpBB team may need to see the branding keynote from this morning. It talked about how it takes a generation to change perception about a brand. Most people I talked to here have a negative reaction to the phpBB brand.

The rest of the night we just hung out at the party hosted by Jive Software. We use OpenFire from those guys. I am not a big Java user on the server. Its just one more different thing to admin in a company that is 99% GNU C apps on the servers. But, Openfire does a damn good job with XMPP.

In closing, O'Reilly Open Source Convention was great. I got some great ideas of stuff we should be doing. I got confirmation of things that we are already doing. And most important, IMO, we got to share with others how we solve problems. As Gopal said in his caching talk, sometimes is better to stop doing stuff and tell others what you are doing (paraphrase).

O'Reilly Open Source Conference Day One

brianlmoon — Thu, 26 Jul 2007 02:52:34 -0500

Day one is complete. Portland is great as always. Its really day 1 1/2 since we got in at 1PM yesterday. That allowed us to go to the MySQL/Zend party last night. Great party by those guys. Touched based with old friends and made some new ones.

I kind of session hopped today. Of note, I attended Andi Gutmans PHP Security talk which really had little to do with PHP. Like Larry Wall's onion metaphor, Andi presented an onion metaphor for security. I stopped in for a while on the SOLR talk. It looks neat. I like that it is a REST interface to Lucene. If we were not using Sphinx already I might take a longer look. But, we like Sphinx and, SOLR and Lucene are Java. Not that there is anything wrong with that, we just don't use Java a lot, so its just one more thing that would be out of the norm. I admit I spent a good bit of time in what is being called the "hallway track" working on some code. Work does not stop just because you are at a conference.

I got to hang out with Jay Pipes of the MySQL Community team a good bit. We talked about the MySQL forums (which or course runs Phorum) and how they want to improve them. They would like to see tagging, user and post rating and some other things. Some good things will come out of that. Hopefully they have some of the tagging stuff done already at MySQL Forge and can contribute that code to Phorum, saving us time.

I hosted the Caching for fun and profit BoF. It was not packed, but it was a good time. The MySQL BoF was at the same time, so we lost some folks to that I am sure. They had beer and pizza. Brad Fitzpatrick did come by and contribute. Thanks Brad. It was mostly the same stuff you get on the memcached mailing list. "How do we expire lots of cache at once?" Questions about different clients. Stuff like that. It kind of turned into a memcached BoF, but I tried to share the dealnews experience with the attendees including our MySQL Cluster pushed caching.

I have met many readers of both dealnews and this blog (hi to you) while here. Glad to know that both my professional work and my personal work are of use to folks. The demographic at this conference is dead on for dealnews. Maybe I can get them to sponsor it next year. That would be cool.

I say every year that I want to present "next year". Something always keeps me from doing it. Usually its just not having time to prep for it. By the time I think about it, the call for papers has passed. I really want to get it done this time. We shall see I suppose.

We went to the Sun party tonight. It was a good time. There was beer that was free as in beer. More hanging with friends and talking about all kinds of stuff. Now, all you Slashdotters sit down. I saw people from the PostgreSQL and MySQL teams drinking beer and having fun together. OMGWTFBBQ!!!1!! See, the people that really matter in those projects don't bicker and fight about which is better. They just drink beer and have a good time together.

Anyhow, I will blog more after day 2. There won't be a day 3 as I have to catch an 11:30 flight back home. That is usually how it goes. Not sure why they book anything on Friday really. Even O'Reilly has its "after party" on Thursday night. Its late, and I need sleep.

MySQL cluster and all dump 1000

brianlmoon — Mon, 23 Jul 2007 01:00:00 -0500

So, I had written a while back: "We currently have a DataMemory of 4GB and IndexMemory of 2GB. Based on the crude methods we have to monitor it, I think we are at about 40% capacity." Boy, I was wrong.

After that post, I started looking at this more in detail because we were considering buying more RAM "just in case". I figured out how to use the super secret command "all dump 1000". The command is not documented in the MySQL documentation that I could find. I did find it in the NDB API documentation before writing this post however. Not sure why I could not find it before.

For those that still don't know how to use it, simply type "all dump 1000" from your management console. Then check your cluster log files on the management server. You will see something like this:



2007-07-20 17:56:47 [MgmSrvr] INFO     -- Node 2: Data usage is 46%(90672 32K pages of total 196608)

2007-07-20 17:56:47 [MgmSrvr] INFO     -- Node 2: Index usage is 1%(2006 8K pages of total 131104)

2007-07-20 17:56:47 [MgmSrvr] INFO     -- Node 3: Data usage is 46%(90724 32K pages of total 196608)

2007-07-20 17:56:47 [MgmSrvr] INFO     -- Node 3: Index usage is 1%(2039 8K pages of total 131104)

2007-07-20 17:56:48 [MgmSrvr] INFO     -- Node 4: Data usage is 43%(86153 32K pages of total 196608)

2007-07-20 17:56:48 [MgmSrvr] INFO     -- Node 4: Index usage is 1%(2016 8K pages of total 131104)

2007-07-20 17:56:48 [MgmSrvr] INFO     -- Node 5: Data usage is 46%(90672 32K pages of total 196608)

2007-07-20 17:56:48 [MgmSrvr] INFO     -- Node 5: Index usage is 1%(2007 8K pages of total 131104)

Anyhow, I ran that and low and behold I saw that we were at about 93% capacity. As you can see above, we have made some changes. This got me to really digging as to what the difference could have been. As far as I can tell, our use of TEXT fields in our data was causing the issue. We have several fields in our data structure that hold data larger than 256 bytes. However, they hardly ever are more than 600 bytes. Based on what I read, it seems that the data after the first 256 bytes would be stored in 2k chunks. So, our litte ~300 byte extras were being stuck into 2k chunks. That caused a huge amount of wasted space.

So, the first thing I realized was that we could use 6GB DataMemory instead of 4GB. The machines have 8GB of RAM in them. We were using only about 200MB of IndexMemory. That leaves well over 1GB of the system.

The second thing I did was carefully analyze our data. I went back to school and started counting bytes. Its easy to get lazy as a developer these days. We just use a data type we know will work and it really is no big deal. But, not in this case. I changed int to mediumint and some to even small and tiny. I realized that there were two TEXT fields I could eliminate altogether. I looked at the varchar fields and made them only as long as they need to be. In the end, we ended up at the numbers you see above, about 46% capacity or 2.76GB. We were at 3.72GB before. The removal of the two TEXT fields may have had more to do with the improvement than anything.

This was all with MySQL 5.0. 5.1 will bring better storage of the large data. For example, all data over 256 bytes, for a row, will be stored together in 2k chunks rather than one 2k chunk per column. That will likely save us 1GB of DataMemory. The other feature of 5.1 is Disk Data Tables. We are currently testing those and I will blog about them when I have more information. Early numbers look good though. We have just set our largest table (the one with the TEXT fields) as a disk data table and our DataMemory is down to just 280MB. Yeah, that is an MB. Our COO is a bit concerned with the performance hit we may see with data on disk.

Oh, one more pointer about all dump 1000. For a quick way to grab my current usage, I use this command from my prompt (on a Mac):

ssh user@mgmserver "ndb_mgm -e 'all dump 1000' && tail -n 1000 /var/lib/mysql-cluster/ndb_1_cluster.log | fgrep usage | tail -n 8";;

Use at your own risk. It may make your server catch fire. =)

RAID is dying?

brianlmoon — Fri, 08 Jun 2007 12:38:20 -0500

There is a bunch of posts on Planet MySQL this week about RAID. This comment from Kevin Burton really kind of made me go "huh?".

You’re thinking too low level. Who cares if the disk fails. The entire shard is setup for high availability. Each server is redundant with 1-2 other boxes (depends on the number of replicas). If you have automated master promotion you’ll never notice any downtime. All the disks can fail in the server and a slave will be promoted to a new master.

Monitoring then catches that you have a failed server and you have operations repair it and put it back into production as a new slave.

Someone has to think low level. The key phrase in there is you have operations repair it and put it back into production as a new slave. This tells me all I need to know. Kevin later states that his company does in fact not operate their own equipment, but uses a provider for all their hosting.

At this point, I think this is a philosophy argument and not a real world application argument at this point. Sure, I guess if I am Google or Yahoo I can do this. But, for the mass majority of web sites running out there, having 4 data centers and "operations" at your beck and call is not a reality. For real people, having a server go down is pain in the ass. Why should I want to spend a full day of labor rebuilding a server because a $200 part broke or just got corrupted. It takes 10 minutes to start a rebuild and maybe another 10 minutes to install a new drive if the rebuild fails.

His other argument is about performance. Sure, its debatable whether RAID is faster or slower. It probably depends on the application. If your RAID is a bottle neck for your application, then you need to address that. For us, its far from the bottleneck so why bother with the downtime of having one (of our 30, not 1000) servers down.

BTW, would you rather admin 30 servers or 1000? I think 30.

I should add that we only use RAID on servers that are used for data storage. Losing data sucks. For web servers we don't use RAID. They do fit the model that Kevin describes. We have a lot of them. If one goes down, its ok. Maybe Kevin's application can fit all its data on one web node. Don't know. I just know its right for us and I don't see a future where I won't want it on our servers. We are even using RAID in our MySQL Cluster servers. Why? Because I don't want to have to wait a day to get a storage node back up and running for a $200 part.

Five months with MySQL Cluster

brianlmoon — Thu, 07 Jun 2007 12:11:49 -0500

So, the whole world changed at dealnews when Yahoo! linked us. We realized that our current infrastructure was not scaling very well. We had to make a change.

The Problem

Even though we were using all sorts of cool techniques, the server architecture was really still just a bunch of web servers all serving the same content. In addition to that, our existing systems as the time used a pull method. When a request came in, memcache was checked, if the data was not there, it was fetched from our main MySQL server. So, when there is no data in the cache or when it expires, this was very bad. Like when Yahoo! hit us. Some cache item would expire and 60,000 users would hit a page and each page would try and create the cache item.

The Solution

I was tasked with two things. Find a way to handle something like the Yahoo burst and finding a way to store the data we need to generate our web pages that was highly available and would scale. For bursting, I wrote a proxy using apache, mod_rewrite, php and memcached. I have reasons I did it this way that are not relevent to this post. Maybe more on that later.

For the data solution, I considered several things: MySQL replication, writing my own replicating memcached client, and other exotic ideas. One of the semi-exotic ideas for us was MySQL Cluster. We had not used it at all. Some things about it made us gun shy. But, we tested it and were very happy with the results.

Initial Test

With the help of Gentoo, getting a cluster up and running was really, really easy. In fact, it seemed too easy. We ran a cluster on some dev boxes at first. We did some generic testing using the PHPTestSuite from the guys at MySQL Performance Blog. What we found was that while the cluster appeared slower at low concurrent connections, it scaled much better than InnoDB (our prefered storage engine) when the concurrent connections grew.

Application Testing

So, we moved to the next step, testing our application. We discovered early on with cluster that we would have to redesign our application. Our DB was highly relational. Almost no data could be put on the site without data from other tables. We used a lot of joins. We learned (later) that joins in the cluster are not a good idea. Neither are sub-selects. So, we wrote some proof of concept scripts for our application. We were very happy. Very few issues were found. Nothing anywhere near show stopping.

Installation

We ordered our servers. Six new Dell dual-core, dual processor Opterons with a lot of memory. Two would become SQL nodes and the other four would be storage nodes. Our data set is not that large compared to a lot of companies. So, we configured the cluster with 4 replicas. Our main goal is high availability and scalability. I could find nothing in my tests or in the manual that indicated this would be bad for scalability and it should be great for HA.

We rewrote our application (basically, our public web site) to use the new cluster and its new table design. We hit our first snag when we tried to seed the data in the cluster. We got errors from the cluster about its transaction logs not being big enough to handle the inserts. Through the manual, forum posts, the mailing list archives and some blogs I was able to find the correct settings for our needs. I remembered back when I first installed the cluster thinking it was too easy. I now realize that getting a cluster running is easy. Making it run well, is a whole other story.

The second snag was with joins. Our test bed for the cluster was not a cluster. We used a group of servers using InnoDB to test against. That was a mistake. Joins did not work at all with the cluster. We had to back up, rewrite some code and redo some tables. In the end, the design is probably faster on InnoDB or cluster.

Everyday Use

We started using the cluster for every day use about a month ago. I guess 5 months is not bad for starting from nothing to live in production. We have been slowly moving applications to it. We take care each time to monitor the cluster and see that its not throwing new errors. So far, so good. We have about 80% of our page views (40% of our page views are our front page) and about 50% of our end user applications using the cluster now. We are doing caching at the proxy level for a lot of this. But, when tested, the new architecture is much more reliable even without the caching proxies. Some things like our forums will never translate to the cluster. But, they have their own dedicated systems already and are non-critical for our business. They could be shut down if there was a problem with them.

Administration

MySQL Cluster is a whole new animal. Its not like monitoring mysqld, apache or other stuff we already use. It took me a while to get the hang of rolling restarts, brining nodes up and down after crashes, etc. We have had just one crashed node since we switched over to production use. The cluster stayed up and kept serving content. We have written a Nagios monitor to keep track of the nodes' status. It uses ndb_mgm and reports any problems to us.

Feedback

Now, as the title says, I have only been using MySQL Cluster for 5 months. If you are reading this and have more experience and are thinking "What a moron!", please tell me. We are still learning.

Update:

Ronald Bradford had some questions on his blog for me. I figured I would just answer them here.

You didn’t mention any specific sizes for data, I’d be interested to know, particularly growth and how you will manage that?

We currently have a DataMemory of 4GB and IndexMemory of 2GB. Based on the crude methods we have to monitor it, I think we are at about 40% capacity. We are using MySQL Cluster purely as a data store for content on our web site. So, we can trim the data store down significantly. If it does not appear on the site, its not in cluster.

You also didn’t mention anything about Disk? MySQL Cluster may be an in-memory database but it does a lot of disk work, and having appropriate disk is important. People overlook that.

Yes, we have U320 15k SCSI drives. We do use RAID 1 on our servers contrary to some opinions. We see a lot of drive failures. About one every 4 months. Sucks to lose a whole machine just because a $200 drive failed.

You didn’t mention anything about timings? Like how does backups for example compare now to previously.

Well, we don't currently back up the cluster data as it is being copied from our main database already. Maybe that is a mistake, I don't know. But, I can't come up with a reason to backup data that is just a copy of another database server. Also, I have written a PHP class that does parallel writing to multiple servers using transactions. Everything we write to the cluster also gets written to an "oh shit" mysql server that users InnoDB. So, in the event we have a total cluster failure, F5 BIG-IP load balancers will send mysql traffic to the InnoDB server.

You didn’t mention version? 5.1 (not GA) is significant improvement in memory utilization due to true varchar support, saving a lot of memory, but as I said not yet production software.

Yeah, I am drooling over 5.1. But, we are using current Gentoo stable, 5.0.38 I believe. 5.1 looks superior in many many ways. I can't wait to upgrade.

Quick script to check user bandwidth usage

brianlmoon — Sun, 15 Apr 2007 01:51:01 -0500

A buddy needed a quick report to see if one of his users was slamming his site. I got a little carried away and wrote a PHP script (plus some awk and grep) to make a little report for him. I am sure it is full of bugs and will bring your server crashing down. So, use at your own risk.



$ ./bwreport.php -h

Usage: bwreport.php [-d YYYYMMDD] [-u URI] [-i HOST/IP] [-r REGEXP] [-v]

-d YYYYMMDD     Date of the logs to parse. If no date provided, yesterday assumed.

-i IP/HOST      Only report log lines with IP/HOST for host part of log line

-r REGEXP       Only report log lines that match REGEXP.  Should be a valid grep regexp

-u URI          Only report log lines with URI match to URI

-v              Verbose mode

http://www.phorum.org/downloads/bwreport.php.gz

Vixie Cron and the new US DST

brianlmoon — Wed, 14 Mar 2007 15:51:34 -0500

So, the new DST changes in the US caused a small stir among system administrators recently. We got all of our servers updated and verified they were working before the even. Or so we thought.

I noticed today that our 3PM Eastern newsletters arrived in my inbox at 3PM alright. However, I am in Central time. My immediate assumption was that we missed the server that sends that email out. Logging in I found the time correct on the server. It had received the appropriate updates thanks to portage. So, what happened? I looked at /etc/crontab and all was fine. I then looked at the system log where cron jobs are logged. Oddly, that log line said the job started at 15:00. I knew that was not correct. I started looking around at other cron jobs on other servers, especially ones that wrote files to disk. Sure enough, every server I checked was doing things an hour behind except one. It just so happens we had restarted cron on this server however last week. We had to shut it down to keep it from causing some errors while we updated the server.

So, long story short, we restarted cron on all the servers. That seems to be the only thing needed. These servers have been running (and crond as well) long before even the announcement of the DST change. I guess vixie cron can't handle time zone rules changing after he starts. For the record, we are using the latest stable version in Gentoo's Portage.

I wish I was as cool as DJB

brianlmoon — Sun, 14 Jan 2007 22:25:09 -0600

I should throw up a fanboy alert right here. You have been warned. =)

I was reading a heated discussion about security (no link, MARC is read only right now) on the PHP internals list this past week. In the middle of it, Zeev Suraski writes: "No remotely accessible software has a perfect track record, perhaps other than qmail." For those that don't know, qmail is the second most used MTA (Mail Transfer Agent) on the internet. It was written by Dan J. Bernstein (DJB). DJB, as I like to refer to him around the office, is a professor at University of Illinois at Chicago. You can read all about him at his web site.

The basis for Zeev's comments is DJB's qmail security guarantee. As Dan writes, he was fed up with security holes in sendmail. So, he decided to do something about them. He just avoided the whole app and wrote his own. Besides being rock solid, the application takes a very intuitive (to me) approach to internet mail. DJB believes in separating jobs into separate daemons that run with separate users and permissions. One daemon accepts incoming mail and puts it in a queue. Another reads that queue and then decides if it is an internal or external delivery. I then hands that to an local or remote daemon responsible for those jobs. Everything has its job. Nice and neat.

DJB did not stop there. He also wrote (IMO) the best darn DNS server ever in djbdns. Like qmail, it has a security guarantee. It uses the same logical design as qmail. Honestly, DNS propagation is a bit of mystery to me. Bind zone files confused the hell out of me. But, djbdns is easy as pie to use.

I have been lucky enough to use qmail for my entire career. The first host I ever signed up with used qmail and it was all I ever wanted to use. When our current systems administrator, a life long sendmail and bind user, came to work for us, I showed him qmail and djbdns. It took a little while, but now he will never go back. Even with the occasional annoyance, its better than the alternative to him.

You do have to adjust to the DJB style. His applications don't have the normal configure, make, make install setup. He is a FreeBSD user. At times there are errors on non FreeBSD systems that are in his opinion flaws of those systems and not qmail. He is usually right. At the least, you can't say he is wrong. djbdns for example does propagate data between hosts "automatically" like bind does. You have to rsync the data somehow yourself. That is a turn off at first for some. Then they realize how much more control that will give them.
He is very diligent when it comes to sticking strictly to whatever RFC exist for each daemon he writes. One guy I know complains that qmail is the only MTA that requires the \r\n at the end of emails. qmail will reject them straight away. As you soon discover, there is a huge community of "patches" to make qmail do all sorts of things. There is a patch for that "feature" as well.

For more on qmail, see qmail.org, a collection of patches, documents and add-ons. The most popular of those documents is likely Life with qmail. It is sort of a noobs guide to qmail.

For more on djbdns, see DJB's page about it.

Is Yahoo!ed a word?

brianlmoon — Fri, 22 Dec 2006 00:03:41 -0600

Everyone has heard of being slashdotted or maybe dugg. But have you ever been Yahoo!ed?

Phones started beeping, mayhem ensued. The first thing we looked at was the database. Is some MyISAM table locked? Is there a hung log processor running? The database was busy, but it looked odd. The web servers were going nuts.

As we soon discoverd, we (dealnews.com) were mentioned in an article on Yahoo!. At 5Pm Eastern, that article made it to be the featured article on the Yahoo! front page. It was there for an hour. We went from our already high Christmas traffic of about 80 req/s for pages and 200 req/s for images to a 130 req/s for pages and 500 req/s for images. We survived with a little tinkering. We have been working on a proxy system and this sounded like as good a time as any to try it out. Thanks to the F5 BIG-IP load balancers, we could send all the traffic from Yahoo! to the proxy system. That allowed us to handle the traffic. Just after 6PM, Yahoo! changed the featured article and things returned to normal.

Until 9PM. It seems the earlier posting by Yahoo! must not have went out to all their users. Because at 9PM the connections came back with a vegance. We started hitting bottleneck after bottleneck. We would up one limit and another would bottleneck would appear. The site was doing ok during this time. Some things like images were loading slow. That was a simple underestimation of having our two image servers set to only 250 MaxClients. Their load was nothing. We upped that and images flowed freely once again. Next we realized that all our memcached daemons were maxed out on connections. So, again, we up that and restart them. That's fixed now. Oh, now that we are not waiting on memcached, the Apache/PHP servers are hitting their MaxClients. We check the load and the servers are not stressed. So, up those limits go. The proxy servers were not doing well using a pool of memcached servers. So, we set them to use just one server each. This means several copies of the same cache, but better access to the data for each server. After all that, we were handling the Yahoo! load.

In the end, it was 300 req/s for pages and 3000 req/s for images. It lasted for over 2 hours. The funny thing is, we have been talking all week about how to increase our capacity before next Christmas. Given our content, this is our busy time. Our traffic has doubled each December for the last 3 years. At one point, during the Yahoo! rush, the incoming traffic was 10MB/s. A year and a half ago, that was the size of our whole pipe with our provider. Luckily we increased that a while back.

The silver lining is that I got to see this traffic first hand for over 2 solid hours. This will help us to design our systems to handle this load and then some all the time in the future. In some ways it was a blessing.

Digg? Slashdot? They can bring traffic for sure. We have been on both several times. But wow, just getting in the third paragraph of an article that is one page deep from the Yahoo! front page can bring you to your knees if you are not ready. But, in this business, I will do it again tomorrow. Bring it on.

Update: Yahoo! put the article on their front page again on the 26th. Both our head sys admin and I were off. No phones went off. We handled 400 req/s for the front pages and 1500 req/s for images. This lasted for 3 hours. Granted, some things were not working. You could not change your default settings for the front page for example. But, all in all, the site performed quite well.

dealnews.com is hiring

brianlmoon — Fri, 11 Aug 2006 13:00:33 -0500

From our jobs page:

As a dealnews web developer, you will help maintain our current stable of deal and price-tracking web sites, and build new features and new web sites as we continue to grow. You'll be part of a small, fast-moving team of developers that are involved at every stage of product development, from concept to rollout.

We use Gentoo, Apache, PHP and MySQL. While that does not need to be your expertise, it is a plus. We have a little Perl and Python thrown in as well. You will need to code on a non-Windows system as we run our development environment on our local machines. Currently, we all use Macs.

A big plus, however, is disc golf. We play weekly as an um, team building excercise. Yeah, team building.

OSCON 2006

brianlmoon — Tue, 01 Aug 2006 13:30:43 -0500

Well, I am back from OSCON 2006. Portland is a really nice city. I hope they keep this conference there forever. I hosted a BoF on memcached. That was fun. There were a lot of people there. I did not expect that. They had something this year that was either new or I had not seen before. It was called OSCAMP. They fall below a BoF in terms of plans and organization. A neat idea. You just go in the room, post on the wall what you are interested in and what time you will be back to talk about it. Other folks show up and you kick it. No approval, no rules. I tried to set one up, but I got in there too late to get my topic noticed.

Rasmus Lerdorf gave a good talk about debugging PHP and making your life easy when using PHP. His title was different, but that was what I took from it. I was glad to see someone near the top of the PHP food chain say that speed matters to him. Having him work at Yahoo! helps too.

As usual, Robert "r0ml" Lefkowitz was great. His keynote compared Open Source to vegetables. He is a great speaker.

I did not have a epiphany at this conference. It's still good to go. There will not always be something to blow you away every time you go to one of these things. It was good to just get away and hob nob with other open source developers.