Ramblings of a web guy

Open Source Web Design

brianlmoon — Mon, 17 Nov 2008 12:00:00 -0600

So, my wife told me that my site design was boring. Yeah, she was right. I am no designer. I just don't have that gene. But, during my work on Wordcraft, I came across some cool places to find designs that are relased under Open Source licenses.

Open Designs - This is arguably the the prettiest of the three. The search, however, is painfully slow because all results return on one page. I guess if you can wait, this is a plus as browsing is easier. Also, you can pick multiple colors and choose by license. They only list XHTML templates (at least as search options). That could be a turn off if you like HTML 4 like me.
Open Web Design - The site itself could use a design overhaul. But, the content is good. The search lets you choose primary and secondary color, a unique feature among these sites. Thumbnails are a bit small though.
Open Source Web Design - Their search is not as powerful as the others, but it does return very fast. The thumbnails are a nice size.

You will find the same content on all three sometimes. But, it comes down to browsing and searching.

I found my new design at one of those. Not sure which, I looked at a lot of them. I did not use the template's HTML exactly as I like HTML 4.0 and wanted a different sidebar than the original author. But, the design is the hard part. So, thanks for Deep Red.

Wordcraft 0.5 available

brianlmoon — Mon, 10 Nov 2008 08:00:00 -0600

Well, I blogged about Wordcraft the other day. I have just been running live on the software for 4 days now. Well, that post had no URI associated with it. It took me two days to figure this out. Oops. Welcome to eating my own dog food. So, running this live with actual users (and a host of bot spam attempts) I am learning a lot and making a lot of commits. So, I may very well roll once or twice a week for the first few weeks.

So, with that, I have packaged 0.5. There are 15 changes in this package. Some features, but mostly bug fixes. So, if you could use a simple blog, give it a try and help me debug it. If you do, please use the Google Code issue tracker. Maybe I can figure out how to have those things emailed to me.

Wordcraft, a simple PHP blogging application

brianlmoon — Thu, 06 Nov 2008 08:00:00 -0600

So, a while back, not sure when, I was listening to the P3 Podcast and Paul mentioned his dislike for Wordpress. He said he wished there was a simple blogging application. I am probably misquoting him horribly. It was an idea that I had been tinkering with. So, I started on Wordcraft in my spare time. Like super spare time. That time between the kids going to bed and me falling asleep. So, it took a while to get it to a usable state.

Up until now, I have used Wordpress.com for my blogging. It works quite well. You can get started quite quickly and it does what most people need. My wife uses Blogger for our family blog. It is, IMO, not as nice as Wordpress.com in some ways. But, it does allow you to edit your styles (for free) and such which is nice.

So, why would I want to reinvent the wheel? I am a control freak and rarely run other people's code. I know, it is a character flaw. I am working on it. So, what did I come up with?

I had some goals when I started on this.

Keep it simple.
Focus on what I am good at doing.

Keeping it simple

I use MySQL. I didn't try to make it work with every possible database. In fact, it only uses the mysqli PHP extenstion. The few objects (CAPTCHA) are all PHP 5 objects. I don't plan to worry about PHP 4. The templates don't use a template language. They use plain old PHP. The are scoped to protect template authors from global scope. There are only 6 files required to make a new template. There are just 589 lines of code in the forward facing scripts. The admin has 2,446.

What am I good at doing?

I write PHP/MySQL code that has to work fast for a living. It is what I get paid to do. I am not a designer. I am not a spam catching wizard. I don't write cool javascript widgets. So, I focused on the PHP/MySQL parts of the code. For templates, I used designs that are released under the Creative Commons license. I use Akismet and the CAPTCHA libraries from Phorum for spam catching. I used the YUI Rich Editor for the admin where I needed a WYSIWYG widget. I even link to the YUI sources that are hosted by Yahoo. No sense taking on that bandwidth or storage.

So, what does it do you ask? Well, here are some of the features:

WYSIWYG editing via YUI.
Comments with optional CAPTCHA and/or Akismet.
Custom pages can be created.
Tagging of posts
Custom publish dates
Automatic Pingback support
Friendly URL support with mod_rewrite
5 Templates in first release. Easy to build more.
Email notifications to authors

There are some things missing of course. Internationalization of both the admin and templates is a big one. There is no current search engine for blog posts. There is no "blog roll" type of feature. There is no date based archive. And I am sure there is more missing. And I am sure there are bugs.

But, if you would like to try out yet another PHP application, I welcome you to give it a try. The code is hosted at Google Code. It is a BSD licensed application.

PHP Appalachia Corrections

brianlmoon — Tue, 14 Oct 2008 23:03:00 -0500

Just got home finally from PHP Appalachia. I enjoyed meeting all the great people.

I presented about what I learned and how we deal with importing large amounts of CSV data into MySQL. I threw my idea onto the wiki at the last minute, made the slides while everyone ate breakfast and I had planned on researching it all (been a few years since I wrote it), but we had no reliable internet. Some claims I made and their corrections.

I said our largest file is about 1.8 million lines. WRONG. Actually it is about 4.6 million. I was correct however that it does finish importing and indexing in about 5 minutes.

I claimed I LOAD DATA INFILE to MyISAM first and then "insert into ... select from" into an InnoDB table for speed reasons. WRONG. In fact, I do that because I need to merge fields from the file sometimes into one field in the databaes. I could not find a way to do that with LOAD DATA INFILE. As to speed. I can't say either way as I have no solid data. Sounds like a good test. MyISAM probably still wins on a LOAD DATA INFILE into a blank, fresh table based on my experience.

Total rows currently indexed is 7.2 million. I did not make a claim, but I thought I would just mention that. I wanted to include that, but did not have Internet. (Damn you Hughes)

Deploying Scalable Websites with Memcached

brianlmoon — Fri, 03 Oct 2008 09:55:00 -0500

I spoke at the MySQL Conference and Expo this year about the architecture we have here at dealnews.com. After my talk, Jimmy Guerrero of Sun/MySQL invited me to give a webinar on how dealnews uses memcached. That is taking place next week, Thursday, October 09, 2008. It is a free webinar. We have used memcached in a variety of ways as we have grown. So, I will be talking about how dealnews used memcached in the past and present.

For more information, visit the MySQL web site.

strtotime() - The PHP, date swiss army knife

brianlmoon — Fri, 19 Sep 2008 22:02:00 -0500

Man, what did I do before strtotime(). Oh, I know, I had a 482 line function to parse date formats and return timestamps. And I still could not do really cool stuff. Like tonight I needed to figure out when Thanksgiving was in the US. I knew it was the 4th Thursday in November. So, I started with some math stuff and checking what day of the week Nov. 1 would fall on. All that was making my head hurt. So, I just tried this for fun.

strtotime("thursday, november ".date("Y")." + 3 weeks")

That gives me Thanksgiving. Awesome. It is cool for other stuff too. At its very basic, it can take a MySQL datetime field and turn it into a timestamp. Very handy for date calculations. It also understands RFC 2822 and ISO 8601 date formats. These are common in HTTP headers and some XML documents like RSS and Atom feeds. Also, PHP can output those two standard formats with the date() function. So, this makes them a good standards compliant way to pass full, timezone specific dates around.

Google Chrome and privacy

brianlmoon — Tue, 02 Sep 2008 14:39:00 -0500

So, Google Chrome is out. If you don't know, it's Google's new browser. I downloaded it on my Windows XP machine and tried it out. I found this curious thing in the options.

So, I thought, I will click "Learn more" to see what they are watching. I get this.

So, I unchecked the box. Let's hope the premature launch is the reason there is no more information out there.

UPDATE: The page comes up now and says:

Information that's sent to Google includes crash reports and statistics on how often you use Google Chrome features. When you choose to accept a suggested query or URL in the address bar, the text you typed and the corresponding suggestion is sent to Google. Google Chrome doesn't send other personal information, such as name, email address, or Google Account information.

So, if you use their suggestions, they know it. And it tracks what features you use. Hmm, I think I will disable.

Shrinking ibdata files after innodb_file_per_table

brianlmoon — Thu, 21 Aug 2008 23:54:00 -0500

Patrick Galbraith wrote in his blog about switching to innodb_file_per_table. For those that don't know about this setting, it places the data for the tables into .ibd files within the database dir instead of storing it in the ibdata files in the main datadir. This is useful if you don't want to babysit your innodb tablespace. At least, that was my main reason for wanting to use it. There is still dictionary data stored in the ibdata file(s) so you can't just remove them.

Anyhow, the delima Patrick wrote about is recovering the space used by the ibdata files in the datadir after you have converted the tables to one file per table. I commented on his blog, but thought it worth a full post to be sure others could find my solution.

Backup your data (cuz, you never know)

Convert all tables to MyISAM

Stop MySQL.

Delete ib* in the datadir

Restart MySQL. MySQL will recreate the files.

Convert all tables to InnoDB

It worked for me. Your mileage may vary. No warranty that you won't lose all your data. Try it on a dev server first.

The benefits of this were that the data was online while we converted the tables. The only downtime was while we shut down MySQL, removed the files and waited on MySQL to recreate a small 10MB ibdata file and the ib_logfiles. I am not 100% sure you have to remove the ib_logfiles, but I did for good measure. I just run with a single 10MB autoextend ibdata file. I think it is at 34MB or so on our main database server now.

dealnews is hiring a Systems Administrator

brianlmoon — Wed, 20 Aug 2008 18:02:00 -0500

We are hiring! dealnews is looking for a full time systems administrator. The developers have been sharing the sys admin load for over 10 years now. But, we really need a dedicated person now. If you are interested, see our jobs page.

Replication is much better than cold backups

brianlmoon — Sun, 17 Aug 2008 01:58:00 -0500

So, I wrote about the begining of our wild database issues. Since then, I have been fighting a cold, coaching little league football and trying to help out in getting our backup solutions working in top shape. That does not leave much time for blogging.

Never again will we have ONLY a cold backup of anything. We were moving nightly full database dumps and hourly backups of critical tables over to that box all day long. Well, when the filesystem fails on both the primary database server and your cold backup server, you question everything. A day after my marathon drive to fix the backup server and get it up and running, the backup mysql server died again with RAID errors. I guess that was the problem all along. In the end, we had to have a whole new RAID subsystem in our backup database server. So, my coworker headed over to the data center to pull the all nighter to get the original, main database server up and running. The filesystem was completely shot. ReiserFS failed us miserably. It is no longer to be used at dealnews.

Well, today at 6:12PM, the main database server stops responding again. ARGH!! Input/Ouput errors. That means RAID based on last weeks experience. We reboot it. It reports memory or battery errors on the RAID card. So, I call Dell. Our warranty on these servers includes 4 hour, onsite service. They are important. While on the phone with Dell, I run the Dell diagnostic tool on the box. During the diagnostic test, the box shuts down. Luckily, the Dell service tech had heard enough. He orders a whole new RAID subsystem for this one as well.

There is one cool thing about the PERC4 (aka, LSI Megaraid) RAID cards in these boxes. They write the RAID configuration to the drives as well as on the card. So, when a new blank RAID card is installed, it finds the RAID config on the drives and boots the box up. Neato. I am sure all the latest cards do it. It was just nice to see it work.

So, box came up, but this time we had Innodb corruption. XFS did a fine job in keeping the filesystem in tact. So, we had to go from backups. But, this time we had a live replicated database that we could just dump and restore. We should have had it all along, but in the past (i.e. before widespread Innodb) we were gun shy about replication. We had large MyISAM tables that would constantly get corrupted on the master or slave and would halt replication on a weekly basis. It was just not worth the hassle. But, we have used it for over a year now in our front end database servers with an all Innodb data set. As of now, only two tables in our main database are not Innodb. And I am trying to drop the need for a Full-Text index on those right now.

So, here is to hoping our database problems are behind us. We have replaced almost everything in one except the chassis. The other has had all internal parts but a motherboard. Kudos to Dell's service. The tech was done with the repair in under 4 hours. Glad to have that service. I recommend it to anyone that needs it.

Database nightmare

brianlmoon — Tue, 05 Aug 2008 09:39:00 -0500

Its 12:30AM (00:30 for you Euros). I am watching The Daily Show on Tivo. All is well. Then the phone beeps.

MySQL Main is critical

SSH? no.

Digi console? no.

About a week ago, we had a mysqldump file that was corrupt. We cleaned it up. My worst fears came to my mind. We tried power cycling it. It did not come back.

While my coworker was dealing with the facilily people, I worked on the backup server. Had to ensure the last full backup was in place and apply the incremental data. Suddenly, my SSH connection dies. OMG. THAT DUMB A** GUY POWER CYCLED THE WRONG BOX!!! --- FS corrupted. Damn you ReiserFS!

By now, it's 4AM. Tech took an hour to get to the rack. It is 20 feet from his cubicle. I get in the car. I am two hours away from the facility. No sleep. It is still dark. I play loud music. I talk to myself. I curse the guy that power cycled the wrong box. The sun comes up and it is easier to drive.

I sit here, waiting on the OS to finish installing so I can restore the backup and incremental data again. Hours of content lost. The content team is hand writing HTML that other developers are rsyncing around to the servers.

The good news? All the work done in 2007 to separate our front end and backend worked. The front end works fine (99%). Just no new content. Well, except for the hand done HTML.

Note to self: Get that main database replication working again. ASAP.

Where Drizzle fits in for me

brianlmoon — Thu, 24 Jul 2008 13:17:00 -0500

So, most of you have heard about Drizzle by now. For those that have not, you can check out many, many blog posts or the Launchpad page.

The thread on Slashdot about Drizzle was quite negative. Most misunderstand what Drizzle is about. SQLite is not a good solution when you have 100 web servers. Let me describe how it I would use it and maybe that will help some understand it.

When it comes to MySQL use, dealnews has two very different use cases. The first is an enterprise storage system that involves content creation, reporting and data warehousing. For that layer of our business, we are using more and more advanced features as they become available. We use triggers and stored procedures. We use complex data types for specific use cases. All those features are a big gain.

The other way that we use MySQL is for serving content to our readers. I have written about this before. For this purpose, we avoid joins, don't use any advanced features. We do use replication, indexes and intelligent queries. We don't (as one slashdot reader claimed) do all of our processing in the code. That would be stupid. If you do that you are ignorant. I will stop talking about that before this becomes a rant. I do believe in letting MySQL do my work for me.

This is where Drizzle fits in. To serve content, I don't need stored procedures, triggers, views or any of that other stuff. The whole database that the front end web servers use is basically a view. It is a denormalized, prepared version of the real data. I store objects. But, I have to be able to sort and filter the data in a way that SQL allows me to do. CouchDB sounds interesting. Maybe one day it will be there. It is sill in the optimization phase.

Now, some say that this is just MySQL 3.x all over again. Well, you clearly have not been listening to the really smart people that are working on Drizzle. They are doing more than just removing the 4.1 and 5.x features from MySQL. They are removing things that don't make sense for this use case. They are adding things that do make sense. They are replacing parts of the code base where there is a better library or way of doing it. At this point, they have no feature requirements to meet. They have no deadlines. They are making what they think the high volume web world and/or cloud computing needs. They are making it plugable: think Apache modules or PHP extensions. So, if you need feature XYZ that was yanked out, you can add it back in (hopefully) via the internal API. There is a lot more going on here than just removing "features".

So, I am cheering on the folks working on Drizzle. I have joined their community and will provide what feedback I can from userland. I am no C++ coder. I can read it. I can debug it. But, writing it or doing heavy lifting is not in my skill set. Hopefully I can contribute

Usability FAIL

brianlmoon — Tue, 22 Jul 2008 09:52:00 -0500

I can't be at OSCON this year. But my colleague Rob is and he just posted a usability post about, of all things, the Double Tree hotel where I am sure a lot of you are staying. Great stuff.

Spreading Open Source (sort of)

brianlmoon — Fri, 11 Jul 2008 21:25:00 -0500

So, I know I just had a kid, but I am at a friends house helping with some computer issues. This is the friend that took my wife to the hospital and sat with her until I got there, after all. She is also the friend that took my other five kids in while we were at the hospital. So, I owe her big time.

First a little backstory. A few years ago, I started installing Firefox and Thunderbird on my non-technical friend's computers. I would label the Firefox icon "Internet" and the Thunderbird icon "Email". This made it simple for them. I would also install OpenOffice on those machines that did not have the full Microsoft Office package and show them that it could do all the same things that they needed MS Office for.

Anyhow, I am helping this friend by installing XP Service Pack 3 and remove some malware that somehow got on here. While waiting on Windows, I notice that my usual pattern of installing FF, TB and OOO.org are all done on this machine. What is cool is that I have never used this computer before. This one is new to me. So, that means this friend sought out Firefox, Thunderbird and OpenOffice all on her own and installed them for her family the same way I always have.

Now, I am not naive enough to think that my friend suddenly understands Open Source. She is not using it because she wants to be a part of the open source movement. But, it does make me feel good to help spread open source if even from the user perspective. It is also a testament to those applications and how far they have come.

Hudson is born!

brianlmoon — Wed, 09 Jul 2008 19:21:00 -0500

Well, it was not as planned, but he is here. Our 6th child, Hudson Bennett Moon came into the world this morning at 8:53AM. All of our children have been born via C section. The plan was to come in at around 10AM on the 16th to deliver Hudson the same way. Well, yesterday my wife starting having some pains that the doctors did not like. They watched her overnight and decided this morning to go ahead and perform the C section this morning. It was a whirlwind. We waited all night to see what was going to happen. Then, at 8:25AM, the nurse came in and said that we were doing it at 8:45 and that my wife was to be wheeled in to the OR in 5 minutes. We franticlly called friends and family. None of them were there. We assumed (you know how that goes) that we would have an hour or something. No such luck. So, by 8:53 he was here and at 9:40 we were all in a room with baby and family. Everyone is healthy. Mom feels better than she did the last 3 weeks.

Click for a slideshow

Caching and TTL behavior

brianlmoon — Wed, 02 Jul 2008 23:56:00 -0500

So, I am working on MemProxy some. Mainly, I am trying to implement more of the Cache-Control header's many options. The one that has me a bit perplexed s-maxage. Particularly when combined with max-age.

s-maxage is the maximum time in seconds an item should remain in a shared cache. So, if s-maxage is set by the application server, my proxy should keep it for that amount of time at the most. Up until now, I have just been looking at max-age. But, s-maxage is the proper one for a proxy to use if it is present. I do not send the s-maxage through because this is a reverse proxy and, IMO, that is proper behavior for an application accelerating proxy. However, I do send forward the max-age value that is set by the application servers. If no max-age is set, I send a default as defined in the script. Also, if no-cache or no-store is set, I send those and a max-age of 0.

My problem arises when max-age is less than s-maxage. Up until now, I have sent a max-age back to the client that represents the time left for the cached item in my proxy's cache. So, if the app server sent back max-age=300 and a request comes in and the cache is found and the cache was created 100 seconds ago, I send max-age-200 back to the client. But, I was only using max-age before. Now, in cases where s-maxage is longer than max-age, I would come up with negative numbers. That is not cool. The easiest solution would be to always send the original max-age back to the client. But, that seems kind of lame.

So, my question is, if you are using an application (HTTP or otherwise) accelerator, what would you expect? If you application set a max-age of 300 would you always expect the end client to receive a max-age of 300? Or should it count down over time? The only experience I have is a CDN. If you watch CDN traffic, the max-age gets smaller and smaller over time until it hits 0. I have not tried sending an s-maxage to my CDN. I don't know what they would do with that. Maybe that is a good test.

UPDATE: Writing this gave me an idea. If the item will be in the proxy cache longer than the max-age ttl, send the full max-age ttl. Otherwise, send the time left in the proxy cache. Thoughts on that?

(thanks for being my teddy bear blogosphere)

Velocity Conference Roundup

brianlmoon — Tue, 01 Jul 2008 01:01:00 -0500

As I said before, I was invited to be on a panel at Velocity Conference. I was delighted to go. I had never been to San Francisco. I have been to Portland and Santa Clara several times. The panel was great. It was the Brian and photo sharing sites show. Seriously, it was me (dealnews.com), John Allspaw of Flickr, Don MacAskill of SmugMug and Farhan Mashraqi of Fotolog. Oh, there was also Shayan Zadeh of Zoosk, a social dating network and Michael Halligan, a consultant from BitPusher. We all had similar ideas. I told my Yahoo story. I told everyone that they should denormalize (or optimize as Farhan prefered) their data to improve performance. Others agreed. I have written about my methods for denormalizing normalized data before. (See pushed cache) Fun was had by all.

I mentioned John Allspaw above. He gave a talk on his own as well. It was good. The slides are on SlideShare. He and I see eye to eye on a lot of things. One thing he says in there that may shock a lot of people is to test using produciton. I agree fully. We could have never been sure our infastructure was ready last year without testing the production servers.

I also learned about Varnish at the conference. It is a super fast reverse proxy. It uses the virtual memory systems of recent kernels to store its cache. The OS worries about moving things from memory to disk based on usage. The claim is that the OSes are better at this than any programmer could do (without copying them of course). It is fast. The developers are proud. And by proud I mean cocky. I have been playing with it. As you know, I have my own little caching proxy solution. Varnish is much faster, as I expected. However, storing cache in memcached is very attractive to me. Varnish can't do that. It would likely slow it down a great deal. MemProxy does do that. Also, because MemProxy is written in PHP and my application layer is PHP, I can do things at the proxy layer to inspect the request and take action. Works well for my use. But, if you are using squid or mod_cache or something, you may want to give Varnish a look.

There was a good bit of information about the client side of performance. There were folks from Microsoft there talking about IE8. It looks like IE8 will catch up with the other browsers in a lot of ways. Yahoo talked about image optimization. Good stuff in there. I use Fireworks and it does a pretty good job of making small images. I am looking more into combining images and making image maps that use CSS. We use a CDN, but fewer connections is better for users.

There was also a lot of great debate. SANs rock! SANs suck! Rails Scales! Rails Sucks! The Cloud is awesome! The Cloud is a lie! (lots of cloud)

I had dinner both nights with guys from Six Apart. Good conversations were had. I don't know if I am a big vegan fan though. I mean, the food was good, but it all kinda tasted the same. Perhaps I ordered poorly. At dinner on Tuesday I met a guy going to work for Twitter soon. He is an engineer that hopefully will be another step toward getting them back to 100% again. Lets keep our fingers crossed.

They did announce that the conference would be held again next year. I am definitely going back. Probably two of us from dealnews will go. OSCON is fun. MySQL conference is too. But, more and more, capacity planning and scaling is what I do. And this conference is all about those topics.

Did you know I am going to be at Velocity?

brianlmoon — Tue, 17 Jun 2008 23:31:00 -0500

Well, neither did I until today. HA!

Velocity is a new O'Reilly conference dedicated to "Optimizing Web Performance and Scalability". It starts next Monday. Yesterday I was contacted by Adam Jacobs of HJK Solutions about taking part in a panel discussion about what happens when success comes suddenly to a web site. I think he thought I was in the bay area. Little did he know I am in Alabama. But, amazingly, I was able to work it all out so I can be there. I wish I had known about this conference ahead of time. It sounds really awesome. Performance has always been something I focus on. I hope to share some and learn at the same time.

So, if you are going to be there, come see our panel.

P.S. Thanks to John Allspaw of Flickr for recommending me to Adam.

An Introduction to MySQL - Birmingham, AL

brianlmoon — Mon, 16 Jun 2008 19:00:00 -0500

I am giving a talk titled "An Introduction to MySQL" here in Birmingham, AL on June 21, 2008 at 3PM.

I love living in Alabama. I was born and raised in Huntsville. However, Birmingham has always seemed a bit behind in technology compared to what I do for a living. There is good reason. The industry here is medical, banking, industrial and utilities. I don't really want my doctors keeping my medical records in an alpha release of anything. Same goes for my banking and utilities. But, as this page shows, the companies here are catching up. So, I am happy to present MySQL to as many people as I can in this town. Hopefully I will help some folks that have not been exposed to MySQL or any open source for that matter.

The event is part of our local Linux user group's (BALU) planned events.

MemProxy 0.1

brianlmoon — Tue, 10 Jun 2008 20:46:00 -0500

MemProxy 0.1 is out! It has taken me a while, but I have finally gotten around to releasing the code that I credited with saving us during a Yahoo! mention. It is a caching proxy "server" that uses memcached for storing the cache. I put server in quotes because it is really just a PHP script that handles the caching and talking to the application servers. Apache and other HTTP servers already do a good job talking HTTP to a vast myriad of clients. I did not see any reason to reinvent the wheel. Here are some of the features that make it different from anything I could find:

Uses memcached for storage

Serves cache headers to clients based on TTL of cached data

Uses custom headers to assemble multiple pieces of cache into one object

Minimal dependencies. Only PHP and pecl/memcached needed.

Small code base. It is just two files, one when settings are cached.

Application agnostic. If the backend is hosted on an HTTP server this can cache it.

Some other things it does that you might expect:

Handles HTTP 1.1 requests to the backend

Allows TTLs set by the standard Cache-Control header

Appears transparent to the client.

Sends proper HTTP error codes relating to proxies/gateways

Allows pages to be refreshed or removed from cache

Allows a page to be viewed from the application server without caching it

more....

You can find the code on Google Code. The code (or something like it rather) has been in use at dealnews for well over a year. But, this is a new code base. It had to be refactored for public consumption. So, there may be bugs.

OOP does not equal portable or shareable

brianlmoon — Fri, 06 Jun 2008 14:29:00 -0500

So, just now, I was reading a good Rails post by Stuart Herbert and nodding my head along. I have not gotten into the Rails bashing fun on my blog, but I do poke fun at it around the office. Then I got to this part:

The OO in Rails continues to leave PHP for dead, and OO brings many advantages to a thriving development community. There are real advantages to being able to share code between both the must-be-real-time web front-end and the non-real time backends, and to be able to easily reuse whatever external open-source libraries save you time and effort.

Now, I have no idea about the first part. I am not an OOP guy. But, what I have issue with is the idea that for code to be reusable, it has to be OOP. So, if I am a college kid or young PHP developer, I would read this and think "Oh, so, to reuse code or share it, I have to be using OOP". Man, this is just so dead wrong and irresponsible. Can someone tell me why only OOP can be reused? Why can't people write sane functions that can be reused? I do it every day. They do it in C all the time. Our front end web servers run the same code base as the cron jobs that do a wide variety of things. They use the same libraries. They use the same objects (yeah, i use them when they are a good idea).

Please, someone explain this to me.

(I have a half written post about how you can write good, maintainable, reusable code without OOP. I have not finished it yet, but I guess I need to. It seems the world is going to OOP hell otherwise.)

in_array is quite slow

brianlmoon — Thu, 05 Jun 2008 02:38:00 -0500

So, we had a cron job hanging for hours. No idea why. So, I started debugging. It all came down to a call to in_array(). See, this job is importing data from a huge XML file into MySQL. After it is done, we want to compare the data we just added/updated to the data in the table so we can deactivate any data we did not update. We were using a mod_time field in mysql in the past. But, that proved to be an issue when we wanted to start skipping rows from the XML that were present but unchanged. Doing that saved a lot of MySQL writes and sped up the process.

So, anyhow, we have this huge array of ids accumulated during the import. So, an in clause with 2 million parts would suck. So, we suck back all the ids in the database that exist and stick that into an array. We then compared the two arrays by looping one array and using in_array() to check if the value was in the second array. Here is a pseudo example that shows the idea:

[sourcecode language='php']

foreach($arr1 as $key=>$i){

if(in_array($i, $arr2)){

unset($arr1[$key]);

}
}

[/sourcecode]

So, that was running for hours with about 400k items. Our data did not contain the value as the key, but it could as the value was unique. So, I added it. So, now, the code looks like:

[sourcecode language='php']

foreach($arr1 as $key=>$i){

if(isset($arr2[$i])){

unset($arr1[$key]);

}
}

[/sourcecode]

Yeah, that runs in .8 seconds. Much better.

So, why were we using in_array to start with if in_array is clearly not the right solution to this problem? Well, it was basic code evolution. Originally, these imports would be maybe 100 items. But, things changed.

FWIW, I tried array_diff() as well. It took 25 seconds. Way better than looping and calling in_array, but still not as quick as a simple isset check. There was refactoring needed to put the values into the keys of the array.

UPDATE: I updated this post to properly reflect that there is nothing wrong with in_array, but simply that it was not the right solution to this problem. I wrote this late and did not properly express this. Thanks to all those people in the comments that helped explain this.

Stupid PHP Tricks: Normalizing SimpleXML Data

brianlmoon — Mon, 02 Jun 2008 21:59:00 -0500

SimpleXML is neat. Some people don't think it is so simple. Boy, use the old stuff. The DOM-XML stuff.

Anyhow, one annoying thing about SimpleXML has to do with caching. When using web services, we often cache the contents we get back. We were having a problem where we would get an error about a SimpleXML node not existing. We were caching the data in memcached which serializes the variable. So, when it unserialized the variable, there were references in there to some SimpleXML nodes that we did not take care of. Basically, a tag like:

bar

is a string. But a tag like:

is an empty SimpleXML Object. That is a little annoying, but I don't feel like digging into the C code and figuring out why. So, we just work around it. We made a recursive function to do the dirty work for us.

function makeArray($obj) {

$arr = (array)$obj;

if(empty($arr)){

$arr = "";

} else {

foreach($arr as $key=>$value){

if(!is_scalar($value)){

$arr[$key] = makeArray($value);

}

}

}

return $arr;

}

That will turn whatever you pass it into an array or empty string if it is empty.

But, while I was hacking around tonight, I came up with another idea. Check out this hackery:

$data = json_decode(json_encode($data));

Yeah! One liner. That converts all the SimpleXML elements into stdClass objects. All other vars are left intact.

Ok, so this is where someone in the comments can tell me about the magic SimpleXML method or magic OOP function I have missed to take care of all this. Go ahead, please make my code faster. I dare you.

Oh Hail!

brianlmoon — Sun, 01 Jun 2008 17:38:00 -0500

We had a huge hail storm today.

It came out of nowhere. It was kind of scary.

Here is a photo that may yield more perspective.

Short Array Syntax for PHP

brianlmoon — Wed, 28 May 2008 09:18:00 -0500

So, I was asked in IRC today about the proposed short array syntax for PHP. For those that don't know, I mean the same syntax that other languages (javascript, perl, python, ruby) all have. Currently in PHP we have this:

$var = array(1,2,3);

The proposed additional syntax is:

$var = [1,2,3];

So, I voted +1 for this feature on the PHP Internals list. A colleague asked me why I voted +1. At first I had no good answer other than it was just a gut feeling. It just feels like a good addition to the language. It is common among web languages and therefore users coming into PHP from other languages may find it more comfortable.

The best thing I could tell him was that it would make arrays fall in line with other data types in PHP. For example, you never write:

$var = int(1);

$var = string(foo);

So, why oh why do we have to have what looks like a function, but in reality is not, for creating an array? It is a language construct and should look like a language construct. I think the [ ] syntax makes more sense when you think about it in those terms.

I say commit it Andi. That seems to be what everyone else does. =)

PHP session cookie refresh

brianlmoon — Tue, 13 May 2008 19:40:00 -0500

I have always had an issue with PHP Sessions. Albeit, a lot of my issues are now invalid. When they were first implemented, they had lots of issues. Then the $_SESSION variable came to exist and it was better. Then memcached came to exist and you could store sessions there. That was better. But, still, after all this time, there is one issue that still bugs me.

When you start a session, if the user had no cookie, they get a new session id and they get a cookie. You can configure that cookie to last for n seconds via php.ini or session_cookie_set_params(). But, and this is a HUGE but for me, that cookie will expire in n seconds no matter what. Let me explain further. For my needs, the cookie should expire in n seconds from last activity. So, each page load where sessions are used should reset the cookie's expiration. This way, if a user leaves the site, they have n seconds to come back and still be logged in.

Consider an application that sets the cookie expiration to 5 minutes. The person clicks around on the site, gets a phone call that lasts 8 minutes and then gets back to using the site. Their session has expired!!!! How annoying is that? The only sites I know that do that are banks. They have good reason. I understand that.

My preference would be to either set an ini value that tells PHP sessions to keep the session active as long as the user is using the site. Or give me access to the internal function php_session_send_cookie(). That is the C function that sends the cookie to the user's browser. Hmm, perhaps a patch is in my future.

In the short term, this is what I do:



setcookie(

ini_get("session.name"),

session_id(),

time()+ini_get("session.cookie_lifetime"),

ini_get("session.cookie_path"),

ini_get("session.cookie_domain"),

ini_get("session.cookie_secure"),

ini_get("session.cookie_httponly")

);

That will set the session cookie with a fresh ttl.

Ok, going to dig into some C code now and see if I can make a patch for this.

Thoughts on the 2008 MySQL Conference and Expo

brianlmoon — Wed, 07 May 2008 12:48:00 -0500

Well, it has been almost a month. I know I am late to the blogosphere on my thoughts. Just been busy.

Again this year, the Phorum team was invited to be a part of the DotOrg Pavilion. What is that? Basically they just give expo floor space to open source projects. It is cool. We had a great location this year. We were right next to the area where they served food and drinks during the breaks. We had lots of traffic and met some of our power users. IMVU.com is getting 1.5 million messages per month in their Phorum install. They did have to customize it to fit into their sharding. But, that is expected. A guy (didn't catch his name) from Innobase came by and told us that they just launced InnoDB support forums on their site using Phorum. Cool. So now MySQL and Innobase use Phorum. I am humbled by the message that sends to me about Phorum.

Speaking of our booth, we were right next to the phpMyAdmin guys. Wow, that product has come a long way. I was checking out the visual database designer they have now. It was neat. I also met the Gentoo MySQL package maintainer. He was in the phpMyAdmin booth.

I was interviewed by WebDevRadio as I already posted. I was also asked to do a short Q&A with the Sun Headlines video team. They used one part of my clip. I won't link to that. No, if you find it good for you. I need to be interviewed some more or something. I did not look comfortable at all.

There were lots of companies with open in their name or slogan. I guess this is expected pandering.

I attended part of the InnoDB talk given by Mark Callaghan of Google. It appears that Google is serious about improving InnoDB on large machines. That is, IMO, good news for anyone that likes InnoDB. If I counted right, they had more than 5 people who at least part of their job is to improve InnoDB.

I gave my two talks. The first had low attendance, but the feedback was nice. It was just after the snack break in the expo hall and I was in the farthest room from the expo hall. That is what I keep telling myself. =) The second was better attended and the feedback seemed good there. I was told by Maurice (Phorum Developer) that I talked too fast and at times sounded like Mr. Mackey from South Park by repeating the word bad a lot. I will have to work on that in the future. I want to do more speaking.

On the topic of my second talk, there seemed to be a lot of "This is how we scaled our site" talks. I for one found them all interesting. Everyone solves the problem differently.

Next year I am thinking about getting more specific with my talk submissions. Some ideas include: PHP, MySQL and Large Data Sets, When is it ok to denormalize your data?, Using memcached (not so much about how it works), Index Creation (tools, tips, etc.).

In closing, I want to give a big thanks to Jay Pipes and Lenz Grimmer from MySQL. Despite Jay's luggage being lost he was still a big help with some registration issues among other things. Both of them helped out the Phorum team a great deal this year. Thanks guys.

Amazon MP3 Store has holes

brianlmoon — Tue, 06 May 2008 11:03:00 -0500

A coworker found out how secure Amazon's MP3 store is. Even big guys like Amazon make errors in their web site security.

So, I clicked purchase and the album immediately started downloading. It was at this point that I had the thought cross my mind: "Did I update my credit card info?"

Well, no, I didn't. Before the album finished downloading, I was trying to change the method of payment. Turns out, for a digital purchase, you can't do such a thing. So, I waited and wondered was was going to come of this...

Example my.cnf files

brianlmoon — Tue, 06 May 2008 10:17:00 -0500

UPDATE: There are some examples being added at the MySQL Forge now.

When I first started installing MySQL for myself, it was quite handy to have the example my.cnf files in the source package. I was a noob to the MySQL configuration. Even after I became more experienced, I would use them as a starting point. However, I now find that they are so behind the times they are not as useful. Here are some of the comments from the files.

my-small.cnf

# This is for a system with little memory (<= 64M) where MySQL is only used
# from time to time and it's important that the mysqld daemon
# doesn't use much resources.

my-medium.cnf

# This is for a system with little memory (32M - 64M) where MySQL plays
# an important part, or systems up to 128M where MySQL is used together with
# other programs (such as a web server)

my-large.cnf

# This is for a large system with memory = 512M where the system runs mainly
# MySQL.

my-huge.cnf

# This is for a large system with memory of 1G-2G where the system runs mainly
# MySQL.

I end up using the large or huge files as a starting point for every server I set up by hand. The small and medium should be renamed underpowered and teeny-tiny. Who has less than 64MB of RAM on a server now? Can you even buy sticks of memory that small in any modern system? Most come with 256MB sticks minimum. And they never come with just one stick.

I will use the large example as a starting point for a server that has 2GB of RAM and will be running an entire site on one server. I use huge for any server that runs only MySQL. And even then, most of them have 4GB of RAM or more.

I don't know if anyone at MySQL has plans on tweaking these files or not. Perhaps those good guys at the MySQL Performance Blog or Percona could create some example my.cnf files. I could put some out there, but I fear their sole purpose would be for someone to point out what I am doing wrong. =P Hey, they work for me. Hmm, maybe this would make a good MySQL Forge section. A whole area of user contributed my.cnf files. They could be architecture specific and everything. What runs best on Solaris? Linux? BSD? Windows? 32-bit? 64-bit?

One thing I would for sure like to see is example files for InnoDB dominant servers. Most of our servers all run primariy InnoDB tables. None of these above examples covers InnoDB. They have comments, but no preconfigured values. I have seen more than one server using InnoDB tables without any custom configuration in their my.cnf. In the end that is the fault of the server admin/owner no doubt.

What do you say? Anyone up for a MySQL Forge section for my.cnf files?

Interview with WebDevRadio

brianlmoon — Sat, 03 May 2008 10:13:00 -0500

While I was at the MySQL Conference, I sat down with Michael Kimsal of WebDevRadio and recapped the two talks that I gave at the conference. I have uploaded the slides so you can follow along if you want.

One to a Cluster - The evolution of the dealnews.com architecture.

MySQL Tips and Tricks - Some simple tips and some of the more advanced SQL we use in Phorum.

Thanks Michael. Any time you need a guest, just let me know.