Ramblings of a web guy (Tag: memcached)

PHP and Memcached: The state of things

brianlmoon — Wed, 23 Jun 2010 11:48:31 -0500

Memcached is the de facto standard for caching in dynamic web sites. PHP is the one of the most widely used languages on the web. So, naturally there is lots of interest in using the two together. There are two choices for using memcached with PHP: PECL/memcache and PECL/memcached. Great names huh? But as of this writing there are issues with the two most popular Memcached libraries for PHP. This is a summary of those issues that I hope will help people being hurt by them and may bring about some change.

PECL/memcache

This is the older of the two and was the first C based extension for using memcached with PHP. Before this all the options were native PHP code. While they worked, they were slower of course. C > PHP. That is just fact. However, there has not been much active development on this code in some time. Yes, they have fixed bugs, but support for new features in the memcached server have not been added to this extension. Newer version of the server suppot a more efficient binary protocol and many new features. In addition, there are some parts of the extension that simply don't work anymore.

The most glaring one is the delete() function. It takes a second parameter that is documented as: "the item will expire after timeout seconds". In fact that was never a feature of memcached. It was completely misunderstood by the original extension authors. When that parameter was supported, it locked the key for timeout seconds and would not allow a new add operation on that key. Second, this feature was completely removed in memcached 1.4.0. So, if you send a timeout to the delete function, you simply get a failure. This is creating a huge support issue in the memcached community (not the PHP/PECL community) with people not being able to delete keys because of bad documentation and unsupported behavior. I sent a documentation patch for the this function to the PHP Docs list. I then modified it based on feedback. But since I have heard nothing about it getting merged. I have a PHP svn account, but even if I do have karma on the docs repository, I don't want to hijack them without the support of the people that work on the docs all the time. If you are reading this and can change the docs, please make the documentation for that function say "DON'T USE THIS PARAMETER, IT HAS BEEN DEPRECATED IN THE MEMCACHED SERVER!" or something.

Not too long ago the extension was officially abandoned by its original maintainers. Some people stepped up and claimed they wanted to see it continue. But, since that time, there have been no releases. There are bugs being closed though so maybe there is good things coming.

A very problematic issue with this extension is with the 3.0 beta release. It needs to just die. It has huge bugs that, IMO, were introduced by the previous maintainers in an effort to bring it up to speed, but never saw them through to make sure the new code worked. In their defense, it is marked as beta on PECL. But, thanks to Google, people don't see the word beta anymore. There are lots of people using this version and when they get bad results they blame the whole memcached world. Really, the new maintainers would do the world a favor if they just removed the 3.x releases from the PECL site.

PECL/memcached

This extension was started by Andrei Zmievski while working at Digg.com as an open source developer. It uses libmemcached, a C++ memcached library that does all the memcached work. This made it quite easy to support the new features in the memcached daemon as it as it was being developed at the same time as the new server. However, it has not had a stable release on PECL in nearly a year except for release to make it compatible with new versions of libmemcached. No bug fixes and no new features. There are currently 28 open bugs on PECL for this extension. Not all of which are bugs. Some are feature requests. The ironic thing is that the GitHub repository for this extension has seen a lot of development. But, none of these bug fixes have made it into the official PECL channel. And some of these bugs are major and others are just huge WTF for a developer.

The most major bug is one that I found. If you use persistent connections with this extension, it basically leaks those connections, not reusing an existing connection but also not closing the ones already made. This uses up the memory in your processes until they crash and it creates an exponential number of connections to your memcached server until it has no more connections available. Andrei does report in the bug that it is fixed in hist GitHub. But, for now, you can't use persistent connections.

The big WTF for a developer is that some functions don't take a numeric key and use it properly.


$mc = new Memcached();

$mc->addServer("localhost", 11211);

$mc->set(123, "yes!");

var_dump($mc->getMulti(array(123)));

?>

The above code should generate:

array(1) {
  [123]=>
  string(4) "yes!"
}

But, in reality, it generates:

bool(false)

The set succeeds, but the getMulti fails. Again, this is being fixed in GitHub, but is not available for release on PECL.

Compatibility

One big issue with these two extensions is that they are not drop in replacements for each other. If you want to move from one to the other, you have to take into consideration some time to convert your code. For instance, the older extension's set() function takes flags as the third parameter. The newer extension does not take a flags parameter at all. The older uses get() with an array to do a multi-get and the newer extension has a separate method for that called getMulti(). There are others as well.

Summary

So, what should you do as a PHP developer? If you are deploying memcached today, I would use the 2.2.x branch of PECL/memcache. It is the most stable. Just avoid the delete bug. It is not as fast and does not have the features. But, it is very reliable for set, get, add.... the basics of memcached. For the long term, it is a bit unclear. PECL/memcached looks to fix a lot of things in 2.0. But, I have not used it yet. In addition the long term growth of the project is a bit in question. Will there be a 2.1, 2.2, etc? I hope so. The other unknown is if the those people fixing the PECL/memcache bugs will keep it up and release a good stable product that supports new features of the server. Again, I hope so. The best scenario would be to have a choice between two fully compatible and feature rich extensions. Keep your fingers crossed.

MySQL Conference Review

brianlmoon — Sat, 17 Apr 2010 10:54:52 -0500

I am back home from a good week at the 2010 O'Reilly MySQL Conference & Expo. I had a great time and got to see some old friends I had not seen in a while.

Oracle gave the opening keynote and it went pretty much like I thought it would. Oracle said they will keep MySQL alive. They talked about the new 5.5 release. It was pretty much the same keynote Sun gave last year. Time will tell what Oracle does with MySQL.

The expo hall was sparse. Really sparse. There were a fraction of the booths compared to the past. I don't know why the vendors did not come. Maybe because they don't want to compete with Oracle/Sun? In the past you would see HP or Intel have a booth at the conference. But, with Oracle/Sun owning MySQL, why even try. Or maybe they are not allowed? I don't know. It was just sad.

I did stop by the Maatkit booth and was embarrassed to tell Baron (its creator) I was not already using it. I had heard people talk about it in the past, but never stopped to see what it does. It would have only saved me hours and hours of work over the last few years. Needless to say it is now being installed on our servers. If you use MySQL, just go install Maatkit now and start using it. Don't be like me. Don't wait for years, writing the same code over and over to do simple maintenance tasks.

Gearman had a good deal of coverage at the conference. There were three talks and a BoF. All were well attended. Some people seemed to have an AHA! moment where they saw how Gearman could help their architecture. I also got to sit down with the PECL/gearman maintainers and discuss the recent bug I found that is keeping me from using it.

I spoke about Memcached as did others. Again, there was a BoF. It was well attended and people had good questions about it. There seemed to be some FUD going around that memcached is somehow inefficient or not keeping up with technology. However, I have yet to see numbers or anything that proves any of this. They are just wild claims by people that have something to sell. Everyone wants to be the caching company since there is no "Memcached, Inc.". There is no company in charge. That is a good thing, IMO.

That brings me to my favorite topic for the conference, Drizzle. I wrote about Drizzle here on this blog when it was first announced. At the time MySQL looked like it was moving forward at a good pace. So, I had said that it would only replace MySQL in one part of our stack. However, after what, in my opinion, has been a lack of real change in MySQL, I think I may have changed my mind. Brian Aker echoed this sentiment in his keynote address about Drizzle. He talked about how MySQL AB and later Sun had stopped focusing on the things that made MySQL popular and started trying to be a cheap version of Oracle. That is my interpretation of what he said, not his words.

Why is Drizzle different? Like Memcached and Gearman, there is no "Drizzle, Inc.". It is an Open Source project that is supported by the community. It is being supported by companies like Rackspace who hired five developers to work on it. The code is kept on Launchpad and is completely open. Anyone can create a branch and work on the code. If your patches are good, they will be merged into the main branch. But, you can keep your own branch going if you want to. Unlike the other forks, Drizzle has started over in both the code and the community. I personally see it as the only way forward. It is not ready today, but my money is on Drizzle five or ten years from now.

Using ini files for PHP application settings

brianlmoon — Tue, 19 Jan 2010 17:24:35 -0600

At dealnews we have three tiers of servers. First is our development servers, then staging and finally production. The complexity of the environment increases at each level. On a development server, everything runs on the localhost: mysql, memcached, etc. At the staging level, there is a dedicated MySQL server. In production, it gets quite wild with redundant services and two data centers.

One of the challenges of this is where and how to store the connection information for all these services. We have done several things in the past. The most common thing is to store this information in a PHP file. It may be per server or there could be one big file like:








if(DEV){


    $server = "localhost";


} else {


    $server = "10.1.1.25";


}





?>

This gets messy quickly. Option two is to deploy a single file that has the settings in a PHP array. And that is a good option. But, we have taken that one step further using some PHP ini trickeration. We use ini files that are loaded at PHP's startup and therefore the information is kept in PHP's memory at all times.

When compiling PHP, you can specify the --with-config-file-scan-dir to tell PHP to look in that directory for additional ini files. Any it finds will be parsed when PHP starts up. Some distros (Gentoo I know) use this for enabling/disabling PHP extensions via configuration. For our uses we put our custom configuration files in this directory. FWIW, you could just put the above settings into php.ini, but that is quite messy, IMO.

To get to this information, you can't use ini_get() as you might think. No, you have to use get_cfg_var() instead. get_cfg_var returns you the setting, in php.ini or any other .ini file when PHP was started. ini_get will only return values that are registered by an extension or the PHP core. Likewise, you can't use ini_set on these variables. Also, get_cfg_var will always reflect the initial value from the ini file and not anything changed with ini_set.

So, lets look at an example.



; db.ini


[myconfig]


myconfig.db.mydb.db     = mydb


myconfig.db.mydb.user   = user


myconfig.db.mydb.pass   = pass


myconfig.db.mydb.server = host

This is our ini file. the group in the braces is just for looks. It has no impact on our usage. Because this is parsed along with the rest of our php.ini, it needs a unique namespace within the ini scope. That is what myconfig is for. We could have used a DSN style here, but it would have required more parsing in our PHP code.










/**


 * Creates a MySQLi instance using the settings from ini files


 *


 * @author     Brian Moon 


 * @copyright  1997-Present dealnews.com, Inc.


 *


 */





class MyDB {





    /**


     * Namespace for my settings in the ini file


     */


    const INI_NAMESPACE = "dealnews";





    /**


     * Creates a MySQLi instance using the settings from ini files


     *


     * @param   string  $group  The group of settings to load.


     * @return  object


     *


     */


    public static function init($group) {





        static $dbs = array();





        if(!is_string($group)) {


            throw new Exception("Invalid group requested");


        }





        if(empty($dbs["group"])){





            $prefix = MyDB::INI_NAMESPACE.".db.$group";





            $db   = get_cfg_var("$prefix.db");


            $host = get_cfg_var("$prefix.server");


            $user = get_cfg_var("$prefix.user");


            $pass = get_cfg_var("$prefix.pass");





            $port = get_cfg_var("$prefix.port");


            if(empty($port)){


                $port = null;


            }





            $sock = get_cfg_var("$prefix.socket");


            if(empty($sock)){


                $sock = null;


            }





            $dbs[$group] = new MySQLi($host, $user, $pass, $db, $port, $sock);





            if(!$dbs[$group] || $dbs[$group]->connect_errno){


                throw new Exception("Invalid MySQL parameters for $group");


            }


        }





        return $dbs[$group];





    }





}





?>

We can now call DB::init("myconfig") and get a mysqli object that is connected to the database we want. No file IO was needed to load these settings except when the PHP process started initially. They are truly constant and will not change while this process is running.

Once this was working, we created separate ini files for our different datacenters. That is now simply configuration information just like routing or networking configuration. No more worrying in code about where we are.

We extended this to all our services like memcached, gearman or whatever. We keep all our configuration in one file rather than having lots of them. It just makes administration easier. For us it is not an issue as each location has a unique setting, but every server in that location will have the same configuration.

Here is a more real example of how we set up our files.



[myconfig.db]


myconfig.db.db1.db         = db1


myconfig.db.db1.server     = db1hostname


myconfig.db.db1.user       = db1username


myconfig.db.db1.pass       = db1password





myconfig.db.db2.db         = db2


myconfig.db.db2.server     = db2hostname


myconfig.db.db2.user       = db2username


myconfig.db.db2.pass       = db2password





[myconfig.memcache]


myconfig.memcache.app.servers    = 10.1.20.1,10.1.20.2,10.1.20.3


myconfig.memcache.proxy.servers  = 10.1.20.4,10.1.20.5,10.1.20.6





[myconfig.gearman]


myconfig.gearman.workload1.servers = 10.1.20.20


myconfig.gearman.workload2.servers = 10.1.20.21

Memcached: What is it and what does it do?

brianlmoon — Wed, 30 Sep 2009 11:49:39 -0500

I spoke at CodeWorks in Atlanta, GA this week. I totally dropped the ball promoting it on my blog. It was a neat venue. Rather than a large conference they are doing a traveling show. Seven cities in 14 days. Many of the presenters are working in every city. Crazy. I was just in Atlanta. It is close to home and easy for me to get to.

I spoke about memcached. I tried to dig a bit deeper into how memcached works. On the mailing list we get a lot of new people that make assumptions about memcached. Most talks I have seen focus on why caching is good, how to use memcached, the performance gain. I kind of assumed everyone knew that stuff already. I guess you could say I gave a talk that was the real FAQs of the project.

Here are the slides. Derick Rethans took video of the talk. When he gets that online I will add it to this post.

Memcached: What is it and what does it do?

View more documents from Brian Moon.

Building PECL/memcache on Mac OS X

brianlmoon — Sun, 31 May 2009 22:23:29 -0500

My coworker Rob ran into an issue building the PECL/memcache extension on his Mac. He did find the solution however. You can read and leave comments on his blog.

Deploying Scalable Websites with Memcached

brianlmoon — Fri, 03 Oct 2008 09:55:45 -0500

I spoke at the MySQL Conference and Expo this year about the architecture we have here at dealnews.com. After my talk, Jimmy Guerrero of Sun/MySQL invited me to give a webinar on how dealnews uses memcached. That is taking place next week, Thursday, October 09, 2008. It is a free webinar. We have used memcached in a variety of ways as we have grown. So, I will be talking about how dealnews used memcached in the past and present.

For more information, visit the MySQL web site.

Where Drizzle fits in for me

brianlmoon — Thu, 24 Jul 2008 13:17:26 -0500

So, most of you have heard about Drizzle by now. For those that have not, you can check out many, many blog posts or the Launchpad page.

The thread on Slashdot about Drizzle was quite negative. Most misunderstand what Drizzle is about. SQLite is not a good solution when you have 100 web servers. Let me describe how it I would use it and maybe that will help some understand it.

When it comes to MySQL use, dealnews has two very different use cases. The first is an enterprise storage system that involves content creation, reporting and data warehousing. For that layer of our business, we are using more and more advanced features as they become available. We use triggers and stored procedures. We use complex data types for specific use cases. All those features are a big gain.

The other way that we use MySQL is for serving content to our readers. I have written about this before. For this purpose, we avoid joins, don't use any advanced features. We do use replication, indexes and intelligent queries. We don't (as one slashdot reader claimed) do all of our processing in the code. That would be stupid. If you do that you are ignorant. I will stop talking about that before this becomes a rant. I do believe in letting MySQL do my work for me.

This is where Drizzle fits in. To serve content, I don't need stored procedures, triggers, views or any of that other stuff. The whole database that the front end web servers use is basically a view. It is a denormalized, prepared version of the real data. I store objects. But, I have to be able to sort and filter the data in a way that SQL allows me to do. CouchDB sounds interesting. Maybe one day it will be there. It is sill in the optimization phase.

Now, some say that this is just MySQL 3.x all over again. Well, you clearly have not been listening to the really smart people that are working on Drizzle. They are doing more than just removing the 4.1 and 5.x features from MySQL. They are removing things that don't make sense for this use case. They are adding things that do make sense. They are replacing parts of the code base where there is a better library or way of doing it. At this point, they have no feature requirements to meet. They have no deadlines. They are making what they think the high volume web world and/or cloud computing needs. They are making it plugable: think Apache modules or PHP extensions. So, if you need feature XYZ that was yanked out, you can add it back in (hopefully) via the internal API. There is a lot more going on here than just removing "features".

So, I am cheering on the folks working on Drizzle. I have joined their community and will provide what feedback I can from userland. I am no C++ coder. I can read it. I can debug it. But, writing it or doing heavy lifting is not in my skill set. Hopefully I can contribute

Caching and TTL behavior

brianlmoon — Wed, 02 Jul 2008 23:56:25 -0500

So, I am working on MemProxy some. Mainly, I am trying to implement more of the Cache-Control header's many options. The one that has me a bit perplexed s-maxage. Particularly when combined with max-age.

s-maxage is the maximum time in seconds an item should remain in a shared cache. So, if s-maxage is set by the application server, my proxy should keep it for that amount of time at the most. Up until now, I have just been looking at max-age. But, s-maxage is the proper one for a proxy to use if it is present. I do not send the s-maxage through because this is a reverse proxy and, IMO, that is proper behavior for an application accelerating proxy. However, I do send forward the max-age value that is set by the application servers. If no max-age is set, I send a default as defined in the script. Also, if no-cache or no-store is set, I send those and a max-age of 0.

My problem arises when max-age is less than s-maxage. Up until now, I have sent a max-age back to the client that represents the time left for the cached item in my proxy's cache. So, if the app server sent back max-age=300 and a request comes in and the cache is found and the cache was created 100 seconds ago, I send max-age-200 back to the client. But, I was only using max-age before. Now, in cases where s-maxage is longer than max-age, I would come up with negative numbers. That is not cool. The easiest solution would be to always send the original max-age back to the client. But, that seems kind of lame.

So, my question is, if you are using an application (HTTP or otherwise) accelerator, what would you expect? If you application set a max-age of 300 would you always expect the end client to receive a max-age of 300? Or should it count down over time? The only experience I have is a CDN. If you watch CDN traffic, the max-age gets smaller and smaller over time until it hits 0. I have not tried sending an s-maxage to my CDN. I don't know what they would do with that. Maybe that is a good test.

UPDATE: Writing this gave me an idea. If the item will be in the proxy cache longer than the max-age ttl, send the full max-age ttl. Otherwise, send the time left in the proxy cache. Thoughts on that?

(thanks for being my teddy bear blogosphere)

Velocity Conference Roundup

brianlmoon — Tue, 01 Jul 2008 01:01:56 -0500

As I said before, I was invited to be on a panel at Velocity Conference. I was delighted to go. I had never been to San Francisco. I have been to Portland and Santa Clara several times. The panel was great. It was the Brian and photo sharing sites show. Seriously, it was me (dealnews.com), John Allspaw of Flickr, Don MacAskill of SmugMug and Farhan Mashraqi of Fotolog. Oh, there was also Shayan Zadeh of Zoosk, a social dating network and Michael Halligan, a consultant from BitPusher. We all had similar ideas. I told my Yahoo story. I told everyone that they should denormalize (or optimize as Farhan prefered) their data to improve performance. Others agreed. I have written about my methods for denormalizing normalized data before. (See pushed cache) Fun was had by all.

I mentioned John Allspaw above. He gave a talk on his own as well. It was good. The slides are on SlideShare. He and I see eye to eye on a lot of things. One thing he says in there that may shock a lot of people is to test using produciton. I agree fully. We could have never been sure our infastructure was ready last year without testing the production servers.

I also learned about Varnish at the conference. It is a super fast reverse proxy. It uses the virtual memory systems of recent kernels to store its cache. The OS worries about moving things from memory to disk based on usage. The claim is that the OSes are better at this than any programmer could do (without copying them of course). It is fast. The developers are proud. And by proud I mean cocky. I have been playing with it. As you know, I have my own little caching proxy solution. Varnish is much faster, as I expected. However, storing cache in memcached is very attractive to me. Varnish can't do that. It would likely slow it down a great deal. MemProxy does do that. Also, because MemProxy is written in PHP and my application layer is PHP, I can do things at the proxy layer to inspect the request and take action. Works well for my use. But, if you are using squid or mod_cache or something, you may want to give Varnish a look.

There was a good bit of information about the client side of performance. There were folks from Microsoft there talking about IE8. It looks like IE8 will catch up with the other browsers in a lot of ways. Yahoo talked about image optimization. Good stuff in there. I use Fireworks and it does a pretty good job of making small images. I am looking more into combining images and making image maps that use CSS. We use a CDN, but fewer connections is better for users.

There was also a lot of great debate. SANs rock! SANs suck! Rails Scales! Rails Sucks! The Cloud is awesome! The Cloud is a lie! (lots of cloud)

I had dinner both nights with guys from Six Apart. Good conversations were had. I don't know if I am a big vegan fan though. I mean, the food was good, but it all kinda tasted the same. Perhaps I ordered poorly. At dinner on Tuesday I met a guy going to work for Twitter soon. He is an engineer that hopefully will be another step toward getting them back to 100% again. Lets keep our fingers crossed.

They did announce that the conference would be held again next year. I am definitely going back. Probably two of us from dealnews will go. OSCON is fun. MySQL conference is too. But, more and more, capacity planning and scaling is what I do. And this conference is all about those topics.

Did you know I am going to be at Velocity?

brianlmoon — Tue, 17 Jun 2008 23:31:32 -0500

Well, neither did I until today. HA!

Velocity is a new O'Reilly conference dedicated to "Optimizing Web Performance and Scalability". It starts next Monday. Yesterday I was contacted by Adam Jacobs of HJK Solutions about taking part in a panel discussion about what happens when success comes suddenly to a web site. I think he thought I was in the bay area. Little did he know I am in Alabama. But, amazingly, I was able to work it all out so I can be there. I wish I had known about this conference ahead of time. It sounds really awesome. Performance has always been something I focus on. I hope to share some and learn at the same time.

So, if you are going to be there, come see our panel.

P.S. Thanks to John Allspaw of Flickr for recommending me to Adam.

MemProxy 0.1

brianlmoon — Tue, 10 Jun 2008 20:46:15 -0500

MemProxy 0.1 is out! It has taken me a while, but I have finally gotten around to releasing the code that I credited with saving us during a Yahoo! mention. It is a caching proxy "server" that uses memcached for storing the cache. I put server in quotes because it is really just a PHP script that handles the caching and talking to the application servers. Apache and other HTTP servers already do a good job talking HTTP to a vast myriad of clients. I did not see any reason to reinvent the wheel. Here are some of the features that make it different from anything I could find:

Uses memcached for storage

Serves cache headers to clients based on TTL of cached data

Uses custom headers to assemble multiple pieces of cache into one object

Minimal dependencies. Only PHP and pecl/memcached needed.

Small code base. It is just two files, one when settings are cached.

Application agnostic. If the backend is hosted on an HTTP server this can cache it.

Some other things it does that you might expect:

Handles HTTP 1.1 requests to the backend

Allows TTLs set by the standard Cache-Control header

Appears transparent to the client.

Sends proper HTTP error codes relating to proxies/gateways

Allows pages to be refreshed or removed from cache

Allows a page to be viewed from the application server without caching it

more....

You can find the code on Google Code. The code (or something like it rather) has been in use at dealnews for well over a year. But, this is a new code base. It had to be refactored for public consumption. So, there may be bugs.

Stupid PHP Tricks: Normalizing SimpleXML Data

brianlmoon — Mon, 02 Jun 2008 21:59:04 -0500

SimpleXML is neat. Some people don't think it is so simple. Boy, use the old stuff. The DOM-XML stuff.

Anyhow, one annoying thing about SimpleXML has to do with caching. When using web services, we often cache the contents we get back. We were having a problem where we would get an error about a SimpleXML node not existing. We were caching the data in memcached which serializes the variable. So, when it unserialized the variable, there were references in there to some SimpleXML nodes that we did not take care of. Basically, a tag like:

bar

is a string. But a tag like:

is an empty SimpleXML Object. That is a little annoying, but I don't feel like digging into the C code and figuring out why. So, we just work around it. We made a recursive function to do the dirty work for us.

function makeArray($obj) {

$arr = (array)$obj;

if(empty($arr)){

$arr = "";

} else {

foreach($arr as $key=>$value){

if(!is_scalar($value)){

$arr[$key] = makeArray($value);

}

}

}

return $arr;

}

That will turn whatever you pass it into an array or empty string if it is empty.

But, while I was hacking around tonight, I came up with another idea. Check out this hackery:

$data = json_decode(json_encode($data));

Yeah! One liner. That converts all the SimpleXML elements into stdClass objects. All other vars are left intact.

Ok, so this is where someone in the comments can tell me about the magic SimpleXML method or magic OOP function I have missed to take care of all this. Go ahead, please make my code faster. I dare you.

PHP session cookie refresh

brianlmoon — Tue, 13 May 2008 19:40:47 -0500

I have always had an issue with PHP Sessions. Albeit, a lot of my issues are now invalid. When they were first implemented, they had lots of issues. Then the $_SESSION variable came to exist and it was better. Then memcached came to exist and you could store sessions there. That was better. But, still, after all this time, there is one issue that still bugs me.

When you start a session, if the user had no cookie, they get a new session id and they get a cookie. You can configure that cookie to last for n seconds via php.ini or session_cookie_set_params(). But, and this is a HUGE but for me, that cookie will expire in n seconds no matter what. Let me explain further. For my needs, the cookie should expire in n seconds from last activity. So, each page load where sessions are used should reset the cookie's expiration. This way, if a user leaves the site, they have n seconds to come back and still be logged in.

Consider an application that sets the cookie expiration to 5 minutes. The person clicks around on the site, gets a phone call that lasts 8 minutes and then gets back to using the site. Their session has expired!!!! How annoying is that? The only sites I know that do that are banks. They have good reason. I understand that.

My preference would be to either set an ini value that tells PHP sessions to keep the session active as long as the user is using the site. Or give me access to the internal function php_session_send_cookie(). That is the C function that sends the cookie to the user's browser. Hmm, perhaps a patch is in my future.

In the short term, this is what I do:



setcookie(

ini_get("session.name"),

session_id(),

time()+ini_get("session.cookie_lifetime"),

ini_get("session.cookie_path"),

ini_get("session.cookie_domain"),

ini_get("session.cookie_secure"),

ini_get("session.cookie_httponly")

);

That will set the session cookie with a fresh ttl.

Ok, going to dig into some C code now and see if I can make a patch for this.

Thoughts on the 2008 MySQL Conference and Expo

brianlmoon — Wed, 07 May 2008 12:48:47 -0500

Well, it has been almost a month. I know I am late to the blogosphere on my thoughts. Just been busy.

Again this year, the Phorum team was invited to be a part of the DotOrg Pavilion. What is that? Basically they just give expo floor space to open source projects. It is cool. We had a great location this year. We were right next to the area where they served food and drinks during the breaks. We had lots of traffic and met some of our power users. IMVU.com is getting 1.5 million messages per month in their Phorum install. They did have to customize it to fit into their sharding. But, that is expected. A guy (didn't catch his name) from Innobase came by and told us that they just launced InnoDB support forums on their site using Phorum. Cool. So now MySQL and Innobase use Phorum. I am humbled by the message that sends to me about Phorum.

Speaking of our booth, we were right next to the phpMyAdmin guys. Wow, that product has come a long way. I was checking out the visual database designer they have now. It was neat. I also met the Gentoo MySQL package maintainer. He was in the phpMyAdmin booth.

I was interviewed by WebDevRadio as I already posted. I was also asked to do a short Q&A with the Sun Headlines video team. They used one part of my clip. I won't link to that. No, if you find it good for you. I need to be interviewed some more or something. I did not look comfortable at all.

There were lots of companies with open in their name or slogan. I guess this is expected pandering.

I attended part of the InnoDB talk given by Mark Callaghan of Google. It appears that Google is serious about improving InnoDB on large machines. That is, IMO, good news for anyone that likes InnoDB. If I counted right, they had more than 5 people who at least part of their job is to improve InnoDB.

I gave my two talks. The first had low attendance, but the feedback was nice. It was just after the snack break in the expo hall and I was in the farthest room from the expo hall. That is what I keep telling myself. =) The second was better attended and the feedback seemed good there. I was told by Maurice (Phorum Developer) that I talked too fast and at times sounded like Mr. Mackey from South Park by repeating the word bad a lot. I will have to work on that in the future. I want to do more speaking.

On the topic of my second talk, there seemed to be a lot of "This is how we scaled our site" talks. I for one found them all interesting. Everyone solves the problem differently.

Next year I am thinking about getting more specific with my talk submissions. Some ideas include: PHP, MySQL and Large Data Sets, When is it ok to denormalize your data?, Using memcached (not so much about how it works), Index Creation (tools, tips, etc.).

In closing, I want to give a big thanks to Jay Pipes and Lenz Grimmer from MySQL. Despite Jay's luggage being lost he was still a big help with some registration issues among other things. Both of them helped out the Phorum team a great deal this year. Thanks guys.

Interview with WebDevRadio

brianlmoon — Sat, 03 May 2008 10:13:00 -0500

While I was at the MySQL Conference, I sat down with Michael Kimsal of WebDevRadio and recapped the two talks that I gave at the conference. I have uploaded the slides so you can follow along if you want.

One to a Cluster - The evolution of the dealnews.com architecture.

MySQL Tips and Tricks - Some simple tips and some of the more advanced SQL we use in Phorum.

Thanks Michael. Any time you need a guest, just let me know.

2008 MySQL Conference

brianlmoon — Mon, 31 Mar 2008 21:46:38 -0500

In just two weeks I will be heading to the 2008 MySQL Conference. I will be speaking this year. My two talks are:

MySQL Hacks and Tricks to Make Phorum Fast
04/16/2008 4:25pm PDT Room: Ballroom A

From One Server to a Cluster
04/16/2008 5:15pm PDT Room: Ballroom C

I have to pull back to back talks. *PHEW* I hope I can hold up. To make it worse, they did not put me in the same room. If I remember right though, those are really close to each other.

Of course, the Phorum team will be in the Expo Hall in the DotOrg pavilion. Just look for the big dog.

Local: Best practices for SQL backed web applications

brianlmoon — Thu, 06 Mar 2008 13:08:35 -0600

When
Tuesday, March 11, 2008 at 12:00 PM

Where
BizTech
515 Sparkman Drive
Huntsville , AL 35816

Details
Brian Moon of dealnews.com will be discussing best practices for writing database backed web based applications. Many users teach themselves SQL and programming on the web. Other developers may have experience in enterprise desktop applications. No matter what your background, there are common mistakes made when deploying web based applications that use a database.

Also, at this event, we will be giving away two copies of NuSphere's PhpED. Plus, everyone who attends can purchase any NuSphere product at 50% off.

Lunch will be served at this event.

Speaking at MySQL Conference 2008

brianlmoon — Wed, 20 Feb 2008 15:12:01 -0600

I had mentioned a while back that I submitted three proposals for the 2008 MySQL Conference. Well, two were accepted.

From one server to a cluster

In the last 10 years, dealnews.com has grown from a single shared hosting account to an entire rack of equipment. Luckily, we started using PHP and MySQL very early in the company's history.

From the early days of growing a forum to surviving Slashdotting, Digging and even a Yahoo! front page mention, we have had to adapt both our hardware and software many times to keep up with the growth.

I will discuss the traps, bottlenecks, and even some big wins we have encountered along the way using PHP and MySQL. From the small scale to using replication and even some MySQL Cluster. We have done many interesting things to give our readers (and our content team) a good experience when using our web site.

MySQL hacks and tricks to make Phorum fast

Phorum is the message board software used by MySQL. One reason they chose Phorum was because of its speed. We have to use some tricks and fancy SQL to make this happen. Things we will talk about in this session include:

Using temporary tables for good uses.

Why PHP and MySQL can be a bad mix with large data sets.

What mysqlnd will bring to the table with the future of PHP and MYSQL.

How Phorum uses full text indexing and some fancy SQL to make our search engine fast.

Forcing MySQL to use indexes to ensure proper query performance.

You can find my conference page here. (as Terry would say, me, me, me!)

Apache Worker and PHP

brianlmoon — Wed, 13 Feb 2008 01:09:32 -0600

The PHP manual basically tells you not to use Apache 2 with a threaded MPM and PHP as an Apache module. In general, it may be good advice. But, at dealnews.com, we have found it very valuable.

Apache threaded MPMs

Well, first, what is an MPM? It stands for Multi-Processing Module. It is the process model that Apache uses for its children process. Each request that comes in is handed to a child. Apache 1 used only one model for this, the prefork model. That uses one process per Apache child. The most commonly used threaded MPM is the Worker MPM. In this MPM, you have several processes that run multiple threads within it. This is the one I will be talking about. You can read more on Apache MPMs at the Apache web site.

Huge memory savings

With the Apache prefork or even FastCGI, each apache/php process allocates its own memory. Most healthy sites I have worked on use about 15MB of memory per apache process. Code that has problems will use even more than this. I have seen some use as much as 50MB of RAM. But, lets stick with healthy. So, a server with 1GB of RAM will only realistically be able to run 50 Apache processes or 50 PHP children for FastCGI if each uses 15MB or RAM. That is 750MB total. That leaves just 256MB for the OS and other applications. Now, if you are Yahoo! or someone else with lots of money and lots of equipment, you can just keep adding hardware. But, most of us can't do that.

As I wrote above, the worker MPM apache uses children (processes) and threads. If you configure it to use 10 child processes, each with 10 threads you would have 100 total threads or clients to answer requests. The good news is, because 10 threads are in one process, they can reuse memory that is allocated by other threads in the same process. At dealnews, our application servers use 25 threads per child. In our experience, each child process uses about 35MB of RAM. So, that works out to about 1.4MB per thread. That is 10% the usage for a prefork server per client.

Some say that you will run out of CPU way before RAM. That was not what we experienced before switching to worker. Machines with 2GB of RAM were running out of memory before we hit CPU as a bottleneck due to having just 100 Apache clients running. Now, with worker, I am happy to say that we don't have that problem.

Building PHP for best success with Worker

This is an important part. You can't use radical extensions in PHP when you are using worker. I don't have a list of extensions that will and won't work. We stick with the ones we need to do our core job. Mainly, most pages use the mysql and memcached extension. I would not do any fancy stuff in a worker based server. Keep a prefork server around for that. Or better yet, do funky memory sucking stuff in a cron job and push that data somewhere your web servers can get to it.

Other benefits like static content

Another big issue you hear about with Apache and PHP is running some other server for serving static content to save resources. Worker allows you to do this without running two servers. Having a prefork Apache/PHP process that has 15MB of RAM allocated serve a 10k jpeg image or some CSS file is a waste of resources. With worker, like I wrote above, the memory savings negate this issue. And, from my benchmarks (someone prove me wrong) Apache 2 can keep up with the lighttpds and litespeeds of the world in terms of requests per second for this type of content. This was actually the first place we used the worker mpm. It may still be a good idea to have dedicated apache daemons running just for that content if you have lots of requests for it. That will keep your static content requests from over running your dynamic content requests.

Some issues we have seen

Ok, it is not without problems (but, neither was prefork). There are some unknown (meaning undiagnosed by us) things that will occasionally cause CPU spikes on the servers running worker. For example, we took two memcached nodes offline and the servers that were connected to them spiked their CPU. We restarted Apache and all was fine. It was odd. We had another issue where a bug in my PHP code that was calling fsockopen() without a valid host name and a long timeout would cause a CPU spike and would not seem to let go. So, it does seem that bad PHP code makes the server more sensitive. So, your mileage may vary.

As with any new technology, you need to test a lot before you jump in with both feet. Anyone else have experience with worker and want to share?

One last tip

We have adopted a technique that Rasmus Lerdorf had mentioned. We decide how many MaxClients a server can run and we configure that number to always run. We set the min and max settings of the Apache configuration the same. Of course, we are running service specific servers. If you only have one or two servers and they run Apache and MySQL and mail and dns and... etc. you probably don't want to do that. But, then again, you need to make sure MaxClients will not kill your RAM/CPU as well. I see lots of servers that if MaxClients was actually reached, they would be using 20GB of RAM. And, these servers only have 2GB of RAM. So, check those settings. If you can, configure it to start up more (all if you can) Apache process rather than a few and make sure you won't blow out your RAM.

Caching for fun and profit, a BoF at OS CON

brianlmoon — Tue, 03 Jul 2007 13:44:27 -0500

Last year at O'Reilly's Open Source Conference I hosted a BoF (Birds of a Feather) about memcached. It was a popular event. So, this year, I decided to broaden the scope to caching in general. Its titled Caching for fun and profit. It will be Wednesday, July 25 from 8:30-9:30pm in Room E141.

Anything goes. We can talk about memcached, Tugela, basic file caching... whatever.

More and more web sites are finding that they need to uses caching to increase their performance. There are those of us that have solved some problems. Others that are new to these techniques have a lot of questions. This BoF is an opportunity for web developers to share their ideas on caching.

Specifically, the dealnews.com dev team can talk about the 3 main types of caching we use and where each is applicable.

Caching and patience

brianlmoon — Sat, 23 Jun 2007 08:56:23 -0500

I am subscribed to the memcached list. Memcached use (and interest in caching in general) has exploded it seems in the last couple of years. There are a lot of people that join the list that not only don't have a lot of experience with memcached, but they don't have experience with caching. We have been caching at dealnews since the beginning. We started using memcached at dealnews in early 2005.

One common concern that new users have is data not being updated the second it goes in the database. This is how it worked before they were caching. They are used to it working that way. So, when they start caching, they miss that instant gratification. We went through this at dealnews. Our content team could write a deal, go to the front page and see it right then. They could then move on with their lives. As we grew and it became apparent that we could do that anymore, we had to make changes. Does the front page really have to be updated the second we write a deal? We discovered the answer was no.

We use three primary techniques to keep our cache as up to date as it needs to be.

Sane TTLs

With tools like memcached, you can provide a TTL (time to live) for each cached item. This ensures that a particular piece of cache will not longer be used after a given time. We cache data for our front page for 2 minutes. It does not sound like a lot I know. But, we have a 84% hit rate on that cache. So, the data is never really that old, but the cache does a wonderful job. For other content that hardly changes, we use a ttl of a day or even an hour. You have to decide per object for your application if this is the right thing to do for you. TTLs are best, IMO, because you know what to expect. A surge in traffic can not force the cache to expire more quickly.

Forcibly updated

If you can't use a TTL, removing, or better yet updating the cache via code is another option. If we have objects that need to be updated, we will usually update the cache rather than simply expiring it. For us, we usually have a function that will return an object from cache, but if its not there, it will make the queries and create the cache. The function will generally have a force option that will recreate the cache for the item even if the cache is found. We gave a talk at Apachecon and wrote a paper that covered this topic in 2001 (see Caching in the Real World on that page). The basics in that paper still hold true for caching today. WARNING! There are a couple actually. If your data is updated constantly and you are doing this on every single insert/update to your database, you are wasting your time. You have to use your cache wisely. Ask yourself, "Does this data have to be real time?" The second warning is that when you come under high load, one expiring item on your page can cause thousands of queries to be run. We experienced a little of this when Yahoo linked us.

Pushed Cache

We have been using this method for a while in our ad serving software. We are now using it more and more. IMO, its the most sure fire way to handle increased load. Basically, you don't have the pages of your web site make SQL requests to the live SQL data in the event no cache is found. That is what I call a pulled cache. Instead, you push the data from your primary database into some caching (or even another, optimized SQL server) for your web site to use. We are actually using MySQL Cluster for this purpose on our web site. The forward facing web site hits only the MySQL Cluster. If the data is not there, its just not there. We have processes on our backend that gather data from our primary database, assemble it for presentation and populate the cluster. The queries that the web site uses to access the cluster are highly optimized. You could do the same with memcached, but memcached is volatile. With cluster, we have high availability and get about the same performance as we did with a fully cached paged.

Exceptions

Of course, there are exceptions. Forums are a good example. For them to be useful, its kind of hard to cache a lot of stuff. With Phorum, we do cache things like user profiles, forum settings and other slow changing items. But, caching messages for any amount of time usually has a low ROI. They update so fast and if users don't see updates they lose interest.