PHP and Memcached: The state of things

Wed, Jun 23, 2010 11:48 AM
Memcached is the de facto standard for caching in dynamic web sites. PHP is the one of the most widely used languages on the web. So, naturally there is lots of interest in using the two together. There are two choices for using memcached with PHP: PECL/memcache and PECL/memcached. Great names huh? But as of this writing there are issues with the two most popular Memcached libraries for PHP. This is a summary of those issues that I hope will help people being hurt by them and may bring about some change.

PECL/memcache

This is the older of the two and was the first C based extension for using memcached with PHP. Before this all the options were native PHP code. While they worked, they were slower of course. C > PHP. That is just fact. However, there has not been much active development on this code in some time. Yes, they have fixed bugs, but support for new features in the memcached server have not been added to this extension. Newer version of the server suppot a more efficient binary protocol and many new features. In addition, there are some parts of the extension that simply don't work anymore.

The most glaring one is the delete() function. It takes a second parameter that is documented as: "the item will expire after timeout seconds". In fact that was never a feature of memcached. It was completely misunderstood by the original extension authors. When that parameter was supported, it locked the key for timeout seconds and would not allow a new add operation on that key. Second, this feature was completely removed in memcached 1.4.0. So, if you send a timeout to the delete function, you simply get a failure. This is creating a huge support issue in the memcached community (not the PHP/PECL community) with people not being able to delete keys because of bad documentation and unsupported behavior. I sent a documentation patch for the this function to the PHP Docs list. I then modified it based on feedback. But since I have heard nothing about it getting merged. I have a PHP svn account, but even if I do have karma on the docs repository, I don't want to hijack them without the support of the people that work on the docs all the time. If you are reading this and can change the docs, please make the documentation for that function say "DON'T USE THIS PARAMETER, IT HAS BEEN DEPRECATED IN THE MEMCACHED SERVER!" or something.

Not too long ago the extension was officially abandoned by its original maintainers. Some people stepped up and claimed they wanted to see it continue. But, since that time, there have been no releases. There are bugs being closed though so maybe there is good things coming.

A very problematic issue with this extension is with the 3.0 beta release. It needs to just die. It has huge bugs that, IMO, were introduced by the previous maintainers in an effort to bring it up to speed, but never saw them through to make sure the new code worked. In their defense, it is marked as beta on PECL. But, thanks to Google, people don't see the word beta anymore. There are lots of people using this version and when they get bad results they blame the whole memcached world. Really, the new maintainers would do the world a favor if they just removed the 3.x releases from the PECL site.

PECL/memcached

This extension was started by Andrei Zmievski while working at Digg.com as an open source developer. It uses libmemcached, a C++ memcached library that does all the memcached work. This made it quite easy to support the new features in the memcached daemon as it as it was being developed at the same time as the new server. However, it has not had a stable release on PECL in nearly a year except for release to make it compatible with new versions of libmemcached. No bug fixes and no new features. There are currently 28 open bugs on PECL for this extension. Not all of which are bugs. Some are feature requests. The ironic thing is that the GitHub repository for this extension has seen a lot of development. But, none of these bug fixes have made it into the official PECL channel. And some of these bugs are major and others are just huge WTF for a developer.

The most major bug is one that I found. If you use persistent connections with this extension, it basically leaks those connections, not reusing an existing connection but also not closing the ones already made. This uses up the memory in your processes until they crash and it creates an exponential number of connections to your memcached server until it has no more connections available. Andrei does report in the bug that it is fixed in hist GitHub. But, for now, you can't use persistent connections.

The big WTF for a developer is that some functions don't take a numeric key and use it properly.
<?php

$mc = new Memcached();

$mc->addServer("localhost", 11211);

$mc->set(123, "yes!");

var_dump($mc->getMulti(array(123)));

?>
The above code should generate:
array(1) {
  [123]=>
  string(4) "yes!"
}
But, in reality, it generates:
bool(false)
The set succeeds, but the getMulti fails. Again, this is being fixed in GitHub, but is not available for release on PECL.

Compatibility

One big issue with these two extensions is that they are not drop in replacements for each other. If you want to move from one to the other, you have to take into consideration some time to convert your code. For instance, the older extension's set() function takes flags as the third parameter. The newer extension does not take a flags parameter at all. The older uses get() with an array to do a multi-get and the newer extension has a separate method for that called getMulti(). There are others as well.

Summary

So, what should you do as a PHP developer? If you are deploying memcached today, I would use the 2.2.x branch of PECL/memcache. It is the most stable. Just avoid the delete bug. It is not as fast and does not have the features. But, it is very reliable for set, get, add.... the basics of memcached. For the long term, it is a bit unclear. PECL/memcached looks to fix a lot of things in 2.0. But, I have not used it yet. In addition the long term growth of the project is a bit in question. Will there be a 2.1, 2.2, etc? I hope so. The other unknown is if the those people fixing the PECL/memcache bugs will keep it up and release a good stable product that supports new features of the server. Again, I hope so. The best scenario would be to have a choice between two fully compatible and feature rich extensions. Keep your fingers crossed.
26 comments
Gravatar for Brian Aker

Brian Aker Says:

Hi!

Thanks for this write up. I've been wondering about this myself, and feeling like I might need to work on a client myself just to get it going again.

Cheers,
   -Brian

Gravatar for Michael Shadle

Michael Shadle Says:

PECL/memcache is what I use right now. Granted, not a lot, but the one thing I wanted to see was a way to add a prefix to keys, as well as a response to come back if a key was missed.

Right now in "userland" PHP code I have to do multiple array cycles to glue the prefix on each key, pass it to memcached, get the result back, and cycle through the result, removing the key prefix.

(Think about memcached servers supporting multiple sites or environments - I give each site/client with their own prefix - like a namespace - so there's little chance of collisions. Of course a cache clear will dump it for everyone, but "oh well")

Also during the array iterations I have to check if I got a result back from memcached or not, if I didn't I need to hit the database. To minimize the amount of queries, I make an array of keys, implode() and then do a single DB query. All of this has to return back the array of data sorted by the key order I provided originally, so it has to maintain that as well.

It sounds like a lot but I've slimmed it down - but it's probably not the most efficient way of doing things and the library itself could handle the prefix work for sure, possibly even other things - without having to do PHP array work over and over.

I could provide an example, and I am sure it is EXTREMELY easy to do inside of the PECL modules.

Ideally the best things from each PECL module would be adopted into one, and I would like to see the "memcached" name be used as the "official" module in the future. There's pros and cons on both (hell, I think it was you who made a blog post once about their differences, IIRC.)

I think I posted once to the PHP mailing list or gave my example code of what I did and why (and asked if there was any way to cut it down more) but this was a couple years ago now. But I would love to see that become more simplified. I'm glad to see you shining a light on this and Brian's brain already spinning on this :)

Gravatar for Gerard

Gerard Says:

Great post Brian.

I was just starting some work with memcache after a long hiatus when your post came up on PHPDeveloper. Perfect timing. Maybe you could make a comment or answer over on Server Fault, which is the top Google result for this question.

http://serverfault.com/questions/63383/memcache-vs-memcached

Gravatar for Tom

Tom Says:

This line-up was just what I've been looking for. Thanks!

Gravatar for Herman Radtke

Herman Radtke Says:

I am maintaining the pecl/memcache extension. I will work with Pierre Joye to get a new release out shortly for the delete bug.

Gravatar for Brian Moon

Brian Moon Says:

Awesome, thank you very much Herman.

Gravatar for Sagar Sonawane

Sagar Sonawane Says:

Hi,

Good work to summarize issues and will surely answer questions of many. But, one thing i really felt was, there is no proper listing of "Right" Combination of libraries per OS!!
I have wasted(? / invested) lot of time in trying various combination to get "latest stable releases of libraries with my os", if it is provided by developers or some really experieced guys over here, then it will surely benefit neophytes like me :)

Its been almost 7-8 months, since im working closely with Memcache, i have used PECL/Memcache and currently experimenting(hope my seniors not reading this..they will be shocked to see im "experimenting") with PECL/Memcached, still im struggling to find correct combination of libmemcached+PECL/Memcached+Memcached(Server)+CentOS/RHEL!!

Also, many blogs are so misleading , because i think they confuse many times without making clear differentiation between "PECL/Memcached PHP Client Library" & "Libmemcached Library" & "Memcached Server".

I have also suggested/asked many queries in memcached google groups.

I hope senior guys like you and others in developers list will continue to help neophytes like me.

Thanks to all!

Gravatar for J. Bruni

J. Bruni Says:

Hi, Brian,

I have linked this post in stackoverflow, here:
http://stackoverflow.com/questions/983925/

But then I read the following note:
http://ca2.php.net/manual/en/memcached.construct.php#93536

And I was wondering if the mentioned issue of re-adding servers was your case...

I'm really interested in the answer. I need to decide if we will use persistent connections with PHP memcached or not...

Thanks!

Gravatar for Brian Moon

Brian Moon Says:

@J There is a very simple test script for this in the bug. http://pecl.php.net/bugs/bug.php?id=17382 It destroys the object completely on every iteration and creates it.

Gravatar for J. Bruni

J. Bruni Says:

But, Brian, the advice in the PHP note is:

"When using persistent connections, it is important to not re-add servers."

Read with care: http://ca2.php.net/manual/en/memcached.construct.php#93536

Your code does not follow this advice...

The object is destroyed, but not the connection (right? it is "persistent"!)

When the connection is recreated (either in the next loop iteration, or in the next script call), it already has that server added. When you call "addServer" on it, it will add the same server a second time!

That's why you should check "getServerList" before adding a server, i.e., only add a server if it is not already there. Thus, the advice: "do not re-add a server"...

Makes sense? Or did I miss something?

Gravatar for Brian Moon

Brian Moon Says:

Well, according the author of the project, that is a bug. The bug is still open and he is attempting to fix it. So, no, I don't consider that to be an acceptable answer.

Gravatar for J. Bruni

J. Bruni Says:

Ok, agreed.

So, we do have a bug.

Anyway, the PHP manual note advice seems useful as a workaround...

I just made a test similar to yours... http://pastebin.com/nZP3MGDT
(only added an "if" before the "addServer" line)

Thanks a lot!

Gravatar for Bill Getas

Bill Getas Says:

Two things. #1 Given that git bests SVN and PECL, github activity is to be commended. This should not present any problem whatsoever. #2 http://us3.php.net/manual/en/memcached.construct.php (see comment by tschundler at gmail dot com 15-Sep-2009 03:43) gives example of how to avoid constantly re-adding servers in case PECL version of memcached is used instead of git version.

Gravatar for CaptTofu

CaptTofu Says:

I was working on getting PECL/memcached to support a newer libmemcached - especially with Brian Aker's changes to have stats be execute/fetch (last time I checked, the number of stats was hard-coded/fixed). I need to revive that effort, unless of course someone got to it first!

Gravatar for OZ

OZ Says:

Memcache can be used with APC - it will give caching bytecode and user's variables. APC works much faster, than Memcache, but for some scripts Memcache can be only choice.

Michael Shadle, to use a prefixes for keys (I call it "tags"), you can try this wrapper: https://github.com/jamm/memory

Comments are disabled for this post.