Ramblings of a web guy

PHP command line progress bar

brianlmoon — Wed, 10 Mar 2010 11:33:34 -0600

Was just looking through some code and came across this function I wrote some time ago. If you do a lot of your processing scripts in PHP like we do, you probably need to know what is going on sometimes. So, I made a progress bar for use on the cli. I thought I would share it. Here is a video of it in action. And the code can be found here.

Developers should write code for production

brianlmoon — Wed, 03 Mar 2010 11:20:02 -0600

Having development and staging environments that reflect production is a key component of DevOps. An example for us is dealing with our CDN.

I can imagine in some dysfunctional, fragmented company, a developer works on a web application and sticks all the images in the local directory with his scripts. Then some operations/deployment guy has to first move the images where they need to be and then change all the code that references those images. If he is lucky, he has a script that does it for him. This is a needless exercise. If you have a development environment that looks and acts like production, this is all handled for you.

Here is an example of how it works for us. We use a CDN for all images, javascript, CSS and more. Those files come from a set of domains: s1.dlnws.com - s5.dlnws.com. So, our dev environments have similar domains. somedev.s5.dlnws.com points to a virtual server. We then use mod_substitute in Apache to rewrite those URLs on the dev machine. Each developer and staging instances will have an Apache configuration such as:

Substitute "s|http://s1.dlnws.com|http://somedev.s1.dev.dlnws.com|in"
Substitute "s|http://s2.dlnws.com|http://somedev.s2.dev.dlnws.com|in"
Substitute "s|http://s3.dlnws.com|http://somedev.s3.dev.dlnws.com|in"
Substitute "s|http://s4.dlnws.com|http://somedev.s4.dev.dlnws.com|in"
Substitute "s|http://s5.dlnws.com|http://somedev.s5.dev.dlnws.com|in"

So our developers put the production URLs for images into our code. When they test on the development environment, they get URLs that point to their instance, not production. No fussing with images after the fact.

In addition to this, we use mod_proxy to emulate our production load balancers. Special request routing happens in production. We need to see that when developing so we don't deploy code that does not work in that circumstance. If the load balancers send all requests coming in to /somedir to a different set of servers, we have mod_proxy do the same thing to a different VirutalHost in our Apache configuration. It is not always perfect, but it gets us as close to production as we can get without buying very expensive load balancers for our development environments.

Of course, we did not come to this overnight. It took us years to get to this point. Maybe it won't take you that long. Keep in mind when creating your development environments to make them work like production. It is neat to be able to write code on your laptop. I did it for years. But, at some point before you send out code for production, the developer should run it on a production like environment. Then deploying should be much easier.

DevOps is the only way it has ever made sense

brianlmoon — Mon, 22 Feb 2010 20:24:32 -0600

DevOps is the label being given to the way we have always done things. This is not the first time this has happened. As it says on my About Me page,

Brian Moon has been working with the LAMP platform since before it was called LAMP.

At some point, not sure when, someone came up with LAMP. I started working on what is now considered LAMP in 1996. I have seen lots of acronyms come and some go. We started using "Remote Scripting" after hearing Terry Chay talk about it at OSCON. The next OSCON, AJAX was all the rage. Technically, we never used AJAX. The X stands for XML. We didn't use XML. What made sense for us was to send back javascript arrays and objects that the javascript interpreter could deal with easily. We wrote a PHP function called to_javascript that converted a PHP array into a javascript array. Sound familiar? Yeah, two years later, JSON was all the rage.

We also have seen the same thing with how we run our development process. We always considered our team to be an agile development team. That is agile with little a. Nowadays, "Agile" with the big A is usually all about how you develop software and not about actually delivering the software. So, I am always perplexed when people ask me if we use "Agile" development. Are they talking little a or big A?

Today I came across the term DevOps on twitter (there is no Wikipedia page yet). We have always had an integrated development and operations team. I could be writing code in the morning and configuring servers in the afternoon. Developers all have some level of responsibility over managing their development environment. They updated their Apache configurations from SVN and make changes as needed for their applications. The development environments are simulated as close as possible to production. Developers roll code to the production servers. It is their responsibility to make sure it works on production. They also roll it when it is ready rather than letting it sit around for days. This means if there is an unforeseen issue, the code is fresh on their minds and the problem can quickly be solved. We have done things this way since 1998. We are not the only ones. The great guys at Flickr gave a great talk last year at Velocity about their DevOps environment. People were amazed at how their teams worked together.

One of the huge benefits of being a DevOps team is that we can utilize the full stack in our application. If we can use the load balancers, Apache or our proxy servers to do something that offloads our application servers, we plan for that in the development cycle. It is a forethought instead of an afterthought. I see lots of PHP developers that do everything in code. Their web servers and hardware are just there to run their code. Don't waste those resources. They can do lots of things for you.

One cool thing about this is that I now have a label to use when people ask us about our team. I can now say we are an agile DevOps team. They can then go look that up and see what it means. Maybe it will lead to less explanation of how we work too. And if we are lucky, maybe we can find people to hire that have been in a similar environment.

So, I welcome all the new people into the "DevOps movement". Adopt the ideas and avoid any rules that books, blogs, "experts" may want to come up with. The first time I see someone list themselves as a DevOps management specialist, I will die a little on the inside. It is not a set of rules, it is a way of thinking, even a way of life. If the process prevents you from doing something, you are using it wrong, IMO.

State of the Browsers and ad blocking

brianlmoon — Mon, 01 Feb 2010 23:25:55 -0600

In my last post about CSS layout and ads, a commenter brought up that the dealnews.com web site did not handle extensions like Ad Block very gracefully. To which I responded that I don't care. To which he responded with download counts. Well, the reason I don't care is that ad impressions when compared to page views on dealnews.com are within 2% of each other. So, at best, less than 2% of users are blocking ads. In reality, that is going to include some DNS failures, network issues, or something else. I would bet our logo graphic has about the same difference. The reality is that normal people don't block ads. In my opinion, if you make your money by working on the web, you shouldn't either. I should add that this site's (my geeky blog) ad views was about 16% lower than the recorded page views. So, geeks block ads more I guess. But, geeks have dominated the web for a long time.

This got me thinking that I had not look at the browser stats very much lately. dealnews has a very odd graph on browser statistics. We do not follow the industry averages. Our audience is dominantly tech savy (that does not mean geeks). Our users don't just use the stuff that is installed on the computer when they get it. This kind of proves my point about ad blocking even more. We have non-moron users and they still don't block ads.

Browser	% of Visits
Internet Explorer	42.34%
Firefox	36.94%
Safari	9.55%
Chrome	8.34%
Mozilla	1.46%
Opera	0.68%
Netscape	0.41%
Avant	0.08%
Camino	0.06%
IE Mobile	0.02%

As you can see, Firefox is very prevalent on our site. We generally test in IE7/8, Firefox 3, Safari and Chrome. I will occasionally test a major change in Opera. Typically, well formed HTML and CSS works fine in Opera so everything is all good.

As for operating systems, Windows still dominates, but we have more Macs than the average site I would guess.

OS	% of Visits
Windows	82.95%
Macintosh	11.27%
iPhone	3.80%
Linux	1.19%
Android	0.17%

Interesting that iPhone beats out Linux. That is just another sign to me that Linux is still not a real choice for real people. Be that a product issue from OEMs or user choice. That is debatable. It is notable that most of our company uses Macs. I don't think we make up a speck of that traffic though. If we did, our home state of Alabama would be our most dominant. It isn't. We are very typical in that regard, California is number one. We only have one employee there.

Tables for layout

brianlmoon — Fri, 29 Jan 2010 10:01:39 -0600

I just read A case for table-based design and was thrilled to know I am not the only one that drowns in div soup from time to time. I do not for the most part use tables for layout, but there are some cases where I just can't make a set of divs do my bidding. The classic example is having a two column layout where the left column LOADS FIRST and is elastic and the right column is a fixed size. The "loads first" is important in a world where the rendering time of pages has become important. Ideally with any page, the most important content would render first for the user. In my case, this fixed column is an ad. As a web developer, I don't care when the ad loads. The ads are a necessary evil in my page layout. I must ensure that they load in an acceptable time frame, but certainly not the first thing on the page. The specific layout I am talking about is that of the top of dealnews.com. It has a fixed size 300x250 ad on the right of the page and the left side is elastic. I fiddled with divs for hours to get that to act the way I wanted it to act. We use the grid CSS from OOCSS.org. Wonderful piece of CSS that is. But, even with that in hand, I could not get the elements to behave, in all browsers, the way I could with a simple two column table where the left column's width is set to 100% and the right column contains a div of width 300 pixels. It was so easy to pull that off. Maybe CSS3 is going to solve this problem? I don't know. If you have the magic CSS that can do what this page does, let me know.

ob_start and HTTP headers

brianlmoon — Fri, 29 Jan 2010 10:00:00 -0600

I was helping someone in IRC deal with some "headers already sent" issues and told them to use ob_start. Very diligently, the person went looking for why that was the right answer. He did not find a good explination. I looked around and I did not either. So, here is why this happens and why ob_start can fix it.

How HTTP works

HTTP is the communication protocol that happens between your web server and the user's browser. Without too much detail, this is broken into two pieces of data: headers and the body. The body is the HTML you send. But, before the body is sent, the HTTP headers are sent. Here is an example of an HTTP request response including headers:

HTTP/1.1 200 OK
Date: Fri, 29 Jan 2010 15:30:34 GMT
Server: Apache
X-Powered-By: PHP/5.2.12-pl0-gentoo
Set-Cookie: WCSESSID=xxxxxxxxxxxxxxxxxxxxxxxxxxxx; expires=Sun, 28-Feb-2010 15:30:34 GMT; path=/
Content-Encoding: gzip
Vary: Accept-Encoding
Keep-Alive: timeout=15, max=99
Connection: Keep-Alive
Transfer-Encoding: chunked
Content-Type: text/html; charset=UTF-8




 
 Ramblings of a web guy
.
.

So, all those lines before the HTML starts have to come first. HTTP headers are where things like cookies and redirection occur. When a PHP script starts to send HTML out to the browser, the headers are stopped and the body begins. When your code tries to set a cookie after this has started, you get the "headers already sent" error message.

How ob_start works

So, how does ob_start help? The ob in ob_start stands for output buffering. ob_start will buffer the output (HTML) until the page is completely done. Once the page is completely done, the headers are sent and then the output is sent. This means any calls to setcookie or the header function will not cause an error and will be sent to the browser properly. You do need to call ob_start before any output occurs. If you start output, it is too late.

The down side

The down side of doing this is that the output is buffered and sent all at once. That means that the time between the user request and the time the first byte gets back to the user is longer than it has to be. However, in modern PHP application design, this is often already the case. An MVC framework for example would do all the data gathering before any presentation is done. So, your application may not have any issue with this.

Another down side is that you (or someone) could get lazy and start throwing setcookie calls in any old place. This should be avoided. It is simply not good programming design. In a perfect world, we would not need output buffering to solve this problem for us.

Using ini files for PHP application settings

brianlmoon — Tue, 19 Jan 2010 17:24:35 -0600

At dealnews we have three tiers of servers. First is our development servers, then staging and finally production. The complexity of the environment increases at each level. On a development server, everything runs on the localhost: mysql, memcached, etc. At the staging level, there is a dedicated MySQL server. In production, it gets quite wild with redundant services and two data centers.

One of the challenges of this is where and how to store the connection information for all these services. We have done several things in the past. The most common thing is to store this information in a PHP file. It may be per server or there could be one big file like:








if(DEV){


    $server = "localhost";


} else {


    $server = "10.1.1.25";


}





?>

This gets messy quickly. Option two is to deploy a single file that has the settings in a PHP array. And that is a good option. But, we have taken that one step further using some PHP ini trickeration. We use ini files that are loaded at PHP's startup and therefore the information is kept in PHP's memory at all times.

When compiling PHP, you can specify the --with-config-file-scan-dir to tell PHP to look in that directory for additional ini files. Any it finds will be parsed when PHP starts up. Some distros (Gentoo I know) use this for enabling/disabling PHP extensions via configuration. For our uses we put our custom configuration files in this directory. FWIW, you could just put the above settings into php.ini, but that is quite messy, IMO.

To get to this information, you can't use ini_get() as you might think. No, you have to use get_cfg_var() instead. get_cfg_var returns you the setting, in php.ini or any other .ini file when PHP was started. ini_get will only return values that are registered by an extension or the PHP core. Likewise, you can't use ini_set on these variables. Also, get_cfg_var will always reflect the initial value from the ini file and not anything changed with ini_set.

So, lets look at an example.



; db.ini


[myconfig]


myconfig.db.mydb.db     = mydb


myconfig.db.mydb.user   = user


myconfig.db.mydb.pass   = pass


myconfig.db.mydb.server = host

This is our ini file. the group in the braces is just for looks. It has no impact on our usage. Because this is parsed along with the rest of our php.ini, it needs a unique namespace within the ini scope. That is what myconfig is for. We could have used a DSN style here, but it would have required more parsing in our PHP code.










/**


 * Creates a MySQLi instance using the settings from ini files


 *


 * @author     Brian Moon 


 * @copyright  1997-Present dealnews.com, Inc.


 *


 */





class MyDB {





    /**


     * Namespace for my settings in the ini file


     */


    const INI_NAMESPACE = "dealnews";





    /**


     * Creates a MySQLi instance using the settings from ini files


     *


     * @param   string  $group  The group of settings to load.


     * @return  object


     *


     */


    public static function init($group) {





        static $dbs = array();





        if(!is_string($group)) {


            throw new Exception("Invalid group requested");


        }





        if(empty($dbs["group"])){





            $prefix = MyDB::INI_NAMESPACE.".db.$group";





            $db   = get_cfg_var("$prefix.db");


            $host = get_cfg_var("$prefix.server");


            $user = get_cfg_var("$prefix.user");


            $pass = get_cfg_var("$prefix.pass");





            $port = get_cfg_var("$prefix.port");


            if(empty($port)){


                $port = null;


            }





            $sock = get_cfg_var("$prefix.socket");


            if(empty($sock)){


                $sock = null;


            }





            $dbs[$group] = new MySQLi($host, $user, $pass, $db, $port, $sock);





            if(!$dbs[$group] || $dbs[$group]->connect_errno){


                throw new Exception("Invalid MySQL parameters for $group");


            }


        }





        return $dbs[$group];





    }





}





?>

We can now call DB::init("myconfig") and get a mysqli object that is connected to the database we want. No file IO was needed to load these settings except when the PHP process started initially. They are truly constant and will not change while this process is running.

Once this was working, we created separate ini files for our different datacenters. That is now simply configuration information just like routing or networking configuration. No more worrying in code about where we are.

We extended this to all our services like memcached, gearman or whatever. We keep all our configuration in one file rather than having lots of them. It just makes administration easier. For us it is not an issue as each location has a unique setting, but every server in that location will have the same configuration.

Here is a more real example of how we set up our files.



[myconfig.db]


myconfig.db.db1.db         = db1


myconfig.db.db1.server     = db1hostname


myconfig.db.db1.user       = db1username


myconfig.db.db1.pass       = db1password





myconfig.db.db2.db         = db2


myconfig.db.db2.server     = db2hostname


myconfig.db.db2.user       = db2username


myconfig.db.db2.pass       = db2password





[myconfig.memcache]


myconfig.memcache.app.servers    = 10.1.20.1,10.1.20.2,10.1.20.3


myconfig.memcache.proxy.servers  = 10.1.20.4,10.1.20.5,10.1.20.6





[myconfig.gearman]


myconfig.gearman.workload1.servers = 10.1.20.20


myconfig.gearman.workload2.servers = 10.1.20.21

Autoloading for fun and profit

brianlmoon — Fri, 08 Jan 2010 09:44:47 -0600

So, as I stated in a previous post, the code base here at dealnews is going through some changes. I have been working heavily on those changes since then. Testing and benchmarking to see what works best. One of those changes is the heavy use of autoloading.

During the holidays, I always like to read the PHP Advent. One of the posts this year was by Marcel Esser titled You Don’t Need All That. It was a great post and echos many things I have said about PHP and web development. In his post, Marcel benchmarks the difference between using an autoloader and using straight require statements. I was not surprised by the result. The autoloading overhead and class overhead is well known to me. It is one thing that kept me from using any heavy OOP (we banned classes on our user facing pages for a long time) in PHP 4 and PHP 5.0. It has gotten a lot better however. Class overhead is very, very small now. Especially when using it as a namespacing for static functions. However, there is one word of caution I would like to add to his statements. When you use require/include statements instead of autoloading, you end up with a file like this:





require "DB.php";

require "Article.php";





function myfunc1() {

    DB::somemethod();

}



function myfunc2(){

    Article::somemethod();

}



?>

That file needs to require two files, but each one is only needed by one function in the file. This is the dilemma we have found ourselves in. We have a file that is filled with functions for building, taking apart, repairing, fetching, or anything else you can think to do with a URL. So, at the top of that file are 13 require statements. This all happened organically over time. But, now we are in a situation where we load lots of files that may not even be needed to begin with. By moving to an autloading system, we will only include the code we need. This saves cycles and file IO.

Again, Marcel's post was dead on. Our front end is written with multiple entry points. We use auto_prepend files to do any common work that all requests need (sessions, loading the application settings, etc). The front page of dealnews.com is about 600 lines of PHP that does the job as quickly as possible. But, we are moving it to use autoloaded code because its require list has grown larger than I would like and some of those requires are not always "required" to generate the page.

One point Marcel did make was about how modern MVC systems are part of the problem. We don't have a highly structured traditional MVC layout of our code. We use a very wide, flat directory structure rather than deep nesting of classes and objects. We also don't do a lot of overloading. Maybe 1% of our classes extend another and never more than one deep. All that makes for much quicker loading objects vs. some of the packaged frameworks like Zend Framework.

So, as Marcel warned, be aware of both your use of autoloading and require statements. Both can be bad when used the wrong way.

Bing ads are such bull crap

brianlmoon — Mon, 21 Dec 2009 12:03:08 -0600

Have you see these Bing ads? The insinuate that web search gives you a lot of crap. But, that Bing can cure that problem. Really? One simple question Bing:

How big is the sun?

Google

Google tells me the mass right off the bat. Neat. The following links are all very relevant too.

Yahoo

Yahoo gives me an interesting alternate query. The links are all relevant too.

Bing

Bing starts well by suggesting some queries.

But, it soon falls apart IMO.

Big Red Sun? A garden design company? That is the most relevant? Really? The third result is way off too. Their related searches clearly seem to indicate that Bing understands what I am looking for, but their results fail badly. Get the log out of your eye Bing.

Code Organization Dilemma

brianlmoon — Wed, 18 Nov 2009 08:00:00 -0600

So, we have been building up our code library at dealnews for 9 years. It was started at the end of PHP3 and the beginning of PHP4. So, we did not have autoloading, static functions, and all that jazz. Classes had lots of overhead in early PHP4 so we started down a pure procedural road in 2000. And for a long time, it was very maintainable. We had 2 or 3 developers for most of this time. We now have 5 or 6 depending on whether we have contractors. There are starting to be too many files and too many functions. We find ourselves adding new files when some new function is created instead of adding it to an existing file because we don't want to have huge files with 100 functions in them. File names and function names are getting longer and more ambiguous. For example, we have a file called url_functions.php. It contains functions to generate URLs for different types of pages on the site, functions to fetch URLs from the web and functions to parse URLs from an article. Those probably don't all belong in one file. But, they got nickle and dimed in there over time. So, now, we are inclined to not add anything to that file and make new files for new semi-URL related functions. Ugh.

It is time to start thinkinb about a reorganization. There are 1,900+ functions in 400+ files in our code library. This is just our library. This does not include the code that actually builds a page and generates output. It does not include our cron jobs or system administration scripts. Yeah, that is a lot. So, where do we go from here? Some things are easy to do. For example, we have a file called string.php. Most all the functions in that file can easily be moved a String class with static functions that can be accessed via an autoloader.

Then we have the various ways we deal with the articles on the web site. I have written about our front end vs. back end system before. What this means for our code base is that we have two ways to deal with an article. One is in our highly relational backend system. The other is in our optimized front end database servers. So, one Article object won't really do. We already have an Article object that serves as an ORM interface for the backend. To access the front end data, we currently have a library of functions (fetch_article for a single, fetch_articles for a set, etc.) but it does not fit with an autoloading environment. It also is not related to the object (the article) and is associated with where the data is stored. New developers don't grok the server infrastructure, so the code organization may not make sense to them. We have about 10 different objects that need both a back end and front end interface.

On the other hand, I really don't want to end up with a class named FrontEndArticle and BackEndArticle. Much less do we want to have stuff like BackEnd_Article where the file is actually in BackEnd/Article.php somewhere. The verbosity becomes overwhelming and hard to read, IMO.

So, what are others doing with huge code bases? I see lots of projects with 100 or so functions/methods in 20-30 files. Frameworks have it easy because they don't have a CEO that wants something on this one page to be different than it is on every other page where that data is used. We have to deal with those types of hacks in an elegant way that can be maintained.

Forums are crap. Can we get some help?

brianlmoon — Mon, 12 Oct 2009 10:01:44 -0500

Amy Hoy has written a blog post about why forums are crap. And she is right. Forum software does not always do a good job of helping people communicate. I have worked with Amy. She did a great analysis of dealnews.com that led to our new design. So, she is not to be ignored.

However, as a software developer (Phorum), I see a lot of problems and no answers. And it is not all on the software. Web site owners use forums to solve problems that they really, really suck at. Ideally, every web site would be very unique for their audience. They would use a custom solution that fits a number of patterns that best solves their problem. However, most web site owners don't want to take the time to do such things. They want a one stop, drop in solution. See the monolith that is vBulletin, scary.

And what if a forum is the best solution? Well, software developers, in general, are not good designers. They don't think like normal people. And they don't see their applications as a whole, but as pieces that do jobs. The forum software market has been run by software developers for over 10 years. Most of them all are still copies of what UBB was 13 years ago. And software (like Phorum) that has tried to be different is shunned by the online communities of the world because they don't work/look/feel like every other forum software on the planet.

So, as software developers, what are we to do? We want to make great software. We want to help our users help their users. But, what we have been doing for 10+ years has only been adequate. As the leader of an open source forum software project, I am open to any and all ideas.

Custom carousel with YUI and OOCSS.org

brianlmoon — Fri, 09 Oct 2009 15:52:19 -0500

YUI has a built in carousel widget. However, it requires fixed widths for all the parts. That does not fit well in a liquid CSS layout. In particular, we have to support people using large fonts at the OS level and large font settings in their browsers. I did not want for our carousel to break down in these cases. The built in YUI widget does. See this normal sized screen shot versus one with large fonts. Now look at dealnews with normal fonts and large fonts.

So, I decided to make my own using YUI and the grid CSS from Object-Oriented CSS. You can see my working example with some explination here.

Memcached: What is it and what does it do?

brianlmoon — Wed, 30 Sep 2009 11:49:39 -0500

I spoke at CodeWorks in Atlanta, GA this week. I totally dropped the ball promoting it on my blog. It was a neat venue. Rather than a large conference they are doing a traveling show. Seven cities in 14 days. Many of the presenters are working in every city. Crazy. I was just in Atlanta. It is close to home and easy for me to get to.

I spoke about memcached. I tried to dig a bit deeper into how memcached works. On the mailing list we get a lot of new people that make assumptions about memcached. Most talks I have seen focus on why caching is good, how to use memcached, the performance gain. I kind of assumed everyone knew that stuff already. I guess you could say I gave a talk that was the real FAQs of the project.

Here are the slides. Derick Rethans took video of the talk. When he gets that online I will add it to this post.

Memcached: What is it and what does it do?

View more documents from Brian Moon.

Wordcraft 0.10 available

brianlmoon — Mon, 10 Aug 2009 00:12:20 -0500

The latest package of Wordcraft, the PHP/MySQL based blog software that runs this site, is available for download from Google Code. Just some minor bug fixes and cosmetic stuff. Its getting a little use in the wild. That is always fun to see.

Forking PHP!

brianlmoon — Thu, 23 Jul 2009 13:17:42 -0500

We use PHP everywhere in our stack. For us, it makes sense because we have hired a great staff of PHP developers. So, we leverage that talent by using PHP everywhere we can.

One place where people seem to stumble with PHP is with long running PHP processes or parallel processing. The pcntl extension gives you the ability to fork PHP processes and run lots of children like many other unix daemons might. We use this for various things. Most notably, we use it run Gearman worker processes. While at the OReilly Open Sourc Convention in 2009, we were asked about how we pulled this off. So, we are releasing the two scripts that handle the forking and some instructions on how we use them.

This is not a detailed post about long running PHP scripts. Maybe I can get to the dos and don'ts of that another time. But, these are the scripts we use to manage long running processes. They work great for us on Linux. They will not run on Windows at all. We also never had any trouble running them on Mac OS X.

The first script, prefork.php, is for forking a given function from a given file and running n children that will execute that function. There can be a startup function that is run before any forking begins and a shutdown function to run when all the children have died.

The second script, prefork_class.php, uses a class with defined methods instead of relying on the command line for function names. This script has the added benefit of having functions that can be run just before each fork and after each fork. This allows the parent process to farm work out to each child by changing the variables that will be present when the child starts up. This is the script we use for managing our Gearman workers. We have a class that controls how many workers are started and what functions they provide. I may release a generic class that does that soon. Right now it is tied to our code library structure pretty tightly.

We have also included two examples. They are simple, but do work to show you how the scripts work.

You can download the code from the dealnews.com developers' page.

What is the problem? AT&T or Apple?

brianlmoon — Fri, 26 Jun 2009 14:20:59 -0500

So, I told you how AT&T messed up my account when I ordered the new iPhone. Well, I go the new one on Friday via FedEx. I then left for Velocity in San Jose, CA on Sunday. The phone was not activated when I left. So, I had to take my old iPhone and my new iPhone 3GS with me. Grrrr. On top of that, I had to keep them both on me because afaik, as soon as the new one started working, the old one would not. Double Grrrrrr.

I was blaming AT&T for this over and over. I tried calling them on Saturday and the guy on the phone said he was instructed to tell me to go to an AT&T store or an Apple Store. I did not have time to find an AT&T store. I tried the Apple store as we were having dinner right near it. But, the wait was an hour!!! Triple Grrrrrrr.

So, on Sunday night, I got this email from Apple:

Dear Apple Customer,

Thank you for your recent Apple Store order. We appreciate your patience and apologize for the inconvenience caused by the delay in your iPhone activation.

We are still resolving the issue that was encountered while activating your iPhone with AT&T. Unfortunately, due to system issues and continued high activation volumes, this could take us up to an additional 48 hours to complete.

On Monday, you'll receive an email from Apple with an iTunes Store credit in the amount of $30. We hope you will enjoy this gift and accept our sincere apologies for the inconvenience this delay has caused.

Thank you for choosing Apple.

Sincerely,
Apple Online Store Team

So, whose fault is this? Is it Apple or AT&T? I had been assuming AT&T all along. But, Apple is giving me the credit.

On Monday morning, at about 11:30CDT, my phone was activated. That is about 62 hours after I hooked it up to iTunes. The phone is great.

I guess I should listen to Louis CK says and just be happy that the future is here and quit bitching.

AT&T is a bleepity bleeping bleep

brianlmoon — Wed, 17 Jun 2009 00:29:38 -0500

So, on June 19th, the new iPhone comes out. I have a first generation iPhone. I have loved it. My area is not well covered by 3G so I did not bother upgrading. But, my two year contract is up next month and just the other day I dropped and cracked my iPhone screen. So, this is a good time to upgrade.

I ordered my new phone via the Apple web site. It took me through a wizard that checked my AT&T account and showed me the options for upgrading. That all went smooth. This all took place on Thursday, June 11, 2009. Well, at 11AM on Saturday, June 13th, something happened to my account. On my account are 4 phones. Mine, my wifes and my two oldest kids. My daughter says from the back seat (did I mention we are on our way to the beach for a week?) "That took them long enough. I got a welcome text from AT&T." She has had her phone since Christmas. Later I noticed I had no data connection. Just bars. I was getting calls. Then later my wife commented that no one was returning her SMS messages. So, we tried sending some to each other. Nothing. My son sees he got a welcome SMS too. OMG, wtf did AT&T do?!?!?!?

We are driving (I mentioned that I think) and have time on our hands, so my wife calls them about it all. Sure enough, they claim that "someone" removed our family unlimited text message plan from the account. Well, that someone would be AT&T. See, I had reduced my iPhone to the lower plan a while back because I was getting the family unlimited text messaging plan when I added my kids phones. But, the new iPhone only has one data plan available. It's a $30/month unlimited data plan. Apparently, me changing the data plan for the new iPhone that is not even shipped yet removed all text messaging for my whole family. The lady claims to have it fixed so we hang up.

Time passes and I still don't have and EDGE connection. My daughter tries to send a MMS picture and is blocked. We call back. Apparently the brilliant person my wife talked to the first time added a pay per SMS feature and NO MMS OR DATA! ARGHHH!!!! We tell the lady, "Look, just put it back like it was on the last bill." That worked, we were happy with that. A few minutes later I had data and we could send pictures again.

So, you would think it is over. But, apparently I don't have voice mail. And now I am really wondering what will happen when I activate my new iPhone this week. Be thinking about me as I am sure it will hose my account again. And I leave town on Sunday to attend Velocity.

Please, please, two years from now, when my contract is up, please let Verizon have the iPhone. Our accounts payable people at dealnews don't like their billing practices, but I have never had a customer service issue with them and their network is the best ever.

Some interesting things to note:

You can't buy or activate an iPhone without a data plan. But, clearly they had mine working without one.
They turned off SMS for my whole family, but, the first thing my kids got after they turned it off was an SMS that I bet they would have charged us for.
We could not send MMS at all for our whole account after the new iPhone was purchased. It is well known that AT&T is not allowing MMS on the new iPhone yet. Guess they control this at the account level?

Building PECL/memcache on Mac OS X

brianlmoon — Sun, 31 May 2009 22:23:29 -0500

My coworker Rob ran into an issue building the PECL/memcache extension on his Mac. He did find the solution however. You can read and leave comments on his blog.

The rise of the GLAMMP stack

brianlmoon — Fri, 22 May 2009 11:34:39 -0500

First there was LAMP. But are you using GLAMMP? You have probably not heard of it because we just coined the term while chatting at work. You know LAMP (Linux, Apache, MySQL and PHP or Perl and sometimes Python). So, what are the extra letters for?

The G is for Gearman - Gearman is a system to farm out work to other machines, dispatching function calls to machines that are better suited to do work, to do work in parallel, to load balance lots of function calls, or to call functions between languages.

The extra M is for Memcached - memcached is a high-performance, distributed memory object caching system, generic in nature, but intended for use in speeding up dynamic web applications by alleviating database load.

More and more these days, you can't run a web site on just LAMP. You need these extra tools (or ones like them) to do all the cool things you want to do. What other tools do we need to work into the acronym? PostgreSQL replaces MySQL in lots of stacks to form LAPP. I guess Drizzle may replace MySQL in some stacks soon. For us, it will likely be added to the stack. Will that make it GLAMMPD? We need more vowels! If you are starting the next must use tool for running web sites on open source software, please use a vowel for the first letter.

Is there a program for finding uses of register_globals?

brianlmoon — Fri, 15 May 2009 12:38:58 -0500

register_globals is going way in PHP6. That is fine with me. Super globals are cool and I have taken to using filter_input_array these days anyhow. However, our code base is now 10+ years old at dealnews. Most of the forward facing code was completely rewritten in the last couple of years due to architecture changes. Many new projects had register_globals turned off via php_admin_flag in Apache. So, that area is not that big of a problem. However, our internal admin areas have not all be rewritten because, well frankly, they still work. Yeah, stuff written for PHP4 in 2000 is still working. KISS helps a lot with that. But, this code, somewhere in there, may still be relying on register_globals. Now, we could go line by line and try and fix it. But, it seems like a program could be written to do this job. I mean, I use jEdit and it can highlight unset vars using the PHPParserPlugin just fine. I bet Zend IDE can do the same. Has anyone written such a tool for the command line? There will be false positives I know. Things like passing a variable by reference to a function would look like a use before set. But, I can deal with those if I don't have to go line by line through tons of old code. What would the rules look like for such an animal? This would be a great project to get off the ground before PHP6 hits. Ideally you could provide a list of variables for it to ignore. We have some globals we set up in prepends and includes.

Scaling for the Expected and Unexpected - Speaking at Velocity

brianlmoon — Thu, 07 May 2009 11:57:27 -0500

Last year I was surprised to be going to Velocity. Read the post, it was an adventure. But, I really like the conference. It is the perfect conference for me. While a good majority of my work is done coding PHP/MySQL apps, I tend to focus on architecture, frameworks, performance and that kind of stuff. So, a web performance and operations conference is just perfect.

Last year, I was on a panel with some great guys. I was able to share just a bit about my experience dealing with the instant success of a web site. This year, my proposal was accepted to talk more about dealing with success of a web site. The talk will be focused on my experience at dealnews.com and from working with power users for Phorum. Here is the summary:

Lots of people talk about scaling and performance. But, are they preparing for all the things that could happen? There are multiple problems and there is not one solution to solve them all.

Everything is running fine and BAM! – your site is linked from the front page of Yahoo! What do you do? How can you handle that sudden rush of traffic. Requests per second are running 5x normal levels. Servers have CPU spikes. Daemons are hitting the maximums. You are running out of bandwidth. How could you have been prepared for this? What are the tools and techniques for this type of sudden rush?

Or, lets say you have just come out of a meeting where everyone discovered that your site is growing in traffic 70% – 80% year over year. That means that 1 million page views this month will be nearly 3 million this time in 2 years. How can you plan for that? You don’t want to redesign the whole architecture every 2 years. What methods could be used to deal with this constant long term growth?

While there is no magic bullet for either of these scenarios, there are techniques used by many sites out there to help you get through these situations. This session will cover some of these techniques and talk about their pros and cons.

I must admit, this if the first time since 2000 that I am a little intimidated to speak at a conference. The people that present and attend Velocity are so awesome. I just hope I don't disappoint.

Net::Gearman and PHP 5.2.9

brianlmoon — Tue, 21 Apr 2009 12:48:53 -0500

I just discovered an incompatibility between Net Gearman and PHP 5.2.9+. json_decode was changed in 5.2.9 to return NULL on invalid JSON strings. Previously, the bare string had been returned if it was not valid JSON. This was nice in a way as you could pass a scalar string to json_decode and not worry about it. But, in reality, it would make debugging a nightmare for JSON.

I have updated my github fork and requested a pull into the main branch. Once that is done a new PEAR release can be done.

The death of die()

brianlmoon — Thu, 16 Apr 2009 23:13:42 -0500

I am calling it. The death of the PHP die function. Now, I have no actual authority to do so. My PHP CVS karma does not extend that far. And I doubt it will actually get removed despite it being nothing more than an alias for exit now.

No, what I would like to call a death to is the usage of die such as:

$conn = mysql_connect($server, $user, $pass) or die("Could
not connect to MySQL, but needed to tell the whole
world");

I don't know who thought that particular usage was good, but they need to .... no, that is harsh. I just really wish they had never done that.

So, what should you use? Well, there are a couple of options depending on what context you are working in and whether or not the failure is actually catastrophic.

Exceptions

If you are using OOP in your PHP code, Exceptions are the logic choice for dealing with errors. I have mixed feelings about them. But, it has more to do with the catching of exceptions than the throwing of them. If you are going to live in a world of exceptions, please catch them and provide useful error messages. The PHP world is not too bad about that, but I have read too many Java error logs full of huge, verbose exception dumps in my life already. Please don't follow that technique in PHP.

trigger_error

The function trigger_error is quite handy. It allows you, a common PHP coder, to create errors just like the core system. So, the error messages are familiar to anyone that is used to seeing PHP errors. So, if your system is configured to log errors and not display them, errors from trigger_error will be treated the same as built in errors.

Also, errors thrown with trigger_error are caught by a custom error handler just like built in errors. They can be logged, printed, whatever you want from that error handler, just like normal PHP errors. There are even several levels of errors you can raise like notices, warnings, errors, and even deprecated. Again, just like the built in PHP errors.

FATAL Errors

trigger_error is also the most suitable way, IMO, to end a script immediately.

$conn = mysql_connect($server, $user, $pass);

if(!$conn) {

    trigger_error("Could not connect to MySQL
database.", E_USER_ERROR);

}

Now that will not be told to the whole world if you have display_errors set to Off as you should in any production environment.

Wordcraft 0.9.1 available

brianlmoon — Wed, 08 Apr 2009 08:00:00 -0500

There are several key changes in Wordcraft 0.9.1. The two big things are:

Tokens on post forms in the admin to help ward off CSRF attacks.
Database schema updates automated.

The first comes as a result of us doing the same work on Phorum recently. I realized I needed the same protection in Wordcraft. The second was done out of neccesity as I changed the datetime fields in the database schema into int fields. Not sure why I ever made them datetime fields. Unix timestamps are much easier to work with. It saves many strtotime() calls and will make eventual time zone settings much easier to implement.

In addition to those two big ones, there were some notable small ones:

HTML 4.01 validation fixes
Ensuring UTF-8 on all encoding function calls
Protection against hitting the back button when writing a post (most annoying on Macs as the back button and the beginning of line keystroke is the same).

And there were other a few other bug fixes.

I will or course need many more testers and users before I can ever declare this software as stable. If you need a simple blog, give it a try.

About Wordcraft
Wordcraft aims to be a simple, lightweight blogging application. Wordcraft is written exclusively for PHP 5+ and MySQL 5.0+ using only the PHP mysqli extension, UTF-8, and HTML 4.01 to achieve that simpleness.

mod_substitute is cool. But, be careful with mod_proxy

brianlmoon — Tue, 07 Apr 2009 20:03:22 -0500

For our development servers, we have always used output buffering to replace the URLs (dealnews.com) with the URL for that development environment. Where we run into problems is with CSS and JavaScript. If those files contains URLs for images (CSS) or AJAX (JS) the URLS would not get replaced. Our solution has been to parse those files as PHP (on the dev boxes only) and have some output buffering replace the URLs in those files. That has caused various problems over the years and even some confusion for new developers. So, I got to looking for a different solution. Enter mod_substitute for Apache 2.2.

mod_substitute provides a mechanism to perform both regular expression and fixed string substitutions on response bodies. - Apache Documentation

Cool! I put in the URL mappings and VIOLA! All was right in the world.

Fast forward a day. Another developer is testing some new code and finds that his XML is getting munged. At first we blamed libxml because we had just been through an ordeal with a bad combination of a libxml compile option and PHP a while back. Maybe we missed that box when we fixed it. We recompiled everything on the dev box but there was no change. So I started to think what was recently different with the dev boxes. So, I turn off mod_substitute. Dang, that fixed it. I looked at my substitution strings and everything looked fine. After cursing and being depressed that such a cool tool was not working, I took a break to let it settle in my mind.

I came back to the computer and decided to try a virgin Apache 2.2 build. I downloaded the source from the web site instead of building from Gentoo's Portage. Sure enough, a simple test worked fine. No munging. So, I loaded up the dev box Apache configuration into the newly compiled Apache. Sure enough, munged XML. ARGH!!

Up until this point, I had configured the substitutions globally and not in a particular virtual host. So, I moved it all into one virtual host configuration. Still broken.

A little more background on our config. We use mod_proxy to emulate some features that we get in production with our F5 BIG-IP load balancers. So, all requests to a dev box hit a mod_proxy virtual host and are then directed to the appropriate virtual host via a proxied request.

So, I got the idea to hit the virtual host directly on its port and skip mod_proxy. Dang, what do you know. It worked fine. So, something about the output of the backend request and mod_proxy was not playing nice. So, hmm. I got the idea to move the mod_substitute directives into the mod_proxy virtual hosts configuration. Tested and working fine. So, basically, this ensures that the substitution filtering is done only after the proxy and all other requests have been processed. I am no Apache developer, so I have not dug any deeper. I have a working solution and maybe this blog post will reach someone that can explain it. As for mod_substitute, here is the way my config looks.

In the VirtualHost that is our global proxy, I have this:

FilterDeclare DN_REPLACE_URLS

FilterProvider DN_REPLACE_URLS SUBSTITUTE resp=Content-Type
$text/

FilterProvider DN_REPLACE_URLS SUBSTITUTE resp=Content-Type
$/xml

FilterProvider DN_REPLACE_URLS SUBSTITUTE resp=Content-Type
$/json

FilterProvider DN_REPLACE_URLS SUBSTITUTE resp=Content-Type
$/javascript

FilterChain DN_REPLACE_URLS

Elsewhere, in a file that is local to each dev host, I keep the actual mappings for that particular host:

Substitute
"s|http://dealnews.com|http://somedevbox.dealnews.com|in"

Substitute
"s|http://dealmac.com|http://somedevbox.dealmac.com|in"

# etc....

I am trying to think of other really cool uses for this. Any ideas?

Best practices for escaping HTML

brianlmoon — Fri, 20 Mar 2009 22:55:00 -0500

I am working on Wordcraft, trying to get the last annoying HTML validation errors worked out. Thinks like ampersands in URLs. In doing so, I am asking myself where the escaping should take place. In the case of Wordcraft, there are several parts to it.

The code that pulls data from the database. Obviously not the right place.
The code that formats data like dates and such. It also organizes data from several data sources into one nice tidy array. Hmm, maybe
The parts of the code that set up the output data for the templates.
The templates themselves.

Now, I am sure 1 is not the place. And I really would not want 4 to be the place. That would make for some ugly templating. Plus, the templates, IMO, should assume the data is ready to be output. So, that leaves the code that does the formatting and the code that does the data setup.

Of those two, I guess the place to do this job is in the data setup. Wordcraft has a $WCDATA array that is available in the scope of the templates. I suppose anything that goes into that array should be escaped as appropriate.

I largely wrote this blog post as a teddy bear exercise. But, I am curious. Where and when do you escape your data for use in HTML documents?

Enhance your 404 Pages with Google

brianlmoon — Thu, 12 Mar 2009 22:04:57 -0500

I was just poking around the Google Webmaster Tools and came across a neat idea. Google has a snippet of javascript you can put on your 404 error pages that will give the user more information based on their indexing of your site. It can offer a user things like a closest match to the URL they were trying to find, an alternative URL, a link to the site's site map and last but not least, a Google search box that searches the site in question. You have to have your site setup in Webmaster Tools to use it.

HTML vs. XHTML and validation

brianlmoon — Tue, 10 Mar 2009 17:08:29 -0500

There is no shortage on the pages on the internet that talk about HTML vs. XHTML. The vast majority of these (in the first few pages of Google) seem to favor XHTML. I don't really have an agenda, so I thought I would post my thoughts on the topic.

I have stated on this blog that I use HTML 4.01 Transistional. I do so because it is easiest for me. Some people argue that XHMTL is easier because there are set rules and if you violate those rules, the documents will not render. Is that a good thing? Perhaps my time in the late 90's has made my mind work differently than newcomers to the World Wide Web.

The browser wars were ugly. And I mean literally ugly. If you wanted to do anything fancy, it required lots of images or compromise. I learned early on that it was ok that the spacing in IE on my PC was larger than IE on the Mac. The fonts were all different sizes from browser to browser and OS to OS. I learned that graceful fallback was part of the web. Even now, dealnews.com looks "adequate" in IE 6. I could make it look perfect. But, the declining traffic from IE6 does not merit my time to fix the errors in IE 6.

So, when I start thinking about HTML vs. XHTML, I want the more flexible of the two. I find syntax like nowrap='nowrap' very annoying in XHTML. Especially since I can't say nowrap='yeswrap' and it mean anything. nowrap=1 I could handle. But, no, it has to be nowrap='nowrap'. Geez.

Ok, ok, this is turning into an XHTML hate post. I don't want to do that. There are some things about XHTML that I do like. I like the self closing tags. My OCD (which I have brought up before) has never liked having an open tag without a closing tag. so, the
format is appealing to me in that sense. I love that XHTML elements should always be lower case. I hate upper case HTML. It just reads funny. Like camel case function names. Some folks on our content team used to use Adobe PageMaker to write up deals. They would copy and paste the HTML from there into our CMS. The output would be pretty ugly.

So, I like parts of both. What is interesting to me is the fact that the "big sites" on the internet don't seem concerned with document types or validation.

Site	DocType	Validates
Google	None	No
Yahoo	HTML 4.01 Strict	No
Live.com (Microsoft)	XHTML 1.0 Transitional	No
MSN.com	XHTML 1.0 Strict	Yes
Facebook	XHTML 1.0 Strict	No
eBay	HTML 4.01 Transitional	No
YouTube	HTML 4.01 Transitional	No
Amazon.com	None	No
Wikipedia	XHTML 1.0 Strict	Yes
MySpace	XHTML 1.0 Transitional	No

So, of the 10 most popular sites on the internet (according to Compete.com), two don't include a document type in their front page at all. Only two of the sites validate according to the W3C. MSN and Wikipedia both validated on their front page with XHTML 1.0 Strict. However, neither is sending a Content-Type of application/xhtml+xml. According to this page, that is a bad thing. And the search results page for XHTML on MSN.com did not validate. Kudos to Wikipedia. Their page on XHTML does validate. Interestingly, they switch to XHTML 1.0 Transitional for that page.

So, is the internet broken? No. The most important validation is that of your users. Can they use the site? Does the site look right in their browser? Most sites have much bigger navigation and content issues than they do document structure.

So, my idea of validation is this: Does it render the same (or damn near) in the browsers that cover 90% of the internet users? If so, then your page validates. The only way to check that is (most likely without SkyNet) the human eye.

The history of PHP eating newlines after the closing tag

brianlmoon — Tue, 03 Mar 2009 21:51:18 -0600

Have you ever noticed that PHP eats the newlines after a closing PHP tag? Not sure what I mean? There is lots on Google about it. Here is an example.

Hello there!




// this is just a dump PHP block



?>

How are you?

becomes:

Hello there!

How are you?

I was talking about this with a coworker tonight. He is trying to generate some XML and, like me and Chis Shiflett, is anal about his output. You see, what happens in modern use of PHP as a template language is something like this:




$subelement = range(1, 10);



?>

That code will output this mess:

So, why does PHP do this? Well, you have to go back 11 years. PHP 3 was emerging. I was just starting to use it for Phorum at the time. There were two reasons.

The first was that you would want the newline after the first closing tag to be removed as it would remove the existence of the PHP block completely. At the time, people were shunned for writing PHP as a tag looking language. ColdFusion was new then too and the PHP community liked to point and laugh at it.

The second case (and this is probably a more legitimate one) was that many editors (some still do this for some insane reason) force every friggin file to end in a newline. We did not have output buffering in those days. It was the stone age man. So, to get around the "Headers already sent" errors, Zeev decided to make the PHP ending tag be "?> with an optional newline". It was a heated debate on the PHP Internals (then php-dev) list. So much that I remembered it and dug it up on MARC.

Heck, now I want to add to it. I would like it please if PHP could remove any leading, non-newline whitespace before an open tag. That would solve this problem. Yeah, more magic! Nothing like it.

To me, the worst alternative to all this is the lack of a closing tag in a file. My OCD just can't deal with that. Please, baby seals cry when you don't use a closing tag.

Wordcraft 0.8 available

brianlmoon — Mon, 23 Feb 2009 08:00:00 -0600

I am pleased to announce the release of Wordcraft 0.8. I have managed to release about once a month since November. I also have actually gotten some feedback and tickets posted. Thanks to those that have tried it out.

I have decided to go back to YUI's Editor. I tried TinyMCE in the last release. But, using it full time I found it messed with my HTML too much for my liking. When I would switch to raw HTML mode and add something like a

 tag, it would
be lost when saving the data back into the WYSIWYG editor.



I also converted the admin HTML to HTML 4.01 Transitional.  I
never use XHTML anymore these days.  So, I was writing invalid
XHTML inadvertantly.



I worked on the session handling some more in this release. 
Users should stay logged in to the admin better now.



I put comment blocks in all the files and documented every
function.  This should help anyone wanting to dig in and help
out.



I fixed several bugs reported by users (or maybe just testers, not
sure).  Thanks for that and keep the feedback
coming.