Using ini files for PHP application settings

At dealnews we have three tiers of servers. First is our development servers, then staging and finally production. The complexity of the environment increases at each level. On a development server, everything runs on the localhost: mysql, memcached, etc. At the staging level, there is a dedicated MySQL server. In production, it gets quite wild with redundant services and two data centers.

One of the challenges of this is where and how to store the connection information for all these services. We have done several things in the past. The most common thing is to store this information in a PHP file. It may be per server or there could be one big file like:

<?php

if(DEV){
    $server = "localhost";
} else {
    $server = "10.1.1.25";
}

?>


This gets messy quickly. Option two is to deploy a single file that has the settings in a PHP array. And that is a good option. But, we have taken that one step further using some PHP ini trickeration. We use ini files that are loaded at PHP's startup and therefore the information is kept in PHP's memory at all times.

When compiling PHP, you can specify the --with-config-file-scan-dir to tell PHP to look in that directory for additional ini files. Any it finds will be parsed when PHP starts up. Some distros (Gentoo I know) use this for enabling/disabling PHP extensions via configuration. For our uses we put our custom configuration files in this directory. FWIW, you could just put the above settings into php.ini, but that is quite messy, IMO.

To get to this information, you can't use ini_get() as you might think.  No, you have to use get_cfg_var() instead. get_cfg_var returns you the setting, in php.ini or any other .ini file when PHP was started. ini_get will only return values that are registered by an extension or the PHP core. Likewise, you can't use ini_set on these variables. Also, get_cfg_var will always reflect the initial value from the ini file and not anything changed with ini_set.

So, lets look at an example.

; db.ini
[myconfig]
myconfig.db.mydb.db     = mydb
myconfig.db.mydb.user   = user
myconfig.db.mydb.pass   = pass
myconfig.db.mydb.server = host


This is our ini file. the group in the braces is just for looks. It has no impact on our usage. Because this is parsed along with the rest of our php.ini, it needs a unique namespace within the ini scope. That is what myconfig is for. We could have used a DSN style here, but it would have required more parsing in our PHP code.

<?php

/**
 * Creates a MySQLi instance using the settings from ini files
 *
 * @author     Brian Moon <brianm@dealnews.com>
 * @copyright  1997-Present dealnews.com, Inc.
 *
 */

class MyDB {

    /**
     * Namespace for my settings in the ini file
     */
    const INI_NAMESPACE = "dealnews";

    /**
     * Creates a MySQLi instance using the settings from ini files
     *
     * @param   string  $group  The group of settings to load.
     * @return  object
     *
     */
    public static function init($group) {

        static $dbs = array();

        if(!is_string($group)) {
            throw new Exception("Invalid group requested");
        }

        if(empty($dbs["group"])){

            $prefix = MyDB::INI_NAMESPACE.".db.$group";

            $db   = get_cfg_var("$prefix.db");
            $host = get_cfg_var("$prefix.server");
            $user = get_cfg_var("$prefix.user");
            $pass = get_cfg_var("$prefix.pass");

            $port = get_cfg_var("$prefix.port");
            if(empty($port)){
                $port = null;
            }

            $sock = get_cfg_var("$prefix.socket");
            if(empty($sock)){
                $sock = null;
            }

            $dbs[$group] = new MySQLi($host, $user, $pass, $db, $port, $sock);

            if(!$dbs[$group] || $dbs[$group]->connect_errno){
                throw new Exception("Invalid MySQL parameters for $group");
            }
        }

        return $dbs[$group];

    }

}

?>


We can now call DB::init("myconfig") and get a mysqli object that is connected to the database we want. No file IO was needed to load these settings except when the PHP process started initially.  They are truly constant and will not change while this process is running.

Once this was working, we created separate ini files for our different datacenters. That is now simply configuration information just like routing or networking configuration. No more worrying in code about where we are.

We extended this to all our services like memcached, gearman or whatever. We keep all our configuration in one file rather than having lots of them. It just makes administration easier. For us it is not an issue as each location has a unique setting, but every server in that location will have the same configuration.

Here is a more real example of how we set up our files.

[myconfig.db]
myconfig.db.db1.db         = db1
myconfig.db.db1.server     = db1hostname
myconfig.db.db1.user       = db1username
myconfig.db.db1.pass       = db1password

myconfig.db.db2.db         = db2
myconfig.db.db2.server     = db2hostname
myconfig.db.db2.user       = db2username
myconfig.db.db2.pass       = db2password

[myconfig.memcache]
myconfig.memcache.app.servers    = 10.1.20.1,10.1.20.2,10.1.20.3
myconfig.memcache.proxy.servers  = 10.1.20.4,10.1.20.5,10.1.20.6

[myconfig.gearman]
myconfig.gearman.workload1.servers = 10.1.20.20
myconfig.gearman.workload2.servers = 10.1.20.21

Forums are crap. Can we get some help?

Amy Hoy has written a blog post about why forums are crap. And she is right. Forum software does not always do a good job of helping people communicate. I have worked with Amy. She did a great analysis of dealnews.com that led to our new design. So, she is not to be ignored.

However, as a software developer (Phorum), I see a lot of problems and no answers.  And it is not all on the software.  Web site owners use forums to solve problems that they really, really suck at.  Ideally, every web site would be very unique for their audience.  They would use a custom solution that fits a number of patterns that best solves their problem.  However, most web site owners don't want to take the time to do such things.  They want a one stop, drop in solution. See the monolith that is vBulletin, scary.

And what if a forum is the best solution? Well, software developers, in general, are not good designers. They don't think like normal people. And they don't see their applications as a whole, but as pieces that do jobs. The forum software market has been run by software developers for over 10 years. Most of them all are still copies of what UBB was 13 years ago. And software (like Phorum) that has tried to be different is shunned by the online communities of the world because they don't work/look/feel like every other forum software on the planet.

So, as software developers, what are we to do? We want to make great software. We want to help our users help their users. But, what we have been doing for 10+ years has only been adequate. As the leader of an open source forum software project, I am open to any and all ideas.

The rise of the GLAMMP stack

First there was LAMP.  But are you using GLAMMP?  You have probably not heard of it because we just coined the term while chatting at work.  You know LAMP (Linux, Apache, MySQL and PHP or Perl and sometimes Python). So, what are the extra letters for?

The G is for Gearman - Gearman is a system to farm out work to other machines, dispatching function calls to machines that are better suited to do work, to do work in parallel, to load balance lots of function calls, or to call functions between languages.

The extra M is for Memcached - memcached is a high-performance, distributed memory object caching system, generic in nature, but intended for use in speeding up dynamic web applications by alleviating database load.

More and more these days, you can't run a web site on just LAMP.  You need these extra tools (or ones like them) to do all the cool things you want to do.  What other tools do we need to work into the acronym?  PostgreSQL replaces MySQL in lots of stacks to form LAPP.  I guess Drizzle may replace MySQL in some stacks soon.  For us, it will likely be added to the stack.  Will that make it GLAMMPD?  We need more vowels!  If you are starting the next must use tool for running web sites on open source software, please use a vowel for the first letter.

Scaling for the Expected and Unexpected - Speaking at Velocity

Last year I was surprised to be going to Velocity.  Read the post, it was an adventure.  But, I really like the conference.  It is the perfect conference for me.  While a good majority of my work is done coding PHP/MySQL apps, I tend to focus on architecture, frameworks, performance and that kind of stuff.  So, a web performance and operations conference is just perfect.

Last year, I was on a panel with some great guys.  I was able to share just a bit about my experience dealing with the instant success of a web site.  This year, my proposal was accepted to talk more about dealing with success of a web site.  The talk will be focused on my experience at dealnews.com and from working with power users for Phorum.  Here is the summary:

Lots of people talk about scaling and performance. But, are they preparing for all the things that could happen? There are multiple problems and there is not one solution to solve them all.

Everything is running fine and BAM! – your site is linked from the front page of Yahoo! What do you do? How can you handle that sudden rush of traffic. Requests per second are running 5x normal levels. Servers have CPU spikes. Daemons are hitting the maximums. You are running out of bandwidth. How could you have been prepared for this? What are the tools and techniques for this type of sudden rush?

Or, lets say you have just come out of a meeting where everyone discovered that your site is growing in traffic 70% – 80% year over year. That means that 1 million page views this month will be nearly 3 million this time in 2 years. How can you plan for that? You don’t want to redesign the whole architecture every 2 years. What methods could be used to deal with this constant long term growth?

While there is no magic bullet for either of these scenarios, there are techniques used by many sites out there to help you get through these situations. This session will cover some of these techniques and talk about their pros and cons.

I must admit, this if the first time since 2000 that I am a little intimidated to speak at a conference.  The people that present and attend Velocity are so awesome.  I just hope I don't disappoint.

Wordcraft 0.9.1 available

There are several key changes in Wordcraft 0.9.1. The two big things are:
  • Tokens on post forms in the admin to help ward off CSRF attacks.  
  • Database schema updates automated.
The first comes as a result of us doing the same work on Phorum recently.  I realized I needed the same protection in Wordcraft.  The second was done out of neccesity as I changed the datetime fields in the database schema into int fields.  Not sure why I ever made them datetime fields.  Unix timestamps are much easier to work with.  It saves many strtotime() calls and will make eventual time zone settings much easier to implement.

In addition to those two big ones, there were some notable small ones:
  • HTML 4.01 validation fixes
  • Ensuring UTF-8 on all encoding function calls
  • Protection against hitting the back button when writing a post (most annoying on Macs as the back button and the beginning of line keystroke is the same).
And there were other a few other bug fixes.

I will or course need many more testers and users before I can ever declare this software as stable.  If you need a simple blog, give it a try.

About Wordcraft
Wordcraft aims to be a simple, lightweight blogging application.  Wordcraft is written exclusively for PHP 5+ and MySQL 5.0+ using only the PHP mysqli extension, UTF-8, and HTML 4.01 to achieve that simpleness.

Wordcraft 0.8 available

I am pleased to announce the release of Wordcraft 0.8.  I have managed to release about once a month since November.  I also have actually gotten some feedback and tickets posted.  Thanks to those that have tried it out.

I have decided to go back to YUI's Editor.  I tried TinyMCE in the last release.  But, using it full time I found it messed with my HTML too much for my liking.  When I would switch to raw HTML mode and add something like a <code> tag, it would be lost when saving the data back into the WYSIWYG editor.

I also converted the admin HTML to HTML 4.01 Transitional.  I never use XHTML anymore these days.  So, I was writing invalid XHTML inadvertantly.

I worked on the session handling some more in this release.  Users should stay logged in to the admin better now.

I put comment blocks in all the files and documented every function.  This should help anyone wanting to dig in and help out.

I fixed several bugs reported by users (or maybe just testers, not sure).  Thanks for that and keep the feedback coming.

Altering several things in a MySQL table at one time

To some kick ass DBA or someone that works on MySQL internals, this is probably a no brainer.  But, for just joe schmoe power user types, we wonder about these things.  We could not find the answer on the internet, so I decided to blog about it.  We expected the result we found, but I think it's good information.

Have you ever had to make changes to several indexes or columns in a table?  With Phorum, we keep a series of scripts that run queries to upgrade the database as we make changes.  Recently I had to delete about 12 keys and add about 7 back in their place.  I initially thought to make two sql statements.  One to delete indexes and one to create new ones.  But, Thomas, one of the other developers, wondered if that was really any better.  So, I decided to run some tests.

First I altered a table with 70k rows, dropping one key.  That took 16 seconds.  I then added a key to that same table.  Again, about 16 seconds.  So, I then dropped a key and added a key in one query.  Again 16 seconds.  So, I decided to go for the ultimate challenge.  I dropped 12 keys and added back 12 keys all in one query.  TADA!  16 or so seconds.  The table was an InnoDB table, so I repeated after converting the table to MyISAM.  Again, all the alters took about the same time, regardless of the number of changes.

Update


I was told in the comments that the data was not big enough for a real test.  I hold to the fact that because, as Eric Bergen pointed out, the most time is taken in copying the data, the data size is not that important here.  However, I tried it out on a table with 750k rows just to see.  Dropping 4 keys on a table with 12 keys took 1min 46sec.  Adding 4 keys to the same table, (now with 8 keys) took 1min 52sec.  Adding 4 keys and dropping 4 keys in one alter table took 1min 49sec.  So, looks like I was right.  For fun, I decided to drop all the keys except the primary.  That only took 28sec.  That is what I guessed.  The slow part is copying the table to a new table.  That includes filling in indexes.  It is much faster to write a table with indexes than it is a table with 12 indexes.  However, that was not my use case and not the reason for my testing.  But, it is good information to attach to this blog post.

Wordcraft 0.7 available

I am happy to announce Wordcraft 0.7.  There are two big changes in this release.  On the front end, I added a simple search.  It just uses a LIKE clause.  But, I figure a lot of blogs never reach 1,000 posts.  Even at 10,000 posts, a LIKE would not be too bad.  On the backend, I have switched the post editor to TinyMCE. YUI's editor is decent, but it needs polish.  Perhaps my time with Wordpress just made me more familiar with it.  TinyMCE does save XHTML.  I put some code in the PHP side to use Tidy if it is available to convert it to HTML 4.01, which I prefer.

Also in this release:

  • Fixed an XSS issue in tag.php.
  • Fixing a parsing issue with anchor tags when doing pingbacks.
  • Fixed an error when the remote site can not be contacted.
  • Fix for pingbacks with nice URLs enabled. Was blocking pingbacks.

Seven Things about me - tagged by Brian DeShong

So, my Haystacks teammate Brian DeShong tagged me in his list of seven.  We won
that trivia contest by the way.  It was a real team effort.

So, here goes my seven things:
  1. I have six kids.  Okay, let that sink in.  Yes, six.  Logan(12), Macy(11), Molly(9), Parker(7), Collin(3), and Hudson(6 months).  I know what causes it.  Yes, it is hard at times.  But, there are those moments when you are sitting in the yard or in the den and all is right in the world.  The best program I will ever write will not compare to what have done with my children.  They are truly my greatest project.  My wonderful wife blogs about them at Moonmania.
  2. I started my career as a Visual Basic programmer.  PHP and VB are very much alike.  Neither is OOP, yet people keep trying to make them so.  You can write truly powerful applications if you know what you are doing.  Much of the outside community thinks poorly of the language.  In both cases I was way too busy making cool things and getting stuff done to care or pay attention.
  3. I play disc golf.  It is a great sport.  It reminds me of the Open Source community in that it is a very community driven sport.  The courses are installed and maintained by the players, usually on public land such as parks.  The tournaments are run by local players.  The sport is largely managed by players.  I maintain the web site of one of the larger regional (maybe the largest) tournament series in the world.  I have played in most cities that I have visited including those I go to for conferences.  We have a group of 9 or so people at dealnews that play on a semi-regular basis when the weather is right.
  4. I never graduated college.  The VB work and then the PHP work got in the way.  It is a regret in some ways.  But, I can't tell you what I would have changed about how my life has gone.  Perhaps I will go back and finish my degree at some point.  Sadly, in this world, there are doors that are closed to you no matter how brilliant you are if you don't have a simple piece of paper.  I know people with that piece of paper that ended up working fast food.  But, they could get an interview where I could not.  They would not get the job, but I can't even get in the door.
  5. I have contributed two PHP internals functions.  mysql_fetch_assoc and the now deprecated set_file_buffer.  In the case of mysql_fetch_assoc, I was fixing what was IMO a bug.  mysql_fetch_array originally returned only an associative array.  Someone decided it should return the data in both associative and numeric forms.  mysql_fetch_row already existed to return numeric keys.  So, I simply copied it and added mysql_fetch_assoc.  That was the real wild wild west days of PHP CVS.  You either had full access or you didn't have any access.  I did not keep my development up on PHP so I lost karma on the core later.  My C is not super polished.  Still I will admit I liked being part of the club.
  6. I watch pro wrestling.  There I said it.  Guilty pleasure.  The Tivo makes it easy.  2 hour show only takes about 40 minutes.  Heh, I skip most of the wrestling.  Not everyone is entertaining.  My great grandmother would sit and watch it every Saturday night.  I was hooked from that point.  I won't get going on it.  I would lose you guys.  FWIW, I am a big fan of mixed martial arts too.  That stuff IS real and very exciting.
  7. I am a compulsive code rewriter and/or write it myself kind of guy.  It is something I struggle with every day.  It is why I started Phorum and Wordcraft.  Luckily, the guys I work with at both dealnews and on Phorum are good programmers.  My biggest problem is opening a file I last worked on 5 years ago.  The way we write web applicaitons has evolved so much in the 11 years I have been doing this.  I am using the same language, but have such different ideas.  I can only imagine what the next 11 years will bring.

I will be tagging these seven.  Forgive me if I get a title wrong.
  1. Brian Aker.  The creators of sites like Facebook and Digg may have had some great ideas.  But, without guys like Brian Aker, they would be nowhere.  He makes the things that make dealnews, Facebook and Digg possible.  And he continues to contribute with Drizzle and the new C based gearmand.
  2. John Allspaw.  John manages the operations team for Flickr.  He is a smart guy.  He and I see eye to eye on a lot of web performance topics.
  3. Joe Stump. Joe is the lead architect for Digg.  I have not met Joe in person.  But, I have heard him interviewed and we are on the same page about a lot of things.  I have started using some of his contributed PHP code lately as well.
  4. Alan Kasindorf aka Dormando.  Alan works for operations at Six Apart.  They bought Danga.  They formerly owned Live Journal.  He has kind of become the care taker of memcached, gearmand and all those other cool things that Danga created.  The community is helping him more each day, but he still does a lot of work for those projects.  Including, but not limited to coding.
  5. Jay Pipes.  I think I saw him on someone else's list.  But, I am putting him down anyway.  Jay has been good to me over the years.  It is cool that someone can chair the MySQL Conference, go around talking to user groups and commit code to fix MySQL bugs.  Jay is great because he and I can disagree on just about everything when it comes to programming then toast our beers.  You need people like that in the world.
  6. John Allen.  I work with John at dealnews.  He has learned the search engine optimization and marketing world the way I learned programming.  John posses skills that I don't.  We butt heads on a regular basis about things.  But, the end result is always better than what existed before we started.  I have learned many things from him.
  7. Daniel Beckham.  Daniel is my partner in crime at dealnews.  We have been a two headed monster (yeah, a monster.  Just ask the CEO.) for 8 or so years now.  Unfortunately, Daniel does not blog or twitter or much of anything like that.  So, we likely won't see his seven things.  Many of the things I have blogged about when it comes to building dealnews' architecture were thought of and done by both of us.  I am just a show off extrovert that needs a lot of attention.  So, I do all the blogging and talking at conferences.  
And, I am told I need to post this for my taggies to follow:
  • Link your original tagger(s), and list these rules on your blog.
  • Share seven facts about yourself in the post - some random, some weird.
  • Tag seven people at the end of your post by leaving their names and the links to their blogs.
  • Let them know they’ve been tagged by leaving a comment on their blogs and/or Twitter.