Ramblings of a web guy

AI Feels Like Working With a Very Fast Junior Developer

brianlmoon — Thu, 19 Mar 2026 11:12:41 -0500

Over the last year I have started to think of AI as something like a very fast junior developer. It can read and understand a codebase quickly and produce a lot of code in a short amount of time, but it needs clear instructions and you still have to review everything it writes. That comparison has ended up being the most accurate way I can describe what using AI in day to day development actually feels like.

Before AI became part of my workflow, most of my day looked like what you would expect for a software engineer. I spent most of my time writing code. The rest of the time was usually spent researching libraries, reading API documentation, or figuring out how to implement something. That research portion often took longer than the actual coding. Context switching between development and meetings was also a constant challenge, and that part has not really changed.

What has changed is how quickly I can move through implementation work. One of the first practical things I used AI for was porting a TypeScript library to PHP. I could have done it manually, but it would have taken a while. Instead I tried letting AI translate the code. To my surprise it worked fairly well. There were a few bugs where it generated incorrect PHP syntax or called the wrong methods, but the overall structure translated better than I expected. It even handled some of the differences in flow between TypeScript and PHP without much trouble. That was the first time AI stopped feeling like a novelty and started feeling like a useful development tool.

These days I use AI regularly to implement new features. My typical workflow is to describe the feature I want and the libraries involved, let the AI draft an implementation, and then iterate from there. It usually takes some back and forth to get things right, especially when the libraries involved are less common. Models tend to perform much better when there are plenty of examples available.

In practice I now write far less code by hand than I used to. Instead I spend more time reviewing and guiding the code AI generates. I still read every line it produces. AI is capable, but it is not careful in the way an experienced engineer is. It will call the wrong methods, apply patterns from unrelated projects, or misunderstand how a library is supposed to be used.

AI also struggles when a project is completely new. If there is no structure to follow, it tends to guess wrong. In those cases it is usually better for me to write the initial groundwork myself. Once the project has some shape and patterns in place, the AI can usually follow them after a little coaching.

Where AI helps the most is with repetitive work. Writing tests is a good example. I used to dread writing tests because they are repetitive and writing good tests can easily take longer than writing the implementation itself. AI handles that kind of work well and can generate a solid starting set of tests very quickly, which I then review and adjust. I still verify the tests carefully because AI has a habit of creating test doubles or mocking things in ways that avoid actually testing the real code, especially when things get complex.

AI has also changed how I approach learning new APIs and libraries. I tend to learn better from examples than from reading documentation. AI is very good at reading documentation and turning it into working examples, which makes it easier for me to start with a working snippet and adapt it from there.

Another task AI has made much easier is porting libraries from other languages into PHP. I have done this several times now. It is something I could have done before, but it would have been very time consuming. AI makes that kind of translation much faster.

The biggest practical change is speed. I can get a lot more code written in a day than I could before. Features move faster, and the code often ends up better documented and better tested because AI generates those pieces alongside the implementation. Because of that speed it has also changed how I estimate work. Tasks that used to take much longer now move fairly quickly, and since I tend to take on as much work as I can fit into a day, faster implementation usually just means I attempt more things.

AI is not perfect. It makes mistakes, it sometimes applies the wrong patterns, and it is not great at starting projects from scratch. But for many everyday development tasks it is genuinely useful. I could probably go back to writing software without AI, but right now I do not really want to. Using AI in development has turned out to be fun in a way I did not expect, and for the moment it has become a normal part of how I write software.

Murmur: A Simple, Self Hosted Platform for Your Own Community

brianlmoon — Sat, 28 Feb 2026 09:59:07 -0600

When I set out to build Murmur the goal was not to reinvent social media but to create a community space that feels purposeful and that you control. Traditional forums worked well for niche interest groups for decades, but modern social platforms are driven by engagement metrics and algorithmic feeds that do not always serve community wellbeing. Murmur embraces a different model: chronological posts, calm interaction, and ownership of your own data.

Murmur is a self hosted, open source social platform that lets you run your own community space on your infrastructure. There are no recommendation algorithms pushing content, no public leaderboards, and no hidden optimization logic. Posts appear in chronological order and you see what you choose to follow. This design makes the experience predictable and easier to moderate without external pressures.

You can like posts, but Murmur does not expose detailed engagement mechanics that encourage competition or gaming the system. There are no public leaderboards and you cannot see who liked a post. The emphasis stays on conversation rather than visible social scoring.

For anyone who has wanted to host a private community, whether for a club, hobby group, alumni network, or special interest, the appeal of owning the platform and the data is significant. With Murmur you host it yourself, you define the rules, and you decide how conversations flow. There is no external service shaping what gets seen or surfaced.

Feature wise, Murmur provides the basic building blocks of a community space. You can create posts and replies with optional attachments, organize discussions with topics, and build personal feeds by following people or topics you care about. One to one private messaging is available between mutual follows, and there are admin controls for registration settings and user management. It supports multiple database backends, flexible storage options, OAuth login providers, theming, and internationalization so it can adapt to different hosting environments.

I chose PHP as the platform for practical reasons. It is the environment I have worked in for more than twenty years and it remains well suited to building web applications that are straightforward to deploy and maintain. A server rendered approach keeps the operational footprint simple. I kept the feed chronological rather than algorithmic both to simplify the product and to lower the cognitive load for moderators and users.

This project was also one of my first serious attempts to build an application with AI assistance. What surprised me was how effectively AI helped with architectural thinking and scaffolding. It was useful in sketching out structure and moving from idea to working code quickly. That said, it did not produce a finished system on its own. There were several rounds where features did not behave the way I expected. It took repeated testing, feedback, and iteration to get the application working the way I wanted. AI accelerated parts of the process, but validation and refinement still required hands on attention.

Murmur is new and does not yet have usage metrics to point to. It exists because I believe there are still people who want to run their own communities on their own terms. If you are looking for a simple, intentional alternative to large social platforms, one that you can host and control yourself, Murmur is a practical starting point.

Phlag: A Practical, Self-Hosted Feature Flag System

brianlmoon — Sun, 22 Feb 2026 13:57:22 -0600

Feature flags seem simple at first, but they tend to grow complicated over time.

You start with a boolean in a config file. Then you need environment overrides. Then kill switches. Then scheduled rollouts. Then temporary debug settings. Before long, you have conditionals scattered throughout the codebase and a shared understanding of how to toggle things that only exists in people’s heads.

At DealNews we had an internal feature flag system that handled basic enable and disable cases. It worked, but we kept running into situations where we wanted something more flexible. So I started looking at alternatives.

Most of the commercial options were focused heavily on experimentation. They support A/B testing, segmentation, analytics, and complex rollout strategies. Those features are useful for certain teams, but they were not what we needed. We were not trying to build an experimentation platform. We just needed reliable feature control and the ability to store configuration values.

The open source landscape did not have many options that matched those needs out of the box. The tools that existed either did not support the scheduling and value storage we wanted or required architectural patterns that did not fit our environment. So we built Phlag.

Phlag has a very narrow purpose. When asked for a feature flag, it returns a value. That value might be a boolean, a number, or a string. That is the entire contract. It does not attempt to track user behavior or run experiments. It simply answers the question: what is the value of this flag right now?

One of the features that was important for us was scheduling. In production environments you often need temporary changes. You may want to enable debug logging during an investigation and ensure it turns off later. You may want to enable functionality for a seasonal promotion. You may need a change to activate during a maintenance window. Without scheduling, someone has to remember to reverse the change. Eventually someone forgets. Phlag allows flags to have start and end times so those temporary changes manage themselves.

Today we primarily use Phlag for feature rollouts and kill switches. That alone provides a lot of operational safety. It allows us to deploy code paths that can be disabled quickly if needed. Phlag also supports numeric and string values, which means it can act as a centralized configuration system rather than just a collection of booleans. We are beginning to use it in that way as well.

Phlag is written in PHP because that matches our team’s expertise, but it is not limited to PHP environments. It exposes a REST API and can be deployed using Docker or Kubernetes. Any system that can make an HTTP request can consume it. In practice, it should work for most teams regardless of their application stack.

I suspect Phlag will be most useful for teams that want a self-hosted option and do not need a full experimentation suite. Commercial feature flag services can become expensive as usage scales, especially when you are paying for capabilities you do not use. Phlag focuses on the core problem of returning a flag value reliably, without adding layers of complexity.

There was not a single dramatic challenge in building Phlag. Keeping the scope small made the architecture straightforward. We focused on the data model, the API, and predictable behavior. We used AI tools to help with the UI, which allowed us to spend more time on the underlying system design.

Recent work has focused on improving the user interface and making the system easier to use. I have looked at efforts like OpenFeature, which are interesting, but they also aim to support a broader set of capabilities than Phlag currently targets. For now, the goal remains the same: keep it focused and reliable.

If you are evaluating feature flag systems and find yourself wanting something simple and self-hosted that just returns a value when asked, Phlag might be worth trying.

Varnish Workspace Error

brianlmoon — Mon, 07 Oct 2024 12:21:55 -0500

We had been having an issue with losing cookies in Varnish. Because of how Varnish works, the way to work around cookies is to store them in headers inbetween states of the request process. We were finally able to get some data out of varnishlog that looked like this.

- ReqHeader DN-Varnish-Offer-Group: 

- LostHeader DN-Varnish-Offer-Sort: price 

- Error out of workspace (req) 

- LostHeader DN-Varnish-Sbtab-Sorts: 

- Error out of workspace (req) 

- LostHeader DN-Varnish-Use-View: 

- Error out of workspace (req) 

- LostHeader DN-Varnish-Ux-Variant: classic_site 

- Error out of workspace (req)

When you first read these errors, you will likely find the settings workspace_client and workspace_backend. Those seem like very logical settings to tweak. However, no matter how big we set them nothing helped. We graph stats coming out of Varnish using the prometheus exporter. We found the metric varnish_main_ws_backend_overflow. That made us believe even more that this was a workspace_backend limit we were hitting. It turns out, there is more to the workspace settings than just this. I read through an old issue on Github and found some folks trying to set other settings related to header size and header limits. In the end, that was our issue. We increased these settings and our overflows disappeared.

http_req_hdr_len = 64k (default is 8k)

http_req_size = 128k (default is 32k)

http_max_hdr=256 (default is 64)

Hopefully this will help someone else that runs up against this.

Disc Golf Discs similar to Remix Discs on Amazon

brianlmoon — Mon, 26 Feb 2024 16:11:38 -0600

Remix Disc Golf is a brand of disc golf discs that I have only been able to find on Amazon. The seller on Amazon is named Disc Golf Goods. On its Amazon store page, they sell MVP, Axiom, Remix and other brands of disc golf equipment. The detailed seller information on Amazon says the "Business Name" is MVP Pro Shop, LLC. It is pretty common knowledge that these discs are manufacturered and sold by MVP. The speculation is that they are molds made for other companies (Mint, Thought Space Athletics, and possibly others) which they are selling under the Remix name on Amazon. Many of the reviews mention the discs have cosmetic defects or look like they have been used. That has led some to think these are factory seconds. The cool thing is, they cost less than any of the MVP brands or third party brands for which they are known to manufacturer discs. The discs sell from $9.95 to $12.95.

One thing people are always trying to figure out is what disc from another brand was renamed for a Remix disc. Well, it's not an exact science. Some of them could be rejected molds. So, while they may be very similar to another disc, it could be a mold that was meant for another disc that was not used for that disc. This is pure speculation based on talking to people in the know for almost 28 years of playing disc golf.

Now, there is a site that already has a feature that lets one search for similar discs. It is called Try Discs. Their recommendation engine seems to favor flight number similarities over measurements. And we all know that flight numbers are kind of made up. I decided to use the PDGA specs for approved discs to find the discs most similar to the Remix discs available on Amazon. I did not limit the search to brands that are known or believed to be manufactured by MVP. Perhaps you have a favorite disc from another brand that is similar to a Remix disc. There are more Remix discs approved by the PDGA than are on thist list. However, they are not for sale anywhere I can find. I am not claiming that any of the discs will fly like one another. I am solely comparing the measuerments has observed by the PDGA.

If you are interested in some reviews of Remix Discs, Pete Collins has some on his YouTube channel.

All values are centimeters except rim configuration. To determine similarity, diameter and inside rim diameter must to be +/- 0.5cm, height, rim depth, and rim thickness must be +/- 0.2cm (it was 0.1cm in an earlier version of the blog post), and rim configuration must be +/- 1.

For details on these specifications, see the PDGA Technical Standards document.

Brand	Name	Diameter	Height	Rim Depth	Inside Rim Diameter	Rim Thickness	Rim Configuration
Remix Disc Golf	Battleship 5 / 4.5 / 0 / 2.5	21.4	1.8	1.4	18.5	1.4	50.5
Clash Discs	Cherry 5 / 5 / -1 / 1	21.4	1.7	1.4	18.5	1.4	51
Clash Discs	Berry 5 / 5 / -1 / 1	21.4	1.7	1.4	18.8	1.3	50.5
Kastaplast	Järn 5 / 3 / 0 / 3	21.3	1.8	1.4	18.8	1.3	51.5
Legacy Discs	Sumo 4 / 2 / 0 / 4	21.3	1.7	1.3	18.7	1.3	50.25
Axiom Discs	Paradox 5 / 4 / -4 / 0	21.5	1.8	1.3	18.9	1.3	50.25
Innova Champion Discs	Gator 5 / 2 / 0 / 4	21.2	1.9	1.4	18.6	1.3	49.5
Innova Champion Discs	Gator3 5 / 2 / 0 / 3	21.2	1.8	1.4	18.2	1.5	49.5
Discmania	MD5 5 / 3 / 0 / 4	21.2	2	1.3	18.2	1.5	51.25
Lone Star Disc	Artemis 4 / 4 / 0 / 2	21.1	2	1.4	18.8	1.2	50.5
Infinite Discs	Ra 5 / 4 / 0 / 3	21.7	1.8	1.4	18.8	1.4	50.25

Brand	Name	Diameter	Height	Rim Depth	Inside Rim Diameter	Rim Thickness	Rim Configuration
Remix Disc Golf	Behemoth 12 / 4.5 / -1 / 3	21.2	1.8	1.2	16.6	2.3	26.5
Mint Discs	Goat 12 / 4 / -1 / 3	21.2	1.7	1.2	16.6	2.3	27
Thought Space Athletics	Synapse 12 / 5 / -1 / 3	21.2	1.6	1.2	16.6	2.3	26.25
Axiom Discs	Time-Lapse 12 / 5 / -1 / 3	21.2	1.7	1.2	16.5	2.3	26.25
Discmania	Astronaut 12 / 6 / -4 / 1	21.2	1.8	1.1	16.6	2.3	26
Prodigy Disc	X4 13 / 5 / -2 / 2	21.2	1.7	1.1	16.6	2.3	27
Disc Golf Association	Hypercane 13 / 4 / 0 / 4	21.2	1.6	1.2	16.7	2.2	26.5
Latitude 64	Recoil 12 / 4 / 0 / 3	21.2	1.8	1.2	16.8	2.2	26
Kastaplast	Rask 14 / 3 / 0 / 4	21.2	1.6	1.1	16.8	2.2	26.5
Lone Star Disc	Curl 11 / 6 / -1 / 2	21.2	1.8	1.1	16.8	2.2	26.75
MVP Disc Sports	Zenith 11 / 5 / -0.5 / 2	21.2	1.7	1.1	16.8	2.2	27

Brand	Name	Diameter	Height	Rim Depth	Inside Rim Diameter	Rim Thickness	Rim Configuration
Remix Disc Golf	Creature 5 / 5 / -3 / 1	21.4	1.8	1.4	18.6	1.4	46.25
Thought Space Athletics	Crux 5 / 4 / 0 / 2	21.4	2	1.4	18.5	1.4	47.25
Discmania	Origin 5 / 5 / -2 / 1	21.4	1.6	1.2	18.8	1.3	46.5
Dynamic Discs	Culprit 4 / 2 / 0 / 3.5	21.3	1.6	1.3	18.9	1.2	46.25
Streamline Discs	Runway 5 / 4 / 0 / 3.5	21.5	1.7	1.4	18.6	1.4	47.25
MVP Disc Sports	Matrix 5 / 4 / -1 / 2	21.5	1.6	1.4	18.7	1.4	46.5
MVP Disc Sports	Vertex 4 / 4 / -2 / 0.5	21.5	1.9	1.3	19	1.2	46.75
MVP Disc Sports	Tensor 4 / 4 / 0 / 2.5	21.5	2	1.3	19.1	1.2	46.75
Doomsday Discs	Wasteland 5 / 4 / 0 / 1	21.2	2	1.2	18.6	1.3	45.5
MVP Disc Sports	Detour 5 / 5 / -2 / 1	21.6	1.6	1.3	18.7	1.4	47.25
Doomsday Discs	Scavenger 5 / 5 / -2 / 1	21.6	1.9	1.2	19	1.3	46.75

Brand	Name	Diameter	Height	Rim Depth	Inside Rim Diameter	Rim Thickness	Rim Configuration
Remix Disc Golf	Eldritch 3 / 3 / 0 / 1	21.2	2	1.5	18.9	1.1	55.75
Prodiscus	Origo 3 / 3 / 0 / 1	21.2	1.9	1.5	19.1	1.1	56.75
Streamline Discs	Stabilizer 3 / 3.5 / 0 / 3	21.2	1.8	1.4	18.8	1.1	56.25
Viking Discs	Rune 2 / 4 / 0 / 0	21.2	2.1	1.6	19	1.1	55.5
Discmania	Logic 3 / 3 / 0 / 1	21.2	1.9	1.3	19	1.1	55
Gateway Disc Sports	Houdini 3 / 3 / 0 / 3	21.2	1.9	1.5	18.8	1.2	55.75
Gateway Disc Sports	Warlock GE 2 / 3 / 0 / 1	21.2	1.9	1.5	19.2	1	56.25
Innova Champion Discs	Aviar3 3 / 2 / 0 / 2	21.2	1.9	1.5	19.2	1	55.75
Prodigy Disc	P Model S 3 / 5 / 0 / 2	21.2	2	1.5	19.2	1	56.25
Disc Golf Association	Steady 2 / 3 / 0 / 2	21.2	2	1.4	19.3	1	56
Gateway Disc Sports	Chief OS 3 / 3 / 0 / 2	21.2	2.1	1.3	19.2	1	55.75

Brand	Name	Diameter	Height	Rim Depth	Inside Rim Diameter	Rim Thickness	Rim Configuration
Remix Disc Golf	Gladius 5 / 5 / -0.5 / 1	21.4	1.9	1.4	18.5	1.4	45.75
Clash Discs	Peach 4 / 5 / -2 / 1	21.4	1.9	1.3	18.6	1.4	45
Doomsday Discs	Despair 5 / 4 / -1 / 1	21.4	2.1	1.4	18.8	1.3	46.25
Discraft	Malta 5 / 4 / 0 / 3	21.3	1.7	1.3	18.6	1.3	45
MVP Disc Sports	Uplink 5 / 5 / -3 / 0.5	21.5	1.8	1.4	18.8	1.3	45
MVP Disc Sports	Vertex 4 / 4 / -2 / 0.5	21.5	1.9	1.3	19	1.2	46.75
Innova Champion Discs	Spider 5 / 3 / 0 / 1	21.2	1.7	1.4	18.6	1.3	45
Doomsday Discs	Wasteland 5 / 4 / 0 / 1	21.2	2	1.2	18.6	1.3	45.5
Prodiscus	STARi 4 / 4 / -2 / 0	21.6	1.9	1.4	18.9	1.3	45
Doomsday Discs	Scavenger 5 / 5 / -2 / 1	21.6	1.9	1.2	19	1.3	46.75
Innova Champion Discs	Hydra 3 / 3 / 0 / 2	21.1	2	1.4	18.3	1.4	44.75

Brand	Name	Diameter	Height	Rim Depth	Inside Rim Diameter	Rim Thickness	Rim Configuration
Remix Disc Golf	Goliath 10.5 / 4.5 / -1 / 2.5	21.2	1.6	1.2	17	2.1	31
RPM Discs/Disc Golf Aotearoa	Tara Iti 10 / 5 / 0 / 2	21.2	1.5	1.2	17	2.1	30
Thought Space Athletics	Animus 11 / 5 / -1 / 2	21.2	1.6	1.2	17	2.1	30
Thought Space Athletics	Mantra 9 / 6 / -2 / 1	21.2	1.7	1.2	17.1	2	32
Discraft	Spectra 12 / 5 / -2 / 2	21.2	1.7	1.2	16.8	2.2	30.75
Latitude 64	Blitz 12 / 3 / 0 / 4	21.2	1.6	1.2	16.8	2.2	30.5
Lone Star Disc	Chupacabra 9 / 3 / 0 / 4	21.2	1.6	1.2	17.2	2	30.5
Mint Discs	Phoenix 9 / 3 / 0 / 4	21.2	1.5	1.2	17.2	2	30.25
Streamline Discs	Trace 11 / 5 / -1 / 2	21.2	1.7	1.2	16.8	2.2	31.25
Dynamic Discs	Sockibomb Felon 9 / 3 / 0.5 / 4	21.2	1.6	1.1	17.1	2	31.5
Discmania	DD 11 / 6 / -3 / 2	21.2	1.7	1.1	16.8	2.2	30

Brand	Name	Diameter	Height	Rim Depth	Inside Rim Diameter	Rim Thickness	Rim Configuration
Remix Disc Golf	Haymaker 9 / 4 / 0 / 4	21.2	1.8	1.2	17.2	2	31.5
Gateway Disc Sports	Ninja 10 / 5 / -1 / 2	21.2	1.9	1.2	17.2	2	31.75
Lone Star Disc	Chupacabra 9 / 3 / 0 / 4	21.2	1.6	1.2	17.2	2	30.5
Gateway Disc Sports	Diablo 9 / 5 / 0 / 2	21.2	1.7	1.2	17.1	2	32.5
Thought Space Athletics	Mantra 9 / 6 / -2 / 1	21.2	1.7	1.2	17.1	2	32
Prodigy Disc	FX-4 9 / 5 / -2 / 1	21.2	1.8	1.2	17.3	2	32.25
Dynamic Discs	Felon 9 / 3 / -0.5 / 4	21.2	1.8	1.1	17.2	2	31.5
Prodigy Disc	Feedback 9 / 5 / -1 / 3	21.2	1.7	1.1	17.2	2	31
Dynamic Discs	Sockibomb Felon 9 / 3 / 0.5 / 4	21.2	1.6	1.1	17.1	2	31.5
Latitude 64	Culverin 9 / 5 / -0.5 / 3	21.2	2	1.1	17	2	31.25
Thought Space Athletics	Coalesce 9 / 5 / 0 / 3	21.2	1.9	1.2	17.4	1.9	32.5

Brand	Name	Diameter	Height	Rim Depth	Inside Rim Diameter	Rim Thickness	Rim Configuration
Remix Disc Golf	Ronin 8 / 4.5 / 0 / 2	21.3	1.9	1.2	17.4	1.9	31.5
Prodigy Disc	H1 9 / 5 / -2 / 2	21.3	1.8	1.2	17.4	1.9	31.75
Lone Star Disc	Dos X 8 / 4 / -1 / 2	21.3	1.9	1.2	17.3	2	32.25
Prodigy Disc	H3 V2 11 / 5 / -1 / 2	21.3	1.8	1.2	17.3	2	30.75
Clash Discs	Soda 7 / 5 / -2 / 2	21.4	2	1.2	17.8	1.8	31.5
Thought Space Athletics	Coalesce 9 / 5 / 0 / 3	21.2	1.9	1.2	17.4	1.9	32.5
Latitude 64	Honor 9 / 5 / 0 / 2.5	21.2	1.8	1.1	17.4	1.9	31.75
Yikun Discs	Hu 9 / 5 / -2 / 2	21.2	1.8	1.1	17.4	1.9	31.75
Discmania	CD1 9 / 5 / -1 / 2	21.2	1.7	1.1	17.3	1.9	31.25
Discraft	Tracker 8 / 4 / -1 / 3	21.2	1.7	1.2	17.7	1.8	32.5
Discmania	Instinct 7 / 5 / 0 / 2	21.2	1.9	1.1	17.6	1.8	32.25

Brand	Name	Diameter	Height	Rim Depth	Inside Rim Diameter	Rim Thickness	Rim Configuration
Remix Disc Golf	Rumble 10 / 5 / -1 / 2	21.2	1.7	1.2	17.1	2	33.75
Gateway Disc Sports	Blade V2 9 / 5 / 0 / 3	21.2	1.7	1.2	17.2	2	34
Legacy Discs	Vengeance 10 / 5 / 0 / 2	21.2	1.7	1.2	17.2	2	33
Prodiscus	Titan 8 / 3 / 0 / 2	21.2	1.6	1.2	17.3	1.9	32.75
MVP Disc Sports	Terra 8 / 5 / 0 / 3	21.2	1.8	1.2	17.4	1.9	32.75
Prodigy Disc	F Model OS+ 10 / 5 / 2 / 4	21.2	1.7	1.1	17.1	2.1	34.25
Streamline Discs	Shift 11 / 5 / -2 / 2	21.2	1.5	1.1	17.1	2.1	32.75
Dynamic Discs	Thief 8 / 5 / -1.5 / 2	21.2	1.8	1.1	17.3	1.9	34.25
Mint Discs	Jackalope 8 / 5 / -2 / 1	21.2	1.5	1.1	17.3	1.9	32.75
Infinite Discs	Aztec 10 / 5 / -1 / 2	21.2	1.5	1.1	16.9	2.1	33
Latitude 64	Spark 7 / 4 / -0.5 / 2	21.2	1.5	1.1	17.4	1.9	34

Brand	Name	Diameter	Height	Rim Depth	Inside Rim Diameter	Rim Thickness	Rim Configuration
Remix Disc Golf	Spartan 9 / 6 / -2 / 1	21.2	1.6	1.2	17	2.1	32.5
Infinite Discs	Aztec 10 / 5 / -1 / 2	21.2	1.5	1.1	16.9	2.1	33
Streamline Discs	Shift 11 / 5 / -2 / 2	21.2	1.5	1.1	17.1	2.1	32.75
Gateway Disc Sports	Diablo 9 / 5 / 0 / 2	21.2	1.7	1.2	17.1	2	32.5
Thought Space Athletics	Mantra 9 / 6 / -2 / 1	21.2	1.7	1.2	17.1	2	32
Legacy Discs	Vengeance 10 / 5 / 0 / 2	21.2	1.7	1.2	17.2	2	33
Prodigy Disc	FX-4 9 / 5 / -2 / 1	21.2	1.8	1.2	17.3	2	32.25
Dynamic Discs	Sockibomb Felon 9 / 3 / 0.5 / 4	21.2	1.6	1.1	17.1	2	31.5
Dynamic Discs	Felon 9 / 3 / -0.5 / 4	21.2	1.8	1.1	17.2	2	31.5
Westside Golf Discs	Sorcerer 13 / 5 / -0.5 / 3	21.2	1.6	1.1	16.7	2.3	32.25
Innova Champion Discs	Mystere 11 / 6 / -2 / 2	21.1	1.6	1.1	16.9	2.1	31.5

Brand	Name	Diameter	Height	Rim Depth	Inside Rim Diameter	Rim Thickness	Rim Configuration
Remix Disc Golf	Torpedo 5 / 5 / 0 / 1	21.4	1.6	1.4	18.5	1.4	42.5
Lone Star Disc	Middy 5 / 5 / 0 / 3	21.4	1.6	1.2	18.8	1.3	42.75
Streamline Discs	Echo 5 / 5 / -1.5 / 1	21.4	1.6	1.3	18.5	1.5	42
Legacy Discs	Badger 6 / 3 / 0 / 3	21.4	1.6	1.3	18.3	1.5	42
Westside Golf Discs	Anvil 4 / 2 / 0 / 4	21.3	1.8	1.2	18.3	1.5	41.75
Thought Space Athletics	Pathfinder 5 / 5 / 0 / 1	21.5	1.7	1.3	18.6	1.4	43
Westside Golf Discs	Tursas VIP Air 5 / 5 / -2 / 1	21.6	1.7	1.3	18.8	1.4	42.5
Westside Golf Discs	Tursas 5 / 5 / -2 / 1	21.7	1.7	1.3	18.9	1.4	42.5
Latitude 64	Anchor 5 / 4 / 0 / 3	21.7	1.8	1.5	18.9	1.4	43.25
Latitude 64	Fuji 4 / 4 / 0 / 2	21.7	1.8	1.2	18.9	1.3	42.75
Dynamic Discs	EMAC Truth 5 / 5 / 0 / 2	21.7	1.8	1.2	18.7	1.5	42.25

Brand	Name	Diameter	Height	Rim Depth	Inside Rim Diameter	Rim Thickness	Rim Configuration
Remix Disc Golf	Troll 7.5 / 4.5 / 0 / 2	21.2	1.8	1.2	17.6	1.8	37
Thought Space Athletics	Votum 8 / 5 / 0 / 3	21.2	1.7	1.2	17.5	1.8	36.25
Discmania	FD1 7 / 4 / 0 / 2	21.2	1.7	1.1	17.6	1.8	36.25
Dynamic Discs	Vandal 9 / 5 / -1.5 / 2	21.2	1.7	1.1	17.4	1.9	36.25
Latitude 64	Fury 9 / 6 / -2 / 2	21.2	1.8	1.1	17.4	1.9	36.75
Streamline Discs	Ascend 6 / 5 / -3 / 0.5	21.2	1.8	1.2	17.8	1.7	38
Innova Champion Discs	Kite 5 / 6 / -3 / 1	21.2	1.9	1.3	17.9	1.6	36.25
Infinite Discs	Exodus 7 / 5 / -1 / 2	21.1	1.8	1.1	17.5	1.8	36.25
Prodiscus	Talisman 9 / 4 / 0 / 2	21.1	1.7	1.2	17.3	1.9	37.75
Innova Champion Discs	Savant 9 / 5 / -1 / 2	21.1	1.6	1.1	17.2	1.9	36.5
Lone Star Disc	The Dome 8 / 6 / -3 / 1	21.1	1.8	1.1	17.2	1.9	36

How I got Disc Golf Network Pro for FREE for 2024

brianlmoon — Tue, 20 Feb 2024 11:48:03 -0600

Do you plan to go to a DGPT event this year? Are you a PDGA member? Then it could be worth it to buy the Disc Golf Network yearly plan.

Disc Golf Network (aka DGN) (the media arm of the Disc Golf Pro Tour) (aka DGPT), announced their new pricing tiers for 2024 earlier this month. It was met with some mixed reviews. Some users of the service had issues using it the first week. Most of those appear to be due to users needing to update the app on their devices or using older streaming devices that do not support the new 60fps stream. They have updated their upgrade guide. I experienced this on one of my Roku devices. I was not surprised to be honest. Many of the Roku apps we use on that device are laggy and crash from time to time. It is over 10 years old. The fact that it has kept working at all is a credit to Roku.

As for the pricing for DGN, there are three tiers: Basic, Standard, and Pro. See the link above for the differences. The pricing ranges from $5.99/mo to $19.99/mo for non-PDGA members. While PDGA members can get Basic for free, Standard for $5.99/mo, and Pro for $12.99/mo. There are also yearly options. Basic for $59.99, Standard for $129.99, and Pro for $239.99 for non-PDGA members. And for PDGA members, Standard for $69.99 and Pro for $139.99. Since Basic is free, there is no yearly option for PDGA members of course. Most people I know that want to consume live disc professional disc golf are PDGA members. While some say you have to factor in the $50 annual PDGA membership cost along with the discounted DGN price, that does not apply to me. I would be renewing my PDGA membership either way. So, I will only be speaking about how and why I chose the option I did based on the discounted PDGA pricing.

The first question I had to ask is what do I want to pay monthly or go ahead and pay for the whole year? The Standard plan annual cost only saves you $2 for the year. Not a compeling reason to do it in my opinion. The annual cost for Pro actually saves more than the cost of a month, $139.99 one time compared to $12.99/mo over 12 months totaling $155.88. There are some ways to save if you change your plan for certain months for certain events or remember to cancel after the DGPT Finale in October. But, let's be real. I won't remember to do that. Most people won't remember to do that. That is why the subscription model is so popular in the USA. That is how gyms stay in business to be honest. If you are the kind of person that likes to manage subscriptions that way, go for it. If you micro manage it completely and only pay for February through October and upgrade the months of the USDGC and European Open, you could get all of the coverage for as low as $88.89 for the year as a PDGA member. I think I did that math right. You are probably saying "Hey, your headline says you are getting it free for the whole year! What gives?" Yes, let me get to that.

Here is why I opted for the full year, Pro plan. It’s $139.99 for the year. The kicker for me is that any yearly plan includes two free general adminssion (aka GA) weekend passes to a Disc Golf Pro Tour event as well as 10% off any other DGPT ticket purchases. As a family, we had already booked an AirBnB for Nashville in April to go watch the Music City Open before this announcement was made. My two sons and I are going for all three days. And two other family members will be joining us for Sunday. I had planned to get the weekend VIP pass for myself. So, altogether, our tickets to the Music City Open were going to cost around $350. However, with the yearly DGN option, I get the GA passes for free. And I get a 10% discount on the other tickets. Those ended up costing me around $210 after the discount. So, my savings on tickets (tickets I had already planned to buy before I knew there was a discount available) is around $140. That is the cost of the yearly plan. If you include all of the decimals in all of the math, I technically am spending 17 cents more on the DGN subscription than I am saving on tickets. Would that make a better headline?

Ticket	Quantity	Regular Price	DGN Discounted Price
3-Day General Admission	2	$116.88 ($58.44/ea)	FREE
Sunday General Admission	2	$71 ($35.50/ea)	$64.12 ($32.06/ea)
3-Day VIP	1	$161.68	$145.62
Total	5	$349.56	$209.74
Savings			$139.82

Having said all that, if you are not really into watching live disc golf and don’t follow the pro scene on social media, the best way to consume the pro tour is next day, post produced coverage on the JomezPro YouTube channel. In the past, their coverage only included the lead group each round. This year, JomezPro are doing more comprehensive coverage of the event as a whole. I am sure this is due in part to having multiple winners last season that were not in the lead group in the final round (aka chase card champions). This meant that the JomezPro coverage didn’t include footage (or not much at least) of the actual winner of the event. They were purchased or merged with Disc Golf Network early last year. However, other media companies covered the second and third groups. This year, they didn’t partner with those media companies in the same way which allows JomezPro to use the footage from all groups. Listening to Jeff Spring on the Staggered Stance podcast, it sounds like they will still be partnering with some of those media companies like Ace Run Pro for some things this year as well.

Update

I forgot to mention that I had a hard time finding information about how to get my free tickets and discounts. After some digging, I found in the DGN support pages. You have to fill out a form which triggers an email which has discount coupon codes and a link to another form you have to fill out to get your free DGPT tickets. So, if you get an annual subscription, be sure and fill out this form to at least get your 20% discount codes for the DGPT Pro Shop and JomezPro Shop. JomezPro has made some cool stamps in the past that I have rarely bought unless they had a sale going. Having a 20% discount code will make me consider them more often when ordering discs online.

This post was not paid for or done as a favor for the DGPT. I am just a disc golf fan that wanted to share my experience.

So, I had a heart attack and bypass surgery

brianlmoon — Sat, 15 Apr 2023 11:38:06 -0500

Hey y’all. So, I had a heart attack and bypass surgery. Here is the full story.

Friday (3/31) and Saturday (4/1) nights I had chest pain that I thought was acid reflux. On Sunday morning, the pain returned. I checked my blood pressure. It was 170/103. I took some anxiety meds, gas x, and aspirin. An hour later it was still high.

So we went to the ER. Blood work showed troponin in my blood. “When heart muscles become damaged, troponin is sent into the bloodstream. As heart damage increases, greater amounts of troponin are released in the blood.” I was then admitted for observation and further testing.

On Monday (4/3) morning they performed an echocardiogram. There were some abnormalities. They then performed a heart catheter. I had 90% blockage in at least one artery. And blockages in several others. I was immediately transferred to UAB hospital.

Later that day I met with a surgeon. After discussing it with him, we decided to do bypass surgery. The long term success rate with this surgery at my age is better than the alternatives. Surgery was booked for Thursday, April 6.

On Tuesday (4/4) and Wednesday (4/5) I just hung out at the hospital. I could have another heart attack at any moment. Going home was not an option. Friends and family visited. I had some hard conversations with family about what to do “just in case”. Those conversations don’t phase me. And the family I spoke to were very practical about it as well.

Early Thursday morning, before dawn, I broke down a little bit. The reality that I could not wake up was hitting me. I knew it was not likely. These procedures are done every day. My doctor would probably do several that day alone. Still, it could have happened. It’s normal for me to have these emotional outbursts alone. The first time I remember it happening was with my great grandmothers death when I was 15. It’s been the same with all my other grandparents’ deaths as well. It’s just how I deal with it.

Then it was time to go. The family that was there followed us down to the waiting room. Once I was in pre-op and settled they said I could have one person come back. They let two coke back. The nurse said we seemed like solid people. I don’t remember a lot about that time. I do remember Deedra deciding to read my chart. Haha. The staff walking by was confused. Then I was off. While still rolling me in, I started to feel woozy. And then black.

I wake up very confused with some voices I know and others I don’t know. I understand their instructions but don’t know how to follow them. I need to breath. Ok. There is something in my mouth. Oh it’s the ventilator of course. They can’t remove it until I breath. There are two people I know there. My ex wife, and mother of my six children, Robin, coaching me on what to do. Good I need that right now. And Amy, my platonic life partner, speaking to me softly and encouraging me to breath. Man, I really need that approach too. If you had asked me what two people I would want in that moment, I would probably not have chosen either of them. And yet, they were the perfect combination at that time. Some (I assume) nurse said “good job” and out came the ventilator. Based on when I knew I went into the OR and how long the surgery took, I would say this is around 2pm. People say they visited with me in recovery. I believe them. Still, I really don’t recall much until 5AM on Friday morning.

Friday was confusing. It’s like my mind was still trying to figure out what happened to my body.

Between Friday and Tuesday (4/11) I had good times and bad times. I eventually got to go home. That is where I am now. My strength has slowly been recovering. The last device attached to my body came off yesterday. It will be several weeks of very limited activity. Mostly I just can’t lift things or drive. Slowly that will be allowed more and more. Then once all restrictions are removed, I can start building up my strength again.

Who am I to call anyone a racist?

brianlmoon — Sun, 07 Jun 2020 19:07:03 -0500

Recently, on Facebook, I was asked "Who are you to call anyone a racist?". To be clear, I did not call anyone a racist. But, really, let's say I did call someone a racist? Who am I to call anyone a racist? After all, I am a 47 year old, white, middle class man. I am, quite possibly, in the absolute sweet spot of my life for white privilege.

As a child, I heard older family members say racist things. Luckily, my parents made it clear to me that they were wrong and I should ignore them. So I have for 40+ years. Some of those racist things were said about my childhood best friend, who is black. Were they nasty, ugly, hateful remarks? No. They didn't use the "N" word. It was subtle things that made me question if I should have a best friend that is black. They found it confusing. They found it odd. It was clear they did not approve.

I was fortunate to have lived in a very diverse neighborhood in Huntsville, AL as a child. My next door neighbor was two years older than me. He was my best friend. I don't know if I was his best friend but he was definitely mine. And like I said, he is black. His father pulled the first tooth I lost. I spent so much time at his house as a child and he spent time at mine. We liked his house better because his parents had converted their two car garage into an awesome den/music hall. I was young and innocent as they say. I didn't know any different. I am so glad that was the case. I have not seen him in years as I moved away from Huntsville when I was eleven years old. A year or so ago, I did find his mother and sisters on Facebook and was able to reconnect with them online. He has a career in the Air Force. I am very proud to have called, and still consider, him my friend.

After living in Tuscumbia, AL for four years where racial topics were basically not discussed despite there being a neighborhood named Richman Hills where the road leading into it was named White City Drive, I moved to a suburb of Birmingham, AL. I have lived in the Birmingham area, including some years in the city proper, for all but two years of my life since I was 15 years old. This is the city where police turned fire hoses and dogs on protesters during the Civil Rights movement.

I found things in Birmingham different than they were in Huntsville. The Birmingham area was and still is highly segregated. It's not by law or force. It's mostly white flight which is itself a form of socioeconomic racism. It was a strange thing for me to realize as I grew older. I found the history of racial injustice to be just below the surface in Birmingham. I did not witness any overtly racist behavior in public. But, it was there, kind of like that subtle language my elders used when I was young. There was a lot of use of the words like "those people" or "they" when referring to black people. Again, I ignored the people saying these things and was silent.

Shortly after President Obama was elected, I was in a barber shop. It was not a local place. I travel for work between Birmingham and Huntsville very regularly. A small town between here and there had a barber shop. It was very convenient for me to stop in there on my way to Huntsville. I had been in there four or five times with little to no issues. Then one day, I am getting my haircut and a customer comes in. He and the barber seemed to know each other socially. They did not seem to be great friends, but they knew each other's names. At some point, the customer says, "You know what Lincoln, Kennedy, and Obama have in common?" I could think of nothing. The barber said "I don't know." The customer says "Nothing yet." I took a minute or so for it to sink in. He was saying that Obama would be assassinated. I was stunned. That was the most overt racism I had faced in years. The barber didn't seem too happy about it, but he didn't say anything about it either. I never went back to that barbershop. Now though, I regret that I ignored it and was silent.

During President Obama’s tenure, I saw posts on Facebook from friends and family, people I shared a meal with, that blamed him for all their problems. I unfollowed some of them. I ignored others. And, I was silent.

In the last three and a half years, I have seen many of those same people post things supporting President Trump and his agenda to divide this country. I unfollowed some of them. I ignored others. And, I was silent.

I have done my best to be a good example to my children the same way my parents were to me. The other day, I told my youngest child “I don’t think I need to say this, but I am going to. Racism is wrong.” His response was “Of course Dad, I know that.” I wanted to be sure I said it out loud though.

I am done being silent. My silence has not helped the world. Who am I to call anyone a racist? I am a 47 year old, white, middle class man. I will not ignore them and I will not be silent any longer.

Masks in Design and How They Relate to HTML/CSS

brianlmoon — Mon, 16 Apr 2018 11:02:36 -0500

I have always struggled understanding the use of masks in design tools like Photoshop, Illustrator, and Sketch. I recently had to work on translating some creative from Sketch into a responsive design. I realized that a mask in these tools is like a containing element in HTML with overflow hidden and the contents being absolutely positioned.

This is not something you would want to do a lot of as there is a lot of the image being loaded that not being shown. But, it is useful in responsive designs.

See the Pen How masks in design tools translate to HTML by Brian Moon (@brianlmoon) on CodePen.

New Responsive Design

brianlmoon — Tue, 08 Aug 2017 03:43:03 -0500

I have not blogged in over a year. Shame on me. I think part of the reason was because my blog template had become quite dated. I tweaked it a bit to make it semi-responsive a while back. But, I have never been happy with it.

So, I decided to build a new, mobile first template from scratch and use lots of modern (relative to my old template) CSS and responsive web design techniques. Things like rem for sizing things, flexbox, text-shadow, and background-size (for the header image). I know it is nothing ground-breaking. And, at the same time, it was nice to shed all the old compatibility layers and work with the code without worrying about fall backs.

For the font, I chose the super-popular Open Sans after reading "Serif or Sans Serif?" by Danielle Stone. I looked at many sans serif fonts. Open Sans just looked the cleanest to me.

I opted to go with an easy to read black text on white background after reading "How the Web Became Unreadable" by Kevin Marks. One great nugget in that article is where Kevin quotes Adam Schwartz:

A color is a color isn’t a color……not to computers…and not to the human eye.

I found this very interesting. I often look at my "black" SUV and think "my car looks kind of brown today".

With the help of the love of my life, Deedra, I found the header graphic, bought it from iStockPhoto and then played with the hue a bit to get it just right.

While testing, I found myself reading my old blog posts. It felt good. I really need to do this more.

Don't say ASAP when you really mean DEADIN

brianlmoon — Tue, 10 May 2016 16:58:24 -0500

I have found that people tend to use the acronym ASAP incorrectly. ASAP stands for As Soon As Possible. The most important part of that phrase to me is As Possible. Sometimes, it's only possible to get something done 3 weeks from now due to other priorities. Or, to do it correct, it will take hours or days. However, some people don't seem to get this concept. Here are a couple of examples I found on the web.

The Problem with ASAP

What ‘ASAP’ Really Means

ASAP is toxic, avoid it As Soon As Possible

ASAP

It's not the fault of those writers. The world in general seems to be confused on this. Not everyone is confused though. I found ASAP — What It REALLY Means which does seem to get the real meaning.

At DealNews, we struggled with the ambiguity surrounding this acronym. To resolve this, we coined our own own phrase and acronym to represent what some people seem to think ASAP means.

DEADIN:
Drop
Everything
And
Do
It
Now

We use this when something needs to be done right now. It can't wait. The person being asked to DEADIN a task needs to literally drop what they are doing and do this instead. This is a much clearer term than ASAP.

With this new acronym in your quiver, you can better determine the importance of a task. Now, when someone asks you to do something ASAP, you can ask "Is next Tuesday OK?" Or you can tell them it will take 10 hours to do it right. If they are okay with those answers, they really did mean ASAP. If they are not, you can ask them if you should "Drop Everything And Do It Now". (Pro tip: It still make 10 hours to to right. Don't compromise the quality of your work.)

Apple Says My Screen Is Third Party

brianlmoon — Mon, 08 Jun 2015 18:45:06 -0500

I have always had the utmost respect for Apple. Even before I used Macs and before the iPhone came out, I knew they were a top notch company.

I have had five iPhones. I have had 6 or 7 MacBook Pros. My kids have Macs. My kids have iPhones. My parents use iPads. I think a lot of Apple products and service... until today.

We took my daughter's hand me down iPhone 5 in to have the ear piece and top button fixed. It's been in the family the whole time. It was never owned by anyone other than family. Last year, I took it in for the Apple Store Battery Replacement Program. That is the last time anyone had it open. In fact, that may have been the last time it was out of its case. More on this later.

After we dropped off the phone today, we were told it was going to be an hour. No problem, we could kill some time. We came back an hour later and the person brought us the phone out and tells us that they refused to work on it because the screen is a 3rd party part. Whoa! What? I tell her that the only place it was ever worked on was in that exact store. She goes to get a manager. I thought, OK, the Apple customer service I know and love is about to kick in. They are going to realize their mistake and this will all be good. Or, even if they still think it's a 3rd party screen, he will come up with some resolution for the problem. Um, no.

He says the same thing (almost verbatim) to me that the previous person said. I again tell him it has only been opened by them. He offers to take it to the back and have a technician open it up again. He was not really gone long enough for that. He comes back, points at some things on the screen and tells me that is how they know it's a 3rd party part. I again, tell him that only the Apple Store has had it open. His response is a carefully crafted piece of technicality that can only come from lawyers and businessmen. It was along the lines of "At some point, this screen has been replaced with a 3rd party screen. I am not saying you are lying. I am not claiming to know how it was replaced. I am only stating that this is a 3rd party screen." What?

So, OK, what now? I mean, it wasn't under warranty. I did not expect to get a new free phone. I was going to pay to have it fixed. Nope. They won't touch it with a ten foot pole. It has a 3rd party part on it. He claims, that because they base their repair fees on being able to refurbish and reuse the parts they pull off of the phone (the phone I own and paid for by the way), they can't offer to repair a phone with parts they can't refurbish. I can't even pay full price, whatever that is. He never gave me a price to pay for a new screen with no discounts.

At this point, I realized I needed to leave. I was so furious. I was furious it was happening. I was furious that the manager had no solution for me. I was furious that he was speaking in legalese.

Just to be clear, I could buy my daughter a new iPhone 6. I am not trying to get something for nothing. I just wanted the phone to work again. One of the things I love about Apple products is how well they hold up. Sure, you have to have some work done on them sometimes. Batteries go bad. Buttons quit working. But, let's be real. My daughter uses this thing for hours a day. I have the data bill to prove it. So, I like that I can have an Apple product repaired when it breaks and it gets a longer life. The alternative is to throw it away.

How did I end up here? I can only come up with one scenario. And the thought that this is what happened upsets me even more. When we took it for the battery replacement last year, they kept it longer than their initial estimate. And the store was dead that day. When they brought it out, the case would not fit on the bottom of the phone. It was like the screen was not on all the way. The person took it back to the back again. They came out later and it seemed to work fine. And I was fine with all of this because it's Apple. I trust(ed) Apple. But, what if, they broke the screen? What if the tech that broke it was used a screen from some returned phone that did have a third party part and no one caught it? Or what if, Apple was knowingly using third party parts?

If I had not just had the battery replaced last year, I would think maybe there was some shenanigans in the shipping when the phone was new. We bought this phone brand new when the iPhone 5 came out. It would not come as a surprise if some devices had been intercepted and taken apart along the shipping lines. Or even in production. But, we just had it serviced at the Apple Store last year. They had no problem with the screen then other than the one they caused when they had to put it back together a second time.

This all sounds too far fetched right? Sadly, there seems to be a trend of Apple denying service to people. All of these people can't be lying. They can't all be out to get one over on Apple.

While waiting for our appointment, I overheard an Apple Genius telling a woman she "may" have had water damage. She didn't tell her she did. She did not claim the woman was lying. She thought she "may" have water damage. I don't know if she did or not. What struck me was the way she told her she "thought it could be" water damage. She told her she had seen lots of bad screens, but none of them (really? not one single screen?) had vertical lines in it like this. It's like she was setting her up to come back later and say "Darn, the tech says it is water damage." Sadly, I find myself doubting that conversation now. It makes me want to take a phone in with horizontal lines and see if I get the same story.

Of course, I know what many, many people will say to this. You will say that if I am really this upset, I should not buy anymore Apple products. And you are right. That is the American way. The free market is the way to get to companies. The thing is, if I bought a Samsung Galaxy, where would I get it fixed? Would my experience be any better? There is not Samsung store. There are no Authorized Samsung repair facilities. So, what would that get me? A disposable phone? Maybe that is what Apple wants. Maybe that is their goal. Deny service to people in hopes it will lead to more sales and less long term use of their devices.

And you know what makes this all even more crappy? One of the reasons he says he knows it is a third party screen is that the home button is lose. It wasn't lose when we brought it in! I was using the phone myself to make sure a back up was done just before we handed it over to the Apple Store. They did that when they opened the screen and decided it was a third pary part. So, now, my daughter's phone not only has no working ear piece and a top button that works only some of the time. Now, her home button spins around. Sigh.

Using socket_connect with a timeout

brianlmoon — Mon, 09 Mar 2015 18:04:15 -0500

TL;DR

I was having trouble with socket connections timing out reliably. Sometimes, my timeout would be reached. Other times, the connect would fail after three to six seconds. I finally figured out it had to do with trying to connect to a routable, non-localhost address. This function is what I finally ended up with that reliably connects to a working server, fails quickly for a server that has an address/port that is not reachable and will reach the timeout for routable addresses that are not up.

I have put a version of my final function into a Gist on Github. I hope someone finds it useful.

Full Story

So, it seems that when you try and connect to an IP that is routable on the network, but not answering, the TCP stack has some built in timeouts that are not obvious. This differs from trying to connect to an IP address that is up, but not listening on a given port. We took a Gearman server down for maintenance and I noticed our warning logs were showing a 3 to 7 second delay between the attempt to queue jobs and the warning log. The timeout we had set was only 100ms. So, this seemed odd.

After a lot of messing around, a coworker pointed out that in production, the failures were happening for an IP that was routable on the network, but that had no host listening on the IP. I had been using localhost and some foreign port for my "failed" server. After using an IP that was local to our LAN but had no host listening on the IP, I was able to recreate it on a dev server. I figured out that if you set the send and receive timeouts really low before calling connect, you can loop while calling connect. You check the error state and timeout. As long as the error is an acceptable one and the timeout is not reached, keep trying until it connects. It works like a charm.

I found several similar examples to this on the web. However, none of them mixed all these techniques.

You can simply set the send and receive timeouts to your actual timeout and it will return quicker. However, the timeouts apply to the packets. And there are retry rules in place. So, I found that a 100ms timeout for each send and receive would wind up taking 500ms or so to actually fail. This was not what I wanted. I wanted more control. So, I set a 100 microsecond timeout during connect. This makes socket_connect return quickly. As long as the socket error is 115 (in progress) or 114 (already trying), we keep calling it. Unless of course our timeout is reached. Then we fail.

It works really well. Should help for doing server maintenance on our Gearman servers.

Most epic ticket of the day

brianlmoon — Mon, 02 Feb 2015 21:02:41 -0600

UPDATE: I should clarify. This ticket is an internal ticket at DealNews. It is about what the defaults on our servers should be. It is not about what the defaults should be in MySQL. The frustration that UTF8 support in MySQL is only 3 bytes is quite real.

This epic ticket of the day is brought to you by Joe Hopkinson.

#7940: Default charset should be utf8mb4
------------------------------------------------------------------------
The RFC for UTF-8 states, AND I QUOTE:

> In UTF-8, characters from the U+0000..U+10FFFF range (the UTF-16
accessible range) are encoded using sequences of 1 to 4 octets.

What's that? You don't believe me?! Well, you can read it for yourself
here!

What is an octet, you ask? It's a unit of digital information in computing
and telecommunications that consists of eight bits. (Hence, __oct__et.)

"So what?", said the neck bearded MySQL developer dressed as Neo from the
Matrix, as he smuggly quaffed a Surge and settled down to play Virtua
Fighter 4 on his dusty PS2.

So, if you recall from your Pre-Intro to Programming, 8 bits = 1 byte.
Thus, the RFC states that the storage maximum storage requirements for a
multibyte character must be 4 bytes, as required.

I know that RFCs are more of GUIDELINE, right? It's not like they could be
considered a standard or anything! It's not like there should be an
implicit contract when an implementor decides to use a label like "UTF-8",
right?

Because of you, we have to strip our reader's carefully crafted emojii.
Because of you, our search term data will never be exact. Because of you,
we have to spend COUNTLESS HOURS altering every table that we have (which
is a lot, by the way) to make sure that we can support a standard that was
written in 2003!

A cursory search shows that shortly after 2003, MySQL release quality
started to tank. I can only assume that was because of you.

Jerk.

* The default charset should be utf8mb4.
* Alter and test critical business processes.
* Change OrderedFunctionSet to generate the appropriate tables.
* Generate ptosc or propagator scripts to update everything else, as needed.
* Curse the MySQL developer who caused this.

Keeping your data work on the server using UNION

brianlmoon — Tue, 08 Jul 2014 10:25:35 -0500

I have found myself using UNION in MySQL more and more lately. In this example, I am using it to speed up queries that are using IN clauses. MySQL handles the IN clause like a big OR operation. Recently, I created what looks like a very crazy query using UNION, that in fact helped our MySQL servers perform much better.

With any technology you use, you have to ask yourself, "What is this tech good at doing?" For me, MySQL has always been excelent at running lots of small queries that use primary, unique, or well defined covering indexes. I guess most databases are good at that. Perhaps that is the bare minimum for any database. MySQL seems to excel at doing this however. We had a query that looked like this:

select category_id, count(*) from some_table

where

    article_id in (1,2,3,4,5,6,7,8,9) and

    category_id in (11,22,33,44,55,66,77,88,99) and

    some_date_time > now() - interval 30 day

group by

    category_id

There were more things in the where clause. I am not including them all in these examples. MySQL does not have a lot it can do with that query. Maybe there is a key on the date field it can use. And if the date field limits the possible rows, a scan of those rows will be quick. That was not the case here. We were asking for a lot of data to be scanned. Depending on how many items were in the in clauses, this query could take as much as 800 milliseconds to return. Our goal at DealNews is to have all pages generate in under 300 milliseconds. So, this one query was 2.5x our total page time.

In case you were wondering what this query is used for, it is used to calculate the counts of items in sub categories on our category navigation pages. On this page it's the box on the left hand side labeled "Category". Those numbers next to each category are what we are asking this query to return to us.

Because I know how my data is stored and structured, I can fix this slow query. I happen to know that there are many fewer rows for each item for article_id than there is for category_id. There is also a key on this table on article_id and some_date_time. That means, for a single article_id, MySQL could find the rows it wants very quickly. Without using a union, the only solution would be to query all this data in a loop in code and get all the results back and reassemble them in code. That is a lot of wasted round trip work for the application however. You see this pattern a fair amount in PHP code. It is one of my pet peeves. I have written before about keeping the data on the server. The same idea applies here. I turned the above query into this:

select category_id, sum(count) as count from 

(

    (

        select category_id, count(*) as count from some_table

        where

            article_id=1 and

            category_id in (11,22,33,44,55,66,77,88,99) and

            some_date_time > now() - interval 30 day

        group by

            category_id

    )

    union all

    (

        select category_id, count(*) as count from some_table

        where

            article_id=2 and

            category_id in (11,22,33,44,55,66,77,88,99) and

            some_date_time > now() - interval 30 day

        group by

            category_id

    )

    union all

    (

        select category_id, count(*) as count from some_table

        where

            article_id=3 and

            category_id in (11,22,33,44,55,66,77,88,99) and

            some_date_time > now() - interval 30 day

        group by

            category_id

    )

    union all

    (

        select category_id, count(*) as count from some_table

        where

            article_id=4 and

            category_id in (11,22,33,44,55,66,77,88,99) and

            some_date_time > now() - interval 30 day

        group by

            category_id

    )

    union all

    (

        select category_id, count(*) as count from some_table

        where

            article_id=5 and

            category_id in (11,22,33,44,55,66,77,88,99) and

            some_date_time > now() - interval 30 day

        group by

            category_id

    )

    union all

    (

        select category_id, count(*) as count from some_table

        where

            article_id=6 and

            category_id in (11,22,33,44,55,66,77,88,99) and

            some_date_time > now() - interval 30 day

        group by

            category_id

    )

    union all

    (

        select category_id, count(*) as count from some_table

        where

            article_id=7 and

            category_id in (11,22,33,44,55,66,77,88,99) and

            some_date_time > now() - interval 30 day

        group by

            category_id

    )

    union all

    (

        select category_id, count(*) as count from some_table

        where

            article_id=8 and

            category_id in (11,22,33,44,55,66,77,88,99) and

            some_date_time > now() - interval 30 day

        group by

            category_id

    )

    union all

    (

        select category_id, count(*) as count from some_table

        where

            article_id=9 and

            category_id in (11,22,33,44,55,66,77,88,99) and

            some_date_time > now() - interval 30 day

        group by

            category_id

    )

) derived_table

group by

    category_id

Pretty gnarly looking huh? The run time of that query is 8ms. Yes, MySQL has to perform 9 subqueries and then the outer query. And because it can use good keys for the subqueries, the total execution time for this query is only 8ms. The data comes back from the database ready to use in one trip to the server. The page generation time for those pages went from a mean of 213ms with a standard deviation of 136ms to a mean of 196ms and standard deviation of 81ms. That may not sound like a lot. Take a look at how much less work the MySQL servers are doing now.

The arrow in the image is when I rolled the change out. Several other graphs show the change in server performance as well.

The UNION is a great way to keep your data on the server until it's ready to come back to your application. Do you think it can be of use to you in your application?

Parenting When Your Kid Is "An Adult"

brianlmoon — Mon, 09 Jun 2014 10:14:41 -0500

When I dropped out of college at 19, I came home to my parents' house. My parents had moved since I had left home. There was no room for me in the new house. I was not there to claim one when they moved in. My Dad and I put up a wall with paneling on it to enclose part of the garage. We cut a hole in the air duct that was in that space. Tada! That was now my bedroom. My room consisted of a concrete floor, three walls with paneling, one concrete block wall, a twin bed (I had a king size when I left 15 months before), and maybe a table. I was not happy. But, that is what was offered to me. I kind of held a grudge about that for a while.

As of right now, my oldest son is 18 years old. He starts college in the fall. I am so very proud of him. He was accepted to an honors program. His grades and testing earned him scholarships. His future is very bright. For this summer, though, he is still home. He has no job. Our attempts to get him to get one have fallen short. He is not motivated to do so. I refuse to go find him one. So, I am giving him one. In exchange for room and board, gas for his car, his car, his car insurance and whatever money is left after those expenses are paid going into his pocket, he will be my assistant. He will fetch his siblings from various places, run errands for me, do extra chores around the house, and anything else I need. To earn his car, he has been doing the "personal driver" service for a while for me. I am expecting more of him this summer though. This arrangement has its good days and bad days.

Today, I suddenly realized why my parents put me in that basement. The bad news for my son is that our basement is darker, dirtier, hotter and a lot less comfortable than the one I lived in at 19 years old. Let's hope I don't get to the point where I want to put him down there.

24 Hour Clock in Thunderbird for Mac OS X

brianlmoon — Wed, 04 Sep 2013 09:37:12 -0500

I like to use a 24 hour clock. I have my iPhone set to a 24 hour clock. I have my Mac's time setting set to a 24 hour clock. I would set my car to a 24 hour clock if I could. It's German. You would think that would be an option, but no.

Thunderbird was not using a 24 hour clock despite me setting my Mac to a 24 hour clock in the Date/Time preferences. I found some mentions of an "International" settings panel in System Preferences, but I did not have anything named that in OS X Lion. I clicked around and found the Language & Text panel. This did the trick. You click Region and then Customize... in Times. The UI was confusing at first and then I realized it worked like iWork cell formatting. You click the hour and a drop down of options appears. I set it to 0-23 and removed the AM/PM part of the formatting and all was good. Thunderbird now uses a 24 hour clock display for all messages. To my surprise, it took effect immediately. All the times in Thunderbird changes as soon as I changed the settings.

Hope this helps others that want to see times this way.

Taking Your Eyes Off The Road

brianlmoon — Fri, 21 Jun 2013 08:00:00 -0500

Source: http://www.flickr.com/photos/viernest/3380560365/

About a year ago, I had a wreck. I totaled my car. I took my eyes off the road to look at my son in the back seat. I put two of my children in danger. Luckily, everything turned out alright.

This week, I have attended the Velocity Conference. It’s not my first time. I have attended all of them but the one last year. Velocity is all about Web Performance and Operations. I attended mostly web and mobile performance tracks. I was quickly reminded (like, first day, first session) of many things I have been wanting to implement to help me know how DealNews.com is doing performance wise. So, like I often do at conferences, I started hacking. This was Tuesday.

By Wednesday morning, I had some stats. Those stats led to more questions. I refactored some of the stats I was collecting. By dinner, I had good data about our page performance. I was pissed.... at myself. As I said before, I didn’t attend Velocity in 2012. In 2012, I attended other things not related to web performance. In doing so, I took my eye off the road. Or in this case, off the performance of DealNews.com.

Now, we still get an A from WebPageTest for first byte. We don’t get any bad scores really. We aren’t doing poorly. The site performance is just no where near where I want it to be. And it is nowhere near where I have been telling people it was. We deliver the first byte in around 500ms for a request that can use cache well. We draw the above the fold in about 1.5 seconds. I have seen way worse sites out there. But, some times, its about yard sticks.

Source: http://www.flickr.com/photos/billhd/3048457153/

You see, if you are measuring using a broken, worn out yard stick, it may not be an actual yard. You need to measure using the the latest greatest, laser cut yard stick. So, when I compare DealNews performance with others, I look to the best of the best.

Amazon, ShopZilla, and others have openly talked about performance and business success being directly correlated. If that is true for DealNews, there is low hanging fruit to improve our business. And apparently that fruit is rotting its been hanging so long.

I have already found 480ms in the header I can trim down. I am not sure yet how much I can reduce it, but it can be faster. I am hoping I can get it down to 100ms. That would be a huge savings as our header currently finishes in about 980ms on average. That would be cutting more than 25% of our header load time completely out. And that is just the first thing I have found.

I saw other good talks that will help me get back on track as well. One talked about premature optimization. Before I put in the new metrics, I had a theory on what was taking up that time. I was wrong. Not totally wrong. That thing is still taking 150ms, so it is next on the list. But, the other issue is clearly more problematic to me since I assumed it was a non-issue and it caught me by surprise.

If you are asking “Brian, how are you doing this?” I am glad you asked. I am using the window.performance.timing object available in new browers. After the onload event fires a script gathers up this data and send it back to our servers in an XHR request. Server side code then takes that data, does a little math where needed and sends it all through StatsD which in turn shoves it in Graphite. That lets me build graphs and get the data as JSON. That second part is key as I will want to put some automated monitoring on this data to keep an eye on when it may go bad again. There were a lot of talks this week about detecting fault or detecting anomalies as well. So, I will put that to good use with the help of a coworker who loves the hard math problems. If you don't have those things in your stack already, SOASTA mPulse appears to be a good option. I was impressed with Philip Tellis from SOASTA in his talk about JavaScript load blocking. Since the mPulse code runs in a JavaScript tag, I was happy to hear he was so concerned with how it affected their user's performance.

I will post anything I think is useful to the general public. Right now, it looks like code and feature bloat. That is not all that interesting.

Being a Better Manager: Communication

brianlmoon — Mon, 06 May 2013 08:17:43 -0500

As I have worked on being a better manager, I have been trying to determine what our strengths and weakness are as a development team. Communication is once place I think we can do a better job.

The problem we have had was how other departments communicated with the development team. The company had been small and now it is large. When the company was small, anyone could talk to anyone. If Bob knows that Tom works on his revenue report, he could just ask him about it. The problem comes when there are 10 Bob’s and 5 Tom’s and people are switching roles in the development team and in other teams. Bob is working on something else and is still coming to Tom with his problems. Tom has no idea how to help him. But, he is a good employee so he tries. He gets distracted from his project and probably does not help Bob all that much in the end. No one is at fault here really. Tom just wants to get his job done. Bob just wants to help a coworker.

When teams start to grow, communication needs to be directed. It’s not some hard and fast rule and people are not punished for talking outside of the chain. However, it helps that when Tom in marketing has a problem, he knows who to talk to every time. By default that is me. It is then my job to know what development resource to tap to solve it. On the other side, the developers have to feel comfortable with telling people “I don’t know. Will you file a ticket or talk to Brian about that?” People are generally helpful. They want to help. I have tried to let people know it is OK to not help if they can’t help or perhaps they are worried about the scope of the problem. I have found that some people click and natural connections will happen. I have no problem with that. There are some people that I expect to reach out to the person they are ultimately writing code for to get feedback. I also stress to them that if the scope of what the person needs changes, we need to talk about it.

From my position, I have to be ready to listen. I want the communication going through me. If I am a jerk, don’t reply to email, or tell people no all day long, that will probably not help me achieve my goals or help the people that need development resources. I have learned to keep a more open mind. I have learned to not say no, but instead say “I don’t think this is a good idea because of these reasons.” And if I think I can modify the idea to something that is workable, I will offer that as an alternative. I feel like this is working well for me and I have been told by others that it is well received.

One place I can not seem to get right is quarterly managers’ meetings. Everyone takes a turn talking about what is going on in their department. When marketing talks, everyone is really interested. Same with financial and sales. When it comes time for me to talk, everyone seems to glass over. I firmly believe it is the content that is the problem. I am going to try a new tactic as recommended by our CEO. Rather than talk about what we did, talk about why we did it. For example, instead of saying we made a change to our code to do X I should talk about the business reasons we spent time on making that change. Instead of “We rolled a new release of the app.” I tell them “Our new app is catching up with the features of the web site.” I am still working on it. Even that still sounds boring to me. The only good news is that technical operations follows me. It is really hard to talk about the fact that we didn’t go down and that we installed new servers sound interesting to non-geeks.

Test Driven Development Conversion

brianlmoon — Mon, 22 Apr 2013 08:00:00 -0500

The concept of unit testing is not new, however it has only gained popularity in the last 10 years or so. When I was asked about it for the first time, I was unsure where it would fit in to my development life. I dismissed it for a long time. I have begun to come around however. I wanted to share how and why.

I have been a professional software developer since 1996. Neither in school nor in practice in the early days of my career was I exposed to the concept of writing tests to confirm portions of my code worked as expected. Testing was always done via user testing. You or someone else would use the program to ensure it produced the expected result. That is how I learned to do my job. And I am good at my job.

My initial reaction to unit testing was negative. I know professional developers that sit in cubicles all day long and write code without any knowledge of the overall application. They are asked to write a function that takes a certain input and returns a certain output. They neither have nor desire additional knowledge about the use of their code. I find those people to be very unattractive as coworkers. They are often out of touch with technology. When I told someone in this line of work I used a Mac, he asked "Do they still make software for those?" He was dead serious. This was in 2005. My initial feelings about unit testing reminded me of these people. Just sitting in cubes, all day, writing functions that took X and returned Y. They required no knowledge about where X came from or where Y was going. Furthermore, I felt like people that must have to have that kind of structure in place to do their jobs must not be capable of understanding the big picture. If they did understand the big picture, they would not need to have these things. They should somehow KNOW how the application worked.

The language I use most (and have for the last 17 years) is PHP. PHP development has always had a haphazard culture around it. I don’t see that as a bad thing. Lots of people all doing things their way. That is often a criticism. People in other language communities where things are a bit more dictated and regimented find it disturbing. I find it refreshing. It’s more real world than the rest to me. But, that is a different blog post.

PHP didn’t come with any sort of testing framework out of the box. And none of the core developers put out a way to test your PHP code. The first tests of any kind that I was aware of for PHP were those built by the PHP QA team. It was targeted at testing the PHP engine by writing PHP code that used the engine. That made sense to me since there are so many moving parts in there and you have lots of people all making changes. It makes sense to have some sanity checks.

The first time I heard about testing your own PHP code was PHPUnit. My first impression was that it was an imitation of JUnit which is the de facto standard way to test Java. I don’t really care for Java. I don’t care for the language. I don’t care for the culture. I don’t care for how a java daemon behaves on my server. In general, I don’t care for the platform. It is my personal preference. It is not right or wrong. It is just how I feel. So, when I see something being copied from Java into PHP, I retreat. That probably makes me a bad person. It is a personality flaw. The end result is that I ignored PHPUnit no matter who said they were using it or why. And the only thing I ever heard about when people talked about testing PHP applications was using PHPUnit.

So, how does someone like me start advocating test driven development as a useful tool? It snuck up on me. A few things had to happen.

The first thing that happened was that we started growing our team. In 2006, we were back down to a two person development team. We had been as large as 5 during the dotcom boom. But, we slowly dwindled back down to two people doing development and systems administration every day as their full time jobs. When there are only two of you, you kind of have to know how everything works. If you don’t, you can’t keep running. As we started to add developers, I realized it was increasingly harder for other people to grok the whole application they way I did. And at first, I thought that was just a matter of time. After a year or so they would get it all. So, we hired more people. We are now at six developers, plus myself and one new one starting in two weeks. On top of that we are actively looking to hire two more as soon as possible. Along the way there have been some others come and go as well. Now, to people in the VC bubbles, this may sound boring. I know places where the team grows 20x in 6 months. I can’t imagine. Our application and code base is huge though. Experienced engineers that are now on the team have told me they have never worked with a code base this large. In the last couple of rounds of hiring, I was asked about testing. My standard answer was that if you wanted to write tests for your code and could develop a sane strategy for deploying those tests, I had no problem with it. As a team however, there was no official policy or systems in place for testing. One of the new hires pointed me to a something that peaked my interest. It was doctest for Python. Essentially, you put a bit of Python code in the doc block for the code you are writing. doctest will scan files for these tests and run them. Partly out of curiosity and partly to hopefully satisfy the guys wanting some way to test their code, I wrote a version of doctest for PHP. To prove it worked, I added a test block to all the methods of an array manipulation class I had worked on recently. I was still not convinced that this was a good use of my time however.

The second thing that happened was I met up with Chris Hartjes (@grmpyprogrammer). Chris is an avid supporter of test driven development. I had heard the term test driven development before meeting Chris, but my testing bias had shielded me from really understanding the concept. He and I talked and later I was interviewed by Chris and Ed Finkler (@funkatron) on the /dev/hell podcast. On that podcast we talked about testing as well as other things. Looking back on those two conversations, I now know what Chris did that most changed how I looked at people who are adamant believers in dynamic unit testing. Chris didn’t make it a personal thing. He acknowledged my perspective. We do have a code base and business that has been working for 10+ years. We do have extensive monitoring and graphing in place that tells us when things are going wrong. He told me that if that was working for me, more power to me. Wait, wasn’t he supposed to tell me how I was doing it wrong? Wasn’t he supposed to tell me I was a cowboy and how unprofessional I am? He did none of those things. So, you know what happened? I kept listening to him.

The third thing that happened was also because of Chris (just so you know now, most of this blog post is indirect or direct praising of Chris). Once I started listening, there was something else I was not hearing from him. I was not hearing how I should be using PHPUnit or any particular product for that matter. He only said that code should be written to satisfy a test. He didn’t seem to care how it was tested. He only cared that it was tested. This was again a breathe of fresh air. In this business where every group likes to tell the other group how wrong they are doing things, having someone that didn’t seem to care how you did it, as long as you did it was awesome. So, again, I kept listening to him.

The fourth thing that happened was when Chris made me realize that I already was a test driven developer. I don't think he realized that he did it. He likely does not remember the conversation. And I probably remember it poorly. For all I know, there were beers and/or whiskey involved. It went something like “So, how do you test your code?” said Chris. I replied “Well, if the web page loads, it worked.” To which he asked “So, you don’t test changes as you make them?” I said “Well, yeah, I write some test code and run it till it works while I am making changes. I just don’t save that code. Once its working I don’t need it.” I don’t think it hit me right then. I think it took days or weeks to sink in. Mother F--ker, I already write code in a test driven manner a lot of the time. I have a test.php script in my home directory on my development server. I reuse that thing all the time to test the code I am working on. Instead of assertions, lots of the time I will just print_r() the output and assert with my eyes. But, the end result is the same. So, why not take that extra step? I had the doctest stuff in place. I could just add my test code there and run the test over and over until it worked like I wanted it to. And what do you know, I kind of liked it.

And what I think was the last straw was when I was merging code onto staging and a test failed. Oh my gosh! I made a change to our code base and my all knowing, all seeing eye did not realize that my changes were going to break something. How is this possible? How did I not see this coming? I know everything in the application don’t I? I architected the whole thing. The reality is, this likely (and by likely I mean, it did) happened before. However, sometime very soon, either a monitor or a human would have noticed and a bug report would have been filed. I would have then fixed it. I would have likely justified the bug as the cost of progress. Now I have a way to help prevent these small bugs from rolling out in the first place. And now that I have this tool, there is no excuse. The only reason this should happen now is that we don’t have enough tests.

We are working on improving our test coverage. I am not to the point yet where I require tests for all code. Perhaps we will get to that point. I don’t know. My team is aware of the tool now. And they are aware that I use it and think it is wise to use it. I told them recently that I am not telling them to use it. But, if they roll a bug, and there was not a test, the fix should probably be written using test driven development.

I have intentionally not talked about what we use other than the a fore mentioned doctest. I hope the message that you take away from what I have learned is that you should be using SOMETHING to test code. That is more important than how you test your code. I also have not talked about how you write code that can be tested. That is another hurdle I have had to (still am) get over. I thought a lot of our code was not testable. Because you know, we are solving problems that no one ever in the history of the world has had to solve, like retrieving a web page, hard stuff. There are better resources to learn how to write code to be more testable, especially for PHP. I highly recommend everything Chris Hartjes has ever said or written about the subject. You can find his thoughts on the topic at http://grumpy-learning.com/, http://grumpy-phpunit.com/, and https://leanpub.com/grumpy-testing.

We will be improving doctest.php. We have lots of ideas. It just fits better for us. If PHPUnit works for you, great, use it. Just search for `unit testing PHP`and find something that works for you. In addition, we are starting to work with Selenium for interactive testing of our site and some commercial products that are really good at testing APIs. I am getting a hard time from some guys at work. I have been critical of testing in the past. I just needed to understand that I already worked this way or thought this way.

College Basketball in 2013

brianlmoon — Tue, 09 Apr 2013 11:51:04 -0500

I am a huge sports fan. I particularly love college sports. We don't have any pro sports teams in Alabama. So, we take our college sports very seriously. Like many sports fans I have watched the NCAA tournament this year. I have to say, if the current trend continues, I don't think I will in the future. I don't like the product.

In particular, I pretty much hate the way Louisville plays basketball. Yes, the won. Kudos to them. I don't blame them really. No, the real problem is the lack of offensive foul calls. Players push and shove each other. Guys going up for a lay up from under the back board are free to bully their way up to the rim including jumping backwards into a well placed defender. The only thing that matters to the NCAA is scoring, at any cost.

Luke Hancock stealing the game from Witchita State by fouling although it was called a jump ball. / Jaime Green/MCT via freep.com

So, getting back to Louisville, Rick Petino and his players have figured out just how far they can push these limits. They are almost playing hockey. But, in basketball, there is no penalty box. You get 5 fouls. If you have a deep bench like Petino, you just plug in another guy and they keep fouling. On the other side you have Luke Hancock, named MVP of the Final Four (geez, talk about bad role models) throwing a pump fake and then leaning into a player in a direction that is not toward the basket or any kind of natural shooting motion, just to draw the foul. Those should be no calls or offensive fouls, IMO.

I coach youth basketball. I teach my kids to not foul. I teach my kids to not touch each other when at all possible. On defense, they need to be in position and ready to move their feet. On offense, don't try and run over people. If you have to foul to stop someone, then we need a better game plan. Or maybe they are just better than we are. I have on occasion praised a kid for being aggressive which led to a foul. These are eight year olds. And some of them are still in a shell about being athletic. But, I never say "foul him". And I never have my kid try and draw a foul the way Luke Hancock does. That is just dirty basketball. I will tell my kids to drive the lane and go for the goal. And "if" you get fouled it's OK. But, never go looking for the foul.

Contrastingly, I watched some of the New York Knicks vs. Oklahoma City Thunder game Sunday. These guys hardly ever touch each other. They play within, what I believe, are the real rules of basketball. The game is a little fast and there is a lot of one on one play that can be tedious. But, it was more fun to watch than NCAA basketball. Maybe all the good basketball players go to the NBA and we are left with scrubs in the NCAA. They are too small (size wise) or to slow to play corner back in football so they end up playing basketball as an "athlete". I hope something is done about this. The game is just getting trashy.

The Web We Lost

brianlmoon — Tue, 19 Mar 2013 19:34:36 -0500

I was reading Chris Shiflett's blog and he mentioned a blog post about the web the way it used to be before Facebook and Twitter. I almost tweeted it. Then I thought, nah, a good old fashion blog post that linked to it was way better.

The Web We Lost - Anil Dash

The tech industry and its press have treated the rise of billion-scale social networks and ubiquitous smartphone apps as an unadulterated win for regular people, a triumph of usability and empowerment. They seldom talk about what we've lost along the way in this transition, and I find that younger folks may not even know how the web used to be.

It is a good read for all us "old timers". He ironically uses Facebook for comments. One really good one that I can't actually link to afaik because... Facebook said:

No sarcasm here.. I legitimately miss the "web-rings" of old. With niche interests, it was a great way to find like minded sites, and I found many of my still bookmarked favorites with those old crappy left and right arrows :)

Yeah, you know, Web Rings weren't that bad. I had sites on a couple of those things. I think they are still around. And I think you could still pull them off and not worry about Google juice and all that. 302 redirects don't pass that crap. Maybe I should write up some quick javascript that indexs the Planet PHP blogs and adds a Planet PHP web ring to your page. Hmmmm.

Reliable Delivery

brianlmoon — Mon, 11 Mar 2013 08:00:00 -0500

There is plenty to read about continuous delivery in terms of rolling out code. In my journey to be a better leader and manager, I have realized there is something we are doing badly. While we are continuously integrating, testing, and deploying code to production, we are not reliably delivering a product for our client. I use client loosely here. The client for the development team at dealnews is the company itself. With every project however, there are people interested in its progress. We were (still are some) underserving those people. Those interested often have no clue of when something will be done. Sometimes things would are done and they do not know it. So, how do we solve this problem?

There was a time when we were a team of two people. If some server issue popped up, it totally derailed whatever project was being worked on. So, we grew accustomed to missing deadlines out of necessity. Now that we are a real team, that excuse is no longer valid.

Part of the problem also lies in the team’s (i.e. me) OSS roots. I really got started doing web development by writing Phorum. In fact, my first job when I was hired full time for, then, dealmac.com was to update the Phorum software to scale better. Banner ads were at an all time high in the late 90s. We made a lot of money off those page views. In OSS, the answer to “When will X be done?” is often “it’s done when it’s done”. Blizzard Entertainment, creators of World of Warcraft, have been quoted as saying that about their products.

Telling someone it will be done when it is done sounds really cool. I feel like a bad ass. It’s art! It’s not about a timeline. I can’t be bothered with your pesky expectations. Except, that is bullshit. In reality, there are people depending on me and my team to get work done. So, that was one of the first things I wanted to change. We have gotten better. Here are some things I have learned.

Think before speaking. If I am in a meeting and talking about a new task or project, I try not to throw out a time frame for completion. I tell them I will get back to them. I try and tell them when I will get back with them. I then gather anyone on the team that needs to have input and evaluate the changes. Once I have a solid answer, I report back to the other department. Very often I worry people will be mad when I say “two weeks” so I say “one week” and hope for a miracle. But if it is going to be two weeks, I need to tell them that. That may be too late. Or it may be not worth it to them to take that much of our time. Of course not all tasks need this kind of time commitment to deciding a time frame. Deciding which do and which don’t is tough sometimes.

I wanted to start communicating deadlines to our developers. We had never had a ticketing system that supported due dates. We had the “done when done” philosophy. I was really worried about adding them. I didn’t want people to feel like they were being micro-managed. After a couple of months, everyone is much happier. Turns out developers really like knowing when things are expected to be done. It also helps to prioritize different tasks. If a developer has 5 things assigned to them, they can look at the due dates to decide which is more important. Because, you know, they are all marked “highest” priority. Developers have the freedom to speak up and say “there is no way I can finish this by that date”. It’s possible I completely misjudged the scale of the change. It is also possible I wrote a horrible ticket and the developer is confused by my 2AM stream of consciousness.

There is another hurdle for me. I have gotten better at managing expectations of other departments and helping developers know what is expected of them. It is better. It is not perfect. It may never be. I am still struggling with doing the same with my own development tasks. I catch myself thinking “Well, that is just how it is when I have to manage and develop.” But that is a total excuse and a cop out. I have to learn to do that better. Managing my own time may be the toughest of all. If and when I figure something out, I will write about it.

Becoming a Better Manager

brianlmoon — Sat, 09 Mar 2013 20:02:47 -0600

I have typically blogged on this site about things I have learned in the web
application world that may help others. In the last year or so, I have been
learning a lot of new things. Most of them are not technical in nature however.
You see, I have moved into a role of being a manager. I am a developing manager.
I still write code. And a lot of my time is dedicated to management as well. I
think this has caused me to stop blogging as much. My mind didn't see these
topics as interesting to what I perceive to be the audience of my blog as things
I have blogged about in the past. The problem is, I miss blogging.

So, going forward, there may be some non-technical things on this blog. My hope
is that someone out there finds them as useful as some of my more technical
blogs posts have been.

Developers and Entropy

brianlmoon — Fri, 13 Jul 2012 12:07:00 -0500

This is a selfish blog post. I read a great blog post titled "Why You Need To Hire Great Developers" but I could not find it in my browser history or chat history. It talks about entropy creators versus entropy reducers and how bad we are at knowing which one someone is during the hiring process. I wanted to mention it here so my followers could read it and so I could find it again when I was looking for it.

Lock Wait Timeout Errors or Leave Your Data on the Server

brianlmoon — Wed, 27 Jun 2012 01:12:34 -0500

If you use MySQL with InnoDB (most everyone) then you will likely see this error at some point. There is some confusion sometimes about what this means. Let me try and explain it.

Let's say we have a connection called A to the database. Connection A tries to update a row. But, it receives a lock wait timeout error. That does not mean that connection A did anything wrong. It means that another connection, call it B, is also updating a row that connection A wants to update. But, connection B has an open transaction that has not been committed yet. So, MySQL won't let you update that row from connection A. Make sense?

The first mistake people may make is looking at the code that throws the error to find a solution. It is hardly ever the code that throws the error that is the problem. In our case, it was code that was doing a simple insert into a table. I had a look at our processing logs around the time that the errors were thrown and I found a job that was running during that time. I then looked for code in that job that updates the table that was locked. This was where the problem lied.

So, why does this happen? Well, there can be very legitimate reasons. There can also be very careless reasons. The genesis of this blog post was some code that appeared to be legitimate at first, but upon further inspection was careless. This is basically what the code did.

Start Transaction on database1
Clear out some old data from the table
Select a bunch of data from database2.table
Loop in PHP, updating each row in its own query to update one column
Select a bunch of data from database2.other_table
Loop in PHP, updating each row in its own query to update another column
Commit database1

This code ran in about 20 minutes on the data set we had. It kept a transaction open the whole time. It appeared legit at first because you can't join the data as there are sums and counts going on that have a one to many relationship which would cause some duplication of the sums and counts. It also looks legit because you are having to pull data from one database into another. However, there is a solution for this. We need to stop pulling all this data into PHP land and let it stay on the server where it lives. So, I changed it to this.

Create temp table on database2 to hold mydata
Select data from database2.table into my temp table
Select data from database2.other_table into my temp table
Move my temp table using extended inserts via PHP from database2 to database1
Start Transaction on database1
Clear out some old data from the table
Do a multi-table bulk update of my real table using the temp table
Commit database1

This runs in 3 minutes and only requires a 90 second transaction lock. Our lock wait timeout on this server is 50 seconds though. However, we have a 3 time retry rule for any lock wait timeout in our DB code. So, this should allow for our current workload to be processed without any data loss.

So, why did this help so much? We are not moving data from MySQL to PHP over and over. This applies to any language, not just PHP. The extended inserts for moving the temp table from one db to another really help. That is the fastest part of the whole thing. It moves about 2 million records from one to the other in about 1.5 seconds.

So, if you see a lock wait timeout, don't think you should sleep longer between retries. And don't dissect the code that is throwing the error. You have to dig in and find what else is running when it happens. Good luck.

Bonus: If you have memory issues in your application code, these techniques can help with those too.

Scaling 101 - We are Failing the Next Generation

brianlmoon — Tue, 26 Jun 2012 00:01:36 -0500

The other day Twitter was down and I had no place to comment on Twitter being down. This got us to talking about scaling at work. I was reminded of the recent slides posted from Instagram about their scaling journey. They are great slides. There is only one problem I have with them. They are just the same slides that you would find from 2000 about scaling.

I have to say, I like Instagram. My daughter has something like 1,000 followers on Instagram. And good for them for being bought by Facebook for a bajillion dollars. This is not a dig on them really. This is a dig on our industry. Why did Instagram have to learn the hard way how to scale their app? I want to point out some of their issues and discuss why its silly they had to learn this the hard way.

Single machine somewhere in LA

Why would anyone deploy an app to the app store when the backend is all on one server in this day and age? I am not big poponent of the cloud, but that has to be better than a single server in a rack somewhere. And L.A.? Go for Dallas or somewhere geographically neutral.

favicon.ico

So, this is one of the biggest mistakes I see in all of web application developement. People use Apache or Nginx or whatever and have a mod_rewrite command that sends ALL requsets into their appliction stack. They do this because they are lazy. They want to write whatever code they want later and just have the request picked up without any work. At dealnews, we don't do that. We have controllers. But, we specify in our Apache config what paths are routed to those controllers. The most general we have is:

RewriteCond %{REQUEST_URI} (\/|.html|.php)$
RewriteRule ^.+$ /torpedo.php/$1 [L]

So, if the request ends with / or .html or .php, send it to the controller. This is the controller our proxy servers use. Any other requests like robots.txt, images, etc. are all served off disk by Apache. No sense having a PHP process handle that. It's crazy. This is not Instagram's fault. They likely followed some examples they found that other developers before them put on the internet. Why are we doing this to people? Is it some rite of passage? I suspect that the problem is actually that 90% of the web does not require any scaling. Only 10% of the sites and services out there have to actually worry about load. So, this never comes up.

So, the next part of their slides basically glorify the scaling process. This is another problem with people in our field. We thrive on the chaos. These are our glory days. Let's face it, most geeks never won the high school football championship. The days when we are faced with a huge scaling challenge are our glory days. I know I have had that problem. And I played sports as a youth. But, nothing is better than my 2006 war story about a Yahoo front page link. Man, I rocked that shit. But, you know what. The fact that I had to struggle through that means I did not do my job. We should have never been facing that issue. It should have just all worked. That is what it does now. It just works. We don't even know when we get a spike now. It just works. The last thing I want in my life now is an unexpected outage that I have to RAGE on to get the site up again. That leaves me feeling like a failure.

Then they realize they are out of the depths. They need to do things they never thought they would need to do. Why not? Why did they not know they would need to do these things? Unexpected growth? Maybe. Why is this not common knowledge though? With all the talk of the cloud being awesome for scaling, why was this not a button on the AWS dashboard that said [SCALE] that you just push? That is what the cloud does right?

In the end, Instagram learned, the hard way again, that you have to build your own architecture to solve your problem. I learned it the hard way. LiveJournal learned it the hard way. Facebook, Twitter, etc. etc. They have all learned the hard way that there is no single solution for massive scale. You have to be prepared to build the architecture that solves your problem in a way that you can manage. But, there are basic building blocks of all scaling that need to be in place. You should never, ever, ever start with an application on a single server that reads and writes directly to the database with no cache in place. Couch, Mongo, blah blah blah. Whatever. They all need cache in front of them to scale. Just build it in from the start.

Instagram was storing images. Why were they surprised when they ran out of room for the images? I just can't fathom that. Its images. They don't compress. You have to put them somewhere. This has to be an education issue. LiveJournal solved this in 2003 with MogileFS and Gearman. Why did they not build their arch on top of that to start with? Poor education, that is why.

One thing they bring up is one that has me bugged is monitoring. There is no good solution for this. Everyone ends up rolling their own solutions using a few different tools. The tools are often the same, but the metrics, and how they are reported and monitored are all different. I think there is a clear need in the industry for some standards in this area.

if you’re tempted to reinvent the wheel ... don't

I did find this slide funny. Reading these slides for me is like seeing them reinvent the whole wheel that is scaling. This has all been done before. Why are they having to learn it the hard way?

don’t over-optimize or expect to know ahead of time how site will scale

I take exception to this slide however. You should have some idea how to scale your app before you deploy it. It is irresponsible and cowboy to deploy and think "oh, we will fix it later". That is true no matter what you are doing. Don't give me the lines about being a start up and all that. It is just irresponsible to deploy something you know won't hold up. For what it is worth, I think these guys really had no clue. And that is because we, as an industry, did not make it known to them what they were up against.

How do we fix this?

Stop comparing stuff you don't understand

brianlmoon — Mon, 25 Jun 2012 23:09:28 -0500

I normally don't do this. When I see someone write a blog post I don't agree with, I often just dismiss it and go on. But, this particular one caught my attention. It was titled PHP vs Node.js: Yet Another Versus. The summary was:

Node.js = PHP + Apache + Memcached + Gearman - overhead

What the f**k? Are you kidding me? Clearly this person has NEVER used memcached or Gearman in a production environment that had any actual load.

Back in the day, when URLs and filesystems had a 1:1 mapping, it made perfect sense to have a web server separate from the language it is running. But, nowadays, any PHP app with attractive URLs running behind the Apache web server is going to need a .htaccess file, which tells the server a regular expression to check before serving up a file. Sound complex and awkward with unnecessary overhead? That’s because it is.

Node has a web server built in. Some people call this a bad thing, I call those people crazy. Having the server built in means that you don’t have the awkward .htaccess config thing going on. Every request is understood to go through the same process, without having to hunt through the filesystem and figure out which script to run.

He believes that PHP inside Apache requires a .htaccess file. Welcome to 1999. I have not used a .htaccess file since then. Anyone that cares at all about scaling Apache would disable .htaccess files. And as for running regexs, how does he propose you decide your code path in the controller of his Node.js code? Something somewhere has to decide what code is going to answer a given request. mod_rewrite is wire speed fast and compiled in C. Javascript nor PHP code could ever beat that.

The official website is quite ugly and outdated.

Really? You choose your tools based on that? I don't know what to say.

Since a PHP process starts, does some boilerplate work, performs the taks the user actually wants, and then dies, data is not persistent in memory. You can keep this data persistent using third party tools like Memcache or traditional database, but then there is the overhead of communicating with those external processes.

He clearly has no understanding of how memcached is supposed to be used. You don't put things in Memcached so you can use it on the next request on this server. You put things in memcached so you can use it in any request on any server in your server pool. If you just have one web server, you can write Perl CGI scripts. Performance and up time is not important to you. If you want to share things across requests in PHP, APC and XCache fill the need very well.

The number one bottleneck with web apps is not the time it takes to calculate CPU hungry operations, but rather network I/O. If you need to respond to a client request after making a database call and sending an email, you can perform the two actions and respond when both are complete.

This sums up the mythical magic of Node.js. People think just because your code is not "running" that somehow the server is not doing anything. The process does not get to go do other shit. No, that would be multi-threaded. And Node.js is not multi-threaded. It is single threaded. That means the process can only be doing one thing at a time. If you are waiting on a DB call, you are waiting. I don't care what world you think you live in. You are waiting on that DB call. How your code is structured is irrelevant to how computers actually work. The event driven nature that is Node.js is much like OOP. You are abstracting yourself from how computers really work. The further you get from that, the less you will be able to control the computer.

Node.js is a very new, unstable, untested platform. If you are going to be building a large corporate scale app with a long lifetime, Node.js is not a good solution. The Node API’s are changing almost daily, and large/longterm apps will need to be rewritten often.

So, if I plan on making a living, don't use Node.js. Got it. We finally agree on something.

Being so new, it doesn’t have a lot of baggage leftover from days of old. Having a server built in, the stack is a lot simpler, there are less points of failure, and there is more control over what you can do with HTTP responses (ever try overwriting the name of the web server using PHP?).

No, why would you? It is not the web server. Apache or nginx would be your web server.

Are you building some sort of daemon? Use Node.
Are you making a content website? Use PHP.
Do you want to share data between visitors? Use Node.
Are you a beginner looking to make a quick website? Use PHP.
Are you going to run a bunch of code in parallel? Use Node.
Are you writing software for clients to run on shared hosts? Use PHP.
Do you want push events from the server to the client using websockets? Use Node.
Does your team already know PHP? Use PHP.
Does your team already know frontend JavaScript? Node would be easier to learn.
Are you building a command line script? Both work.

Yes, of course you would not build a daemon in PHP. Do you plan to share that same data across servers? He already told us Node is not multi-threaded, so how can it run code in parallel? Websockets have a ton of their own pain to deal with that is not even related to Node vs. PHP. Things like proxies. If I was building a command line tool and wanted to use Javascript, I would just use V8.

Listen, I write code in PHP and JavaScript all day. I also use some Ruby, Lua and even dabble in C. I am not a language snob. Use what works for you. I do however take exception when people write about things they clearly have no idea about. He claims to have written a lot of PHP. He clearly has never deployed a lot of PHP in environments that matter. If you are building small sites that don't have a lot of traffic, you can use anything. If you are building massive sites that have to scale, any technology is going to require a full understanding of what it takes to scale it out. I leave you with this wisdom that I am reminded of by his blog post.

PHP Coding Standards

brianlmoon — Fri, 25 May 2012 19:27:56 -0500

Update: Matthew Weier O'Phinney, one of the core members of the group, has cleared up the naming history in the comments.

During the /dev/hell podcast at Tek12, someone asked the guys their opinion about PSR. I did not know what PSR was by that name. A quick search lead me to the Google Group named PHP Standards Working Group. I had vaguely remembered a consortium of frameworks, libraries and applications that were organizing to attempt to make their projects cooperate better. But, this did not sound like the same project. Another search and I found the PHP Framework Interoperability Group on Github. A bit more searching led me to a post where apparently the PHP FIG changed their name at some point citing people not knowing what FIG meant. But, this is not a history post. The group had done some work on setting a standard for auto loaders in PHP. This is a very good thing and much needed. That is a real thing that impacts real developers.

The person asking the question had asked about PSR1 and PSR2. These are the first two standards proposals in the group and they deal with coding standards. There were mixed feelings in the room about the proposals. I asked (being me, probably with very little tact) why in 2012 were a group of really smart people still discussing coding standards such as tabs vs. spaces. Because this is what immediately came to mind for me.

Source: http://xkcd.com/927/

There are already coding standards for PHP and any other language out there. Why does anyone need to make a new one? For Phorum we chose the PEAR standard (ok, with 2 minor modifications). On top of that, every one of the projects in this group already have coding standards. Why not just pick one of those? Are 10 projects that currently have their own standards going to actually all change to something else? I highly doubt it. My guess is that, at best, they will all end up with a modified version of the groups standards.

This reminds me a lot of Open Source licenses. There are tons of these things. And in the end, most (GPL has its issues I know) of the open source licenses represent the same idea. I suppose you could say that most of all of them fall into GPL like or BSD like. Anyhow, I quit worrying about having my own license years ago. I now just use a BSD style license that you can generate with several online BSD license generators.

When I voiced my concern about what is, in my opinion, a waste of very smart people's time, my good friend Cal Evans (He has bled in my car. So, I think he is my friend. And I hope he feels the same.) said that I was misunderstanding the point of the group. It was a group of projects that were collaborating to try and use similar standards and practices to make the PHP OSS community better. And that is exactly what I thought PHP FIG was. However, the group name is now "PHP Standards Working Group". That reminds me of the W3C HTML Working Group. And in my mind that means a group that is deciding the future of a technology. In addition the proposal being discussed is titled "PSR-1, a standard coding convention for PHP". If you pair that with the name of the group, it sounds very authoritative. And I don't think that is by accident. If I was heading up such an effort, I would hope that every PHP developer on the planet would follow it too. If you saw Terry Chay's keynote at the PHP Community Conference last year, he talked about frameworks and platforms. He pointed out that the reason people like Facebook were sharing their data center technology was in hopes that people would start using it and it would become common. Thus meaning the equipment they are custom building would be cheaper and people they hire would already be familiar with it. But, if the point of the group is *only* cooperation between lage OSS PHP projects, I wish they would pick a name that is more indicative of that. As it stands, when I landed on the page, my immediate assumption was that this groups intention was to dictate to the rest of the PHP world how to write their PHP code.

In the end, cooperation is good. And if these guys want to cooperate I say more power to them. I just hope they get into really good things soon. Like, can we talk about a maximum number of files, functions or classes used for any one single page execution? *That* would be valuable to the PHP community. I can deal with funny formatting. I can't deal with poorly performing code that his dragged down in abstracton and extension. Or how about things like *never* running queries inside loops that are reading results from another query. That would be a great thing to make examples of and show people the best practices. Tabs vs. spaces? That should have been solved 10 years ago. When in doubt, PHP code should do what the PHP core does. This is PHP we are talking about. Would it not make sense to have the people who write PHP writing code that is somewhat similar in style to those that make PHP? C and PHP syntax are very, very similar. So, why don't we all just refer to the PHP CODING_STANDARDS file when in doubt and not even worry with the little stuff that does not affect performance?

So, as of a few minutes ago, I have joined the group. If for no other reason, just to see what is discussed. Perhaps I should follow the advice I give people when they ask for features in my projects and do something about my issues and worries.

Living in the Prove It Culture

brianlmoon — Tue, 06 Mar 2012 22:45:18 -0600

Engineering cultures differ from shop to shop. I have been in the same culture for 13 years so I am not an expert on what all the different types are. Before that I was living in Dilbert world. The culture there was really weird. The ideas were never yours. It was always some need some way off person had. A DBA, a UI "expert" and some product manager would dictate what code you wrote. Creativity was stifled and met with resistance.

I then moved to the early (1998) days of the web. It was a start up environment. In the beginning there were just two of us writing code. So, we thought everything we did was awesome. Then we added some more guys. Lucky for us we mostly hired well. The good hires where type A personalities that had skills we didn't have. They challenged us and we challenged them. On top of that, we had a CEO who had been a computer hacker in his teens. So, he had just enough knowledge to challenge us as well. Over the years we kept hiring more and more people. We always asked in the interview if the person could take criticism and if they felt comfortable defending their ideas. We decided to always have a white board session. We would ask them questions and have them work it out on a white board or talk it out with us in a group setting. The point of this was not to see if they always knew the answer. The point was to see how they worked in that setting. Looking back, the hires that did not work out also did not excel in that phase of the interview. The ones that have worked out always questioned our methods in the interview. They did not belittle our methods or dismiss them. They just asked questions. They would ask if we had tried this or that. Even if we could quickly explain why our method was right for us, they still questioned it. They challenged us.

When dealing with people outside the engineering team, we subconsciously applied these same tactics. The philosophy came to be that if you came to us with an idea, you had to throw it up on the proverbial wall. We would then try to knock it down. If it stuck, it was probably a good idea. Some people could handle this and some could not. The ones that could not handle that did not always get their ideas pushed through. It may not mean they were bad ideas. And that is maybe the down side of this culture. But, it has worked pretty well for us.

We apply this to technology too. My first experience on Linux was with RedHat. The mail agent I cut my teeth on was qmail. I used djbdns. When Daniel Beckham, our now director of operations, came on, he had used sendmail and bind. He immediately challenged qmail. I went through some of the reasons I prefered it. He took more shots. In the end, he agreed that qmail was better than sendmail. However, his first DNS setup for us was bind. It took a few more years of bind hell for him to come around to djbdns.

When RedHat splintered into RedHat Enterprise and Fedora, we tried out Fedora on one server. We found it to be horribly unstable. It got the axe. We looked around for other distros. We found a not very well known distro that was known as the ricer distro of the Linux world called Gentoo. We installed it on one server to see what it was all about. I don't remember now whose idea it was. Probably not mine. We eventually found it to be the perfect distro for us. It let us compile our core tools like Apache, PHP and MySQL while at the same time using a package system. We never trusted RPMs for those things on RedHat. Sure, bringing a server online took longer but it was so worth it. Eventually we bought in and it is now the only distro in use here.

We have done this over and over and over. From the fact that we all use Macs now thanks to Daniel and his willingness to try it out at our CEO's prodding to things like memcached, Gearman, etc. We even keep evaluating the tools we already have. When we decided to write our own proxy we discounted everything we knew and evaluated all the options. In the end, Apache was known and good at handling web requests and PHP could do all we needed in a timely, sane manner. But, we looked at and tested everything we could think of. Apache/PHP had to prove itself again.

Now, you might think that a culture of skepticism like this would lead to new employees having a hard time getting any traction. Quite the opposite. Because we hire people that fit the culture, they can have a near immediate impact. We have a problem I want solved and a developer that has been here less than a year suggested that Hadoop may be a solution, but was not sure we would use it. I recently sent this in an email to the whole team in response to that.

The only thing that is *never* on the table is using a Windows server. If you can get me unique visitors for an arbitrary date range in milliseconds and it require Hadoop, go for it.

You see, we don't currently use Hadoop here. But, if that is what it takes to solve my problem and you can prove it and it will work, we will use it.

Recently we had a newish team member suggest we use a SAN for our development servers to use as a data store. Specifically he suggested we could use it to house our MySQL data for our development servers. We told him he was insane. SANs are magical boxes of pain. He kept pushing. He got Dell to come in and give us a test unit. Turns out it is amazing. We can have a hot copy of our production database on our dev slices in about 3 minutes. A full, complete copy of our production database in 3 minutes. Do you know how amazing that is? Had we not had the culture we do and had not hired the right person that was smart enough to pull it off and confident enough to fight for the solution, we would not have that. He has been here less than a year and has had a huge impact to our productivity. There is talk of using this in production too. I am still in the "prove it" mode on this. We will see.

I know you will ask how our dev db works, here you go:

1. Replicate production over VPN to home office

2. Write MySQL data on SAN

3. Stop replication, flush tables, snapshot FS

4. Copy snapshot to a new location

5. On second dev server, umount SAN, mount new snapshot

6. Restart MySQL all around

7. Talk in dev chat how bad ass that is

We had a similar thing happen with our phone system. We had hired a web developer that previously worked for a company that created custom Asterisk solutions. When our propietary PBX died, he stepped up and proved that Asterisk would work for us. Not a job for a web developer. But he was confident he could make it work. It now supports 3 offices and several home bound users world wide. He also had only been here a short time when that happened.

Perhaps it sounds like a contradiction. It may sound like we just hop on any bandwagon technology out there. But no. We still use MySQL. We are still on 5.0 in fact. It works. We are evaluating Percona 5.5 now. We tried MySQL 5.1. We found no advantage and the Gentoo package maintainer found it to be buggy. So, we did not switch. We still use Apache. It works. Damn well. We do use Apache with the worker MPM with PHP which is supposedly bad. But, it works great for us. But, we had to prove it would work. We ran a single node with worker for months before trusting it. Gearman was begrudgingly accepted. The idea of daemonized PHP code was not a comforting one. But once you write a worker and use it, you feel like a god. And then you see the power. Next thing you know, it is a core, mission critical part of your infrastructure. That is how it is with us now. In fact, Gearman has went from untrusted to the go to tech. When someone proposes a solution that does not involve Gearman, someone will ask if part of the problem can be solved using Gearman and not whatever idea they have. There is then a discussion about why it is or is not a good fit. Likewise, if you want to a build a daemon to listen on a port and answer requests, the question is "Why can't you just use Apache and a web service?" And it is a valid question. If you can solve your problem with a web service on already proven tech, why build something new?

This culture is not new. We are not unique. But, in a world of "brogramming" where "engineers" rage on code that is awesome before it is even working and people are said to be "killing it" all the time, I am glad I live in a world where I have to prove myself everyday. I am the most senior engineer on the team. And even still I get shot down. I often pitch an idea in dev chat and someone will shoot it down or point out an obvious flaw. Anyone, and I mean anyone, on the team can question my code, ideas or decisions and I will listen to them and consider their opinion. Heck, people outside the team can question me too. And regularly do. And that is fine. I don't mind the questions. I once wrote here that I like to be made to feel dumb. It is how I get smarter. I have met people that thought they were smarter than everyone else. They were annoying. I have interviewed them. It is hard to even get through those interviews.

Is it for everyone? Probably not. It works for us. And it has gotten us this far. You can't get comfortable though. If you do foster this type of culture, there is a risk of getting comfortable. If you start thinking you have solved all the hard problems, you will look up one day and realize that you are suffering. Keep pushing forward and questioning your past decisions. But before you adopt the latest and greatest new idea, prove that the decisions your team makes are the right ones at every step. Sometimes that will take a five minute discussion and sometimes it will take a month of testing. And other times, everyone in the room will look at something and think "Wow that is so obvious how did we not see it?" When it works, it is an awesome world to live in.