PostGIS 1.4 hot on the heels of PostgreSQL 8.4

PostgreSQL 8.4 has come out, and while I am a bit disappointed that PostGIS 1.4 has not come out for fear that we've missed a bit of the PostgreSQL 8.4 momentum, I am happy that we are nearing closer and just maybe we'll have it out by end next week. We now have a PostGIS 1.4RC1 http://postgis.refractions.net/download/postgis-1.4.0rc1.tar.gz tar ball as well as experimental binary builds of this for windows user's running PostgreSQL 8.3 http://postgis.refractions.net/download/windows/pg83/experimental/postgis/ or PostgreSQL 8.4 http://postgis.refractions.net/download/windows/pg84/experimental/postgis/. Please give both a try.

Working in the Cathedral Really?

As Paul duly noted in his blog entry Working in the Cathedral the model for PostGIS development is morphing, but I wouldn't call this morphing process one that is entirely toward the Cathedral model. Unlike the perceived Cathedral model, I would like to think we will have more frequent releases and beta releases, perhaps parallel experimental builds and most importantly, more fun. The main idea being making it much easier for mere mortals and fake mortals to taste test the cookies in the oven while they are cooking. By fake I mean unit tests, build bots, and computer generated people where the fear of destruction is removed. I feel this is the similar model PostgreSQL goes by or is trying to achieve.

Let me just say for one that I respect Eric Raymond a great deal, but I think that while most of his observations about Cathedral and Bazaar are accurate, some of his conclusions are a bit flawed. Hopefully I won't start too many fist fights from this statement. The main distinctions I see between the two and why Bazaar seems to many more efficient has nothing to do with the model at all. Its all about the people. If you get a group of people all passionate about the work they are doing it really doesn't matter which model you are using (Cathedral or Bazaar). Most likely you will succeed. A dispassionate mess of bazaar workers working together will create dysfunctional code that no one wants to use or work on. You'd be much better with a cathedral one where there is one passionate leader. I also think the success of the Bazaar and Open Source in general is probably over exaggerated. When no one is getting directly paid, its hard to quantify a loss. If 100,000 volunteers work on a project that is free and fun, most likely they aren't keeping a timesheet of how much time they are expending. In a cathedral model a lot of your utility is driven by monetary benefit, which means your time is either over exaggerated or at the very least precise. We know we lost $1,000,000 because we paid people $1,000,000 to build this thing.

First where did this cathedral model come from? it came from engineers and scientists. While I'm not a practicing engineer and I just have a degree in mechanical engineering, I still feel very close to my engineering roots. I'm not talking about that ill-defined term pompous people throw around "I've got a BS in Software Engineering", which to me is so badly abused not to be even worth talking about. I'm talking about real engineering; things like Mechanical Engineering, Civil Engineering, and Electrical Engineering. In engineering you do a lot of planning and have stringent requirements of what is acceptable and what is not. You have very seasoned engineers and technicians in the field who have amazing powers of intuition and new engineers and technicians straight out of school who are are honored to work with these people. You have a lot of smoke tests and things of that sort that stress test a system. This is because while building a bridge or plane might be fun, you can't afford to have bridges falling or planes crashing on people. One of those scandals and you are out of business. Yes some things also are not fun but need to be done. Having rules in place and stringent guidelines on what should and how should it be built is inherently not fun.

In engineering things like building virtual models are becoming more respected and more in vogue, because it allows you to test things out without killing real people or destroying real equipment or using real material in the process. It in essense allows you to be more care-free in designing thus making a better, cheaper and more fun project to work on.

In software the cathedral model was designed to get people to work on things that aren't fun. Even with "fun projects" you have a lot of stuff people consider as not fun and this will not change. In software the cathedral is a self-fulfilling prophecy. It managed to categorize tasks and projects such as building financial packages, documentation and testing as not fun such that the people who would have enjoyed these things decided -- software is not fun but pays the bills thus creating an institution of dispassionate people interested in the industry for the perceived money and scaring off passionate people on to other fulfilling things.

The bazaar model is an equally self-fulfilling prophecy. It encouraged a hacker mentality focused on only doing "fun stuff that scratches my own itch", that doesn't pay much but is fun. It too managed to give the vision of bleeding edge running code stuff is fun and documentation/testing is not important because people we care about will dig into the pie where they fit. These are the same people that whine about where did all the women go and why people aren't biting even though its free. Not to say that women are less worthy hackers, but that they are more apt to put a price on these harder to quantify things, thus making their utility function less compatible with open source or the bazaar model.

So what's my point. The point is that not only did these models define a concept of fun/not fun, they also defined the very meaning of fun not fun to people. No one wants to work on something they perceive of as not fun unless they are getting a ton of money for it. Glory is also a kind of fun because no one wants to work on something that they don't get any glory for or monetary income from. You probably don't want to fill your coffers with people just interested in the monetary value and dispassionate about your product, because they will find ways to maximize profit while minimize level of effort they need to do. In the same vein you don't want to grossly undervalue the things that are hard to quantify.

This gets to the point of a more obnoxious problem. This is the devaluation of testing and documentation. These two things are things where people agree its important, but can't put a price on. A cathedral model has an easier time putting a price on this, because part of the documenation is the specification itself that drives the model. In a bazaar we may have a "this will be a cool feature lets throw it in". The tragedy of this is that people put a price of 0 on something they can't price and don't see standing on its own in a grocery list and thus the work of testers and documenters is seemed as useless which makes the work not glamorous and no fun and in a pure volunteer model, you also get no money.

To demonstrate if you were to put a price on say introducing function X into a product - to figure out the value of it -- all you need to do is look at the price of that in a similar product. We need only look at say Oracle to figure out the value of x. However for a simple feature documentation, primarily user documentation, aside from its existence is not important to document it well and testing is a similar thing. How do you price the documentation of a complex feature X? The documentation/testing of X may be more valuable than the programming that went into X or it may not be needed at all. A pure model driven by volunteers has a bit of a harder time overcoming this problem than a paid project unless they can quantify the value of these or make the work more fun and glamorous. This means figuring out the importance of documentation on each item, glamorizing it, encouraging hackers who understand the function into documenting with cathedral like injections :).