This month, my title may seem obvious to some people and will undoubted excite some impassioned rebuttals. It is not my intention to take sides in the ongoing Free vs. Open debate. In fact, I think that both terms are useful and they both express aspects this movement. The vital fact that I am trying to convey is how far the idea extends beyond software.
To that end, I am going to examine a project that is not software and in fact pre-dates computers as we know them today. I am going to discuss how the points that Eric Raymond made in The Cathedral and the Bazaar applied to a project communicated my snail mail, published books, and person-to-person. The same factors that make open/free software projects work were at work then in a very different context. And in fact, I believe that they shed light on the manner in which human knowledge pushes forward. It is my hope that this insight will prove useful in organizing a wider range of open source projects that have nothing to do with software.
I've dragged this on long enough. It is time for me to pull aside the curtain and introduce tonight's subject. I have chosen a subject so far from free software that it will throw the similarities into sharp relief. It is my contention that the creation and development of Esperanto constituted an independent discovery of the principles that drive the free/open source movement. I will be talking mostly about the history of the movement and very little about the structure of the language itself. The later simply isn't relevant to this discussion. I've provided some links at the end for anyone interested in learning more about Esperanto as a language. Scratching An Itch
Eric Raymond provides a series of lessons for us in The Cathedral and the Bazaar that explain in small steps why the bazaar model works so well. The first of these is:
1. Every good work of software starts by scratching a developer's personal itch.
I believe that this is true for much more than just software. Esperanto certainly started out this way. It's creator, Dr. Zamenhof, grew up in a part of what is now Poland. He saw a great deal of ethnic conflict, and felt that the communication barriers among groups speaking different languages contributed significantly to this. To solve this problem, which was very immediate to him, he set out to create a language which was easy to learn as a second language for communication between people of different mother tongues. That is, Esperanto was designed to be a second language for anyone.
Reuse
Eric's second lesson is:
2. Good programmers know what to write. Great ones know what to rewrite (and reuse).
Dr. Zamenhof was certainly aware of that. To achieve his goal of creating an easy second language, he had to make Esperanto enough like other languages that it could fulfil the same purposes. He drew on the common roots found in five different European languages for the roots of his words. This had two effects. First, it meant that many of the words were easily recognized by speakers of many other languages. Second, it saved him time in creating an initial core vocabulary.
That second point meant that a single individual could bootstrap the creation of an entire language. If you ever wonder about the size of a project that can be achieved by the open source bazaar model, compare anything you are interested in examining to the size of a living language.
Many other artificial languages have been created over the years with more radical departures from existing languages. None have ever succeeded to the extent that Esperanto has. Now would be as good a time as any for me to say why I think Esperanto is successful since most people assume that it failed. In fact, many people believe that it is dead.
First, Esperanto is not dead. It is spoken around the world. The only reliable estimate ever made places the number of Esperanto speakers at somewhere between 500 thousand and 2 million. That number rivals the number of speakers of many other languages. There are even a small number of native speakers. They are the children of Esperantists and they grow up at least bilingual.
Second, even if no one spoke it today, it achieved Zamenhof's goal for what the language would be. It is a complete written and spoken language that can be used in any aspect of human endeavour. It did not reach that point through its creator along. Many other people have contributed along the way.
Third, it is a living language that has survived its creator. Words continue to be added to its vocabulary. In fact, although there is an Academy that approves words as a part of the official vocabulary, their function is largely performed after the fact. They recognize what is already in common use.
Finally, when the subject of artificial languages is raised at all, the one that most people can name is Esperanto. Hardly anyone remembers the dozens of other attempts that have been made. Two of the most successful rival projects were Volapük and Ido, both of which are historical footnotes today.
Throw One Away
Eric's third lesson for us was:
3. ``Plan to throw one away; you will, anyhow.'' (Fred Brooks, ``The Mythical Man-Month'', Chapter 11)
Not much remains of the initial prototypes of Esperanto that Dr. Zamenhof created. Certainly someone familiar with Esperanto today can recognize them. But he created an initial version and tested it ruthlessly by using it for translations and to compose poetry. He changed it as he went until he fealt he had a version that was ready for wider distribution.
Eric's fourth lesson was:
4. If you have the right attitude, interesting problems will find you.
As I said, Zamenhof used the language. Once he published his initial version of it, under the pseudonym Doktoro Esperanto from which the language now takes its name, he wrote letters and articles in it. He encouraged others to translate into it. He considered this a way of testing for what new vocabulary it needed. It was a way of determining what the future features would be.
Eric's fifth lesson fits right in as well:
5. When you lose interest in a program, your last duty to it is to hand it off to a competent successor.
There is no direct correspondence here. Zamenhof never lost interest in his language. However, he took a bold step at the first Esperanto Conference. There was a splinter faction proposing changes to the language. Many of these changes keep resurfacing from other sources even today. However, they would have removed some useful features of the language, and they would have made the language as it existed prior to that an outmoded dialect. That was a large step.
Zamenhof could have bowed to the will of a vocal portion of the Esperanto community and introduced changes that could have been detrimental to the language. He could also have refused and probably alienated a sizeable fraction of the supporters that Esperanto had acquired at that point. Instead, he turned over the future development of his language to the Esperanto community itself. He effectively made his language open source.
The splinter group created a derivative language called Ido. Its name means offspring in Esperanto. It was intended as a replacement for Esperanto with the reforms that had been proposed. It did gather a number of followers. However, support for Esperanto continued to grow while the enthusiasm for Ido continued to fall off. My own belief is that there was little inhibition to forking new derivatives from Ido, whereas Esperanto continued to evolve more slowly, but its community accepted that there was a definition to the project. Incompatible changes were not made.
Community Participation
Eric's next lession is:
6. Treating your users as co-developers is your least-hassle route to rapid code improvement and effective debugging.
I've already said that Zamenhof encouraged Esperantists to use this new language for a variety of projects. His initial hook for getting people to learn was a form in his initial book. On it, readers promised to learn Esperanto when 10 million others had also promised to. He never received 10 million responses. And most of the people reading that wouldn't have waited.
Zamenhof realized that there was a minimum size that the Esperanto community had to attain to become self-sustaining. This leads to some interesting observations. He over-estimated the size that was required. There have never been 10 million Esperantists, yet the language celebrated its hundredth anniversary a decade and a half ago. Zamenhof himself died in 1917. His language continued to grow without him.
Frequent Releases
Eric's next piece of advice is much more important for software than for other types of projects:
7. Release early. Release often. And listen to your customers.
The need for releases stems from a type of project which can be said to be either working or not working. For other kinds of projects, there are other signs that the project is alive and well. Perhaps this should be worded, "Publish early. Publish often." Add to the content of your project on a regular basis. And your customers, your users, are an excellent source of suggestions. The correlary that this leads to is:
8. Given a large enough beta-tester and co-developer base, almost every problem will be characterized quickly and the fix obvious to someone.
Given enough people contributing to a project, they will find what it needs and the work will get done. Coming back to my running example, Esperanto has a computer dictionary of terminology that could not have existed at the time the language was created. This dictionary exists because the need existed, and an Esperantist with the skills and knowledge filled that need.
Making It Obvious
Eric observed:
9. Smart data structures and dumb code works a lot better than the other way around.
Again, Esperanto does provide an example of this. Esperanto's grammar is extremely flexible in many respects. In particular, the part of speech of a word is indicted by a tag at the end. Words ending in "o" are nouns. Words ending in "a" are adjectives. There are a handful of useful little words that ignore this pattern. I think of them as bare roots without an ending. Any root can be used as any part of speech if it makes sense. So from the word kato which means cat, you can derive the word kata which means catlike or feline. Also, compound words are permitted whenever the compound makes sense.
The direct result of this is that in Esperanto conversations, it is normal to hear words that can't be found in any Esperanto dictionary. That means that Esperantists simply use the building blocks (smart data structures) to construct the words they need. Rather than trying to preconstruct in a hard-coded way every word or expression that speakers and writers of the language would need, Esperanto contains the tools for making them. How very open source.
10. If you treat your beta-testers as if they're your most valuable resource, they will respond by becoming your most valuable resource.
11. The next best thing to having good ideas is recognizing good ideas from your users. Sometimes the latter is better.
These observations illustrate that at some point in the growth of a project, the community it has attracted is more important than the original creator. If this is not the case, it is clear that the project will only survive as long as its creator keeps it going. It is not yet self-sustaining.
Inside of Every Big Project Is a Small Project Struggling to Get Out
13. ``Perfection (in design) is achieved not when there is nothing more to add, but rather when there is nothing more to take away.''
Esperanto's grammar was originally stated as 16 rules. Now, it is clear that the grammar of a living language is a bit more complex than that. Nonetheless, it is small and simple. The idea was to allow the speakers and writers of the language as much freedom as possible while constraining the structure of the language enough to allow mutual intelligibility. This leads directly the next observation:
14. Any tool should be useful in the expected way, but a truly great tool lends itself to uses you never expected.
In something as big as a language, it is certain that people will express things that the creator of the language had not envisioned. That is the entire point. But it meant that the language had to be flexible enough to express them.
15. When writing gateway software of any kind, take pains to disturb the data stream as little as possible -- and *never* throw away information unless the recipient forces you to!
This illustrates one of the places where Esperant retained a piece of complexity that many people have wanted to remove over the years. Esperanto has an accusative case for the direct object of a verb. This is tagged by an "n" at the end. Thus, Man bites dog is Homo mordas hundon. Most Esperantists forget that "n" at least part of the time. What it accomplishes though is to make every possible ordering of those three words unambiguous! This is invaluable in translation, and to make speakers of languages which don't share the speaker's word order more comfortable.
Necessary Precondition
Eric goes on to talk about the necessary preconditions for bazaar style development. Some of those observations are quite telling:
It would be very hard to originate a project in bazaar mode.
Zamenhof created his initial grammar and dictionary alone. He published a langauge which was not complete, but was useful. You could ask for and given directions, write some poetry, introduce yourself and so forth. There is a necessary core to provide structure and attract an initial community.
You need to be able to present is a plausible promise.
This was precisely what Zamenhof did with his request that people promise to learn his language when there were 10 million people willing to do it. What he had said, in effect, was that he would undertake to find 10 million people to form a language community, if they would all agree to start using his language together.
This is a variant of the observation that the value of a network rises proportionally to the square of the number of nodes in it (people or computers). Languages and networks exist as a means of communication. Once a certain threshold was reached, people who had little interest in computers wanted e-mail addresses. When enough people will communicate with you through some channel, it is worth being accessible through that channel. And when enough people are interested in a project, it is worthwhile to sustain that project.
There is another kind of skill not normally associated with software development which I think is as important as design cleverness to bazaar projects -- and it may be more important. A bazaar project coordinator or leader must have good people andcommunications skills.
Within a corporation, people can be motivated to contribute to a project because they are paid to do it. Every open source project I've ever seen is sustained by volunteers. There are a few rare individuals who are paid to do it. The overwhelming majority are unpaid and doing something they love in their spare time. Think about how to excite that sort of passion in others.
Eric restates his first lesson in different terms:
18. To solve an interesting problem, start by finding a problem that is interesting to you.
Joseph Campbell found another way of stating this. He advised us all, "Follow your bliss." Simply put, find something you love and do it. Your passion is likely to be infectious. You will find others who share your interest and people will point you to opportunities to pursue it.
19: Provided the development coordinator has a medium at least as good as the Internet, and knows how to lead without coercion, many heads are inevitably better than one.
Just as Zamenhof overestimated the size of the community necessary to sustain a language project, I think Eric Raymond has overestimated the resources necessary to sustain a software project.