A project model for the FreeBSD Project

Open source projects distinguish themselves from normal, commercial projects (also called closed source projects), in that the source code is distributed either alone or upon request, either for free or for a nominal fee to cover the distribution costs. With the internet, the distribution costs are often so amortised that most open source projects distribute both its binaries and source for free.

An important philosophy of the open source movement that comes from the Free Software Foundation and university projects such as BSD is that open source software should be self-hosting. This means that the operating system, the compilers and the utilities needed to make, compile and run the product should be open source. FreeBSD fits into this by being an open source operating system and providing most essential compilers and utilities, either their own or made by other open source projects.

[Hars et al, 2001] talks about "the open source development model", a term that in magazines and by many open source supporters is used in contrast to software process models. However, comparing these are meaningless. The open source development model refers to a significant detail in the project, that the source code will be available. The equivalent detail in commercial projects is that the project in some fashion contributes to the financial gain of the company that creates it. It makes as much sense to contrast software process models to the open source model as it makes to contrast them to the commercial model.

Other features common to open source projects is that they have internet based communities and the authors give the public the right to redistribute and modify the source code free of charge. [Jørgensen, 2001] discovered in his research that 43% of the developers had been paid to some extent for their involvement in their last project. In [Hars et al, 2001], 50% of the surveyed open source developers were paid to some extent. Although both surveys have a relatively small group of respondents, the fact that companies such as Sun, Ximian, IBM and Trolltech have contributed many projects to the open source community suggests that many developers are being paid for their time. The myth that participants do not get direct compensation for their work is therefore not necessarily correct.

However, a large part of the open source community does not get paid for their contribution. [Hars et al, 2001] has studies what motivates people to contribute to open source projects in general. They find that students and hobby programmers, the group that does not get paid for their involvement at all, the fun of having it as a hobby and expanding their skills, capabilities and knowledge are the most motivating factors. The gratification of such a hobby is explained by referring to Maslow's pyramid and the need for a stable, usually high evaluation of oneself. The increased human capital promises a future reward of being able to get a good job in a business which the participant is interested in.

According to [Raymond, 2000], all good programs come from scratching someone's itch. However, only a little more than a third of the respondents to [Hars et al, 2001] claimed that a personal need for the product was a main reason for participation.

The open source community prides itself on writing secure, well-reviewed software. [Raymond, 2000] quotes Linus Torvalds saying that "given enough eyes, all bugs are shallow". This is a paradigm in the community and a common belief that has many examples where problems have been discovered and fixed within hours of having been discovered. We will revisit this review in the verification techniques used by the FreeBSD Project to see if this holds true for this project.

As defined in our introduction, a software process model is a simplified description of the software development process. In being an abstraction, it can fit many software development processes whereas methodology usually only fits one project. Software process models help us both classify development efforts and serve as a basis for developing a methodology for new projects or projects whose methodologies are being re-engineered.

As outlined in our definition of a software process, it can consist of multiple sub-processes. A software process model is often presented as a structured set of sub-processes, so before we outline the most common software process models, a description of common sub-processes is in place.

There exist many software process models. The most used models, that again are parents to many more refined models, are the stepwise, waterfall, evolutionary, spiral, transform, incremental and agile models. This section will briefly summarise these

Just like two people living on an island can live together fine, a society of many individuals needs rules and regulations in order to function well. While two people can use a technique or method to do a job, [Cockburn, 1998] argues that for 20 people to do the job together, a methodology is needed. With an increased number of people involved, the need for communication becomes greater. Two people talking together use little time to synchronise their work, but 20 people who have to talk to everyone to be synchronised will spend all their time talking. A methodology greatly reduces the amount of communication needed to be done in order for the project to work satisfactory.

A methodology is the realization of a software process model within a project. While the model is just that, a model of how a project can be structured, the methodology is how this project is organised with all the details involved.

A methodology is to a software process model what an object is to a class in object-oriented programming. That is, it is an implementation of the model followed. [Cockburn, 1998] identifies nine elements that are usually described in methodologies. The elements [Cockburn, 1998] lists are:

Methodology weight refers to how much is specified in the methodology. For instance, if coding standards are very thorough, this adds to the methodology weight and allows for less tolerance for variations than if more coding standards issues were unstated. This would make it more open and allow for higher variation tolerance. More thorough standards are more precise, and precision thus adds to methodology weight. [Cockburn, 1998]

Projects can have different criticality level. The levels of criticality are classified by the result of product failure. The four levels that [Cockburn, 1998] and others refer to are (1) nuisance, (2) loss of a recoverable sum of money, (3) irreversible loss of a large sum of money and (4) loss of life. Cockburn's 4th principle is that the more critical the product made by the project, the heavier methodology is called for. The additional weight is added as a result of greater precision and less tolerance for variations.

The methodology scope is the range of roles, activities and project deliverables the methodology attempts to cover. The deliverables follow the project's life cycle. [Cockburn, 1998]

Figure 2.2. Scope of a Methodology

It is important not to confuse project activities with the core organisation's activities. While handling requests for vacation is a part of the core organisation's activities, planning how to handle people being unavailable is part of the project manager's activities.

[Osterweil, 1987] writes in his article that software processes are software. In a follow-up paper, [Turk et al, 2000] shows that software process models are software too. Software processes are based on a process model and are key parts of a methodology and a project model. Thus, methodologies and project models can be considered to be software.

Osterweil notes that software processes are dynamic and therefore much harder to comprehend and reason about than static process descriptions. This thesis will look at the processes in FreeBSD as a snapshot in time, thus looking at them as static so that they can be analysed. In reality, discussing the processes contributes to their change or reenforcement.

Because the project model is considered software, we can use a software process model to create a project model. We will discuss the software process model used in Section 3.4.

Since software processes and software process models are considered software, they are victims of the same quality criteria as software. One of these to be particularly careful about is "rotting". Rotting is referred to when code becomes obsolete because identified problems are not fixed or the context in which the software is used evolves while the software is not updated. The same is true for methodologies and project models. If they are out of date, the information becomes unreliable. When parts of the information is unreliable, the users of the documents do not know which parts they can rely upon or not, making them not use it at all. This is actually worse than not having a methodology or project model as the trust in future, updated and maintained models is at stake. Examples of changes that happen are: new roles are created, roles merged, responsibilities broadened or split, techniques changed and communication lines altered. Changes in organisational goals can influence project methodologies even more radically.

[Osterweil, 1987] argues for formalising processes through programming them in a language where human tasks are represented as functions or procedures. In my experience, many people who have programming jobs do open source development on their spare time because it allows them to do the work they find interesting without the bureaucracy and demands they find at work. The processes described in the project model will therefore be descriptive, accompanied by a flow model, rather than normative as a programmed function implies.

[Osterweil, 1997] talks about process evaluation, saying that because process representation is often unavailable, process testing has to review the artifacts produced by the process. Although there are some guidelines available, this is the first written down project model for the FreeBSD Project. To evaluate it properly would require a study comparing what it says happens to what actually happens. This is outside the scope of the project. We have decided, like [Osterweil, 1997] suggests, to investigate the artifacts produced.

Brooks' Law, "adding manpower to a late software project makes it later" has since it was first published in 1975 been treated in many studies, but with the republishing in 1995 and discussing these studies, the author finds it as true today as it was in 1975. [Brooks, 1995]

Brooks' Law is of particular interest to the open source community as it depends on new actors coming in and studying the source code. Netscape did a bold attempt to draw on the benefits of the open source community when they released the source code to their browser in January 1998. The company decided to use an open source model for the continued development of their browser. But from releasing a new browser many times a year, it would take more than two years until they launched their first Mozilla-based browser in November 2000. [Wilson, 2002] Projects such as OpenOffice, a continuation of Sun's StarOffice, have had a similar fate of becoming late.

[Brooks, 1995] also discusses the "Second System Effect" and argues that the second system built is likely to include all the frills and ideas that the architect did not have the time for during the implementation of the first system.

Many open source implementations stress design less as they are re-implementations of systems found in the closed source world. For example, Unix systems were well understood when the FreeBSD Project started and browsers well understood when the Mozilla project was initiated. While it is often not the second time the developers build such a system, they are very familiar with the products they are copying, and inclined to make "corrections" where they see them fit. This has led many open source products to look more complex and feature rich than the products they copy. Yet they use not their time for true innovation, and this allows the closed source world to gain a technical advantage, forcing the open source community to play catch-up.

The open source community often talks about quality in terms of execution speed, fault tolerance and features available. The articles that immediately seemed in line both with this and the history of operating system development were the "Fuzz" articles [Miller et al, 2000].

In the Fuzz articles, the authors made tools that went systematically through common tools in multiple operating systems and exposed them to random input data and checked if the programs either went into an endless loop or crashed. They did this first in 1990, then repeated the test in 1995 and 2000. In 2000 they found that they had crashed all tested Win32 programs, 15-43% of all tested programs on available, commercial UNIX systems and 9% of the tested programs on the open source UNIX system Linux.

They found that testing the developed product repeatedly also said something about the methodologies used. During the tests they supplied error reports and even bug fixes to the vendors of the operating systems. They were therefore surprised to see that many of the bugs reported in 1990 were still in place for the modern systems in 2000.

Although robustness is a good way of measuring the quality of a product, I found that investigating only this dimension and trying to use this dimension to say something about the quality of the project model was not persuading. It may be that FreeBSD is very robust, but that ignores important issues as implementing new features, having regular releases etc.

GQM was first used for a set of projects in the NASA Goddard Space Flight Center environment and a widely used technique today, for instance used in evaluating projects with the Capability Maturity Model (CMM) for finding conformance with its key process areas [Basili et.al, 1994].

GQM assumes that measurement is only useful if it is backed by a goal, and is thus a top-down approach to measuring quality. It is divided into a conceptual level, operational level and quantitative level.

The "conceptual level" states the goal. The goal is defined for an object, that can be a product, process or resource. An example of a goal for a lamp is that we should be able to behave indoors as if we were outdoors in daylight.

At the "operational level", questions are used to characterise the way the achievement of the goal is going to be assessed. A question characterises the quality of an object from a viewpoint, and it can therefore be favourable to have multiple questions to a goal. An example of a question to our example goal is whether people can read a book as fast indoors with the lamp turned on as they can outdoors.

At the "quantitative level" we find metrics for our questions. Metrics can be measured either objectively or subjectively, and multiple metrics can be associated with one question. An example of an objective metric to our question is measuring how many words per minute were read by the test subject indoors and outdoors. An example of a subjective metric is asking the test subjects to fill out a form about how they found reading indoors with the lamp turned on as opposed to reading outdoors.

This is the approach chosen to measure quality in this thesis as it allows the investigation of many dimensions of the FreeBSD Project. In order to use it, we must be careful to have reasons for the goals selected as they must not be simply something selected out of the interest of the author, but be goals of practical value to the project.

There has not been written many articles on the development of the FreeBSD Project when it comes to software engineering. As the largest source of how things have been done in the FreeBSD project are its mailing lists, these have been studied to understand how processes in the project have worked. As noted in section 8.6.4, FreeBSD has 68 mailing lists of which 62 are both publicly available and archived. Mailing lists has been the primary form of communication since the projects' inception.

One particular feature of the FreeBSD mailing lists is that they are archived so that they serve well as research material. But this trait can deter people from writing. In order to allow discussions that are not wanted to be archived, a closed mailing list for committers has been created in which they can discuss anything openly without the outside world gaining access to it. It is considered bad manners to share opinions voiced in this mailing list in public forums without public consent [Lambert, 2002].

However, the amount of correspondence on these lists is too much to possibly study within the time constraint of this research. It has therefore been necessary to narrow down the scope of the search to periods of time on a low number of lists where it is most likely that there will be found evidence of processes being in use. I have selected the lists through recommendations by my interviewees.

The cutdown has been made on the basis of interviews with senior developers. When searching for a topic, the lists they recommended have been queried for. Then a period of time of at least five days before and after the hit have been read as well as the thread the corresponding mail was a part of. When researching things with a known date, the same approach has been taken. The result is that many snapshots in time on selected lists have been chosen

The mailing lists that have been studied are:

In the appendix is a set of mail threads that I have found of particular interest. As threads of mail can change topic as the thread evolves, the list has been ordered such that the first mail on the topic I have found of interest has been listed.

I have not been able to find any mailing list archives on the development of BSD at Berkeley, but I have deemed this unnecessary because of the coverage of the development process by Kirk McKusick's many articles on the topic.

As discussed in Section 2.3.4, project models can be considered software, and we can thus use the same tools we would use to develop software to develop a project model. In order to build a project model for the FreeBSD project, a software process model is needed.

I have chosen an evolutionary model as shown in the figure in the beginning of this chapter. The scope and purpose has been defined in the first chapter of this thesis. The top of the figure is research that leads to either notes or argumentation, and this leads back to further research. After my research has been completed, I have restructured my notes into the project model and supporting chapters.

In the development of the methodology, the current activities will be undertaken: analysis, design, implementation, verification and maintenance. Analysis will draw upon the research done to say one activity is done in one particular setting. Design will based on the analysis generalise how similar problems are being solved. Implementation will integrate the design into the methodology saying how similar problems should be solved. The methodology will be verified by comparing it to the evidence found in the analysis and discussing the methodology with active committers of the FreeBSD project. Maintenance will be the activity of keeping the project and methodology in sync once the project has taken the methodology into use. As an evolutionary model, the maintenance part will include analysis, design, implementation and verification. All the four other stages will produce output that will affect the other stages so that each process is a continuous, ongoing process.

From Section 1.1, we find here are three main requirements for the project model:

For the project model to be of use to the project, it should be founded upon details of what the current state of the project is so that it can be used for setting goals. It should also provide easy access to the people and groups listed. It should be easy to read and should not require significant knowledge in order to understand. It should be a document new members of the project can read in order to familiarise themselves with the project, and a document project members can refer to when working on other domains than they usually work with.

With high academic quality, we will mean that it should contain clear, precise definitions of words that are not necessarily precise and clear. It should be backed up with references and be correct about the facts. It should follow guidelines for project models to ensure it contains what project workers expect to find in a project model.

In order to set goals, it is important to know the current project state. This is requires gathering data to generate descriptive statistics.

In order to make access easy, email addresses and names of the groups and people mentioned will be listed.

While the project model should be correct factually and have references, it should not be cluttered with references. This thesis will support the project model by providing more in-depth observations and references to literature.

As discussed by [Hohmann, 1997], every process has an outcome. In identifying the processes in the project, it is important to see the process' purpose, the activities, the associated roles and what the outcome of the process is.

The implementation of the project model is available in Chapter 8.

In order to verify the project model, it is sent to the interviewees and senior project managers for proof-reading and commenting with the sanction of the FreeBSD Documentation Project. After being revised, it is sent to the project members for further comments. Finally it is incorporated into FreeBSDs documentation.

The product specific goals come as a response to the users wishes. The users in the project can communicate directly to the developers through mailing lists. Product specific goals have been selected by going through the mailing list archives and noting requests that have often been voiced.

The project specific goals are administrative goals that emphasis on developer productivity rather than having to have developers perform administrative tasks. Most goals refer to managing commit privileges and distributing written code. When these goals fail, developers have to spend time making workarounds for having code added and find alternative means of distributing the updated code. These goals have been selected on the basis of interviews with developers.

Up until now, FreeBSD has been very little process oriented. It has good communications with its users through mailing lists, but mailing lists are not well suited for bug management. This is confirmed by [Swanson, 1976] who stresses the importance of a maintenance database. To handle this, they have had to create problem reporting routines for problems identified by users. The following goal is built upon the documents describing these, [FreeBSD, 2002C] [FreeBSD, 2002D]

The "quantitative" research has consisted of gathering data from CVS logs, Problem Reports and classifying news announcements. This information has been easily available through a net browser and the CVS tool, and have been very consistent. The only major inconsistencies have been when some of the machines responsible for handling the logs and reports have had an incorrect time. This has not been discovered for many entries and has not caused significant errors.

The "qualitative" research has consisted of email interviews, reading mailing list correspondence, internet chat and regular interviews.

I have found it difficult to get in contact with people in projects I did not know well. [Mann et. al, 2000: 28] have found that many regard contact with outsiders as threatening and time consuming. I have also tried enlisting altruistic support that is perceived common in the open source and academic community [Hars et al, 2001]. The success of a good response has been perceived as dependant on how much the person's interests stand to gain from the output of my research.

I have had problems with people being gone from discussions for long periods of time because of other issues in life such as vacations hindering them in replying.

It has been difficult to get people to use their time to answer questions. Also, many of the interviewees that did respond would stop responding to follow-up questions. Because of the distance between people and the low resources available for transportation and phone, I have only been able to interview two committers by phone and in person. The interviews have been interesting and the interviewees answered most of my questions.

Monitoring mailing lists has proved to be a very time-consuming task, but has helped provide a more coloured view of the project and its processes than I got from interviews. The two have thus been complementary. Because of time constraints and the vast amount of both daily incoming and archived messages, I have only been able to read snapshots in time from selected lists.

In my research, I have only been in contact with people that have access to the net and the time to work on the project and engage in discussions. I have not been in contact with people that have been marginalised from the project because of any of these issues. I have thus not found how the project model includes or excludes people that cannot have regular access to the net.

I have drawn upon [Mann et. al, 2000] for handling consent. When interviewing people, I always started the interview by presenting what I was researching and clearly stating that I was asking them questions for my research. After my research was completed, I emailed all my interviewees an email with the notes I'd taken and asked if they minded anything in my notes so that I could include them with my research data. I have thus been explicit about gathering concent from my interviewees. However, I do not have signatures or such, except for the occational PGP-signed message. I have not explicitly sought consent when siting public mailing lists as people posting on such lists are aware that they are saying this in a public fora.

For the three last months of the thesis, the project model, most of the data and my thesis has been available to a larger and larger part of the community. The project model was presented to the community at large 1,5 months before the thesis was submitted and the thesis was presented two weeks before it was submitted. I have thus made sure to give people access to the data I have gathered about them.

During my research I have obeyed general nettiquette as described in [Mann et. al, 2000: 59-62] and the rules regarding netiquette of the Department of Informatics at the University of Oslo.

The FreeBSD documentation project was started January 6th, 1995, on an initiative of Jordan K. Hubbard, one of the FreeBSD project's founders [Hubbard, 1995A]. The main motivation for its inception was to gain a selling point: FreeBSD had so far been used mostly by its developers, while Linux, at that time a new contender, was gaining a lot of new users and it was felt that better documentation was what helped them.

The project was set up with a manager, four team leads and regular members [Hubbard, 1995B]. At the present, the project has grown to an estimate of 32 committers [Interview with Nik Clayton], which is roughly 10% of the FreeBSD committers. However, very few committers contribute to the documentation project alone. ^[2] Many of the contributers often contribute to other parts of the FreeBSD project as well. The members are structured in one project leader, several teams responsible for translations into different languages, and a documentation release engineering team [FreeBSD, 2003B].

The documentation release engineering team consists of all the team leaders as well as a member from FreeBSDs release engineering team. Its purpose is to have up to date documentation for each FreeBSD branch at release time and make sure documentation is correct for the different branches. ^[3]

More on what the Documentation project does can be found in section 8.7.2.

Bluetooth is a standard for how wireless devices should form an ad hoc local network, including devices up to ten meters away. The standard details a stack is similar to the ISO network stacks.

The implementation of the Bluetooth stack started in a classic open source way: "one day I just thought that it would be nice thing to do, and since I had spare time I started to work on it" [Interview with Maksim Yevmenkin]. The project was founded upon the Bluetooth Specification Book version 1.1 and was developed for FreeBSD-CURRENT.

The first step was to go through the specification and try to make sense of it all and get the big picture. This step can be seen as requirements analysis. This led to a primary TODO-list.

The next step was to identify the different parts of the system and see how they should interface each other and FreeBSD. This step included drawing a layer graph of the stack and go into each node and find out what needed to interface it and how it should be interfaced. Answering these questions gave the overall interface and algorithm design. The specification also gave many guidelines as to how the implementation should be done to obtain standards conformance. Based on these guidelines and the answers found, the TODO-list was refined as a list of needed functionality.

The implementation consisted of many short cycles that took an item from a TODO list and implemented it. The items were prioritised in the following order:

In order to verify the implementation, testing was used and at different stages in the process announcements were made giving people access to the code for testing and review.

For testing, a Bluetooth device simulator was created that would interface the code. After obtaining Bluetooth-enabled hardware, the hardware was used for ad hoc testing. "Sometimes i do some crazy things like pulling out the PC-CARD while [the] stack is active, connecting both devices to the same laptop and have them ping each other for 24 hours etc. to see what will happen" [Interview with Maksim Yevmenkin].

The development went through six months of snapshots. Very few comments were made on the actual code itself. People with Bluetooth devices would download the code and try it out and report back any errors. Interoperability was thus tested mainly through random combinations of devices the developer had and what devices the people trying out the code had.

During the development, the design changed. But as it was developed and designed by one person, this was not a problem and even expected. The design documents, consisting of pictures, explanatory text and proof of concept, were never released for review nor added to the project when the Bluetooth implementation was committed to FreeBSD-CURRENT. The tests were also not committed as they were deemed to be too special-case for the developer and of little or no use to anyone else.

Being committed, the Bluetooth project is now a part of FreeBSD and updates are committed directly into FreeBSD's repository. It has thus ceased to be a subproject.

OpenPAM is a project that implements Pluggable Authentication Modules (PAM). It replaces LinuxPAM that was used up to FreeBSD version 4.8. Much like Kerberos, it allows for a single signin to cooperating computers in a network. However, rather than to dictate a technology that should be used, it allows for multiple authentication modules to interoperate.

PAM originates from Sun's Solaris, and exists in a standards draft from the Open Group called "X/Open Single Sign-On Service" or XSSO [Open Group, 1997].

The preliminary implementation of PAM into FreeBSD was the integration of LinuxPAM by Juniper Networks. Juniper donated its work to the FreeBSD project so that they would not have to maintain the code themselves. The continued integration of PAM was started on contract from Enitel A/S. Further funding was sought through Network Associates Laboratories (NAI) that had received $1.2 million contract from the US Navy's Space and Warfare Systems Command to develop security extentions for FreeBSD.

After some time of trying to fix bugs in LinuxPAM, it was deemed better by its maintainer to reimplement PAM. Leading up to this decision, LinuxPAM had undergone much testing with a set of headers based on the standard. These headers were now used to generate a code skeleton and was implemented during the next month.

The requirements for OpenPAM were following the XSSO standard, ensuring cross-platform compatibility (especially focusing on Solaris and Linux) and a few extentions deemed useful for many applications. During implementation, further requirements to extentions were discovered, and after the initial integration with FreeBSD, further feature suggestions were made by developers in the FreeBSD community.

There was done very little design. Solaris' implementation was studied carefully where the XSSO standard was unclear.

During implementation, most parts were either basic enough to be trivial to write or consisted of multiple basic parts put together. Much debug code was added to make bug tracking easy.

Verification consisted of writing tests, implementing the framework in other applications and code review. For testing, test modules and a test program were written. After this, OpenPAM was integrated with existing userland utilities that require authentication. During this implementation, further debugging code was added to make bug tracking simpler.

Review was sought during the initial three releases. The mailing lists "freebsd-audit" and "freebsd-arch" were noticed about the implementation and the project web site, but very little feedback was given.

After two weeks of availability and requests for review, OpenPAM was integrated with FreeBSD-CURRENT. After ten more days of availability in the project, it replaced LinuxPAM after a notice on the freebsd-current mailing list.

When the switch-over from LinuxPAM to OpenPAM had happened, people started responding. The responses, however, were very little constructive with regards to making the code better. Only a handful patches have been submitted since the code was integrated February 23, 2002, until the time of writing.

Although OpenPAM has been committed into FreeBSD, it still operates as an independent subproject to facilitate integration with other operating systems. When it is updated, the updates are committed to FreeBSD.

Developers have their own development methods. [Jørgensen, 2001] has a figure [Section 6.1] describing how each individuals own work is integrated into the project. The main part of this figure is "Code". The method each developer has is for the coding or the making of their contribution. During my interviews, not one developer has described the way they work with coding the same as the next developer. The development models they used ranged from the traditional waterfall model to agile methods.

As each developer up until the delivered code has his own method, there is no standard way of gathering requirements and analysing them and prioritising among them. For the Bluetooth stack implementation discussed in Section 4.6.2, the primary requirements were to implement the "Bluetooth Specification Book" version 1.1, and a TODO list was kept of bugs, missing functionality and suggested features. For the similar projects, such as the implementation of smart card support, standard documents (ISO/IEC 7816) were also the primary source for requirements. All the people I have interviewed have kept TODO lists.

Maintenance work does also gather requirements. As noted by [Jørgensen, 2001], FreeBSD is maintenance-centric. Requirements are gathered by the maintainers through feedback the maintainers get through Problem Reports. These Problem Reports, held in the GNATS maintenance database, serve as a TODO-list together with the maintainers wishes and ideas.

As noted in the requirements section, design notes are not kept systematically. There are no design standards nor requirements to how design should be done. Therefore, the outputs of the design processes are lost.

As noted by many authors on open source, many of the successful open source projects have reimplemented something that has been developed before. Much of the design is already done and tacitly understood by the people involved in the project. When implementing standards, much design work has also already been done. Some standards, such as the Bluetooth specification, even include implementation guidelines. Many people in the FreeBSD Project and the open source world view such standards and previous implementations as design enough.

The only recognised design documents is [McKusick et.al, 1997]. People wanting an overview of the kernel, either for understanding it out of interest or for contributing to its development are referred to this book. The reason for this is that the design is regarded to be relatively stable after more than 20 years of operating system development. FreeBSD has very much stuck to the file structure and interfaces of BSD 4.4, although much of the code has been altered or replaced and many bits have been added. This book provides an introduction and understanding of how the kernel is organised and what the various parts do and how they interface one-another. An update of this book is in the works that will describe FreeBSD 5.0, but for the first ten years of the project, this book has provided the stability of design.

Developers who are starting to work in a new area of the code should be careful to study the commit logs of the files in that area. The commit logs will often explain why a certain design has been chosen should there be multiple options or choices made that are different from what would be expected. This ensures that developers do not make the same mistakes as have been done before or reimplement disputed or unwanted code.

Open source projects have in particular been credited for being well verified, in particular through review. The prime example of this is Eric Raymond's quote of Linus Torvalds: "Given enough eyeballs, all bugs are shallow". [Raymond, 2000] The FreeBSD Project uses testing and review to verify its product.

An important aspect of maintenance is removing old code. This is done in the FreeBSD project by mailing a message to the developer community putting it out for vote.

For instance, in the development of FreeBSD 5.0, Ruslan Ermilov suggested renaming the library libexpat to libbsdxml because the XML code, originally based upon expat, deviated so much from the original code that leaving it with the original name, which originally was done to credit the expat library, was misleading. [Ermilov, 2003]

Although deprecated code should stay with FreeBSD for at least one major version after the decision to deprecate it has been made, there are exceptions to this rule. One example of this is Poul-Henning Kamp's argument [Kamp, 2003] for deprecating the previous systems for the new disk handling system GEOM and device handling system DEVFS. [FreeBSD, 2001: 9.4]

Major releases (.0 releases) are releases of the development branch. They are likely to include radical technological changes whereas minor releases (.X releases) are not. Jordan K. Hubbard wrote that "a ".0 release" translates to [...] New, interesting, sort of usable, buggy in the extreme." [Hubbard, 1994] Therefore it is necessary for more testing and bug fixing before releasing a major release than a minor release.

Examples of such new radical changes is that version 2.0 was based on a newer codebase from Berkeley than 1.0 was, 3.0 was in transition of changing the binary format, 4.0 included support for IPv6 networking along with security features such as OpenSSL and OpenSSH and the possibility to jail applications, and 5.0 has a new disk handling system and has a brand new SMP handling system. 6.0 is planned to among other features include kernel thread handling for userland threads.

Minor releases are usually releases of the production branch (-STABLE). Sometimes they may be of the development release if the development branch has been through a stabilisation process following up to a major release and this is the subsequent release.

A "release candidate" is a version aimed at early adopters that would like to try out a new version. It has been through stabilisation, but it is not yet tested enough to be considered an actual release. Even though features that are integrated into -STABLE have been through much testing in -CURRENT, the production branch (-STABLE) should not be considered an ongoing release candidate. Changes to release candidates require the explicit approval of the Release Engineering team, and release candidates are usedfor much more widespread testing. Also, release candidates are used to see that bugs reported in one release candidate are gone in the next.

An example of the benefits of having separate release candidates is that before an upcoming release, many committers will merge code from the -CURRENT to the -STABLE branch. This is possibly the most unstable time for the -STABLE branch and thus a bad time to upgrade systems in a production environment. Those following the -STABLE branch are assumed to understand this and should be careful when release engineering periods begin to evaluate if they really would like to upgrade their systems at that time.

A security branch is a branch that is identical with a release, but has security patches applied to it. Each node along the branch is called a patchlevel. People following a security branch are ensured that nothing else will change in their system, so that no programs will change behaviour. This is particularly useful in a production environment as server needs a higher degree of predictability yet must be updated with the latest security patches at all times.

As mentioned, before a code freeze for an upcoming release starts, many committers will merge those parts of their code they deem stable from the -CURRENT to the -STABLE branch. Because of the inconsistencies that can occur during such a mass-merge, this is not a recommended period of time to upgrade the system. However, not upgrading may be unacceptable as some changes such as security fixes are necessary to have installed as soon as possible in production environments. Security branches have solved this problem, and administrators can track a security branch rather than the -STABLE branch.

As mentioned in Section 3.5.1.3, new releases should be made regularly. Release history also makes it easier to see whether the development model and the release engineering process works.

The following table shows the release dates for every version and how long time it was since the previous release, measured in days.

Table 6.2 shows the releases of FreeBSD since its first release in November 1993 to and including its latest release before April 1st, being in January 2003. It shows how many days have passed between each release as a measure to how regular the releases are and provides the version number of the release.

For the project lifetime, this gives an average of 96,2 days between every release with a standard deviation of 62,9 days. If we look at minor releases, not including security releases since January 1st, 2000, a release is made every 107,4 days, with a standard deviation of 41,7 days.

Figure 6.3. Days between releases

Figure 6.3 shows the amount of days passed between each release of FreeBSD in the time period November 1993 to January 2003. This shows how regular the releases are. The closer the bars are to the neighbouring bars, the more regular are the releases.

Because the only condition for when a FreeBSD release is made is the time, it would seem that a FreeBSD release is no more than a snapshot of the evolving project in time. However, because of the release engineering process used, a version is indeed a well tested and trustworthy product. The user community has made it clear that it does not tolerate reduced security as a result of rushing to meet a self-imposed release deadline. A result of this was that the release of FreeBSD 4.6.2 followed shortly after 4.6 to solve numerous issues.

Putting the evolutionary model, the new feature development model and the version tree together, we get the following model. At the top we have the new feature development that is followed by the integration model proposed by [Jørgensen, 2001]. This model is used for maintenance changes as well. The third last step in Jørgensen's model is the commit living in the -CURRENT branch. Then it is merged from current into the -STABLE branch. A release along the -STABLE branch marks the beginning of a security branch. Finally, older versions are not supported any more.

Figure 6.4. The overall development model

In conclusion, the FreeBSD Project provides multiple versions of FreeBSD intended for audiences with different priorities between the newest features and rock solid stability and security. This means that the newest minor release or update along a security branch is already old in terms of being well tested when it is released.

We have also seen that new versions of FreeBSD are released on a regular basis and that the release model fits in with the overall development model.

An "activity" is an element of work performed during the course of a project [PMI, 2000]. It has an output and leads towards an outcome. Such an output can either be an input to another activity or a part of the process' delivery.

A "process" is a series of activities that lead towards a particular outcome. A process can consist of one or more sub-processes. An example of a process is software design.

A "hat" is synonymous with role. A hat has certain responsibilities in a process and for the process outcome. The hat executes activities. It is well defined what issues the hat should be contacted about by the project members and people outside the project.

An "outcome" is the final output of the process. This is synonymous with deliverable, that is defined as "any measurable, tangible, verifiable outcome, result or item that must be produced to complete a project or part of a project. Often used more narrowly in reference to an external deliverable, which is a deliverable that is subject to approval by the project sponsor or customer" by [PMI, 2000]. Examples of outcomes are a piece of software, a decision made or a report written.

When saying "FreeBSD" we will mean the BSD derivative UNIX-like operating system FreeBSD, whereas when saying "the FreeBSD Project" we will mean the project organisation.

There is no defined model for how people write code in FreeBSD. However, Niels Jørgenssen has suggested a model of how written code is integrated into the project.

Figure 8.3. Jørgenssen's model for change integration

The "development release" is the FreeBSD-CURRENT ("-CURRENT") branch and the "production release" is the FreeBSD-STABLE branch ("-STABLE") [Jørgensen, 2001].

This is a model for one change, and shows that after coding, developers seek community review and try integrating it with their own systems. After integrating the change into the development release, called FreeBSD-CURRENT, it is tested by many users and developers in the FreeBSD community. After it has gone through enough testing, it is merged into the production release, called FreeBSD-STABLE. Unless each stage is finished successfully, the developer needs to go back and make modifications in the code and restart the process. To integrate a change with either -CURRENT or -STABLE is called making a commit.

Jørgensen found that most FreeBSD developers work individually, meaning that this model is used in parallel by many developers on the different ongoing development efforts. A developer can also be working on multiple changes, so that while he is waiting for review or people to test one or more of his changes, he may be writing another change.

As each commit represents an increment, this is a massively incremental model. The commits are in fact so frequent that during one year ^[6] , 132148 commits were made, making a daily average of 360 commits.

Within the "code" bracket in Jørgensen's figure, each programmer has his own working style and follows his own development models. The bracket could very well have been called "development" as it includes requirements gathering and analysis, system and detailed design, implementation and verification. However, the only output from these stages is the source code or system documentation.

From a stepwise model's perspective (such as the waterfall model), the other brackets can be seen as further verification and system integration. This system integration is also important to see if a change is accepted by the community. Up until the code is committed, the developer is free to choose how much to communicate about it to the rest of the project. In order for -CURRENT to work as a buffer (so that bright ideas that had some undiscovered drawbacks can be backed out) the minimum time a commit should be in -CURRENT before merging it to -STABLE is 3 days. Such a merge is referred to as an MFC (Merge From Current).

It is important to notice the word "change". Most commits do not contain radical new features, but are maintenance updates.

The only exceptions from this model are security fixes and changes to features that are deprecated in the -CURRENT branch. In these cases, changes can be committed directly to the -STABLE branch.

In addition to many people working on the project, there are many related projects to the FreeBSD Project. These are either projects developing brand new features, sub-projects or projects whose outcome is incorporated into FreeBSD ^[7]. These projects fit into the FreeBSD Project just like regular development efforts: they produce code that is integrated with the FreeBSD Project. However, some of them (like Ports and Documentation) have the privilege of being applicable to both branches or commit directly to both -CURRENT and -STABLE.

There is no standards to how design should be done, nor is design collected in a centralised repository. The main design is that of 4.4BSD. ^[8] As design is a part of the "Code" bracket in Jørgenssen's model, it is up to every developer or sub-project how this should be done. Even if the design should be stored in a central repository, the output from the design stages would be of limited use as the differences of methodologies would make them poorly if at all interoperable. For the overall design of the project, the project relies on the sub-projects to negotiate fit interfaces between each other rather than to dictate interfacing.

The releases of FreeBSD is best illustrated by a tree with many branches where each major branch represents a major version. Minor versions are represented by branches of the major branches.

In the following release tree, arrows that follow one-another in a particular direction represent a branch. Boxes with full lines and diamonds represent official releases. Boxes with dotted lines represent the development branch at that time. Security branches are represented by ovals. Diamonds differ from boxes in that they represent a fork, meaning a place where a branch splits into two branches where one of the branches becomes a sub-branch. For example, at 4.0-RELEASE the 4.0-CURRENT branch split into 4-STABLE and 5.0-CURRENT. At 4.5-RELEASE, the branch forked off a security release called RELENG_4_5.

Figure 8.4. The FreeBSD release tree

The latest -CURRENT version is always referred to as -CURRENT, while the latest -STABLE release is always referred to as -STABLE. In this figure, -STABLE refers to 4-STABLE while -CURRENT refers to 5.0-CURRENT following 5.0-RELEASE. [FreeBSD, 2002E]

A "major release" is always made from the -CURRENT branch. However, the -CURRENT branch does not need to fork at that point in time, but can focus on stabilising. An example of this is that following 3.0-RELEASE, 3.1-RELEASE was also a continuation of the -CURRENT-branch, and -CURRENT did not become a true development branch until this version was released and the 3-STABLE branch was forked. When -CURRENT returns to becoming a development branch, it can only be followed by a major release. 5-STABLE is predicted to be forked off 5.0-CURRENT at around 5.1-RELEASE or 5.2-RELEASE. It is not until 5-STABLE is forked that the development branch will be branded 6.0-CURRENT.

A "minor release" is made from the -CURRENT branch following a major release, or from the -STABLE branch.

Following and including, 4.3-RELEASE^[9], when a minor release has been made, it becomes a "security branch". This is meant for organisations that do not want to follow the -STABLE branch and the potential new/changed features it offers, but instead require an absolutely stable environment, only updating to implement security updates. ^[10]

Each update to a security branch is called a "patchlevel". For every security enhancement that is done, the patchlevel number is increased, making it easy for people tracking the branch to see what security enhancements they have implemented. In cases where there have been especially serious security flaws, an entire new release can be made from a security branch. An example of this is 4.6.2-RELEASE.

To summarise, the development model of FreeBSD can be seen as the following tree:

Figure 8.5. The overall development model

The tree of the FreeBSD development with ongoing development efforts and continuous integration.

The tree symbolises the release versions with major versions spawning new main branches and minor versions being versions of the main branch. The top branch is the -CURRENT branch where all new development is integrated, and the -STABLE branch is the branch directly below it.

Clouds of development efforts hang over the project where developers use the development models they see fit. The product of their work is then integrated into -CURRENT where it undergoes parallel debugging and is finally merged from -CURRENT into -STABLE. Security fixes are merged from -STABLE to the security branches.

The official hats in the FreeBSD Project are hats that are more or less formalised and mainly administrative roles. They have the authority and responsibility for their area. The following illustration shows the responsibility lines. After this follows a description of each hat, including who it is held by.

Figure 8.6. Overview of official hats

All boxes consist of groups of committers, except for the dotted boxes where the holders are not necessarily committers. The flattened circles are sub-projects and consist of both committers and non-committers of the main project.

The Core team has the responsibility of giving and removing commit privileges to contributors. This can only be done through a vote on the core mailing list. The ports and documentation sub-projects can give commit privileges to people working on these projects, but have to date not removed such privileges.

Normally a contributor is recommended to core by a committer. For contributors or outsiders to contact core asking to be a committer is not well thought of and is usually rejected.

If the area of particular interest for the developer potentially overlaps with other committers' area of maintainership, the opinion of those maintainers is sought. However, it is frequently this committer that recommends the developer.

When a contributor is given committer status, he is assigned a mentor. The committer who recommended the new committer will, in the general case, take it upon himself to be the new committers mentor.

When a contributor is given his commit bit, a PGP-signed email is sent from either Core Secretary, Ports Manager or [email protected] to both [email protected], the assigned mentor, the new committer and core confirming the approval of a new account. The mentor then gathers a password line, SSH 2 public key and PGP key from the new committer and sends them to Admin. When the new account is created, the mentor activates the commit bit and guides the new committer through the rest of the initial process.

Figure 8.7. Process summary: adding a new committer

When a contributor sends a piece of code, the receiving committer may choose to recommend that the contributor is given commit privileges. If he recommends this to core, they will vote on this recommendation. If they vote in favour, a mentor is assigned the new committer and the new committer has to email his details to the administrators for an account to be created. After this, the new committer is all set to make his first commit. By tradition, this is by adding his name to the committers list.

Recall that a committer is considered to be someone who has committed code during the past 12 months. However, it is not until after 18 months of inactivity have passed that commit privileges are eligible to be revoked. [FreeBSD, 2002H] There are, however, no automatic procedures for doing this. For reactions concerning commit privileges not triggered by time, see section 8.5.8.

Figure 8.8. Process summary: removing a committer

When Core decides to clean up the committers list, they check who has not made a commit for the past 18 months. Committers who have not done so have their commit bits revoked.

It is also possible for committers to request that their commit bit be retired if for some reason they are no longer going to be actively committing to the project. In this case, it can also be restored at a later time by core, should the committer ask.

Roles in this process:

[FreeBSD, 2000A] [FreeBSD, 2002H] [FreeBSD, 2002I]

A CVSup mirror is a replica of the official CVSup master that contains all the up-to-date source code for all the branches in the FreeBSD project, ports and documentation.

Adding an official CVSup mirror starts with the potential CVSup Mirror Site Admin installing the "cvsup-mirror" package. Having done this and updated the source code with a mirror site, he now runs a fairly recent unofficial CVSup mirror.

Deciding he has a stable environment, the processing power, the network capacity and the storage capacity to run an official mirror, he mails the CVSup Mirror Site Coordinator who decides whether the mirror should become an official mirror or not.

In making this decision, the CVSup Mirror Site Coordinator has to determine whether that geographical area needs another mirror site, if the mirror administrator has the skills to run it reliably, if the network bandwidth is adequate and if the master server has the capacity to server another mirror.

If CVSup Mirror Site Coordinator decides that the mirror should become an official mirror, he obtains an authentication key from the mirror admin that he installs so the mirror admin can update the mirror from the master server.

Figure 8.9. Process summary: adding a CVSup mirror

When a CVSup mirror administrator of an unofficial mirror offers to become an official mirror site, the CVSup coordinator decides if another mirror is needed and if there is sufficient capacity to accommodate it. If so, an authorisation key is requested and the mirror is given access to the main distribution site and added to the list of official mirrors.

Roles involved in this process:

Tools used in this process:

The committing of new or modified code is one of the most frequent processes in the FreeBSD project and will usually happen many times a day. Committing of code can only be done by a "committer". Committers commit either code written by themselves, code submitted to them or code submitted through a problem report.

When code is written by the developer that is non-trivial, he should seek a code review from the community. This is done by sending mail to the relevant list asking for review. Before submitting the code for review, he should ensure it compiles correctly with the entire tree and that all relevant tests run. This is called "pre-commit test". When contributed code is received, it should be reviewed by the committer and tested the same way.

When a change is committed to a part of the source that has been contributed from an outside Vendor, the maintainer should ensure that the patch is contributed back to the vendor. This is in line with the open source philosophy and makes it easier to stay in sync with outside projects as the patches do not have to be reapplied every time a new release is made.

After the code has been available for review and no further changes are necessary, the code is committed into the the development branch, -CURRENT. If the change applies for the -STABLE branch or the other branches as well, a "Merge From Current" ("MFC") countdown is set by the committer. After the number of days the committer chose when setting the MFC have passed, an email will automatically be sent to the committer reminding him to commit it to the -STABLE branch (and possibly security branches as well). Only security critical changes should be merged to security branches.

Delaying the commit to -STABLE and other branches allows for "parallel debugging" where the committed code is tested on a wide range of configurations. This makes changes to -STABLE to contain fewer faults and thus giving the branch its name.

Figure 8.10. Process summary: A committer commits code

When a committer has written a piece of code and wants to commit it, he first needs to determine if it is trivial enough to go in without prior review or if it should first be reviewed by the developer community. If the code is trivial or has been reviewed and the committer is not the maintainer, he should consult the maintainer before proceeding. If the code is contributed by an outside vendor, the maintainer should create a patch that is sent back to the vendor. The code is then committed and the deployed by the users. Should they find problems with the code, this will be reported and the committer can go back to writing a patch. If a vendor is affected, he can choose to implement or ignore the patch.

Figure 8.11. Process summary: A contributor commits code

The difference when a contributor makes a code contribution is that he submits the code through the send-pr program. This report is picked up by the maintainer who reviews the code and commits it.

Hats included in this process are:

[FreeBSD, 2001] [Jørgensen, 2001]

Core elections are held at least every two years. ^[11] Nine core members are elected. New elections are held if the number of core members drops below seven. New elections can also be held should at least 1/3 of the active committers demand this.

When an election is to take place, core announces this at least 6 weeks in advance, and appoints an election manager to run the elections.

Only committers can be elected into core. The candidates need to submit their candidacy at least one week before the election starts, but can refine their statements until the voting starts. They are presented in the candidates list. When writing their election statements, the candidates must answer a few standard questions submitted by the election manager.

During elections, the rule that a committer must have committed during the 12 past months is followed strictly. Only these committers are eligible to vote.

When voting, the committer may vote once in support of up to nine nominees. The voting is done over a period of four weeks with reminders being posted on "developers" mailing list that is available to all committers.

The election results are released one week after the election ends, and the new core team takes office one week after the results have been posted.

Should there be a voting tie, this will be resolved by the new, unambiguously elected core members.

Votes and candidate statements are archived, but the archives are not publicly available.

Figure 8.12. Process summary: Core elections

Core announces the election and selects an election manager. He prepares the elections, and when ready, candidates can announce their candidacies through submitting their statements. The committers then vote. After the vote is over, the election results are announced and the new core team takes office.

Hats in core elections are:

[FreeBSD, 2000A] [FreeBSD, 2002B] [FreeBSD, 2002G]

Within the project there are sub-projects that are working on new features. These projects are generally done by one person [Jørgensen, 2001] . Every project is free to organise development as it sees fit. However, when the project is merged to the -CURRENT branch it must follow the project guidelines. When the code has been well tested in the -CURRENT branch and deemed stable enough and relevant to the -STABLE branch, it is merged to the -STABLE branch.

The requirements of the project are given by developer wishes, requests from the community in terms of direct requests by mail, Problem Reports, commercial funding for the development of features, or contributions by the scientific community. The wishes that come within the responsibility of a developer are given to that developer who prioritises his time between the request and his wishes. A common way to do this is maintain a TODO-list maintained by the project. Items that do not come within someone's responsibility are collected on TODO-lists unless someone volunteers to take the responsibility. All requests, their distribution and follow-up are handled by the GNATS tool.

Requirements analysis happens in two ways. The requests that come in are discussed on mailing lists, both within the main project and in the sub-project that the request belongs to or is spawned by the request. Furthermore, individual developers on the sub-project will evaluate the feasibility of the requests and determine the prioritisation between them. Other than archives of the discussions that have taken place, no outcome is created by this phase that is merged into the main project.

As the requests are prioritised by the individual developers on the basis of doing what they find interesting, necessary or are funded to do, there is no overall strategy or priorisation of what requests to regard as requirements and following up their correct implementation. However, most developers have some shared vision of what issues are more important, and they can ask for guidelines from the release engineering team and technical review board.

The verification phase of the project is two-fold. Before committing code to the current-branch, developers request their code to be reviewed by their peers. This review is for the most part done by functional testing, but also code review is important. When the code is committed to the branch, a broader functional testing will happen, that may trigger further code review and debugging should the code not behave as expected. This second verification form may be regarded as structural verification. Although the sub-projects themselves may write formal tests such as unit tests, these are usually not collected by the main project and are usually removed before the code is committed to the current branch. ^[12]

It is an advantage to the project to for each area of the source have at least one person that knows this area well. Some parts of the code have designated maintainers. Others have de-facto maintainers, and some parts of the system do not have maintainers. The maintainer is usually a person from the sub-project that wrote and integrated the code, or someone who has ported it from the platform it was written for. ^[13] The maintainer's job is to make sure the code is in sync with the project the code comes from if it is contributed code, and apply patches submitted by the community or write fixes to issues that are discovered.

The main bulk of work that is put into the FreeBSD project is maintenance. [Jørgensen, 2001] has made a figure showing the life cycle of changes.

Figure 8.13. Jørgenssen's model for change integration

Here "development release" refers to the -CURRENT branch while "production release" refers to the -STABLE branch. The "pre-commit test" is the functional testing by peer developers when asked to do so or trying out the code to determine the status of the sub-project. "Parallel debugging" is the functional testing that can trigger more review, and debugging when the code is included in the -CURRENT branch.

As of this writing, there were 275 committers in the project. When they commit a change to a branch, that constitutes a new release. It is very common for users in the community to track a particular branch. The immediate existence of a new release makes the changes widely available right away and allows for rapid feedback from the community. This also gives the community the response time they expect on issues that are of importance to them. This makes the community more engaged, and thus allows for more and better feedback that again spurs more maintenance and ultimately should create a better product.

Before making changes to code in parts of the tree that has a history unknown to the committer, the committer is required to read the commit logs to see why certain features are implemented the way they are in order not to make mistakes that have previously either been thought through or resolved.

FreeBSD comes with a problem reporting tool called "send-pr" that is a part of the GNATS package. All users and developers are encouraged to use this tool for reporting problems in software they do not maintain. Problems include bug reports, feature requests, features that should be enhanced and notices of new versions of external software that is included in the project.

Problem reports are sent to an email address where it is inserted into the GNATS maintenance database. A Bugbuster classifies the problem and sends it to the correct group or maintainer within the project. After someone has taken responsibility for the report, the report is being analysed. This analysis includes verifying the problem and thinking out a solution for the problem. Often feedback is required from the report originator or even from the FreeBSD community. Once a patch for the problem is made, the originator may be asked to try it out. Finally, the working patch is integrated into the project, and documented if applicable. It there goes through the regular maintenance cycle as described in section maintenance. These are the states a problem report can be in: open, analyzed, feedback, patched, suspended and closed. The suspended state is for when further progress is not possible due to the lack of information or for when the task would require so much work that nobody is working on it at the moment.

Figure 8.14. Process summary: problem reporting

A problem is reported by the report originator. It is then classified by a bugbuster and handed to the correct maintainer. He verifies the problem and discusses the problem with the originator until he has enough information to create a working patch. This patch is then committed and the problem report is closed.

The roles included in this process are:

[FreeBSD, 2002C]. [FreeBSD, 2002D]

[FreeBSD, 2001] has a number of rules that committers should follow. However, it happens that these rules are broken. The following rules exist in order to be able to react to misbehaviour. They specify what actions will result in how long a suspension the committer's commit privileges.

[Lehey, 2002]

For the suspensions to be efficient, any single core member can implement a suspension before discussing it on the "core" mailing list. Repeat offenders can, with a 2/3 vote by core, receive harsher penalties, including permanent removal of commit privileges. (However, the latter is always viewed as a last resort, due to its inherent tendency to create controversy). All suspensions are posted to the "developers" mailing list, a list available to committers only.

It is important that you cannot be suspended for making technical errors. All penalties come from breaking social etiquette.

Hats involved in this process:

The FreeBSD project has a Release Engineering team with a principal release engineer that is responsible for creating releases of FreeBSD that can be brought out to the user community via the net or sold in retail outlets. Since FreeBSD is available on multiple platforms and releases for the different architectures are made available at the same time, the team has one person in charge of each architecture. Also, there are roles in the team responsible for coordinating quality assurance efforts, building a package set and for having an updated set of documents. When referring to the release engineer, a representative for the release engineering team is meant.

When a release is coming, the FreeBSD project changes shape somewhat. A release schedule is made containing feature- and code-freezes, release of interim releases and the final release. A feature-freeze means no new features are allowed to be committed to the branch without the release engineers' explicit consent. Code-freeze means no changes to the code (like bugs-fixes) are allowed to be committed without the release engineers explicit consent. This feature- and code-freeze is known as stabilising. During the release process, the release engineer has the full authority to revert to older versions of code and thus "back out" changes should he find that the changes are not suitable to be included in the release.

There are three different kinds of releases:

For releases of the -STABLE-branch, the release process starts 45 days before the anticipated release date. During the first phase, the first 15 days, the developers merge what changes they have had in -CURRENT that they want to have in the release to the release branch. When this period is over, the code enters a 15 day code freeze in which only bug fixes, documentation updates, security-related fixes and minor device driver changes are allowed. These changes must be approved by the release engineer in advance. At the beginning of the last 15 day period a release candidate is created for widespread testing. Updates are less likely to be allowed during this period, except for important bug fixes and security updates. In this final period, all releases are considered release candidates. At the end of the release process, a release is created with the new version number, including binary distributions on web sites and the creation of a CD-ROM images. However, the release isn't considered "really released" until a PGP-signed message stating exactly that, is sent to the mailing list freebsd-announce; anything labelled as a "release" before that may well be in-process and subject to change before the PGP-signed message is sent. ^[14].

The releases of the -CURRENT-branch (that is, all releases that end with ".0") are very similar, but with twice as long timeframe. It starts 8 weeks prior to the release with announcement of the release time line. Two weeks into the release process, the feature freeze is initiated and performance tweaks should be kept to a minimum. Four weeks prior to the release, an official beta version is made available. Two weeks prior to release, the code is officially branched into a new version. This version is given release candidate status, and as with the release engineering of -STABLE, the code freeze of the release candidate is hardened. However, development on the main development branch can continue. Other than these differences, the release engineering processes are alike.

.0 releases go into their own branch and are aimed mainly at early adopters. It is not until .1 versions are released that the branch becomes -STABLE and -CURRENT targets the next major version.

Most releases are made when a given date that has been deemed a long enough time since the previous release comes. A target is set for having major releases every 18 months and minor releases every 4 months. The user community has made it very clear that security and stability cannot be sacrificed by self-imposed deadlines and target release dates. For slips of time not to become to long with regards to security and stability issues, extra dicipline is required when committing changes to -STABLE.

Figure 8.15. Process summary: release engineering

These are the stages in the release engineering process. Multiple release candidates may be created until the release is deemed stable enough to be released.

[FreeBSD, 2002E]

Concurrent Versions System or simply "CVS" is a system to handle multiple versions of text files and tracking who committed what changes and why. A project lives within a "repository" and different versions are considered different "branches".

CVSup is a software package for distributing and updating collections of files across a network. It is consists of a client program, cvsup, and a server program, cvsupd. The package is tailored specifically for distributing CVS repositories, and by taking advantage of CVS' properties, it performs updates much faster than traditional systems.

GNATS is a maintenance database consisting of a set of tools to track bugs at a central site. It supports the bug tracking process for sending and handling bugs as well as querying and updating the database and editing bug reports. The project uses one of its many client interfaces, "send-pr", to send "Problem Reports" by email to the projects central GNATS server. The committers have also web and command-line clients available.

Mailman is a program that automates the management of mailing lists. The FreeBSD Project uses it to run 17 general lists, 45 technical lists and 6 limited lists. It is also used for many mailing lists set up and used by other people and projects in the FreeBSD community. General lists are lists for the general public, technical lists are mainly for the development of specific areas of interest, and closed lists are for internal communication not intended for the general public. The majority of all the communication in the project goes through these 68 lists [FreeBSD, 2003A, Appendix C].

Perforce is a commercial software configuration management system developed by Perforce Systems that is available on over 50 operating systems. It is a collection of clients built around the Perforce server that contains the central file repository and tracks the operations done upon it. The clients are both clients for accessing the repository and administration of its configuration.

Pretty Good Privacy, better known as PGP, is a cryptosystem using a public key architecture to allow people to digitally sign and/or encrypt information in order to ensure secure communication between two parties. A signature is used when sending information out many recipients, enabling them to verify that the information has not been tampered with before they received it. In the FreeBSD Project this is the primary means of ensuring that information has been written by the person who claims to have written it, and not altered in transit.

Secure Shell is a standard for securely logging into a remote system and for executing commands on the remote system. It allows other connections, called tunnels, to be established and protected between the two involved systems. This standard exists in two primary versions, and only version two is used for the FreeBSD Project. The most common implementation of the standard is OpenSSH that is a part of the project's main distribution. Since its source is updated more often than FreeBSD releases, the latest version is also available in the ports tree.

A "port" is a set of meta-data and patches that are needed to fetch, compile and install correctly an external piece of software on a FreeBSD system. The amount of ports have grown at a tremendous rate, as shown by the following figure.

Figure 8.16. Number of ports added between 1996 and 2003

Figure 8.16 is taken from the FreeBSD web site. It shows the number of ports available to FreeBSD in the period 1995 to 2003. It looks like the curve has first grown exponentionally, and then since the middle of 2001 grown linerly.

As the external software described by the port often is under continued development, the amount of work required to maintain the ports is already large, and increasing. This has led to the ports part of the FreeBSD project gaining a more empowered structure, and is more and more becoming a sub-project of the FreeBSD project.

Ports has its own core team with the Ports Manager as its leader, and this team can appoint committers without FreeBSD Core's approval. Unlike in the FreeBSD Project, where a lot of maintenance frequently is rewarded with a commit bit, the ports sub-project contains many active maintainers that are not committers.

Unlike the main project, the ports tree is not branched. Every release of FreeBSD follows the current ports collection and has thus available updated information on where to find programs and how to build them. This, however, means that a port that makes dependencies on the system may need to have variations depending on what version of FreeBSD it runs on.

With an unbranched ports repository it is not possible to guarantee that any port will run on anything other than -CURRENT and -STABLE, in particular older, minor releases. There is neither the infrastructure nor volunteer time needed to guarantee this.

For efficiency of communication, teams depending on Ports, such as the release engineering team, have their own ports liaisons.

The FreeBSD Documentation project was started January 1995. From the initial group of a project leader, four team leaders and 16 members, they are now a total of 44 committers. The documentation mailing list has just under 300 members, indicating that there is quite a large community around it.

The goal of the Documentation project is to provide good and useful documentation of the FreeBSD project, thus making it easier for new users to get familiar with the system and detailing advanced features for the users.

The main tasks in the Documentation project are to work on current projects in the "FreeBSD Documentation Set", and translate the documentation to other languages.

Like the FreeBSD Project, documentation is split in the same branches. This is done so that there is always an updated version of the documentation for each version. Only documentation errors are corrected in the security branches.

Like the ports sub-project, the Documentation project can appoint documentation committers without FreeBSD Core's approval. [FreeBSD, 2003B].

The Documentation project has a primer . This is used both to introduce new project members to the standard tools and syntaxes and acts as a reference when working on the project.

"Methodology size" is a measure of how many control elements a methodology contains. Control elements are roles, activities, outcomes, standards and quality requirements described in the methodology. In Section 8.4 we find 24 roles. Section 8.5 lists 9 processes with a total of 59 activities. Both activities and processes have deliverables, and none of the activities or processes in the project model have more than one delivery. Thus there are 68 deliveries. The tools used in the process model give the standard of how the outcomes will be. Thus there are 7 standards followed. There are no quality requirements. Quality is upheld through review.

"Precision" is how precise the methodology is described. In Section 2.3.4 we decided rather than to formalise the processes completely, to describe the processes so that they can be executed as the people involved see fit rather than to use the project model as a straight jacket. This is in line with the research aim. The project model is thus of low precision.

"Tolerance" is a measure of how much deviation is allowed from the standards defined in the methodology. There is no defined tolerance except for that the tools have to understand the data. Many of the tools have been designed to be very tolerant, so with that respect there is a high tolerance on data variation.

The "specific density" of the methodology is a measure that can be understood as precision divided by tolerance. High specific density means high precision and low tolerance, while low specific density is the opposite. Since tolerance in this project model has a 1:1 relationship to precision, we find that the project model is of medium density.

[Smevold, 2001] describes "methodology weight" as methodology size multiplied with specific density, without detailing further the metrics used for this. The project model weight consists thus of the two factors methodology size, given by the vector [24,9,59,68,7], and the specific density, medium. While it probably gives little meaning comparing this attribute with other methodologies, it is an attribute that can be used to track the project model over time.

"Project size" is the number of people the methodology needs to coordinate. The project model uses most time at coordinating the 275 committers in the project, but also describes how the contributers and users can interface the project.

"Problem size" says how many elements there are in the problem domain and how complex these are to solved. The project model has not discussed this.

Cockburn claims in his principle two that "A larger methodology is needed when more people are involved". We found the methodology size in the previous section. He divides methodologies into "little-m" and "big-M" methodologies where little-m are referring to methodologies as described in popular object oriented methodology books, and big-M are methodologies as used in large corporations. In such terms, this methodology is closer to a little-m methodology.

Being most similar to a little-m methodology is surprising as we saw in Section 1.2 and Section 8.2 that the number of people in the project has risen and the problem size broadened. This should indicate that a medium weight methodology is needed.

As noted in the introduction when discussing motivations, more people coming to the project means more opinions of what the problem domain is. This adds on both axis in the model seen in Section 1.2, and thus gives us an increased methodology size.

This is also in line with Brooks' Law (discussed in Section 2.3.5) as Cockburn's third principle says that "A relatively small increase in methodology size or specific density adds a relatively large amount to the cost of the project". [Cockburn, 1998] Brooks' discusses his law with regard to an incease in cost [Brooks, 1995: 274-275]. He finds that although sometimes cost is increased without an increase in time, but in the general case his law holds true. This is very true in the FreeBSD Project where close to all costs is developer time. An increase in people leads to an increase in methodology size that costs more developer time and makes the project later.

[Cockburn, 1998] divides methodologies into four classes based upon the problem size and criticality of the system developed. In his figure, however, he allows for many other attributes as well. A reduced figure, taking only these two variables into account is as follows

Figure 9.1. Methodology weight needed

Figure 9.1 shows the relation between what criticality and problem size the project faces and what kind of methodology it should have. For instance, development of pace makers has high criticality, because should the pace maker stop, the patients heart risks stopping. This warrants a heavy. Similarly, development of a Mars exploration vehicle has a large problem size, and thus should have a heavy methodology.

What is interesting about the FreeBSD project is that it lands in two categories of criticality: the -STABLE branch has a high criticality due to its use in production environments, and the -CURRENT branch has a low criticality as it is intended for hobby developers. However, both products are so intertwined that making a separate methodology for each does not make sense. The compromise made has been to have a high presision and low tolerance with regard to the release engineering process.

It is common in many organizations to have a written down or online methodology that is consulted for how sub-processes should be done. Some methodologies, such as Rational's "Unified Process" are designed to be consulted for each step. [Smevold, 2001] found that methodologies were likely to be used more actively in the project work if they were available on the project members computers. In accordance with this, the project model is now available for both project members and people outside the project as a stand-alone, web-published document.

[Smevold, 2001] classifies methodologies in monumental and agile methodologies. (See Section 2.3.2.7 for agile models). The project model for the FreeBSD Project resembles the monumental methodologies when it comes to administration and project structure, but with its highly incremental development model, it resembles agile methodologies more when it comes to development.

Since methodologies are software and are developed like software, they need to be debugged. How this should be done is not well defined. [Cockburn, 2000] suggests reflection sessions every 4-8 weeks asking "What did we learn?" and "What can we do better?" This may work well for project with clearly defined increments, but how is this done in a project with no set increments, such as the FreeBSD project? It may be done through an ongoing discussion on a mailing list, but this will rarely include the entire project group such as it would in the reflection sessions suggested by Cockburn.

[Kitchenham et al, 1997] discusses evaluation of project models and proposes the DESMET evaluation methodology. It uses formal experiments, case studies and surveys as means to evaluate a project model. Just like a methodology needs a methodology to support its creation, it needs to be evaluated. DESMET has successfully been applied to evaluate its own methodology. While this would probably be a very scientific correct way of evaluating it, it has a very high cost in the amount of time used, and I cannot say that the benefits it can offer would be worth the time invested.

Formalising the project model in accordance with [Osterweil, 1987] would make the project model maintainable through regular debugging just like any piece of software. The tradeoff, however, is that the project members would feel more restricted in how they should do their work. Since many committers have expressed that they are in the project because it is fun, reducing the amount of enjoyment in the project would be counter-productive. However, tasks that are not development related, should be automated as much as possible so that the committers can do what they do the best and enjoy the most: develop software.

Except for [McKusick et.al, 1997], [FreeBSD, 2001B] and the standard set of tools, there are not many standard templates for how things should look.

The project does not commit to many standards either. With every release comes a list of hardware supported and a list of what has been implemented since last time. As discussed in Section 6.4, there are no lists saying what standards are implemented and to what degree different standards are implemented. Also, the project does not maintain a list of standards it aims to implement and when it aims to have them implemented. From this, it seems the project does not commit to having standards implemented, and if standards are indeed implemented, this is good but not guaranteed to be true in the next version.

This is quite the opposite impression that the FreeBSD project wants to give. It has indeed implemented many standards. For example, [Open Group, 1997] was implemented through the OpenPAM project. There is also an active standards mailing list. FreeBSDs also is reputed to be very standard compliant. Thus the impression the project model and its outcomes gives regarding standards is wrong. The model should be updated with a target list containing what standards are going to be implemented to what extent.

As we saw in the first section of this chapter, we have created a set of quality goals, problemised them through questions and made them measurable by defining metrics. In this way, we have said something about quality instead of "going poof" as discussed in Section 2.4.

Is this enough? We have seen in this chapter that the problem domain increases with more people in the project, and people join regularaly the project. Clearly, there are and will be more project goals.

Goals should be handled by the project. First, goals should be identified. These can be identified through a tracking system where people submit goals important to them. Then these goals could be rated among the project members. Also, it'd be beneficial if the users could rate the goals to see the difference between the two groups. Then a set of goals could be extracted from the list to form official project goals. These would probably be the best set of goals to use in measuring the project model quality over time.

At the moment, quality goals are ensured only through peer review and mailing list discussions. Project participants have been upset when their idea of quality has been reduced. This has led to a number of disagreements that core has had to mediate in. Also, it means that those who are most articulate about their opinions get heared the most.

But the way goals are defined today is in line with Section 7.1. Rather than to discuss the goals, the ones who implement their personal goals as code get the actual say in what quality is.

The discussion between quality goals and standards is closely linked. If a set of goals are identified, a list of targeted standards is easy to make. But if goals remain vague ideas that are only clear once they are implemented, such a list is probably not in order.

There are three main outcomes of the development processes: code, documentation and releases. Code and documentation are seen as source code that are used to build releases. Source code is the only outcome from the development processes that is harvested. Is this beneficial?

The different developers and sub-projects use each their own development method or methodology. The only level they interface one-another on is source-code and by mailing lists and private email. The software engineering community has long investigated the benefits of cooperating at design level and making design repositories for projects that consist of many subprojects. The same is true for requirements, verification documents and tests. However, to cooperate on these levels means less freedom for the individual developer which again may limit their willingness to contribute. Even so, this possibility should be investigated further.

At the moment, testing in FreeBSD happens very random. This makes tracking errors harder as it requires very experienced users in order to find an error.

In order to structurise testing more, there should be a central way for interested testers to list their hardware. This would allow device driver developers to contact interested testers and notify them about upcoming drivers for testing. Such a database could also give an indication of what popular hardware lacks driver support.

Juli Mallet made a test environment she could use to test the reimplementation of xargs. When PAM was reimplemented, no such testing was done. Rather, a test set of headers was used to compile applications with to see if they integrated well with LinuxPAM, and when they did not and the maintenance of LinuxPAM became to cumbersome, a reimplementation was done. What kinds of testing is most beneficial while at the same time being easy to maintain should be investigated. A test set allowing and automating multiple kinds of testing (regression testing, unit testing, structural testing) would be beneficial to aid continued testing.

The project model focusses on "what does a developer do?". That already implies that we know who the developers are, which we answer in the project model. What is not answered is "who are the users?" [Lehey, 2002] notes that Core's focus is not on the users, and the culture is in my experience very committer-centric.

Is this a bad thing? After all, the committers are real power-users of FreeBSD. Some committers are employed by major companies that need the expertice on FreeBSD or need features implemented. We found in Section 4.1 that by far most contributors are contributors through sending problem reports. There are also very many users active on mailing lists (see Section 5.2). This indicates that there are many users that want to be included and have opinions, needs and take their time to contribute in one way or another.

Other than problem reports and mailing lists, users make also up a lot of the community around the project. Examples of this are online communities, providing among other things services such as peer support and how-to articles (see FreeBSD China Community, The FreeBSD Diary, Defcon1 and BSDVault), FreeBSD User Groups (such as Norwegian BSD User Group, Home Unix User Group and Daibou *BSD Users Group), News sites (such as Dæmonnews, BSD Today and BSDForums.org) and commercial services (such as FreeBSD Services, BSD Mall and BSD Consulting).

With such a diverse community, a committer-centric environment is probably not the most favourable environment. Ways to include the users more in the project should be investigated. The software engineering community has written much on such issues, and inspiration could be sought from this community.

The FreeBSD Foundation is a non-profit organisation dedicated to supporting the FreeBSD Project. It does this in terms of fund raising, represent the FreeBSD Project in legal contracts or other arrangements that require a recognized legal entity.

The FreeBSD Foundation interfaces the FreeBSD Project in that one member is a member of the core team. As the core team is elected every two years, this will not necessarily continue to be the case.

The FreeBSD Foundation is not included in the project model as it is not considered a part of the project, but a useful, outside entity.

In November 2002, a technical review board was established for a trial period of three months. Its aim is to make desicions with regards to what technical direction the project is going to take. It handles issues like what technical solutions are the wisest and in the case of multiple solutions, which one to choose on technical grounds.

The FreeBSD Foundation does already much administrative work and is an important legal entity. With the Technical Review Board doing the technical work of Core, Core remains as a conflict resolving and commit privilege assigning entity. An alternative organisation, in line with the division between development processes and administrative processes, would be to have the foundation to handle administrative issues and the review board to handle development issues. The legal person in charge of the foundation could be a secretary for the foundation with the foundation's board being elected like Core is today. The technical review board could be elected in the same way, at the same time. This could bring back the way the fun to the FreeBSD Project that the two members that resigned from the first elected core were missing.

The project model has thus a problem of how the FreeBSD Foundation should be integrated, and in that process needs to discuss the management roles in the FreeBSD Project. Alternatives, of which I have outlined one, should be investigated.

The most interesting findings are the following:

The first finding is a summary of many findings and shows that the FreeBSD Project is a healthy project. The second requires a continued debate about how the project should be run to resolve the issues found.

The conclusion is therefore that the research aim has been met, and that during the research, many findings regarding the FreeBSD Project has been made.

This research has provided a detailed insight into the FreeBSD Project lays the ground for a study between this and other open source projects to gain a better understanding of the open source community so we can use this knowledge to improve software development efforts.