Date: Tue, 21 Jan 2003 19:47:39 -0800 (PST) From: John Polstra To: Niklas Johannes Saers Subject: Re: CVSup Mirror Site Coordinator Niklas Johannes Saers wrote: > Hi, my name is Niklas Saers and I'm writing my master thesis on how > the FreeBSD project works. As a part of this, I'm writing a FreeBSD > methodology, and with that regard I'd like to ask you about your role > as CVSup Mirror Site Coordinator. I have been unable to dig up much > information regarding this role, and I was wondering what kinds of tasks > and responsabilities come along with it? My main responsibility is to make decisions about whether a proposed new mirror site that somebody has offered to run should become an official FreeBSD mirror site. We have one master site (cvsup-master.freebsd.org) which provides the content to all of the public mirror sites. CVSup-master is not publicly accessible; only the official mirrors can access it. This is enforced by an authentication system which prevents anybody else from accessing the site. The process generally begins when somebody writes to me and says, "I would like to offer a new CVSup mirror site in Slovenia. May I please have access to cvsup-master.freebsd.org?" Then I have to consider a number of things: - Does Slovenia need another mirror site? - Does the person offering the new mirror have the skills necessary to run a mirror reliably? - Is the equipment and Internet bandwidth the person is offering adequate? And I have to balance those factors against the other side of the equation: - Is cvsup-master.freebsd.org too loaded to serve another mirror site? If I decide that a proposed mirror is a good idea, then I get an authentication key from the maintainer of the mirror and install it on CVSup-master, thereby granting them access. > Also, was this a role given by core, a role you took upon yourself > or one suggested by someone else? It is a role I took upon myself. That is very typical of the FreeBSD project. Somebody sees a problem; he develops a solution for it; he makes that solution available publicly; with luck, people like his solution and start using it; and suddenly the person who developed the solution is in charge of that entire problem domain. Here is what happened in my case. When I became involved with the FreeBSD project in 1995, the sources were distributed using a program called "sup". It was very slow, and there were only a few (about 5) mirror sites where one could connect with sup. The mirror sites were entirely ad hoc. Every one was different. Some of them were very poorly managed. Each one had its own set of configuration files. As a result, if you updated your sources from one mirror site and then later updated them again from a different mirror, the chances were good that your entire source tree would get deleted. In 1995-1996 I took a 1-year sabbatical from my job. I was looking for a computer project to do, just for fun. I decided that sup was awful and that FreeBSD needed a better tool. So I conceived of and implemented CVSup, a more intelligent replacement for sup. Once it was working pretty well, I told a couple of core team members about it. They started using it, liked it a lot, and told more people about it. Soon, a few people contacted me and said they would like to provide CVSup mirror sites. I helped them to set up the mirror sites. But after setting up a couple of them, I became concerned that the CVSup mirrors would end up having the same problems that the sup mirror sites used to have: namely, that each one would be a little bit different, and that that would cause problems for the users. So, I took some time and developed a package called "cvsup-mirror". With this package, anybody could set up a new mirror site simply by typing "pkg_add cvsup-mirror.tgz" and answering a few simple questions. I designed the cvsup-mirror package such that each mirror site was self-updating. In other words, the configuration of the site itself was updated automatically from cvsup-master.freebsd.org when necessary. This ensured that all of the mirror sites remained consistent, without the need for any manual actions by the maintainers of the sites. Because it was now so easy to set up a mirror site, we soon had enough of them to create a very heavy load on the master server (which at that time was freefall.freebsd.org, the main FreeBSD development machine). The core team allocated a new machine, cvsup-master.freebsd.org, to handle that load. (By that time, I had become a member of the core team.) We decided that access to cvsup-master should be restricted to mirror sites and committers only. So everybody who wanted access had to get a key from me. At that point I became, in effect, the CVSup Mirror Site Coordinator. I did the job for several years, and then I decided it was so much work that I deserved a title. So I added myself to the FreeBSD Handbook as the CVSup Mirror Site Coordinator. :-) Nobody objected. Everybody knew that I was in charge of the mirror sites, and they all wondered why I hadn't been added to the Handbook long ago. This is very typical of how things work in FreeBSD. 1. Notice a problem 2. Provide a solution 3. Tell people about it 4. They like it! 5. You are in charge of it. Here is another very common approach, which _doesn't_ work: 1. Notice a problem 2. Whine that "somebody" should do "something" about it 3. Nothing happens 4. Whine some more 5. Nothing happens [...] In short, people get responsibility in the FreeBSD project when they provide solutions that work. Words count for very little. I hope this is the kind of information you are looking for. John -- Date: Sun, 26 Jan 2003 22:00:18 -0800 (PST) From: John Polstra To: Niklas Johannes Saers Subject: Re: CVSup Mirror Site Coordinator Hi Niklas, > this was indeed what I was looking for and heaps more. :) Good! > Only a few details, when you say authentication key, you mean a > SSH(2?) key? No. I invented my own scheme which was basically an adaptation of HTTP digest authentication. When I did that work (around 1996 or 1997), ssh was still considered to be a "munition" by the US government. I didn't want to use it because of the export problems. > How/when do you decide to take away the authority to be an official > CVS mirror? On what grounds does this happen? (has it happened?) I have never taken that authority away so far, although I have denied quite a few requests from people who have wanted to start new mirrors. If I ever take away cvsup-master access from somebody it will be because the mirror delivered bad or incomplete content for a long time, and the maintainer was totally unresponsive; or because the maintainer abused his access key by using it for other purposes and didn't stop when I warned him. Really, though, every one of the maintainers is basically a generous person who wants to give something back to the project. People like that aren't very much trouble at all. :-) When I have denied requests to create new mirrors, it has been because that country already had plenty of mirrors; or because I felt that the donor's hardware and/or bandwidth wasn't adequate for the job. Regarding the first reason: cvsup-master's capacity is a finite resource, so I try not to waste it where it isn't needed. > When you say you can have a master configuration at the > cvsup-master, does this include binaries or do admins have to update > CVSup themselves? The administrators have to update CVSup themselves. I am pretty security-conscious, and if I were an administrator I would never allow an automatic program to update my software. It's just too dangerous. (The cvsup update processes that run on the mirrors don't even run as root, so they cannot install new software. They run as a user "cvsupin" which doesn't have write permission anywhere except where the mirror content is stored.) Luckily, the maintainers are pretty attentive, and I don't have to nag them very much when an upgrade is needed. You might recall that when the Unix date representation became >= 1000000000 on September 8, 2001, it triggered a nasty bug in CVSup which caused problems on all the mirrors. I released an emergency upgrade, and almost all of the maintainers installed it very quickly. It was a strange time. I was feeling very sorry for myself, as if I had the worst crisis in the world; and then suddenly thousands of people died in New York and Washington, DC (where my wife was at the time), and I got a heavy dose of perspective. The configuration files which are updated automatically are just the cvsupd configuration. These files essentially define the various CVSup collections and their contents. For security reasons, each mirror has a "refuse" file (see cvsup(1)) which makes it impossible for these automatic updates to do anything dangerous. > Apart from that, I'd like to say that I love using CVSup myself. :) > Thanks for writing it and maintaining the tree. :) Thanks, that's nice to hear! Designing and implementing CVSup was one of the most fun times I've ever had. Managing the mirrors is not as fun, but I take a lot of pride in the quality of the result. That's really what the FreeBSD project is all about: doing something that you can take pride in for a long time. John