Received in an SMS from a friend: 'I just walked past a bloke with pasty white skin and a ponytail wearing a t-shirt that said 'I do it with Ubuntu'.'

Wouter asserts that Ubuntu has more than 30 full-time employees. I know it wasn't his number (Scott came up with it and should've been a bit more clear), but it's badly wrong, anyway. Canonical has more than 30 employees, but Ubuntu ... the full-time distribution team hardly breaks double figures. That's pretty comparable to the number of people working on the m68k port (albeit the latter are not funded full-time). But it's an interesting point nonetheless, and thanks to the assumption that Canonical does nothing other than Ubuntu, people seem to assume that Ubuntu was produced by a team of 30, 40, whatever; not even close. That team was barely ten.

(Note: Some of the graphs in here aren't entirely in step with the text -- we didn't really think about blogging it until very late in the game, so the graphs linked to are more a rough idea of where we were at the time, not exhaustive.)

There's nothing like a challenge.

Along with the rest of Canonical and a whole bunch of excellent community guys, Thom May and I have been at the Ubuntu conference at Mataro since last Sunday. While imbibing large amounts of Coke, Thom decided that boot was probably too slow, with an eye on the promised forty-second desktop. He put bootchart on his laptop, and there the madness started.

Originally, we believed our boot was around 20 seconds without readahead, and felt pretty good about ourselves. But once we moved bootchart into the right place (starting at the top of rc2 is not a useful metric), we had a more realistic view -- around 1min for a boot to the gdm login screen -- and set to work. One thing we found was that gdm slept. A lot. After I took to the gdm source with a very large axe, we no longer had a huge gulf in our boot process with time elapsing, but no disk or CPU usage.

We had already replaced large swathes of the massive shell horror that is known as hotplug with grepmap, so it was time to look at other things. cupsys in particular was a huge disk hog, so some fine-tuning from Scott James Remnant soon set that in order. But we had kicked the more obnoxious sleeps out of gdm, and we were still looking at well over fifteen seconds from X starting through to gdm actually prompting you for your login, which sucked. I stopped laughing at Thom when he told me the X server startup really was too slow. So I straced it.

Output of strace -e file (i.e. show all file accesses) when starting X:
5702 xorg-trace-file

In July, I got very frustrated with the old, crufty, and generally horrid MetroLink loader in XFree86/X.Org, which was also used in Debrix (being a fork of X.Org), and threw it away.
daniels@catsby:~/x/xorg/xc/programs/Xserver/hw/xfree86/loader% wc -l *.[ch] | tail -1
13540 total
daniels@catsby:~/x/debrix/debrix--devel/hw/xorg/loader% wc -l *.[ch] | tail -1
897 total

It seemed like some major surgery was needed on the loader, or at least to beat the more obnoxious parts out of it. For instance, stat()ing the Radeon driver forty-one times. I don't even have a Radeon in this machine; it's i855-based. Turns out that the loader was running a regular expression over every single file in /usr/X11R6/lib/modules, and then stat()ing them, for every module load. It is now no longer doing so:
daniels@catsby:~/public_html% wc -l xorg.trace-1040
1904 xorg.trace-1040

With a lazy 3798 file accesses gone, there was no longer a massive disk I/O hit as X started, but it was still not enormously quick. Further beating of gdm ensued, and we discovered that scaling a 1600x1200 pixmap down and then overlaying a transparent PNG really, really hurt. Enabling autologin let us fly through the process, but this was far too much of a security risk to consider, so we went back to measuring gdm proper.

While we were at it, in an inspired move, Scott James Remnant set to work on cupsd, which really was a horrific disk hog, so soon set that in order.

Hotplug was still doing a lot of work (but no disk I/O) for a very long time, so we decided to parallelise hotplug and readahead, so we could have pure CPU grunt work interleaved with smashing the disk. This seems to have worked very effectively, and has shaved quite some time off our accesses. Starting some parts of rc2 in parallel seems to have worked very well, also; gdm starts at 14 (before most services that you don't need for a desktop).

At present, the biggest bottleneck we have is hotplug; Scott is working on replacing the hotplug init script with moving through udevd, and replacing many of our init scripts with hotplug triggers, so we can more effectively parallelise most of our boot process. Thom is, at present, working on moving all the readahead files somewhere where they can be streamed quickly; at the moment, we are getting flayed by seek()s. Beyond this, gdm looks like it needs some serious work, but we believe we are at a very strong position, especially after beating more file accesses out of X (this is our current bootchart, at time of writing).

These tests were done (by both myself and Thom) on IBM ThinkPad X40s, with Pentium M 1.2GHz CPus and low-speed hard drives. One of the larger blockers is i855, which takes forever to initialise through VBE: 'profiling' the X server by throwing in time information with all the logs showed us nothing was usefully slow (in terms of low-hanging fruit), but video initialisation still takes a good four or so seconds. It'll be interesting to measure the results on standard desktop disks (not 15kRPM SCSI or such) and a chipset we have the full video BIOS information for. However, everything we have done thus far is totally applicable to other systems -- hotplug still runs in full, so you can dump the exact same stuff we're using on a totally different system (even PowerPC, if you like), and it will still work perfectly fine, with no modification.

Current statistics: booting through a full, typical startup to GDM login screen, including hotplug -- 42 seconds (warty: 90 or more); file accesses when starting the X server -- 1093 (warty: at least 5700); files read in by readahead: 1000 (total size: 58M).

(Update: X is now down to 538 opens.)

71 S Nov 09 Fabio M. Di Nit ( 0) Accepted xorg 6.8.1-0ubuntu1 (source)

Michael Banck wonders about the Ubuntu development model. Two things I can provide constructive answers to are:

• NMUs: no, we don't have maintainership as such. However, we do have unofficial areas -- for example, myself and Fabio take care of X, I do most of the stuff relating to ppp and also discover1 (including the data list), et al. Doing pppoe without a PPPoE setup has been interesting, but oh well.
• Debian: for most all of us, Debian was an after-work thing anyway (or, at least, a small part of work). Nothing's changed here; the only thing that's changed for me is that I haven't gotten around to setting up a chroot yet (only got DSL bumped up to unlimited recently), so I don't have a Debian machine to build on: I get all my uploads sponsored right now.

... in which our hero performs an install without a CD-ROM drive, seeded almost entirely from /var/cache/apt/archives.

I last installed my desktop ages ago, and since then I've dist-upgraded almost every package that's installed. Installing my X40 has been fun: our bandwidth is incredibly limited by transfer, so instead of grabbing an ISO (not like it has a CD-ROM drive anyway), I decided to attempt a netboot install from /var/cache/apt/archives.

First, I set up apt-move to move /var/cache/apt/archives to a mirror directory, and to mirror something replicating the standard Ubuntu archive. Secondly, I set up dhcpd3 to netboot the contents of pxeboot.tar.gz (found on http://archive.ubuntu.com/ubuntu/dists/warty/main/daily-installer-i386), to the trusty laptop.

Now, here comes the fun. Once this was done, I needed to wget the main/debian-installer/binary-i386 Packages{,.gz,.bz2} file, and manually drop the sums/sizes into the Release file, which needed to be altered to specify the right component/suite. Once this file was there, a small for script got every single udeb specified in the d-i Packages file, and dropped it into the pool structure. Once all these things were satisfied, I was on my way to a netboot install, with minimal bandwidth consumed; thanks to Colin Watson for patiently putting up with my stupid questions.