Space-efficient reproducible builds using schroot, sbuild and LVM

My setup for schroot and sbuild is based on an idea I saw on Planet Debian a while ago. (Sorry, I can't remember whose idea it was, and I can't find it with Google - if I've stolen your idea, please let me know and I'll give you proper credit!)

Instead of having persistent chroot environments which can get broken by buggy package builds, I maintain several clean chroot environments on an LVM volume. schroot sets up snapshots of that volume for normal use; it's also possible to use the underlying "source" volume, mainly to upgrade it.

I use this configuration on Collabora's build box, and also on my laptop.

The naive thing to do would be to allocate a logical volume for each chroot. However, each LV needs enough space to hold a minimal base system, plus the largest package build you ever want to do (including its build dependencies), since the snapshot used for the actual build can't get bigger than the base LV.

So, you can actually minimize the space requirements by putting all the base systems on the same LV - then this LV needs enough space for all the base systems combined, plus the largest package build you'll do. Allow 200-250 MiB per base chroot, so if you want to support Debian stable, testing and unstable and Ubuntu LTS, current and current+1, you'll need 1.5 GiB plus the space for the build - double that if you want x86-64.

However, if you have a build environment which is basically the same as another, you can share the base system using some schroot hooks I describe later. For instance, I don't have a base for Debian experimental - I just construct it as needed, by upgrading from unstable.

Some rather outdated statistics: on Collabora's x86-64 build box, the base chroots at one point occupied 1.6 GiB of a 5 GiB LV, with the following sizes (just after an apt-get clean):

Debian etch i386: 213M
Debian etch x86-64: 216M
Debian sid i386: 216M
Debian sid x86-64: 221M
Ubuntu dapper i386: 207M
Ubuntu edgy i386: 196M
Ubuntu edgy x86-64: 154M

On my laptop I originally allocated 3 GiB for the base system, which should in practice be plenty for an i386-only environment. I now use 5 GiB, since I have plenty of disk.

You'll also need some free space in the same volume group - the maximum you can theoretically need is the free space on your base LV, plus the size of the largest base system, all multiplied by the number of simultaneous builds you'll do. This is to allocate snapshot LVs in. In practice you can get away with rather less than that.

Setting up the logical volumes

Having decided how much space you want in the base, set up an LV for your base system (here I use 5 GiB in the volume group "vg", naming the resulting LV "/dev/vg/schroot"):

sudo lvcreate -L5G -nschroot vg

format it:

sudo mkfs.ext3 -Lschroot /dev/vg/schroot

and temporarily mount it somewhere so you can create the base systems:

sudo mount /dev/vg/schroot /mnt

Setting up Debian base systems with `cdebootstrap`

cdebootstrap is part of the Debian installer. Replace MIRROR with your local Debian mirror.

For use on a laptop with potentially wobbly networking, you probably want to cache installed packages with a local instance of approx, in which case replace MIRROR with localhost:9999.

If you have a Debian mirror on your LAN, giving it the DNS alias mirror is extremely convenient.

cd /mnt
sudo cdebootstrap --flavour=build lenny lenny http://MIRROR/debian
sudo cdebootstrap --flavour=build squeeze squeeze http://MIRROR/debian
sudo cdebootstrap --flavour=build sid sid http://MIRROR/debian

Very basic chroot configuration

At this point you want to set up the absolute basics in each schroot: /etc/resolv.conf, /etc/hosts and possibly /etc/sudoers.

In distributions where Recommends are installed by default, you probably also want to put this in /etc/apt/apt.conf:

APT::Install-Recommends "false";

If you'll be using the chroot for development rather than just builds, you'll probably want to add a deb-src line to /etc/apt/sources.list.

After that, unmount /mnt before continuing.

Configuring `schroot`

Here's what to put in /etc/schroot/chroot.d/NAME for each chroot:

[sid]
# Optional. The chroot with alias 'default' is used if you just type
# 'schroot' without the -c option.
aliases=unstable,default
# Whatever you like
description=Debian sid
# Relative to what was /mnt until a moment ago
location=/sid
# Below here is standard for all the chroots
# Adjust according to the space available and size of builds you'll do
lvm-snapshot-options=--size 2G
device=/dev/vg/schroot
type=lvm-snapshot
priority=3
groups=sbuild,root
root-groups=sbuild,root
source-groups=sbuild,root
source-root-groups=root
run-setup-scripts=true
run-exec-scripts=true
# if you used XFS, you'll also need:
#mount-options=-o nouuid

If you want more than one chroot sharing a base system, make the name in square brackets different, but keep the location the same. For instance, my experimental chroot is headed [experimental] but uses location=/sid.

If most of your builds will be quite small, but you occasionally do one that needs more space, it may be worthwhile setting the snapshot size to be quite small, but then adding chroots like [sid-large] that use a larger snapshot for the same base system.

The schroot.conf from my laptop 'carbon' is available as an example.

Adding hooks

Put this in /etc/schroot/setup.d/60append-apt-sources:

#!/bin/sh
# /etc/schroot/setup.d/60append-apt-sources

if [ $1 = "setup-start" ] || [ $1 = "setup-recover" ]; then

  NAME=$(echo "${CHROOT_NAME}" | sed -e 's/-[a-z0-9][a-z0-9][a-z0-9][a-z0-9][a-z0-9][a-z0-9][a-z0-9][a-z0-9]-[a-z0-9][a-z0-9][a-z0-9][a-z0-9]-[a-z0-9][a-z0-9][a-z0-9][a-z0-9]-[a-z0-9][a-z0-9][a-z0-9][a-z0-9]-[a-z0-9][a-z0-9][a-z0-9][a-z0-9][a-z0-9][a-z0-9][a-z0-9][a-z0-9][a-z0-9][a-z0-9][a-z0-9][a-z0-9]//g')

  EXTRA_APT_SOURCES="/etc/schroot/sources.list.d/${NAME}.sources.list"
  APT_PREFS="/etc/schroot/sources.list.d/${NAME}.preferences"

  if [ "$AUTH_VERBOSITY" = "verbose" ]; then
    echo "Checking for auxiliary apt sources in $EXTRA_APT_SOURCES" >&2
  fi
  if [ -e "$EXTRA_APT_SOURCES" ]; then
    if [ "$AUTH_VERBOSITY" = "verbose" ]; then
      echo "... extra apt sources found" >&2
    fi
    cat "$EXTRA_APT_SOURCES" >> "${CHROOT_PATH}/etc/apt/sources.list"
  fi

  if [ "$AUTH_VERBOSITY" = "verbose" ]; then
    echo "Checking for apt preferences in $APT_PREFS" >&2
  fi
  if [ -e "$APT_PREFS" ]; then
    if [ "$AUTH_VERBOSITY" = "verbose" ]; then
      echo "... apt preferences found" >&2
    fi
    install -m644 "$APT_PREFS" "${CHROOT_PATH}/etc/apt/preferences"
  fi

fi

and this in /etc/schroot/setup.d/80apt-get-update:

#!/bin/sh
# /etc/schroot/setup.d/80apt-get-update

if [ $1 = "setup-start" ]; then
    if : || [ "$AUTH_VERBOSITY" = "verbose" ]; then
        chroot "${CHROOT_PATH}" apt-get update >&2 || true
    else
        chroot "${CHROOT_PATH}" apt-get update >/dev/null || true
    fi
fi

chmod them both to be executable.

Using the sources.list hook

Create a directory /etc/schroot/sources.list.d and place files in it named after a chroot, with .sources.list appended. For instance, in /etc/schroot/sources.list.d/experimental.sources.list write:

# /etc/schroot/sources.list.d/experimental.sources.list
deb http://MIRROR/debian experimental main
deb-src http://MIRROR/debian experimental main

with MIRROR replaced as above. Now that sources.list fragment will be appended to /etc/apt/sources.list in [experimental] chroots only. apt's default pinning behaviour means the experimental packages will only be used where selected by a versioned dependency.

If you want to force in experimental versions of particular packages, you can also do so with a file like this:

# /etc/schroot/sources.list.d/experimental.preferences
Package: libdbus-glib-1-dev
Pin: release a=experimental
Pin-Priority: 990

Package: libdbus-glib-1
Pin: release a=experimental
Pin-Priority: 990

Entering the source chroots

You can enter the source chroots with a command like:

schroot -c sid-source

Typically you'll want to do this as root, to perform updates or set up configuration. Some things you probably want to do on each chroot to start with:

outside% sudo schroot -c sid-source
inside# apt-get upgrade
inside# apt-get install devscripts vim-tiny sudo fakeroot aptitude \
        build-essential
inside# apt-get clean

Changes made here will be reflected in all subsequent snapshots.

For Sarge, or if you want a more capable editor in the chroot, install vim instead of vim-tiny.

Entering a snapshot

Enter a snapshot with a command like:

schroot -c sid

Any changes you make will be lost after the snapshot is closed.

Create a persistent snapshot with:

schroot --begin-session -c sid > my-sid-session-id

You can then resume it with:

schroot --run-session -c `cat my-sid-session-id`

and end it with:

schroot --end-session -c `cat my-sid-session-id`

Using sbuild

You'll need a bit of extra setup for sbuild: Using sbuild as a Debian maintainer describes how I use it.

Build a package in a single-use snapshot with:

sbuild foo-1.2.3.dsc

or to override the distribution in the .dsc file,

sbuild -c experimental foo-1.2.3.dsc