Building Debian packages with local dependencies

Written by Barry Warsaw in technology on Mon 28 November 2016. Tags: ubuntu, debian, packaging,

Way back in 2011 I wrote an article describing how you can build Debian packages with local dependencies for testing purposes. An example would be a new version of a package that has new dependencies. Or perhaps the new dependency isn't available in Debian yet. You'd like to test both packages together locally before uploading. Using sbuild and autopkgtest you can have a high degree of confidence about the quality of your packages before you upload them.

Here I'll describe some of the improvements in those tools, and give you simplified instructions on how to build and test packages with local dependencies.

What's changed?

Several things have changed. Probably the biggest thing that simplifies the procedure is that GPG keys for your local repository are no longer needed.

Another thing that's improved is the package testing support. It used to be that packages could only be tested during build time, but with the addition of the autopkgtest tool, we can also test the built packages under various scenarios. This is important because it more closely mimics what your package's users will see. One thing that's cool about this for Python packages is that autopkgtest runs an import test of your package by default, so even if you don't add any explicit tests, you still get something. Of course, if you do want to add your own tests, you'll need to recreate those default tests, or check out the autodep8 package for some helpers.

I've moved the repository of scripts over to git.

The way you specify the location of the extra repositories holding your local debs has changed. Now, instead of providing a directory on the local file system, we're going to fire up a simple Python-based HTTP server and use that as a new repository URL. This won't be completely automatic as you'll have to start and stop that server around your builds, but that's probably fine.

One-time setup

First a couple of setup steps, then later I'll provide the recipe for the builds. Please adjust the directories to your own taste, and be sure to edit the ~/ubuntu/repo/scan.sh file if you do.

$ cd ~/ubuntu
$ git clone https://git.launchpad.net/~barry/+git/repotools repo
$ mkdir debs
$ mkdir logs

Add the following to your ~/.sbuildrc file:

$external_commands = {
    "pre-build-commands" => [
        ['/home/barry/ubuntu/repo/scan.sh'],
    ],
    "chroot-setup-commands" => [
        ['/repo/prep.sh'],
    ],
};

Note that while you might have to change the path for the pre-build-commands script, the chroot-setup-commands path is correct for finding the script inside your chroot.

Add the following to your /etc/schroot/default/fstab file:

# Additions according to https://wiki.ubuntu.com/SimpleSbuild
/home/barry/ubuntu/scratch      /scratch        none rw,bind 0 0
/home/barry                     /home/barry     none rw,bind 0 0

# Expose local apt repository to the chroot.
/home/barry/ubuntu/repo         /repo           none rw,bind 0 0

If you're using apt-cacher-ng for more efficient local package downloads, you'll need to make a small adjustment to your proxy file. This is because you want localhost connections to the Python HTTP server we're going to start up to go directly through to the server and not your cache.

If this describes your setup, for each of your chroots living in /var/lib/schroot/chroots/, edit the file <chroot>/etc/apt/apt.conf.d/02proxy. The actual file may be different; it's up to you. my.local.cache names your local apt cache hostname.

Acquire::http { "Proxy::127.0.0.1" "DIRECT" }
Acquire::http { Proxy "http://my.local.cache:3142" }

Doing the build

Now we're ready to build the package with our local dependencies. I'll assume you've already built your local dependencies and have the .deb files for them laying around. Start by copying them to the ~/ubuntu/debs directory you created earlier.

Now, in a separate shell:

$ cd ~/ubuntu/debs
$ python3 -m http.server 8000 --bind 127.0.0.1

This starts a very simple HTTP server vending files from the debs directory over localhost. It's going to act like an actual Debian package server. You'll want to leave this server running until you're done with your builds. Then just hit control-C to kill it when you're done.

Now we're ready to build the source package. In this example, I put the source package in build-area/ so again, your directories may be different.

The trick here is that we're going to set up a special local repository that sbuild is going to use. (I'm wrapping the lines for readability.)

$ sbuild -d unstable \
  --extra-repository="deb [trusted=yes] http://127.0.0.1:8000 ./" \
  build-area/foo_9.0.1-1.dsc

That's it! As the build progresses, you should notice your Python HTTP server output the local dependency files the build downloads, as long as your local debs have a higher version number than what's in the platform archive.

autopkgtest

You can run autopkgtest against your new dependency, but I've found it better to build the binary packages first using sbuild, and then run autopkgtest over the built tree. Assuming the sbuild process left your debs (e.g. for the just built foo package) in the current working directory, you will also need your newer local dependencies. Copy them from ~/ubuntu/debs and then run:

$ autopkgtest -B *.deb build-area/foo_9.0.1-1.dsc -- schroot sid-amd64

seasoning to taste as necessary.

Clean up

Don't forget to kill the Python HTTP server you started, and remove the local debs from the directory. You don't want them polluting subsequent builds (though they will be ignored if the version numbers are lower than what's in the archive).


Comments

comments powered by Disqus