Using and contributing to Debian's Rebootstrap

Davor Ocelic, docelic@spinlocksolutions.com, Oct 2019

Rebootstrap is a project aimed at automating Debian bootstrapping for new architectures, or simply to rebuild all packages for an existing architecture from scratch.
Before rebootstrap, this was typically done only once for a particular architecture, and from there packages were rebuilt/upgraded individually rather than rebuilt en-masse.

Rebootstrap's homepage is https://wiki.debian.org/HelmutGrohne/rebootstrap.
Automated builds are at https://jenkins.debian.net/view/rebootstrap. The system is not always up to date and not all builds are always running.
Configurations for Jenkins are on https://salsa.debian.org/qa/jenkins.debian.net.
Valuable and almost mandatory resource is also IRC channel #debian-bootstrap @ irc.OFTC.net.

If you want to help with a new architecture, first make sure that an architecture very similar to it already works. If it doesn't, work on that architecture first.
(For example, before new architecture musl-linux-amd64 would be added, problems present in existing musl-linux-arm64 should be solved.)

Also, if trying to start with something, try gcc9 variants as they may be simpler.

Running bootstrap.sh

The main (and only) rebootstrap's script is called bootstrap.sh.

The idea is that bootstrap.sh is to be copied into any container and started from there. It will do everything on its own to set up the environment and start chosen build.
Also, it is intended that the container in which bootstrap.sh ran (chroot or other) is to be disposed after each build.
Therefore, bootstrap.sh should never be executed on the host system as it modifies system files. Run it inside chroot or any other containment.

bootstrap.sh runs on Debian. Therefore, assuming chroot option, you can create a Debian chroot with: 

# This will give you a clean chroot template in directory ./chroot-template/
# (stretch==9, buster==10):
wget https://ftp-master.debian.org/keys/release-9.asc -qO- > ./debian-9.key
debootstrap --keyring=./debian-9.key stretch chroot-template http://ftp.debian.org/debian

And you can run the build "manually" with: 

cp -a ./chroot-template ./chroot
cp bootstrap.sh ./chroot/
chroot ./chroot
./bootstrap.sh

However, since builds take a long time (especially as we fix more and more packages and so builds run longer and longer before failing), you want to do a couple things to improve its speed.

  1. Prepare a clean chroot which you will always just copy to create a new chroot, rather than e.g. run 'debootstrap' every time.
    Assuming you've already created a clean chroot template as shown above, a viable script which removes old chroot, makes a new one, and runs build can be as follows:

    First a script to run bootstrap.sh once inside chroot: 
    #!/bin/bash
# Call this bootstrap-run.sh

export DEB_BUILD_OPTIONS="nocheck noddebs parallel=16"
./bootstrap.sh HOST_ARCH=musl-linux-arm64 GCC_VER=9
    Then a script to actually do the whole thing (destroy/create chroot, run build): 
    #!/bin/bash
# Call this run.sh

cp chroot/var/cache/apt/archives/*.deb chroot-template/var/cache/apt/
mkdir -p chroot-template/tmp/repo
cp -a chroot/tmp/repo/* chroot-template/tmp/repo/

rm -rf chroot
cp -a chroot-template chroot
cp bootstrap.sh bootstrap-run.sh chroot/

echo START
echo DATETIME `date`
chroot ./chroot ./bootstrap-run.sh 2>&1 | tee build.log
echo DATETIME `date`
echo END
  2. Once bootstrap.sh is ran inside chroot, it will go downloading a bunch of Debian binary packages. Therefore, to save on download time, you can periodically copy /var/cache/apt/archives/*deb from the disposable chroot into your chroot template, so that the packages are already downloaded when bootstrap.sh goes to install them. (This is a basic approach though, better options include using apt-cacher-ng or squid-deb-proxy.)
  3. By default, built packages are placed in /tmp/repo/. You can also periodically copy /tmp/repo/* contents from the disposable chroot into the chroot template. NOTE: sometimes this works, sometimes it'll break.

Producing patches

Rebootstrap will only consider patches which can be upstreamed. (Upstream being either Debian packaging, or upstream source code.)

If/when a build fails, you will have the build log. Note that the build log should typically be read from the bottom to top in search of error. However, note also that often times the 'configure' log is printed after the actual error. Therefore, when looking from the bottom up, you should first reverse-search to "config.log" and then look for errors above it.

After you determine what the error is, you can cd to /tmp/buildd/*/ to find the unpacked sources which caused the error. What you want to do then is modify/fix the source and then somehow produce the diff.

To do so, you can e.g. fix the sources, increase package version (dch -i, from devscripts package), commit changes (dpkg-source --commit), build the sources (debian/rules clean; dpkg-source -b ., OR dpkg-buildpackage -S -uc -us), then debdiff the .dsc files, and report a bug against a Debian package with attaching the debdiff. You can also just compare two source directories (one original, one patched) using diff -ru.

The bug should have tags: patch, user: helmutg@debian.org, usertags: rebootstrap, severity: normal, x-debbugs-cc: debian-cross@lists.debian.org.
The best way to do this is to use reportbug to submit a bug. During the bug text editing, the pseudoheaders will be listed at the top. Leave the first two as prepared by reportbug, and then modify others to look like this: 

Package: ...
Version: ...
Severity: normal
Tags: patch
User: helmutg@debian.org
Usertags: rebootstrap
X-Debbugs-Cc: debian-cross@lists.debian.org
(If the patch is suitable for upstream (original author), tag it with 'upstream' as well. If you have forgotten to do so, you can always do it later with command 'bts tag NNNNN + upstream'.)

After you exit the editor, before sending the bug, remember to choose 'a' and attach your patch file.

For a quick test of whether your patch is valid, you can open bootstrap.sh and e.g. if the package is openssl, you search for 'openssl', then add a function 'patch_openssl' just next to it. E.g.: 

patch_openssl() {
	echo "... description ..."
	drop_privs patch -p1 <<EOF
... RAW CONTENT OF YOUR DIFF/PATCH ...
EOF
}

add_automatic openssl
When adding local patches, remember that e.g. all '$' may need to be escaped and such.

Common issues

Missing profiles or defines

Sometimes, the only thing that is missing are some literal defines or build targets, such as in this case: 

--- a/Configurations/20-debian.conf	2019-10-04 22:03:40.000000000 +0000
+++ b/Configurations/20-debian.conf	2019-10-04 22:36:16.596000000 +0000
@@ -132,6 +130,24 @@
 		cflags => add("-DL_ENDIAN"),
 	},
 
+	"debian-musl-linux-arm64" => {
+		inherit_from => [ "linux-aarch64", "debian" ],
+	},
+	"debian-musl-linux-armhf" => {
+		inherit_from => [ "linux-armv4", "debian" ],
+	},
+	"debian-musl-linux-i386" => {
+		inherit_from => [ "linux-elf", "debian" ],
+	},
+	"debian-musl-linux-mips" => {
+		inherit_from => [ "linux-mips32", "debian" ],
+		cflags => add("-DB_ENDIAN"),
+	},
+	"debian-musl-linux-mipsel" => {
+		inherit_from => [ "linux-mips32", "debian" ],
+		cflags => add("-DL_ENDIAN"),
+	},
+
 	"debian-nios2" => {
 		inherit_from => [ "linux-generic32", "debian" ],
 	},

Gettext

One very typical issue are errors about gettext or missing .mo files. Older autotools versions assume GNU gettext and access its internal symbols for detecting supported gettext features/compatibility.
But e.g. musl libc implements full GNU gettext compatibility, but it does not have these internal symbols, so the checks fail.
This particular issue has been fixed in newer versions, so if this is the issue a package is having, you can solve it with increasing minimum versions to this: 

# From https://hadrons.org/~guillem/tmp/gettext-bump.patch
---
 configure.ac |    3 ++-
 1 file changed, 2 insertions(+), 1 deletion(-)

--- a/configure.ac
+++ b/configure.ac
@@ -37,7 +37,8 @@ AC_SYS_LARGEFILE
 AM_PROG_AR
 LT_INIT
 
-AM_GNU_GETTEXT_VERSION([0.18.2])
+AM_GNU_GETTEXT_REQUIRE_VERSION([0.19.8])
+AM_GNU_GETTEXT_VERSION([0.19.6])
 AM_GNU_GETTEXT([external])
 
 AC_ARG_ENABLE([debug],

Missing symbols

If the package has debian/*.symbols files (see man dpkg-gensymbols) and some symbols are reported as not found, you can use directives to exclude them, such as: 

diff -ru libevent-2.1.8-stable/debian/libevent-2.1-6.symbols libevent-2.1.8-stable,musl/debian/libevent-2.1-6.symbols
--- libevent-2.1.8-stable/debian/libevent-2.1-6.symbols 2017-07-31 16:13:49.000000000 +0000
+++ libevent-2.1.8-stable,musl/debian/libevent-2.1-6.symbols  2019-10-06 19:15:19.232000000 +0000
@@ -362,7 +362,7 @@
  event_set_mem_functions@Base 2.1.8-stable
  event_sock_err@Base 2.1.8-stable
  event_sock_warn@Base 2.1.8-stable
- event_strlcpy_@Base 2.1.8-stable
+ (arch=!musl-linux-any)event_strlcpy_@Base 2.1.8-stable
  event_warn@Base 2.1.8-stable
  event_warnx@Base 2.1.8-stable
  evhttp_accept_socket@Base 2.1.8-stable
diff -ru libevent-2.1.8-stable/debian/libevent-core-2.1-6.symbols libevent-2.1.8-stable,musl/debian/libevent-core-2.1-6.symbols
--- libevent-2.1.8-stable/debian/libevent-core-2.1-6.symbols  2017-07-31 16:13:49.000000000 +0000
+++ libevent-2.1.8-stable,musl/debian/libevent-core-2.1-6.symbols 2019-10-06 19:15:25.912000000 +0000
@@ -306,7 +306,7 @@
  event_set_mem_functions@Base 2.1.8-stable
  event_sock_err@Base 2.1.8-stable
  event_sock_warn@Base 2.1.8-stable
- event_strlcpy_@Base 2.1.8-stable
+ (arch=!musl-linux-any)event_strlcpy_@Base 2.1.8-stable
  event_warn@Base 2.1.8-stable
  event_warnx@Base 2.1.8-stable
  evmap_check_integrity_@Base 2.1.8-stable

__BEGIN_DECLS / __END_DECLS

If this macro is not defined and causes a syntax error, you can use libbsd (preferrably in overlay mode), or if this happens in one or two places only, you can replace it literally with e.g.: 

Index: tcp_wrappers_7.6/tcpd.h
===================================================================
--- tcp_wrappers_7.6.orig/tcpd.h
+++ tcp_wrappers_7.6/tcpd.h
@@ -11,7 +11,9 @@
 #include 
 #include 
 
-__BEGIN_DECLS
+#ifdef __cplusplus
+extern "C" {
+#endif
 
 /* Structure to describe one communications endpoint. */
 
@@ -252,6 +254,8 @@ extern char *fix_strtok();
 extern char *my_strtok();
 #endif
 
-__END_DECLS
+#ifdef __cplusplus
+}
+#endif

Random list of open issues