Topics

Boot-strap [was Re: [draws and udrc] Download Issues]

Stuart Longland VK4MSL
 

On 3/11/19 3:57 pm, Basil Gunn wrote:
Now that standard Raspbian supports the UDRC/DRAWS, I think things
should improve since NWDR's servers only have to carry a small handful
of packages and not a whopping great big SD card image.
If you want a whopping great big SD card image then use the "W3DJS
Raspberry Pi Ham Radio Image v2.0". It's 17GB and has a lot stuff.
https://groups.io/g/RaspberryPi-4-HamRadio/topic/w3djs_raspberry_pi_ham_radio/39671852?p=,,,20,0,0,0::recentpostdate%2Fsticky,,,20,2,0,39671852
Note that they solved their slow download problem by using bitTorrent.

Our image is much smaller @ 2GB. List of NWDR Installed Programs:
https://github.com/nwdigitalradio/n7nix/blob/master/docs/IMAGE_README.md
Heh, I guess it's all relative. :-)

I'm used to a minimal image that basically just gets the machine booted
with all the hardware usable.

I suppose in UDRC/DRAWS terms, that'd be the necessary drivers loaded
and some scripts configured that at least set sane mixer levels. To
that, you add whatever packet or digital mode software, desktop
environments, etc.

Raspbian Lite weighs in at around 2GB uncompressed. Images I've made¹
for TS-7670 industrial computers (basically the result of `debootstrap`)
would basically install about 200-300MB worth of data on the root fs.

I recognise the standard NWDR image is basically aiming to be a complete
radio station out-of-the-box, so somewhat different rationale there.

Coding up a boot-strap script which can be run on stock Raspbian images
to set everything up would not be a difficult job.
I use a (unsupported) script to build and compress the published image. There is even
some documentation on how to run it. Search for "how the NWDR images are
created" in the markdown files in the docs directory.

The script runs for about 5 to 6 hours, IF there are no problems. It
usually takes 1 1/2 to 2 days to verify the image. I would rather do
other things than spin up boot images so looking forward to seeing your
boot-strap script.
Right, one thing I've observed is the Raspberry Pi 3, as powerful as it
is, isn't famous for its ability as a compiler work horse, mine at least
gripes about temperature. Also power supply fluctuations if I use the
standard Raspberry Pi PSU; the one on the UDRC is much better. However,
this doesn't get around other constraints on the Pi.

So my thinking on this is to split the problem up:

1. make a set of scripts that will generate a build environment that
will run on a conventional x86-64 Linux host and produce Debian packages
that will be installable on the Raspberry Pi

2. build each of the packages to be installed on a Pi+UDRC/DRAWS as a
separate Debian package and arrange these into an APT repository

3. create a script which adds the aforementioned APT repository, asks
some questions of the user as to what their intended use case is, then
installs package sets which achieve those aims.
https://wiki.debian.org/tasksel might be a way to achieve this. This
script can also do some last-minute set-up, but should not need to do
any heavy compilation.

Step (1) I've more-or-less already managed. Docker make available
standard Debian images for ARMv7 platforms, and under Linux, one can
combine QEMU user emulation and binfmt-misc to allow execution of ARMHF
binaries on an x86-64 host.

You need a statically linked version of `qemu-arm`; on Debian
derivatives, `apt-get install -y qemu-user-static` will have you sorted.
The other puzzle piece is `binfmt-support`; which can be installed with
`apt-get install -y binfmt-support`.

On my Gentoo host, I was able to get `qemu-arm` by compiling the
app-emulation/qemu package with USE=static and QEMU_USER_TARGETS=arm.
(Actually, I think I might've either nicked Debian's binary or built it
myself. I can't remember now. I have a binary in /opt/qemu/usr/bin.)

`binfmt-support` was found here:
https://github.com/qemu/qemu/blob/master/scripts/qemu-binfmt-conf.sh

The attached script was then able to create a Docker environment for
building the Raspberry Pi packages:

$ DEBIAN_MAIN_MIRROR=http://mirror.internode.on.net/pub/debian \
QEMU_BIN=/opt/qemu-static/usr/bin/qemu-arm-static \
sh mkbuildenv.sh

That'll create a docker image (nwdr/draws-buildimg) which we can use to
compile what we need. For example, building direwolf:

$ mkdir /tmp/work
$ docker run -v /tmp/work:/tmp/work --rm -ti nwdr/draws-buildimg bash
root@c7f1747ae8eb:/# cd /tmp/work
root@c7f1747ae8eb:/tmp/work# apt-get build-dep -y direwolf
Reading package lists... Done
Building dependency tree
Reading state information... Done
The following NEW packages will be installed:
dh-systemd libbluetooth3 libdbus-1-3 libgps-dev libgps23 pkg-config

root@c7f1747ae8eb:/tmp/work# apt-get source -y direwolf
Reading package lists... Done

root@c7f1747ae8eb:/tmp/work# ls
direwolf-1.4+dfsg direwolf_1.4+dfsg-1.debian.tar.xz
direwolf_1.4+dfsg-1.dsc direwolf_1.4+dfsg.orig.tar.gz
root@c7f1747ae8eb:/tmp/work# wget -O direwolf_1.5.orig.tar.gz
https://github.com/wb2osz/direwolf/archive/1.5.tar.gz
--2019-11-15 10:43:18--
https://github.com/wb2osz/direwolf/archive/1.5.tar.gz
Resolving github.com (github.com)... 13.236.229.21
Connecting to github.com (github.com)|13.236.229.21|:443... connected.
HTTP request sent, awaiting response... 302 Found
Location: https://codeload.github.com/wb2osz/direwolf/tar.gz/1.5 [following]
--2019-11-15 10:43:19--
https://codeload.github.com/wb2osz/direwolf/tar.gz/1.5
Resolving codeload.github.com (codeload.github.com)... 52.63.100.255
Connecting to codeload.github.com
(codeload.github.com)|52.63.100.255|:443... connected.
HTTP request sent, awaiting response... 200 OK
Length: unspecified [application/x-gzip]
Saving to: 'direwolf_1.5.orig.tar.gz'

2019-11-15 10:43:33 (1.29 MB/s) - 'direwolf_1.5.orig.tar.gz' saved
[17996557]

root@c7f1747ae8eb:/tmp/work# tar -xzvf direwolf_1.5.orig.tar.gz
direwolf-1.5/
direwolf-1.5/.gitattributes

direwolf-1.5/xmit.h
root@c7f1747ae8eb:/tmp/work# cp -a direwolf-1.4+dfsg/debian direwolf-1.5
root@c7f1747ae8eb:/tmp/work# cd direwolf-1.5
root@c7f1747ae8eb:/tmp/work/direwolf-1.5# rm -fr debian/patches
-- at this point I should update debian/changelog too ! --
root@c7f1747ae8eb:/tmp/work/direwolf-1.5# dpkg-buildpackage -us -uc -b
dpkg-buildpackage: info: source package direwolf
dpkg-buildpackage: info: source version 1.4+dfsg-1
dpkg-buildpackage: info: source distribution unstable
dpkg-buildpackage: info: source changed by Iain R. Learmonth
<irl@...>
dpkg-buildpackage: info: host architecture armhf
dpkg-source --before-build .
debian/rules clean
dh clean --with=systemd
debian/rules override_dh_auto_clean
make[1]: Entering directory '/tmp/work/direwolf-1.5'
make -f Makefile.linux clean
make[2]: Entering directory '/tmp/work/direwolf-1.5'
rm -f direwolf decode_aprs text2tt tt2text ll2utm utm2ll aclients atest
log2gpx gen_packets ttcalc kissutil cm108 gen_fff tune.h
fsk_fast_filter.h *.o *.a direwolf.desktop
make[2]: Leaving directory '/tmp/work/direwolf-1.5'

make[1]: Leaving directory '/tmp/work/direwolf-1.5'
dh_systemd_start
dh_perl
dh_link
dh_strip_nondeterminism
dh_compress
debian/rules override_dh_fixperms-arch
make[1]: Entering directory '/tmp/work/direwolf-1.5'
dh_fixperms
chmod 644 debian/direwolf/usr/share/direwolf/dw-icon.png
make[1]: Leaving directory '/tmp/work/direwolf-1.5'
dh_missing
dh_strip
dh_makeshlibs
dh_shlibdeps
dh_installdeb
dh_gencontrol
dh_md5sums
dh_builddeb
dpkg-deb: building package 'direwolf-dbgsym' in
'../direwolf-dbgsym_1.4+dfsg-1_armhf.deb'.
dpkg-deb: building package 'direwolf' in '../direwolf_1.4+dfsg-1_armhf.deb'.
dpkg-genbuildinfo --build=binary
dpkg-genchanges --build=binary >../direwolf_1.4+dfsg-1_armhf.changes
dpkg-genchanges: info: binary-only upload (no source code included)
dpkg-source --after-build .
dpkg-source: info: unapplying fix_documentation_path.patch
dpkg-buildpackage: info: binary-only upload (no source included)
root@c7f1747ae8eb:/tmp/work/direwolf-1.5# chown -R 1000:1000 /tmp/work
root@c7f1747ae8eb:/tmp/work/direwolf-1.5# exit
$ ls -l /tmp/work/
total 19828
drwxr-xr-x 10 stuartl stuartl 8192 Nov 15 20:41 direwolf-1.4+dfsg/
-rw-r--r-- 1 stuartl stuartl 4570 Nov 15 21:03
direwolf_1.4+dfsg-1_armhf.buildinfo
-rw-r--r-- 1 stuartl stuartl 2143 Nov 15 21:04
direwolf_1.4+dfsg-1_armhf.changes
-rw-r--r-- 1 stuartl stuartl 336804 Nov 15 21:03
direwolf_1.4+dfsg-1_armhf.deb
-rw-r--r-- 1 stuartl stuartl 8732 Sep 15 2017
direwolf_1.4+dfsg-1.debian.tar.xz
-rw-r--r-- 1 stuartl stuartl 1743 Sep 15 2017 direwolf_1.4+dfsg-1.dsc
-rw-r--r-- 1 stuartl stuartl 801382 Sep 15 2017
direwolf_1.4+dfsg.orig.tar.gz
drwxrwxr-x 9 stuartl stuartl 8192 Nov 15 21:04 direwolf-1.5/
-rw-r--r-- 1 stuartl stuartl 17996557 Nov 15 20:43 direwolf_1.5.orig.tar.gz
-rw-r--r-- 1 stuartl stuartl 1108088 Nov 15 21:03
direwolf-dbgsym_1.4+dfsg-1_armhf.deb

The test cases took a little while but the nice thing about this is you
can have several packages building in parallel on a modern system.

There's a few rough edges, but I think that could be worked around to
build the packages more-or-less autonomously… maybe via some CI system
like Jenkins, Atlassian Bamboo or similar… and the resulting binaries
uploaded to a repository.

That would reduce the amount of work needed at the actual boot-strap end
quite a bit.

Regards,
--
Stuart Longland (aka Redhatter, VK4MSL)

I haven't lost my mind...
...it's backed up on a tape somewhere.

1. https://bne.vrt.com.au/technologicsys/ -- this is armel rather than
armhf, but the same principles apply