Showing posts with label embedded. Show all posts
Showing posts with label embedded. Show all posts

Friday, 28 August 2009

Keeping old branches building when the world has moved on

I recently upgraded the system used to build our software to Debian GNU/Linux x86-64. The main software is cross-compiled for an embedded target but there are various host programs and unit tests that are compiled natively. Parts of the software are quite old and don't compile successfully for 64-bit. Luckily -m32 mostly solved this problem so everything keeps on compiling. The cross compiler is compiled as 64-bit so it can take advantage of the extra registers to hopefully improve performance.

Unfortunately -m32 didn't solve everything. The software required many libraries and not all of these were included in the various 32-bit library compatibility packages provided by Debian. The best solution was to modify the build system to build these libraries too. This makes us less reliant on the operating system versions and let me use the same versions as we do with buildroot for the target which reaped other benefits too. One day I'd like to move to a world where the only part of the host system that is used is the compiler to compile own host and cross compilers.

Eventually the trunk is all fine and everything builds happily. But what about older branches? The changes were quite invasive and applying them to supposedly stable branches didn't give me a warm fuzzy feeling. The logical solution was to build them in an entire 32-bit environment. I considered using something like VirtualBox or QEMU but using a completely separate host would have caused confusion for all the developers.

In the end I discovered schroot and followed the instructions. It was very quick and easy to set up. I added various bind mounts to /etc/fstab to ensure that /home was mounted and it was then easy to set a 32-bit environment build going with just a single command. The instructions in the link are clear so I won't go over them again. I made use of linux32 to ensure that uname gave the expected result.

But, I thought it could be even simpler.

Each branch knows what build environments it supports. This can easily be indicated via a file in the top level directory. If the file doesn't exist then some sensible defaults can be used. Once this information exists it is easy to write a script which reads this, checks against the current native build environment and either runs the build directly or inside the chroot. Something like this:
 top=.
while [ ! / -ef "${top}" -a ! -f "${top}/Make.rules" ]; do
top=${top}/..
done

if [ / -ef "${top}" ]; then
echo "Not in source tree."
exit 1
fi

# Defaults
HOST_BUILD_ENVIRONMENTS=i686

if [ -f "${top}/BuildInfo" ]; then
source "${top}/BuildInfo"
fi

this_env="`uname -m`"
use_chroot=
for e in ${HOST_BUILD_ENVIRONMENTS}
if [ "${e}" = "{this_env}" ]; then
use_chroot=
break
elif [ "${e} = "i686" ]; then
use_chroot="$chroot_i686"
fi
done

if [ -n "${use_chroot}" ]; then
echo "Compiling in ${use_chroot} chroot"
schroot -p -q -c ${use_chroot} -d `pwd` -- linux32 make "$@"; then
else
make "$@"
fi

The actual script is more complex than this because it contains better error handling. It also deals with make -C dir and deciding whether -jN can be passed too.

This can easily be extended to support building under different chroots for older operating system versions for example.

Thursday, 30 April 2009

FAT alternative

An interesting opinion has appeared in relation to TomTom's settlement with Microsoft regarding the viability of file systems other than FAT.
Jim Zemlin wrote in his response to the settlement:

The technology at the heart of this settlement is the FAT filesystem. As acknowledged by Microsoft in the press release, this file system is easily replaced with multiple technology alternatives.

This was also quoted again by Groklaw in their article. But is this actually true? The major benefit that FAT brings that other file systems do not is its ubiquity. It's supported without the need to install third party code on Windows, MacOSX and Linux along with countless other devices and operating systems. No other single file system has such cross platform support.
If you're developing an embedded device with internal storage (e.g. a PTP camera or MTP media player with built-in flash memory) then you can get away with using whichever file system you like (and I've worked on products which used ext2 and reiserfs in this situation.) Unfortunately as soon as you start using removable storage or need to make your built-in storage available as a block device over USB mass storage class or similar then you need to be interoperable. Being interoperable means using FAT or if you are very lucky and you have a knowledgeable user base a file system such as ext3 which can be supported on Windows and MacOSX with a little work.

FAT's simplicity makes it even more useful for interoperability. It's lack of support for ownership and ACLs means that you can just plug a USB key in and start reading and writing to it immediately. A more advanced file system such as ext3 just gets in the way if your UID doesn't match between different machines or you give the key to someone else. This problem is less of a worry for an embedded device which may just run as root anyway or can be hacked to work around this. On the desktop there may be a case here for supporting mounts in a special “ignore permissions” or “everybody squash” mode to solve this problem.

This topic has become important to be because recently I've been looking into alternatives to FAT for a completely different reason: resilience. FAT, and in particular FAT as implemented on Linux, is highly prone to corruption if someone pulls the USB key or the power cable. Other file systems such as ext3 are much more capable of dealing with this.

Wednesday, 21 January 2009

NFS mount yields: RPC: failed to contact portmap

I do most of my embedded software development whilst running from an NFS mounted root directory. I was therefore rather confused when I was unable to mount a different path on the same server. The following just appeared very slowly:

~# mount server:/path /mnt
portmap: server localhost not responding, timed out
RPC: failed to contact portmap (errno -5).
portmap: server localhost not responding, timed out
RPC: failed to contact portmap (errno -5).
lockd_up: makesock failed, error=-5
portmap: server localhost not responding, timed out
RPC: failed to contact portmap (errno -5).
mount: Mounting server:/path on /mnt failed: Input/output error

The cause is simple. My device isn't running a port mapper daemon. But why should I run such a daemon? The kernel can mount my root filesystem without one!

In order to mount the filesystem the kernel is trying to talk to the local port mapper daemon in order to find the local lock daemon - I don't have one of those either. The problem can easily be fixed by passing “-o nolock”

~# mount -o nolock server:/path /mnt

Friday, 14 November 2008

Why you shouldn't use the tool chain supplied by your embedded Linux vendor

Linux is understandably popular as an operating system for embedded systems. SoC vendors in particular like to supply a Linux "distribution" that is tailored for their platform. This "distribution" usually includes the binaries for an appropriate toolchain. In my opinion you should stop using this toolchain long before you are ready to release your product.

The binary toolchain supplied by your vendor is easy, it might even be statically linked making it even easier to use. Just plonk it in the right place and everything magically works.

But,

1. One day you're going to have a bug. That bug will either require some debugging in files that are supplied as part of the toolchain (ld.so, glibc) or require you to make some modifications to the toolchain in order to help investigate the problem. You might even find a toolchain bug that you need to fix. This sort of problem is almost guaranteed to occur at a point when it is deemed too dangerous to switch to a self-compiled tool chain. That's if your lucky and you have the source code for the tool chain and it actually compiles for you.

2. In order to comply with the GPL you need to release working source code for certain bits of the tool chain anyway. How can you be sure that this stuff actually compiles unless you have done so yourself?

3. You need to support your product long after your vendor has moved on to another generation of chips. Will their toolchain still work on whatever host operating system you are using then?

So, at the very least you should get the source code for the toolchain from your vendor and then compile it yourself. Use the version you compiled yourself. This leaves you in a much better position when the unexpected occurs. If your vendor won't give you the source for a toolchain they've given you in binary form then find another vendor that understands software licensing.

Of course you could just compile your own tool chain from scratch and use that but creating cross-compilation toolchains is certainly not easy - perhaps that subject is worthy of a future post.

Tuesday, 5 August 2008

mipsel-linux-strip: Not enough room for program headers, try linking with -N

I was rather confused when I started getting this error when attempting to strip a Linux MIPS shared library when I moved to a new toolchain that used binutils-2.18:

BFD: st4lu6Am: Not enough room for program headers, try linking with -N
mipsel-linux-strip: st4lu6Am: Bad value

Other shared libraries could be successfully stripped.

The clue was that these shared libraries were generated with an earlier toolchain that used an older version of binutils.

It turns out that this is caused by binutils-2.18 wanting to add a NULL segment even when the binary didn't originally have one. Applying the fix makes the problem go away.

I only really mention this because the fix description doesn't contain the error message I saw thus making it hard to Google for a solution.

Friday, 18 April 2008

Cross-compiling boost 1.34.x and 1.35.0

There seem to be lots of people asking how to cross-compile boost and very few answers. One of the better answers works for v1.33.x but breaks with v1.34.0.

After digging around for a while trying to make it work I was finally given the answer by the esteemed Peter Hartley who had managed to cross-compile boost 1.34 as part of his Just The Linux distribution. His solution seems to work for v1.35.0 too.

The trick that had eluded me until that point was to tell both the user-config.jam file and bjam about the cross compiler.

Something like:
echo "using gcc : : nicearch-linux-g++ ;" > user-config.jam
make BJAM_CONFIG="-sGXX=nicearch-linux-g++" install
If, like me, you want to only generate static libraries and support multiple builds in the same tree then you might need a bit more cleverness:
build=/tmp/nicearch/build
staging=/tmp/nicearch/staging
CXX=nicearch-linux-g++
CC=nicearch-linux-gcc
mkdir -p $build $staging
echo "using gcc : : $CXX ;" > $build/user-config.jam
bjam --toolset=gcc -sGXX=$CXX -sGCC=$CC \
--prefix=$staging --build-dir=$build \
--user-config=$build/user-config.jam --without-python \
variant=release link=static threading=multi
This should be relatively easy to turn into a buildroot package file but I'm no longer using buildroot to build boost so I didn't need to.

Friday, 21 December 2007

DBUS: "Could not get password database information for UID of current process"

Recently I've been cross compiling DBUS and HAL to make them work on an embedded device with buildroot. It's been a bit of a journey with only sparse documentation available. There are lots of dependencies on things that often aren't present on embedded devices and it's not always easy to work out what they are. So, I'm going to try and document a few of the things I've come across here in the hope that someone finds it useful.

There are various reports of people getting this message but never any responses:

Starting system message bus: Could not get password database information for UID of current process: User "???" unknown or no memory to allocate password entry

This message appears despite /etc/passwd and /etc/group existing and looking correct.

The clue is in the message. It can't read this information because it is using glibc to do it. Glibc provides this information from various sources of which the passwd and group files are only one. It is necessary to enable the glibc back-end that reads the files (libnss_files) along with the library containing the generic functions (libnsl). This can be done in buildroot by enabling the BR2_GLIBC_NSL and BR2_GLIBC_NSS_FILES configuration options. Even better they can automatically be enabled when DBUS support is selected in buildroot by modifying the start of packages/dbus/Config.in to look something like:
config BR2_PACKAGE_DBUS
bool "dbus"
default n
select BR2_PACKAGE_EXPAT
select BR2_GLIBC_NSL
select BR2_GLIBC_NSS_FILES

I'm not sure why the lack of libnsl doesn't manifest itself as an error from ld.so - if it had then it would have been far more obvious what the cause was.