/sbin/init from buildroot hangs

Ed Blackmond
I have a problem and am in need of some suggestions on how to debug.  

I'm reviving an old ARM project and want to switch to buildroot because I
think it will help me keep things up to date going forward.  I have gone
through the buildroot configuration process, built the toolchain (binutils
2.19.1, gcc-4.3.4 uClibc-0.9.30.1).  As long as I compile with shared
libraries I can run them on my old system (2.6.13 kernel, some old version
of uClibc).  Statically linked files fail with an illegal instruction on
the old system.

I have built a new version using 2.6.23 using buildroot to do it for me.  I
can load this kernel using tftp from uboot and it initializes itself and
mounts the root filesystem from my server (I watched the mount request and
the kernel reports a successful mount of the root filesystem).  When it
tries to exec init (a link to busybox) it just hangs.  At this point I can
ping the system and characters typed on the serial console are echoed.  It
isn't completely dead.

I tried replacing /sbin/init with an executable that creates a file in the
root filesystem as well as print output to the console.  Neither happen.  I
can see the root filesystem from my server and no file is created.

I suspect a library incompatibility issue.  The illegal instruction seems
to be a big hint.  I configured the buildroot with generic_arm.

Any help or suggestions of how to instrument this to help track down the
problem would be greatly appreciated.

Thanks,
Ed

Ed Blackmond
This turned out to be incompatible OABI/EABI configuration between kernel
and libraries.