Monthly Archives: May 2018

Debugging the debugger

I use gdb quite often, but until recently I’ve never really needed to understand how it works or debug it before. I thought I’d document a recent issue I decided to take a look at – perhaps someone else will find it interesting or useful.

We run the rust testsuite when building rustc packages in Ubuntu. When preparing updates to rust 1.25 recently for Ubuntu 18.04 LTS, I hit a bunch of test failures on armhf which all looked very similar. Here’s an example test failure:

---- [debuginfo-gdb] debuginfo/borrowed-c-style-enum.rs stdout ----

NOTE: compiletest thinks it is using GDB with native rust support
executing "/<<BUILDDIR>>/rustc-1.25.0+dfsg1+llvm/build/armv7-unknown-linux-gnueabihf/stage2/bin/rustc" "/<<BUILDDIR>>/rustc-1.25.0+dfsg1+llvm/src/test/debuginfo/borrowed-c-style-enum.rs" "-L" "/<<BUILDDIR>>/rustc-1.25.0+dfsg1+llvm/build/armv7-unknown-linux-gnueabihf/test/debuginfo" "--target=armv7-unknown-linux-gnueabihf" "-C" "prefer-dynamic" "-o" "/<<BUILDDIR>>/rustc-1.25.0+dfsg1+llvm/build/armv7-unknown-linux-gnueabihf/test/debuginfo/borrowed-c-style-enum.stage2-armv7-unknown-linux-gnueabihf" "-Crpath" "-Zmiri" "-Zunstable-options" "-Lnative=/<<BUILDDIR>>/rustc-1.25.0+dfsg1+llvm/build/armv7-unknown-linux-gnueabihf/native/rust-test-helpers" "-g" "-L" "/<<BUILDDIR>>/rustc-1.25.0+dfsg1+llvm/build/armv7-unknown-linux-gnueabihf/test/debuginfo/borrowed-c-style-enum.stage2-armv7-unknown-linux-gnueabihf.gdb.aux"
------stdout------------------------------

------stderr------------------------------

------------------------------------------
NOTE: compiletest thinks it is using GDB version 8001000
executing "/usr/bin/gdb" "-quiet" "-batch" "-nx" "-command=/<<BUILDDIR>>/rustc-1.25.0+dfsg1+llvm/build/armv7-unknown-linux-gnueabihf/test/debuginfo/borrowed-c-style-enum.debugger.script"
------stdout------------------------------
GNU gdb (Ubuntu 8.1-0ubuntu3) 8.1.0.20180409-git
Copyright (C) 2018 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.  Type "show copying"
and "show warranty" for details.
This GDB was configured as "arm-linux-gnueabihf".
Type "show configuration" for configuration details.
For bug reporting instructions, please see:
<http://www.gnu.org/software/gdb/bugs/>.
Find the GDB manual and other documentation resources online at:
<http://www.gnu.org/software/gdb/documentation/>.
For help, type "help".
Type "apropos word" to search for commands related to "word".
Breakpoint 1 at 0xcc4: file /<<BUILDDIR>>/rustc-1.25.0+dfsg1+llvm/src/test/debuginfo/borrowed-c-style-enum.rs, line 61.

Program received signal SIGSEGV, Segmentation fault.
0xf77c9f4e in ?? () from /lib/ld-linux-armhf.so.3

------stderr------------------------------
/<<BUILDDIR>>/rustc-1.25.0+dfsg1+llvm/build/armv7-unknown-linux-gnueabihf/test/debuginfo/borrowed-c-style-enum.debugger.script:10: Error in sourced command file:
No symbol 'the_a_ref' in current context

------------------------------------------

error: line not found in debugger output: $1 = borrowed_c_style_enum::ABC::TheA
status: exit code: 0
command: "/usr/bin/gdb" "-quiet" "-batch" "-nx" "-command=/<<BUILDDIR>>/rustc-1.25.0+dfsg1+llvm/build/armv7-unknown-linux-gnueabihf/test/debuginfo/borrowed-c-style-enum.debugger.script"
stdout:
------------------------------------------
GNU gdb (Ubuntu 8.1-0ubuntu3) 8.1.0.20180409-git
Copyright (C) 2018 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.  Type "show copying"
and "show warranty" for details.
This GDB was configured as "arm-linux-gnueabihf".
Type "show configuration" for configuration details.
For bug reporting instructions, please see:
<http://www.gnu.org/software/gdb/bugs/>.
Find the GDB manual and other documentation resources online at:
<http://www.gnu.org/software/gdb/documentation/>.
For help, type "help".
Type "apropos word" to search for commands related to "word".
Breakpoint 1 at 0xcc4: file /<<BUILDDIR>>/rustc-1.25.0+dfsg1+llvm/src/test/debuginfo/borrowed-c-style-enum.rs, line 61.

Program received signal SIGSEGV, Segmentation fault.
0xf77c9f4e in ?? () from /lib/ld-linux-armhf.so.3

------------------------------------------
stderr:
------------------------------------------
/<<BUILDDIR>>/rustc-1.25.0+dfsg1+llvm/build/armv7-unknown-linux-gnueabihf/test/debuginfo/borrowed-c-style-enum.debugger.script:10: Error in sourced command file:
No symbol 'the_a_ref' in current context

------------------------------------------

thread '[debuginfo-gdb] debuginfo/borrowed-c-style-enum.rs' panicked at 'explicit panic', tools/compiletest/src/runtest.rs:2891:9
note: Run with `RUST_BACKTRACE=1` for a backtrace.

The failing tests are all running some commands in gdb, and the inferior (tracee) is crashing inside the dynamic loader (/lib/ld-linux-armhf.so.3) before running any rust code.

I managed to recreate this test failure on an armhf box, but when I installed the debug symbols for the dynamic loader (contained in the libc6-dbg package) so that I could attempt to debug these crashes, the failing tests all started to pass.

A quick search on the internet shows that I’m not the first person to hit this issue – for example, this bug reported in April 2016. According to the comments, the workaround is the same – installing the debug symbols for the dynamic loader (by installing the libc6-dbg package). This obviously isn’t right and I don’t particularly like walking away from something like this without understanding it, so I decided to spend some time trying to figure out what is going on.

This first thing I did was to load the missing debug symbols manually in gdb after hitting the crash, in order to hopefully get a useful backtrace:

$ gdb build/armv7-unknown-linux-gnueabihf/test/debuginfo/borrowed-c-style-enum.stage2-armv7-unknown-linux-gnueabihf
...
(gdb) run                                           
Starting program: /home/ubuntu/src/rustc/rustc-1.25.0+dfsg1+llvm/build/armv7-unknown-linux-gnueabihf/test/debuginfo/borrowed-c-style-enum.stage2-armv7-unknown-linux-gnueabihf

Program received signal SIGSEGV, Segmentation fault.
0xf77c9f4e in ?? () from /lib/ld-linux-armhf.so.3
(gdb) info sharedlibrary
From        To          Syms Read   Shared Object Library                                                
0xf77c7a40  0xf77dadd0  Yes (*)     /lib/ld-linux-armhf.so.3                                             
0xf771ce90  0xf778e288  No          /home/ubuntu/src/rustc/rustc-1.25.0+dfsg1+llvm/build/armv7-unknown-linux-gnueabihf/test/debuginfo/../../stage2/lib/rustlib/armv7-unknown-linux-gnueabihf/lib/libstd-42d13165275d0302.so
0xf76c91f0  0xf76d394c  No          /lib/arm-linux-gnueabihf/libgcc_s.so.1
0xf75dad80  0xf7687a90  No          /lib/arm-linux-gnueabihf/libc.so.6
0xf75b1a14  0xf75b2410  No          /lib/arm-linux-gnueabihf/libdl.so.2
0xf759c810  0xf759edf0  No          /lib/arm-linux-gnueabihf/librt.so.1
0xf757a210  0xf7585214  No          /lib/arm-linux-gnueabihf/libpthread.so.0
(*): Shared library is missing debugging information.
(gdb) add-symbol-file ~/libc6-syms/usr/lib/debug/lib/arm-linux-gnueabihf/ld-2.27.so 0xf77c7a40
add symbol table from file "/home/ubuntu/libc6-syms/usr/lib/debug/lib/arm-linux-gnueabihf/ld-2.27.so" at
        .text_addr = 0xf77c7a40
(y or n) y
Reading symbols from /home/ubuntu/libc6-syms/usr/lib/debug/lib/arm-linux-gnueabihf/ld-2.27.so...done.
(gdb) bt full
#0  dl_main (phdr=<optimized out>, phnum=<optimized out>, user_entry=<optimized out>, auxv=<optimized out>) at rtld.c:2275
        cnt = 1
        afct = 0x0
        head = <optimized out>
        ph = <optimized out>
        mode = <optimized out>
        main_map = <optimized out>
        file_size = 4294899100
        file = <optimized out>
        has_interp = <optimized out>
        i = <optimized out>
        prelinked = <optimized out>
        rtld_is_main = <optimized out>
        tcbp = <optimized out>
        __PRETTY_FUNCTION__ = <error reading variable __PRETTY_FUNCTION__ (Cannot access memory at address 0x15810)>
        first_preload = <optimized out>
        r = <optimized out>
        rtld_ehdr = <optimized out>
        rtld_phdr = <optimized out>
        cnt = <optimized out>
        need_security_init = <optimized out>
        count_modids = <optimized out>
        preloads = <optimized out>
        npreloads = <optimized out>
        preload_file = <error reading variable preload_file (Cannot access memory at address 0x157fc)>
        rtld_multiple_ref = <optimized out>
        was_tls_init_tp_called = <optimized out></details>
#1  0xf77d76d0 in _dl_sysdep_start (start_argptr=start_argptr@entry=0xfffef6b1, dl_main=0xf77c872d <dl_main>) at ../elf/dl-sysdep.c:253
        phdr = <optimized out>
        phnum = <optimized out>
        user_entry = 4197241
        av = <optimized out>
#2  0xf77c8260 in _dl_start_final (arg=0xfffef6b1) at rtld.c:414
        start_addr = <optimized out>
        start_addr = <optimized out>
#3  _dl_start (arg=0xfffef6b1) at rtld.c:521
        entry = <optimized out>
#4  0xf77c7b90 in ?? () from /lib/ld-linux-armhf.so.3
        library_path = <error reading variable library_path (Cannot access memory at address 0x28920)>
        version_info = <error reading variable version_info (Cannot access memory at address 0x28918)>
        any_debug = <error reading variable any_debug (Cannot access memory at address 0x28914)>
        _dl_rtld_libname = <error reading variable _dl_rtld_libname (Cannot access memory at address 0x298a8)>
        _dl_rtld_libname2 = <error reading variable _dl_rtld_libname2 (Cannot access memory at address 0x298b4)>
        tls_init_tp_called = <error reading variable tls_init_tp_called (Cannot access memory at address 0x29898)>
        audit_list = <error reading variable audit_list (Cannot access memory at address 0x298a4)>
        preloadlist = <error reading variable preloadlist (Cannot access memory at address 0x2891c)>
        _dl_skip_args = <error reading variable _dl_skip_args (Cannot access memory at address 0x2994c)>
        audit_list_string = <error reading variable audit_list_string (Cannot access memory at address 0x29968)>
        __stack_chk_guard = <error reading variable __stack_chk_guard (Cannot access memory at address 0x28968)>
        _rtld_global = <error reading variable _rtld_global (Cannot access memory at address 0x29060)>
        _rtld_global_ro = <error reading variable _rtld_global_ro (Cannot access memory at address 0x28970)>
        _dl_argc = <error reading variable _dl_argc (Cannot access memory at address 0x28910)>
        __GI__dl_argv = <error reading variable __GI__dl_argv (Cannot access memory at address 0x29894)>
        __pointer_chk_guard_local = <error reading variable __pointer_chk_guard_local (Cannot access memory at address 0x28964)>

You can grab the glibc source and see that the dynamic loader ends up here in elf/rtld.c:

if (__glibc_unlikely (GLRO(dl_naudit) > 0))
  {
    struct link_map *head = GL(dl_ns)[LM_ID_BASE]._ns_loaded;
    /* Do not call the functions for any auditing object.  */
    if (head->l_auditing == 0)
      {
        struct audit_ifaces *afct = GLRO(dl_audit);
        for (unsigned int cnt = 0; cnt < GLRO(dl_naudit); ++cnt)
          {
            if (afct->activity != NULL) // ##CRASHES HERE##
              afct->activity (&head->l_audit[cnt].cookie, LA_ACT_CONSISTENT);

            afct = afct->next;
          }
      }
  }

The reason for the crash is that afct is NULL:

(gdb) p $_siginfo
$1 = {si_signo = 11, si_errno = 0, si_code = 1, _sifields = {_pad = {0, 56, 19628232, 19628288, -156661788, 0, 80, 19811416, -156663808, -157316581, 104, 1073741824, 19811416, 96, 19811408, 80, -156661788, 14, 
      19551104, 96, 104, 13358248, 19551584, 19552160, 19552232, 19811488, 32, 128, 64}, _kill = {si_pid = 0, si_uid = 56}, _timer = {si_tid = 0, si_overrun = 56, si_sigval = {sival_int = 19628232, 
        sival_ptr = 0x12b80c8}}, _rt = {si_pid = 0, si_uid = 56, si_sigval = {sival_int = 19628232, sival_ptr = 0x12b80c8}}, _sigchld = {si_pid = 0, si_uid = 56, si_status = 19628232, si_utime = 19628288, 
      si_stime = -156661788}, _sigfault = {si_addr = 0x0}, _sigpoll = {si_band = 0, si_fd = 56}}}
(gdb) p afct
$2 = (struct audit_ifaces *) 0x0

A quick look through the dynamic loader code shows that this condition should be impossible to hit.

As the crash doesn’t happen with debug symbols, I thought I would attempt to debug it without the symbols. First of all, I set a breakpoint at the start of dl_main by specifying it at offset 0xcec in the .text section:

$ gdb build/armv7-unknown-linux-gnueabihf/test/debuginfo/borrowed-c-style-enum.stage2-armv7-unknown-linux-gnueabihf
...
(gdb) starti
Starting program: /home/ubuntu/src/rustc/rustc-1.25.0+dfsg1+llvm/build/armv7-unknown-linux-gnueabihf/test/debuginfo/borrowed-c-style-enum.stage2-armv7-unknown-linux-gnueabihf

Program stopped.
0xf77c7b80 in ?? () from /lib/ld-linux-armhf.so.3
(gdb) info sharedlibrary
From        To          Syms Read   Shared Object Library
0xf77c7a40  0xf77dadd0  Yes (*)     /lib/ld-linux-armhf.so.3
(*): Shared library is missing debugging information.
(gdb) break *0xf77c872c
Breakpoint 1 at 0xf77c872c
(gdb) cont
Continuing.

Program received signal SIGSEGV, Segmentation fault.
0xf77da458 in ?? () from /lib/ld-linux-armhf.so.3

Huh? It’s now crashed at a different place, without hitting our breakpoint at the start of dl_main. Loading the debug symbols again shows us where:

(gdb) add-symbol-file ~/libc6-syms/usr/lib/debug/lib/arm-linux-gnueabihf/ld-2.27.so 0xf77c7a40
add symbol table from file "/home/ubuntu/libc6-syms/usr/lib/debug/lib/arm-linux-gnueabihf/ld-2.27.so" at
        .text_addr = 0xf77c7a40
(y or n) y
Reading symbols from /home/ubuntu/libc6-syms/usr/lib/debug/lib/arm-linux-gnueabihf/ld-2.27.so...done.
(gdb) bt
#0  ?? () at ../sysdeps/arm/armv7/multiarch/memcpy_impl.S:654 from /lib/ld-linux-armhf.so.3
#1  0xf77c871e in handle_ld_preload (preloadlist=<optimized out>, main_map=0x0) at rtld.c:848
#2  0x00000000 in ?? ()
Backtrace stopped: previous frame identical to this frame (corrupt stack?)

This doesn’t make much sense, but the fact that setting a breakpoint has altered the program flow is our first clue.

On Linux, gdb interacts with the inferior using the ptrace system call. The next thing I wanted to try was running gdb in strace in order to capture the ptrace syscalls, so that I could compare differences afterwards and see if I could find any more clues.

I created the following simple gdb command file:

file /home/ubuntu/src/rustc/rustc-1.25.0+dfsg1+llvm/build/armv7-unknown-linux-gnueabihf/test/debuginfo/borrowed-c-style-enum.stage2-armv7-unknown-linux-gnueabihf
run
quit

I then ran gdb with this file inside strace, with the symbols for the dynamic loader installed. Here’s the log up until the point at which gdb calls PTRACE_CONT:

$ strace -t -eptrace gdb -quiet -batch -nx -command=~/test.script
13:08:35 --- SIGCHLD {si_signo=SIGCHLD, si_code=CLD_EXITED, si_pid=21136, si_uid=1000, si_status=0, si_utime=0, si_stime=0} ---
...
13:08:35 ptrace(PTRACE_GETREGS, 21137, NULL, 0xffd6afec) = 0
13:08:35 ptrace(PTRACE_GETSIGINFO, 21137, NULL, {si_signo=SIGTRAP, si_code=SI_USER, si_pid=21137, si_uid=1000}) = 0
13:08:35 --- SIGCHLD {si_signo=SIGCHLD, si_code=CLD_TRAPPED, si_pid=21137, si_uid=1000, si_status=SIGTRAP, si_utime=0, si_stime=0} ---
13:08:35 ptrace(PTRACE_CONT, 21137, 0x1, SIG_0) = 0
13:08:35 --- SIGCHLD {si_signo=SIGCHLD, si_code=CLD_TRAPPED, si_pid=21137, si_uid=1000, si_status=SIGTRAP, si_utime=0, si_stime=0} ---
13:08:35 ptrace(PTRACE_GETREGS, 21137, NULL, 0xffd6afec) = 0
13:08:35 ptrace(PTRACE_GETSIGINFO, 21137, NULL, {si_signo=SIGTRAP, si_code=SI_USER, si_pid=21137, si_uid=1000}) = 0
13:08:35 --- SIGCHLD {si_signo=SIGCHLD, si_code=CLD_TRAPPED, si_pid=21138, si_uid=1000, si_status=SIGSTOP, si_utime=0, si_stime=0} ---
13:08:35 ptrace(PTRACE_SETOPTIONS, 21138, NULL, PTRACE_O_TRACESYSGOOD) = 0
13:08:35 ptrace(PTRACE_SETOPTIONS, 21138, NULL, PTRACE_O_TRACEFORK) = 0
13:08:35 ptrace(PTRACE_SETOPTIONS, 21138, NULL, PTRACE_O_TRACEFORK|PTRACE_O_TRACEVFORKDONE) = 0
13:08:35 ptrace(PTRACE_CONT, 21138, NULL, SIG_0) = 0
13:08:35 --- SIGCHLD {si_signo=SIGCHLD, si_code=CLD_TRAPPED, si_pid=21138, si_uid=1000, si_status=SIGTRAP, si_utime=0, si_stime=0} ---
13:08:35 ptrace(PTRACE_GETEVENTMSG, 21138, NULL, [21139]) = 0
13:08:35 ptrace(PTRACE_KILL, 21139)     = 0
13:08:35 --- SIGCHLD {si_signo=SIGCHLD, si_code=CLD_KILLED, si_pid=21139, si_uid=1000, si_status=SIGKILL, si_utime=0, si_stime=0} ---
13:08:35 ptrace(PTRACE_SETOPTIONS, 21138, NULL, PTRACE_O_EXITKILL) = 0
13:08:35 ptrace(PTRACE_KILL, 21138)     = 0
13:08:35 --- SIGCHLD {si_signo=SIGCHLD, si_code=CLD_TRAPPED, si_pid=21138, si_uid=1000, si_status=SIGCHLD, si_utime=0, si_stime=0} ---
13:08:35 ptrace(PTRACE_KILL, 21138)     = 0
13:08:35 --- SIGCHLD {si_signo=SIGCHLD, si_code=CLD_KILLED, si_pid=21138, si_uid=1000, si_status=SIGKILL, si_utime=0, si_stime=0} ---
13:08:35 ptrace(PTRACE_SETOPTIONS, 21137, NULL, PTRACE_O_TRACESYSGOOD|PTRACE_O_TRACEFORK|PTRACE_O_TRACEVFORK|PTRACE_O_TRACECLONE|PTRACE_O_TRACEEXEC|PTRACE_O_TRACEVFORKDONE|PTRACE_O_EXITKILL) = 0
13:08:35 ptrace(PTRACE_GETREGSET, 21137, NT_PRSTATUS, [{iov_base=0xffd6b3b4, iov_len=72}]) = 0
13:08:35 ptrace(PTRACE_GETVFPREGS, 21137, NULL, 0xffd6b298) = 0
13:08:35 ptrace(PTRACE_GETREGSET, 21137, NT_PRSTATUS, [{iov_base=0xffd6b36c, iov_len=72}]) = 0
13:08:35 ptrace(PTRACE_PEEKTEXT, 21137, 0x411efc, [NULL]) = 0
13:08:35 ptrace(PTRACE_PEEKTEXT, 21137, 0x411efc, [NULL]) = 0
13:08:35 ptrace(PTRACE_PEEKTEXT, 21137, 0xf77c9a44, [0x4c18bf00]) = 0
13:08:35 ptrace(PTRACE_PEEKTEXT, 21137, 0xf77c9a44, [0x4c18bf00]) = 0
13:08:35 ptrace(PTRACE_PEEKTEXT, 21137, 0xf77c9a44, [0x4c18bf00]) = 0
13:08:35 ptrace(PTRACE_PEEKTEXT, 21137, 0xf77c9a44, [0x4c18bf00]) = 0
13:08:35 ptrace(PTRACE_PEEKTEXT, 21137, 0xf77c9a44, [0x4c18bf00]) = 0
13:08:35 ptrace(PTRACE_POKEDATA, 21137, 0xf77c9a44, 0x4c18de01) = 0
13:08:35 ptrace(PTRACE_PEEKTEXT, 21137, 0xf77c9ef8, [0xf00cbf00]) = 0
13:08:35 ptrace(PTRACE_PEEKTEXT, 21137, 0xf77c9ef8, [0xf00cbf00]) = 0
13:08:35 ptrace(PTRACE_PEEKTEXT, 21137, 0xf77c9ef8, [0xf00cbf00]) = 0
13:08:35 ptrace(PTRACE_PEEKTEXT, 21137, 0xf77c9ef8, [0xf00cbf00]) = 0
13:08:35 ptrace(PTRACE_PEEKTEXT, 21137, 0xf77c9ef8, [0xf00cbf00]) = 0
13:08:35 ptrace(PTRACE_POKEDATA, 21137, 0xf77c9ef8, 0xf00cde01) = 0
13:08:35 ptrace(PTRACE_PEEKTEXT, 21137, 0xf77cb5b8, [0x603cbf00]) = 0
13:08:35 ptrace(PTRACE_PEEKTEXT, 21137, 0xf77cb5b8, [0x603cbf00]) = 0
13:08:35 ptrace(PTRACE_PEEKTEXT, 21137, 0xf77cb5b8, [0x603cbf00]) = 0
13:08:35 ptrace(PTRACE_PEEKTEXT, 21137, 0xf77cb5b8, [0x603cbf00]) = 0
13:08:35 ptrace(PTRACE_PEEKTEXT, 21137, 0xf77cb5b8, [0x603cbf00]) = 0
13:08:35 ptrace(PTRACE_POKEDATA, 21137, 0xf77cb5b8, 0x603cde01) = 0
13:08:35 ptrace(PTRACE_PEEKTEXT, 21137, 0xf77cb220, [0x4639bf00]) = 0
13:08:35 ptrace(PTRACE_PEEKTEXT, 21137, 0xf77cb220, [0x4639bf00]) = 0
13:08:35 ptrace(PTRACE_PEEKTEXT, 21137, 0xf77cb220, [0x4639bf00]) = 0
13:08:35 ptrace(PTRACE_PEEKTEXT, 21137, 0xf77cb220, [0x4639bf00]) = 0
13:08:35 ptrace(PTRACE_PEEKTEXT, 21137, 0xf77cb220, [0x4639bf00]) = 0
13:08:35 ptrace(PTRACE_POKEDATA, 21137, 0xf77cb220, 0x4639de01) = 0
13:08:35 ptrace(PTRACE_PEEKTEXT, 21137, 0xf77d5310, [0xbf00e71c]) = 0
13:08:35 ptrace(PTRACE_PEEKTEXT, 21137, 0xf77d5310, [0xbf00e71c]) = 0
13:08:35 ptrace(PTRACE_PEEKTEXT, 21137, 0xf77d5310, [0xbf00e71c]) = 0
13:08:35 ptrace(PTRACE_PEEKTEXT, 21137, 0xf77d5310, [0xbf00e71c]) = 0
13:08:35 ptrace(PTRACE_PEEKTEXT, 21137, 0xf77d5310, [0xbf00e71c]) = 0
13:08:35 ptrace(PTRACE_POKEDATA, 21137, 0xf77d5310, 0xde01e71c) = 0
13:08:35 ptrace(PTRACE_PEEKTEXT, 21137, 0xf77d5bb0, [0x6d7bbf00]) = 0
13:08:35 ptrace(PTRACE_PEEKTEXT, 21137, 0xf77d5bb0, [0x6d7bbf00]) = 0
13:08:35 ptrace(PTRACE_PEEKTEXT, 21137, 0xf77d5bb0, [0x6d7bbf00]) = 0
13:08:35 ptrace(PTRACE_PEEKTEXT, 21137, 0xf77d5bb0, [0x6d7bbf00]) = 0
13:08:35 ptrace(PTRACE_PEEKTEXT, 21137, 0xf77d5bb0, [0x6d7bbf00]) = 0
13:08:35 ptrace(PTRACE_POKEDATA, 21137, 0xf77d5bb0, 0x6d7bde01) = 0
13:08:35 ptrace(PTRACE_PEEKTEXT, 21137, 0xf77d5d90, [0xe681bf00]) = 0
13:08:35 ptrace(PTRACE_PEEKTEXT, 21137, 0xf77d5d90, [0xe681bf00]) = 0
13:08:35 ptrace(PTRACE_PEEKTEXT, 21137, 0xf77d5d90, [0xe681bf00]) = 0
13:08:35 ptrace(PTRACE_PEEKTEXT, 21137, 0xf77d5d90, [0xe681bf00]) = 0
13:08:35 ptrace(PTRACE_PEEKTEXT, 21137, 0xf77d5d90, [0xe681bf00]) = 0
13:08:35 ptrace(PTRACE_POKEDATA, 21137, 0xf77d5d90, 0xe681de01) = 0
13:08:35 ptrace(PTRACE_PEEKTEXT, 21137, 0xf77c9a44, [0x4c18de01]) = 0
13:08:35 ptrace(PTRACE_POKEDATA, 21137, 0xf77c9a44, 0x4c18bf00) = 0
13:08:35 ptrace(PTRACE_PEEKTEXT, 21137, 0xf77cb5b8, [0x603cde01]) = 0
13:08:35 ptrace(PTRACE_POKEDATA, 21137, 0xf77cb5b8, 0x603cbf00) = 0
13:08:35 ptrace(PTRACE_PEEKTEXT, 21137, 0xf77cb220, [0x4639de01]) = 0
13:08:35 ptrace(PTRACE_POKEDATA, 21137, 0xf77cb220, 0x4639bf00) = 0
13:08:35 ptrace(PTRACE_PEEKTEXT, 21137, 0xf77d5bb0, [0x6d7bde01]) = 0
13:08:35 ptrace(PTRACE_POKEDATA, 21137, 0xf77d5bb0, 0x6d7bbf00) = 0
13:08:35 ptrace(PTRACE_CONT, 21137, 0x1, SIG_0) = 0

First of all, notice that there are several PTRACE_POKEDATA calls. These are used by gdb to write to memory locations in the process that we’re debugging, eg, to set breakpoints. For more information about how breakpoints work in gdb, this blog post has some good information. Basically, gdb writes an invalid instruction to the breakpoint location and this causes a SIGTRAP when executed, which is intercepted by gdb. When you continue over the breakpoint, gdb writes the original instruction back, single-steps over it, re-writes the invalid instruction and then continues execution.

This is an obvious way in which gdb can interfere with our process and make it crash, so I focused on these calls. I’ve filtered them out below:

13:08:35 ptrace(PTRACE_POKEDATA, 21137, 0xf77c9a44, 0x4c18de01) = 0
13:08:35 ptrace(PTRACE_POKEDATA, 21137, 0xf77c9ef8, 0xf00cde01) = 0
13:08:35 ptrace(PTRACE_POKEDATA, 21137, 0xf77cb5b8, 0x603cde01) = 0
13:08:35 ptrace(PTRACE_POKEDATA, 21137, 0xf77cb220, 0x4639de01) = 0
13:08:35 ptrace(PTRACE_POKEDATA, 21137, 0xf77d5310, 0xde01e71c) = 0
13:08:35 ptrace(PTRACE_POKEDATA, 21137, 0xf77d5bb0, 0x6d7bde01) = 0
13:08:35 ptrace(PTRACE_POKEDATA, 21137, 0xf77d5d90, 0xe681de01) = 0
13:08:35 ptrace(PTRACE_POKEDATA, 21137, 0xf77c9a44, 0x4c18bf00) = 0
13:08:35 ptrace(PTRACE_POKEDATA, 21137, 0xf77cb5b8, 0x603cbf00) = 0
13:08:35 ptrace(PTRACE_POKEDATA, 21137, 0xf77cb220, 0x4639bf00) = 0
13:08:35 ptrace(PTRACE_POKEDATA, 21137, 0xf77d5bb0, 0x6d7bbf00) = 0

Notice that the first 7 of these write the same 2-byte sequence – 0xde01. These are breakpoints in code that is running in Thumb mode (see arm_linux_thumb_le_breakpoint in gdb/arm-linux-tdep.c in the gdb source code). 0xde01 in Thumb mode is an undefined instruction.

(Note that the write to 0xf77d5310 is actually a breakpoint at 0xf77d5312, as 0xde01 appears in the 2 higher order bytes and this is little-endian).

We aren’t inserting any breakpoints ourselves – these breakpoints are set automatically by gdb to monitor various events in the dynamic loader during startup. This is something I wasn’t aware of before debugging this.

It may be useful to know how gdb determines the addresses on which to set breakpoints at startup. The dynamic loader exports various events as SystemTap probes, and data about these is stored in the .note.stapsdt ELF section. We can inspect this using readelf:

$ readelf -n /lib/ld-linux-armhf.so.3

Displaying notes found in: .note.gnu.build-id                                                            
  Owner                 Data size       Description                          
  GNU                  0x00000014       NT_GNU_BUILD_ID (unique build ID bitstring)                      
    Build ID: 3f3b9b4bfea2654f2cedf6db2d120b4e3a39ea7e                      

Displaying notes found in: .note.stapsdt                                                                 
  Owner                 Data size       Description 
  stapsdt              0x00000032       NT_STAPSDT (SystemTap probe descriptors)                         
    Provider: rtld                                           
    Name: init_start                                
    Location: 0x00002a44, Base: 0x00017b9c, Semaphore: 0x00000000                                        
    Arguments: -4@.L1204 4@[r7, #52]                      
  stapsdt              0x0000002e       NT_STAPSDT (SystemTap probe descriptors)                         
    Provider: rtld                                  
    Name: init_complete                                              
    Location: 0x00002ef8, Base: 0x00017b9c, Semaphore: 0x00000000                                        
    Arguments: -4@.L1207 4@r4                                           
  stapsdt              0x0000002e       NT_STAPSDT (SystemTap probe descriptors)                         
    Provider: rtld
    Name: map_failed
    Location: 0x00004220, Base: 0x00017b9c, Semaphore: 0x00000000
    Arguments: -4@[sp, #20] 4@r5
  stapsdt              0x00000035       NT_STAPSDT (SystemTap probe descriptors)
    Provider: rtld
    Name: map_start
    Location: 0x000045b8, Base: 0x00017b9c, Semaphore: 0x00000000
    Arguments: -4@[r7, #252] 4@[r7, #72]
  stapsdt              0x0000003c       NT_STAPSDT (SystemTap probe descriptors)
    Provider: rtld
    Name: map_complete
    Location: 0x0000e020, Base: 0x00017b9c, Semaphore: 0x00000000
    Arguments: -4@[fp, #20] 4@[r7, #36] 4@r4
  stapsdt              0x00000036       NT_STAPSDT (SystemTap probe descriptors)
    Provider: rtld
    Name: reloc_start
    Location: 0x0000e09e, Base: 0x00017b9c, Semaphore: 0x00000000
    Arguments: -4@[fp, #20] 4@[r7, #36]
  stapsdt              0x0000003e       NT_STAPSDT (SystemTap probe descriptors)
    Provider: rtld
    Name: reloc_complete
    Location: 0x0000e312, Base: 0x00017b9c, Semaphore: 0x00000000
    Arguments: -4@[fp, #20] 4@[r7, #36] 4@r4
  stapsdt              0x00000037       NT_STAPSDT (SystemTap probe descriptors)
    Provider: rtld
    Name: unmap_start
    Location: 0x0000ebb0, Base: 0x00017b9c, Semaphore: 0x00000000
    Arguments: -4@[r7, #104] 4@[r7, #80]
  stapsdt              0x0000003a       NT_STAPSDT (SystemTap probe descriptors)
    Provider: rtld
    Name: unmap_complete
    Location: 0x0000ed90, Base: 0x00017b9c, Semaphore: 0x00000000
    Arguments: -4@[r7, #104] 4@[r7, #80]
  stapsdt              0x00000029       NT_STAPSDT (SystemTap probe descriptors)
    Provider: rtld
    Name: setjmp
    Location: 0x0001201c, Base: 0x00017b9c, Semaphore: 0x00000000
    Arguments: 4@r0 -4@r1 4@r14
  stapsdt              0x00000029       NT_STAPSDT (SystemTap probe descriptors)
    Provider: rtld
    Name: longjmp
    Location: 0x00012088, Base: 0x00017b9c, Semaphore: 0x00000000
    Arguments: 4@r0 -4@r1 4@r4
  stapsdt              0x00000031       NT_STAPSDT (SystemTap probe descriptors)
    Provider: rtld
    Name: longjmp_target
    Location: 0x000120ba, Base: 0x00017b9c, Semaphore: 0x00000000
    Arguments: 4@r0 -4@r1 4@r14

GDB uses this information to map events to breakpoint addresses. You can read a bit more about gdb’s linker interface here, and more about userspace SystemTap probes here.

With a base address of 0xf77c7000, we can look at the PTRACE_POKEDATA calls and see that the addresses map to these probes:

  • 0xf77c9a44 => init_start
  • 0xf77c9ef8 => init_complete
  • 0xf77cb5b8 => map_start
  • 0xf77cb220 => map_failed
  • 0xf77d5312 => reloc_complete
  • 0xf77d5bb0 => unmap_start
  • 0xf77d5d90 => unmap_complete

This is consistent with the probe_info array in gdb/solib-svr4.c in the gdb source code.

I then ran gdb inside strace again, this time without the symbols for the dynamic loader installed. Here’s the log up until the point at which the inferior process crashes:

$ strace -t -eptrace gdb -quiet -batch -nx -command=~/test.script
13:01:50 --- SIGCHLD {si_signo=SIGCHLD, si_code=CLD_EXITED, si_pid=21098, si_uid=1000, si_status=0, si_utime=0, si_stime=0} ---
...
13:01:50 ptrace(PTRACE_GETREGS, 21099, NULL, 0xffb84a9c) = 0                                                                                                                                                       
13:01:50 ptrace(PTRACE_GETSIGINFO, 21099, NULL, {si_signo=SIGTRAP, si_code=SI_USER, si_pid=21099, si_uid=1000}) = 0
13:01:50 --- SIGCHLD {si_signo=SIGCHLD, si_code=CLD_TRAPPED, si_pid=21099, si_uid=1000, si_status=SIGTRAP, si_utime=0, si_stime=0} ---
13:01:50 ptrace(PTRACE_CONT, 21099, 0x1, SIG_0) = 0                                                      
13:01:50 --- SIGCHLD {si_signo=SIGCHLD, si_code=CLD_TRAPPED, si_pid=21099, si_uid=1000, si_status=SIGTRAP, si_utime=0, si_stime=0} ---
13:01:50 ptrace(PTRACE_GETREGS, 21099, NULL, 0xffb84a9c) = 0
13:01:50 ptrace(PTRACE_GETSIGINFO, 21099, NULL, {si_signo=SIGTRAP, si_code=SI_USER, si_pid=21099, si_uid=1000}) = 0
13:01:50 --- SIGCHLD {si_signo=SIGCHLD, si_code=CLD_TRAPPED, si_pid=21100, si_uid=1000, si_status=SIGSTOP, si_utime=0, si_stime=0} ---
13:01:50 ptrace(PTRACE_SETOPTIONS, 21100, NULL, PTRACE_O_TRACESYSGOOD) = 0
13:01:50 ptrace(PTRACE_SETOPTIONS, 21100, NULL, PTRACE_O_TRACEFORK) = 0
13:01:50 ptrace(PTRACE_SETOPTIONS, 21100, NULL, PTRACE_O_TRACEFORK|PTRACE_O_TRACEVFORKDONE) = 0
13:01:50 ptrace(PTRACE_CONT, 21100, NULL, SIG_0) = 0
13:01:50 --- SIGCHLD {si_signo=SIGCHLD, si_code=CLD_TRAPPED, si_pid=21100, si_uid=1000, si_status=SIGTRAP, si_utime=0, si_stime=0} ---
13:01:50 ptrace(PTRACE_GETEVENTMSG, 21100, NULL, [21101]) = 0
13:01:50 ptrace(PTRACE_KILL, 21101)     = 0
13:01:50 --- SIGCHLD {si_signo=SIGCHLD, si_code=CLD_KILLED, si_pid=21101, si_uid=1000, si_status=SIGKILL, si_utime=0, si_stime=0} ---
13:01:50 ptrace(PTRACE_SETOPTIONS, 21100, NULL, PTRACE_O_EXITKILL) = 0
13:01:50 ptrace(PTRACE_KILL, 21100)     = 0
13:01:50 --- SIGCHLD {si_signo=SIGCHLD, si_code=CLD_TRAPPED, si_pid=21100, si_uid=1000, si_status=SIGCHLD, si_utime=0, si_stime=0} ---
13:01:50 ptrace(PTRACE_KILL, 21100)     = 0
13:01:50 --- SIGCHLD {si_signo=SIGCHLD, si_code=CLD_KILLED, si_pid=21100, si_uid=1000, si_status=SIGKILL, si_utime=0, si_stime=0} ---
13:01:50 ptrace(PTRACE_SETOPTIONS, 21099, NULL, PTRACE_O_TRACESYSGOOD|PTRACE_O_TRACEFORK|PTRACE_O_TRACEVFORK|PTRACE_O_TRACECLONE|PTRACE_O_TRACEEXEC|PTRACE_O_TRACEVFORKDONE|PTRACE_O_EXITKILL) = 0
13:01:50 ptrace(PTRACE_GETREGSET, 21099, NT_PRSTATUS, [{iov_base=0xffb84e64, iov_len=72}]) = 0
13:01:50 ptrace(PTRACE_GETVFPREGS, 21099, NULL, 0xffb84d48) = 0
13:01:50 ptrace(PTRACE_GETREGSET, 21099, NT_PRSTATUS, [{iov_base=0xffb84e1c, iov_len=72}]) = 0
13:01:50 ptrace(PTRACE_PEEKTEXT, 21099, 0x411efc, [NULL]) = 0
13:01:50 ptrace(PTRACE_PEEKTEXT, 21099, 0x411efc, [NULL]) = 0
13:01:50 ptrace(PTRACE_PEEKTEXT, 21099, 0xf77c9a44, [0x4c18bf00]) = 0
13:01:50 ptrace(PTRACE_PEEKTEXT, 21099, 0xf77c9a44, [0x4c18bf00]) = 0
13:01:50 ptrace(PTRACE_POKEDATA, 21099, 0xf77c9a44, 0xe7f001f0) = 0
13:01:50 ptrace(PTRACE_PEEKTEXT, 21099, 0xf77c9ef8, [0xf00cbf00]) = 0
13:01:50 ptrace(PTRACE_PEEKTEXT, 21099, 0xf77c9ef8, [0xf00cbf00]) = 0
13:01:50 ptrace(PTRACE_POKEDATA, 21099, 0xf77c9ef8, 0xe7f001f0) = 0
13:01:50 ptrace(PTRACE_PEEKTEXT, 21099, 0xf77cb5b8, [0x603cbf00]) = 0
13:01:50 ptrace(PTRACE_PEEKTEXT, 21099, 0xf77cb5b8, [0x603cbf00]) = 0
13:01:50 ptrace(PTRACE_POKEDATA, 21099, 0xf77cb5b8, 0xe7f001f0) = 0
13:01:50 ptrace(PTRACE_PEEKTEXT, 21099, 0xf77cb220, [0x4639bf00]) = 0
13:01:50 ptrace(PTRACE_PEEKTEXT, 21099, 0xf77cb220, [0x4639bf00]) = 0
13:01:50 ptrace(PTRACE_POKEDATA, 21099, 0xf77cb220, 0xe7f001f0) = 0
13:01:50 ptrace(PTRACE_PEEKTEXT, 21099, 0xf77d5310, [0xbf00e71c]) = 0
13:01:50 ptrace(PTRACE_PEEKTEXT, 21099, 0xf77d5314, [0x4620e776]) = 0
13:01:50 ptrace(PTRACE_PEEKTEXT, 21099, 0xf77d5310, [0xbf00e71c]) = 0
13:01:50 ptrace(PTRACE_PEEKTEXT, 21099, 0xf77d5314, [0x4620e776]) = 0
13:01:50 ptrace(PTRACE_PEEKTEXT, 21099, 0xf77d5310, [0xbf00e71c]) = 0
13:01:50 ptrace(PTRACE_POKEDATA, 21099, 0xf77d5310, 0x1f0e71c) = 0
13:01:50 ptrace(PTRACE_PEEKTEXT, 21099, 0xf77d5314, [0x4620e776]) = 0
13:01:50 ptrace(PTRACE_POKEDATA, 21099, 0xf77d5314, 0x4620e7f0) = 0
13:01:50 ptrace(PTRACE_PEEKTEXT, 21099, 0xf77d5bb0, [0x6d7bbf00]) = 0
13:01:50 ptrace(PTRACE_PEEKTEXT, 21099, 0xf77d5bb0, [0x6d7bbf00]) = 0
13:01:50 ptrace(PTRACE_POKEDATA, 21099, 0xf77d5bb0, 0xe7f001f0) = 0
13:01:50 ptrace(PTRACE_PEEKTEXT, 21099, 0xf77d5d90, [0xe681bf00]) = 0
13:01:50 ptrace(PTRACE_PEEKTEXT, 21099, 0xf77d5d90, [0xe681bf00]) = 0
13:01:50 ptrace(PTRACE_POKEDATA, 21099, 0xf77d5d90, 0xe7f001f0) = 0
13:01:50 ptrace(PTRACE_POKEDATA, 21099, 0xf77c9a44, 0x4c18bf00) = 0
13:01:50 ptrace(PTRACE_POKEDATA, 21099, 0xf77cb5b8, 0x603cbf00) = 0
13:01:50 ptrace(PTRACE_POKEDATA, 21099, 0xf77cb220, 0x4639bf00) = 0
13:01:50 ptrace(PTRACE_POKEDATA, 21099, 0xf77d5bb0, 0x6d7bbf00) = 0
13:01:50 ptrace(PTRACE_CONT, 21099, 0x1, SIG_0) = 0
13:01:50 --- SIGCHLD {si_signo=SIGCHLD, si_code=CLD_TRAPPED, si_pid=21099, si_uid=1000, si_status=SIGSEGV, si_utime=0, si_stime=0} ---

Focusing again on the PTRACE_POKEDATA calls:

13:01:50 ptrace(PTRACE_POKEDATA, 21099, 0xf77c9a44, 0xe7f001f0) = 0
13:01:50 ptrace(PTRACE_POKEDATA, 21099, 0xf77c9ef8, 0xe7f001f0) = 0
13:01:50 ptrace(PTRACE_POKEDATA, 21099, 0xf77cb5b8, 0xe7f001f0) = 0
13:01:50 ptrace(PTRACE_POKEDATA, 21099, 0xf77cb220, 0xe7f001f0) = 0
13:01:50 ptrace(PTRACE_POKEDATA, 21099, 0xf77d5310, 0x1f0e71c) = 0
13:01:50 ptrace(PTRACE_POKEDATA, 21099, 0xf77d5314, 0x4620e7f0) = 0
13:01:50 ptrace(PTRACE_POKEDATA, 21099, 0xf77d5bb0, 0xe7f001f0) = 0
13:01:50 ptrace(PTRACE_POKEDATA, 21099, 0xf77d5d90, 0xe7f001f0) = 0
13:01:50 ptrace(PTRACE_POKEDATA, 21099, 0xf77c9a44, 0x4c18bf00) = 0
13:01:50 ptrace(PTRACE_POKEDATA, 21099, 0xf77cb5b8, 0x603cbf00) = 0
13:01:50 ptrace(PTRACE_POKEDATA, 21099, 0xf77cb220, 0x4639bf00) = 0
13:01:50 ptrace(PTRACE_POKEDATA, 21099, 0xf77d5bb0, 0x6d7bbf00) = 0

We see writes to the same 7 addresses where breakpoints were set during the first run, but now there’s a different byte sequence and an extra write. This time, gdb is writing a 4-byte sequence – 0xe7f001f0 to 6 addresses. These are breakpoints for code running in ARM mode (see eabi_linux_arm_le_breakpoint in gdb/arm-linux-tdep.c in the gdb source code). The 2 writes to 0xf77d5310 and 0xf77d5314 are a single breakpoint at 0xf77d5312 (there are 2 writes because it is not on a 4-byte boundary).

Checking the ARMv7 reference manual shows that 0xe7f001f0 is an undefined instruction in ARM mode. However, this byte sequence is decoded as the following valid instructions in Thumb mode:

lsl r0, r6, #7
b #-16

So, it takes the contents of r6, does a logical shift left by 7, writes it to r0 and then does an unconditional branch backwards by 16 bytes. This is quite likely going to cause our program (in this case, the dynamic loader) to go off the rails and crash with a less than useful stacktrace, which is the behaviour we’re seeing.

Why is this happening?

The next step was to figure out why gdb is inserting the ARM breakpoint instruction sequence instead of the Thumb one. To do this, I needed to understand where the breakpoints are written, and grepping the source code suggests the PTRACE_POKEDATA calls happen in inf_ptrace_peek_poke in gdb/inf-ptrace.c (actually, you won’t find PTRACE_POKEDATA here – it’s PT_WRITE_D which is defined in /usr/include/sys/ptrace.h).

Running gdb inside gdb with the dynamic loader debug symbols installed and setting a breakpoint on inf_ptrace_peek_poke shows me the call stack. Note that I set a breakpoint by line number, as inf_ptrace_peek_poke is inlined and it was the only way I could get the conditional breakpoint to work:

$ gdb --args gdb --command=~/test.script
...                                                                                                                                                                
(gdb) break ./gdb/inf-ptrace.c:578 if writebuf != 0x0                                                                                                                                                              
Breakpoint 1 at 0x51218: file ./gdb/inf-ptrace.c, line 578.                                                                                                                                                        
(gdb) run                                                                                                                                                                                                          
Starting program: /usr/bin/gdb --command=\~/test.script                                                                                                                                                            
Cannot parse expression `.L1207 4@r4'.                                                                                                                                                                             
warning: Probes-based dynamic linker interface failed.                                                   
Reverting to original interface.                                                                                                                                                                                   

[Thread debugging using libthread_db enabled]                                                                                                                                                                      
Using host libthread_db library "/lib/arm-linux-gnueabihf/libthread_db.so.1".                                                                                                                                      
GNU gdb (Ubuntu 8.1-0ubuntu3) 8.1.0.20180409-git                                                         
Copyright (C) 2018 Free Software Foundation, Inc.                                                        
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>                                                                                                                                      
This is free software: you are free to change and redistribute it.                                                                                                                                                 
There is NO WARRANTY, to the extent permitted by law.  Type "show copying"                                                                                                                                         
and "show warranty" for details.                                                                                                                                                                                   
This GDB was configured as "arm-linux-gnueabihf".                                                        
Type "show configuration" for configuration details.                                                     
For bug reporting instructions, please see:                                                              
<http://www.gnu.org/software/gdb/bugs/>.                                                                 
Find the GDB manual and other documentation resources online at:
<http://www.gnu.org/software/gdb/documentation/>.
For help, type "help".
Type "apropos word" to search for commands related to "word".

Breakpoint 1, inf_ptrace_xfer_partial (ops=<optimized out>, object=<optimized out>, annex=<optimized out>, readbuf=0x0, writebuf=0x6b29fc <arm_linux_thumb_le_breakpoint>     "\001\336", offset=4152138308, len=2,
    xfered_len=0xfffeee18) at ./gdb/inf-ptrace.c:578
578     ./gdb/inf-ptrace.c: No such file or directory.
(gdb) p/x offset
$1 = 0xf77c9a44
(gdb) bt
#0  inf_ptrace_xfer_partial (ops=<optimized out>, object=<optimized out>, annex=<optimized out>, readbuf=0x0, writebuf=0x6b29fc <arm_linux_thumb_le_breakpoint> "\001\336", offset=4152138308, len=2,
    xfered_len=0xfffeee18) at ./gdb/inf-ptrace.c:578
#1  0x00457512 in linux_xfer_partial (ops=0x8306e0, object=<optimized out>, annex=0x0, readbuf=0x0, writebuf=0x6b29fc <arm_linux_thumb_le_breakpoint> "\001\336", offset=4152138308, len=2, xfered_len=0xfffeee18)
    at ./gdb/linux-nat.c:4280
#2  0x004576da in linux_nat_xfer_partial (ops=0x8306e0, object=TARGET_OBJECT_MEMORY, annex=<optimized out>, readbuf=0x0, writebuf=0x6b29fc <arm_linux_thumb_le_breakpoint> "\001\336", offset=4152138308, len=2,
    xfered_len=0xfffeee18) at ./gdb/linux-nat.c:3908
#3  0x005f64f4 in raw_memory_xfer_partial (ops=ops@entry=0x8306e0, readbuf=readbuf@entry=0x0, writebuf=writebuf@entry=0x6b29fc <arm_linux_thumb_le_breakpoint> "\001\336", memaddr=4152138308, len=len@entry=2,
    xfered_len=xfered_len@entry=0xfffeee18) at ./gdb/target.c:1064
#4  0x005f6a98 in target_xfer_partial (ops=ops@entry=0x8306e0, object=object@entry=TARGET_OBJECT_RAW_MEMORY, annex=annex@entry=0x0, readbuf=readbuf@entry=0x0,
    writebuf=writebuf@entry=0x6b29fc <arm_linux_thumb_le_breakpoint> "\001\336", offset=4152138308, len=<optimized out>, xfered_len=xfered_len@entry=0xfffeee18) at ./gdb/target.c:1298
#5  0x005f7030 in target_write_partial (xfered_len=0xfffeee18, len=2, offset=<optimized out>, buf=0x6b29fc <arm_linux_thumb_le_breakpoint> "\001\336", annex=0x0, object=TARGET_OBJECT_RAW_MEMORY, ops=0x8306e0)
    at ./gdb/target.c:1554
#6  target_write_with_progress (ops=0x8306e0, object=object@entry=TARGET_OBJECT_RAW_MEMORY, annex=annex@entry=0x0, buf=buf@entry=0x6b29fc <arm_linux_thumb_le_breakpoint> "\001\336", offset=4152138308,
    len=len@entry=2, progress=progress@entry=0x0, baton=baton@entry=0x0) at ./gdb/target.c:1821
#7  0x005f70d2 in target_write (len=2, offset=2, buf=0x6b29fc <arm_linux_thumb_le_breakpoint> "\001\336", annex=0x0, object=TARGET_OBJECT_RAW_MEMORY, ops=<optimized out>) at ./gdb/target.c:1847
#8  target_write_raw_memory (memaddr=memaddr@entry=4152138308, myaddr=myaddr@entry=0x6b29fc <arm_linux_thumb_le_breakpoint> "\001\336", len=len@entry=2) at ./gdb/target.c:1473
#9  0x00590c8e in default_memory_insert_breakpoint (gdbarch=<optimized out>, bp_tgt=0x9236f0) at ./gdb/mem-break.c:66
#10 0x004de3aa in bkpt_insert_location (bl=0x923698) at ./gdb/breakpoint.c:12525
#11 0x004e8426 in insert_bp_location (bl=bl@entry=0x923698, tmp_error_stream=tmp_error_stream@entry=0xfffef07c, disabled_breaks=disabled_breaks@entry=0xfffeefec,
    hw_breakpoint_error=hw_breakpoint_error@entry=0xfffeeff0, hw_bp_error_explained_already=hw_bp_error_explained_already@entry=0xfffeeff4) at ./gdb/breakpoint.c:2553
#12 0x004e9556 in insert_breakpoint_locations () at ./gdb/breakpoint.c:2977
#13 update_global_location_list (insert_mode=insert_mode@entry=UGLL_MAY_INSERT) at ./gdb/breakpoint.c:12177
#14 0x004ea0a0 in update_global_location_list_nothrow (insert_mode=UGLL_MAY_INSERT) at ./gdb/breakpoint.c:12215
#15 0x004ea484 in create_solib_event_breakpoint_1 (insert_mode=UGLL_MAY_INSERT, address=address@entry=4152138308, gdbarch=gdbarch@entry=0x0) at ./gdb/breakpoint.c:7555
#16 create_solib_event_breakpoint (gdbarch=gdbarch@entry=0x928fb0, address=address@entry=4152138308) at ./gdb/breakpoint.c:7562
#17 0x004497bc in svr4_create_probe_breakpoints (objfile=0x933888, probes=0xfffef148, gdbarch=0x928fb0) at ./gdb/solib-svr4.c:2089
#18 svr4_create_solib_event_breakpoints (gdbarch=0x928fb0, address=<optimized out>) at ./gdb/solib-svr4.c:2173
#19 0x00449c5c in enable_break (from_tty=<optimized out>, info=<optimized out>) at ./gdb/solib-svr4.c:2465
#20 svr4_solib_create_inferior_hook (from_tty=<optimized out>) at ./gdb/solib-svr4.c:3057
#21 0x0056bba6 in post_create_inferior (target=0x801084 <current_target>, from_tty=from_tty@entry=0) at ./gdb/infcmd.c:469
#22 0x0056c736 in run_command_1 (args=<optimized out>, from_tty=0, run_how=RUN_NORMAL) at ./gdb/infcmd.c:665
#23 0x00465334 in cmd_func (cmd=<optimized out>, args=<optimized out>, from_tty=<optimized out>) at ./gdb/cli/cli-decode.c:1886
#24 0x006062a6 in execute_command (p=<optimized out>, p@entry=0x880f10 "run", from_tty=0) at ./gdb/top.c:630
#25 0x00548760 in command_handler (command=0x880f10 "run") at ./gdb/event-top.c:583
#26 0x00606a66 in read_command_file (stream=stream@entry=0x874ae0) at ./gdb/top.c:424
#27 0x004684e2 in script_from_file (stream=stream@entry=0x874ae0, file=file@entry=0xfffef7d0 "~/test.script") at ./gdb/cli/cli-script.c:1592
#28 0x004639bc in source_script_from_stream (file_to_open=0xfffef7d0 "~/test.script", file=0xfffef7d0 "~/test.script", stream=0x874ae0) at ./gdb/cli/cli-cmds.c:568
#29 source_script_with_search (file=0xfffef7d0 "~/test.script", from_tty=<optimized out>, search_path=<optimized out>) at ./gdb/cli/cli-cmds.c:604
#30 0x0058821a in catch_command_errors (command=0x463a89 <source_script(char const*, int)>, arg=0xfffef7d0 "~/test.script", from_tty=1) at ./gdb/main.c:379
#31 0x00588ea0 in captured_main_1 (context=<optimized out>) at ./gdb/main.c:1125
#32 captured_main (data=<optimized out>) at ./gdb/main.c:1147
#33 gdb_main (args=<optimized out>) at ./gdb/main.c:1173
#34 0x004343ac in main (argc=<optimized out>, argv=<optimized out>) at ./gdb/gdb.c:32

Frame 9 (default_memory_insert_breakpoint) looks like it will probably be interesting to us. Taking a look at what it does:

int
default_memory_insert_breakpoint (struct gdbarch *gdbarch,
                  struct bp_target_info *bp_tgt)
{
  CORE_ADDR addr = bp_tgt->placed_address;
  const unsigned char *bp;
  gdb_byte *readbuf;
  int bplen;
  int val;

  /* Determine appropriate breakpoint contents and size for this address.  */
  bp = gdbarch_sw_breakpoint_from_kind (gdbarch, bp_tgt->kind, &bplen);

  /* Save the memory contents in the shadow_contents buffer and then
     write the breakpoint instruction.  */
  readbuf = (gdb_byte *) alloca (bplen);
  val = target_read_memory (addr, readbuf, bplen);
  if (val == 0)
    {
       ...
      bp_tgt->shadow_len = bplen;
      memcpy (bp_tgt->shadow_contents, readbuf, bplen);

      val = target_write_raw_memory (addr, bp, bplen);
    }

  return val;
}

The call to gdbarch_sw_breakpoint_from_kind appears to return the bytes written for our breakpoint. gdbarch_sw_breakpoint_from_kind delegates to arm_sw_breakpoint_from_kind in gdb/arm-tdep.c. (The gdbarch_ functions provides a way for architecture independent code in gdb to call functions specific to the architecture associated with the target). Taking a look at what this does:

static const gdb_byte *
arm_sw_breakpoint_from_kind (struct gdbarch *gdbarch, int kind, int *size)
{
  struct gdbarch_tdep *tdep = gdbarch_tdep (gdbarch);

  switch (kind)
    {
    case ARM_BP_KIND_ARM:
      *size = tdep->arm_breakpoint_size;
      return tdep->arm_breakpoint;
    case ARM_BP_KIND_THUMB:
      *size = tdep->thumb_breakpoint_size;
      return tdep->thumb_breakpoint;
    case ARM_BP_KIND_THUMB2:
      *size = tdep->thumb2_breakpoint_size;
      return tdep->thumb2_breakpoint;
    default:
      gdb_assert_not_reached ("unexpected arm breakpoint kind");
    }
}

So, arm_breakpoint_from_kind returns an ARM, Thumb or Thumb2 breakpoint instruction sequence depending on the value of kind. If we switch to frame 9, we should be able to inspect the value of kind:

(gdb) f 9
#9  0x00590c8e in default_memory_insert_breakpoint (gdbarch=<optimized out>, bp_tgt=0x9236f0) at ./gdb/mem-break.c:66
66      ./gdb/mem-break.c: No such file or directory.
(gdb) p bp_tgt->kind
$2 = 2

2 is ARM_BP_KIND_THUMB, so this appears to check out. Moving further up the stack, we find that kind is determined in frame 10 (bkpt_insert_location in gdb/breakpoint.c). Let’s have a look at what that does:

static int
bkpt_insert_location (struct bp_location *bl)
{
  CORE_ADDR addr = bl->target_info.reqstd_address;

  bl->target_info.kind = breakpoint_kind (bl, &addr);
  bl->target_info.placed_address = addr;

  if (bl->loc_type == bp_loc_hardware_breakpoint)
    return target_insert_hw_breakpoint (bl->gdbarch, &bl->target_info);
  else
    return target_insert_breakpoint (bl->gdbarch, &bl->target_info);
}

This calls breakpoint_kind, which delegates to arm_breakpoint_kind_from_pc in gdb/arm-tdep.c via gdbarch_breakpoint_kind_from_pc. arm_breakpoint_kind_from_pc maps the breakpoint address to an instruction set and returns one of three values – ARM_BP_KIND_ARM, ARM_BP_KIND_THUMB or ARM_BP_KIND_THUMB2. From looking at arm_breakpoint_kind_from_pc, we can see the most interesting part is a call to arm_pc_is_thumb. Let’s have a look at how this works:

int
arm_pc_is_thumb (struct gdbarch *gdbarch, CORE_ADDR memaddr)
{
  struct bound_minimal_symbol sym;
  char type;
  ...

  /* If bit 0 of the address is set, assume this is a Thumb address.  */
  if (IS_THUMB_ADDR (memaddr))
    return 1;

So, first of all it checks whether bit 0 of the breakpoint address is set. Looking at the SystemTap probes in .note.stapsdt from our earlier readelf output, we can see that this is not the case for any probe. Following on:

  /* If the user wants to override the symbol table, let him.  */
  if (strcmp (arm_force_mode_string, "arm") == 0)
    return 0;
  if (strcmp (arm_force_mode_string, "thumb") == 0)
    return 1;

  /* ARM v6-M and v7-M are always in Thumb mode.  */
  if (gdbarch_tdep (gdbarch)->is_m)
    return 1;

We’re not forcing the mode and this isn’t ARM v6-M or v7-M, so, continuing:

  /* If there are mapping symbols, consult them.  */
  type = arm_find_mapping_symbol (memaddr, NULL);
  if (type)
    return type == 't';

arm_find_mapping_symbol tries to find a mapping symbol associated with the breakpoint address. Mapping symbols are a special type of symbol used to identify transitions between ARM and Thumb instruction sets (see this information). Breaking here in gdb shows that there isn’t a mapping symbol associated with the init_start probe:

(gdb) break ./gdb/arm-tdep.c:434
Breakpoint 2 at 0x43ec3c: file ./gdb/arm-tdep.c, line 434.
(gdb) run
...

Breakpoint 2, arm_pc_is_thumb (gdbarch=gdbarch@entry=0x928fb0, memaddr=memaddr@entry=4152138308) at ./gdb/arm-tdep.c:434
434     ./gdb/arm-tdep.c: No such file or directory.
(gdb) p/x memaddr
$2 = 0xf77c9a44
(gdb) p type
$3 = 0 '\000'

So, continuing to the next step:

  /* Thumb functions have a "special" bit set in minimal symbols.  */
  sym = lookup_minimal_symbol_by_pc (memaddr);
  if (sym.minsym)
    return (MSYMBOL_IS_SPECIAL (sym.minsym));

lookup_minimal_symbol_by_pc tries to map the breakpoint address to a function symbol. MSYMBOL_IS_SPECIAL(sym.minsym) expands to sym.minsym->target_flag_1 and is 1 if bit 0 of the symbol’s target address is set, indicating that the function is called in Thumb mode (see arm_elf_make_msymbol_special in gdb/arm-tdep.c for where this is set). Breaking here in gdb shows that this succeeds:

(gdb) break ./gdb/arm-tdep.c:439
Breakpoint 3 at 0x43ec54: file ./gdb/arm-tdep.c, line 439.
(gdb) cont
Continuing.

Breakpoint 3, arm_pc_is_thumb (gdbarch=gdbarch@entry=0x928fb0, memaddr=memaddr@entry=4152138308) at ./gdb/arm-tdep.c:439
439     in ./gdb/arm-tdep.c
(gdb) p sym.minsym
$4 = (minimal_symbol *) 0x964278
(gdb) p *sym.minsym
$5 = {mginfo = {name = 0x952ff8 "dl_main", value = {ivalue = 5932, block = 0x172c, bytes = 0x172c <error: Cannot access memory at address 0x172c>, address = 5932, common_block = 0x172c, chain = 0x172c}, 
    language_specific = {obstack = 0x0, demangled_name = 0x0}, language = language_auto, ada_mangled = 0, section = 10}, size = 10516, filename = 0x949a48 "rtld.c", type = mst_file_text, created_by_gdb = 0, 
  target_flag_1 = 1, target_flag_2 = 0, has_size = 1, hash_next = 0x0, demangled_hash_next = 0x0}
(gdb) p sym.minsym->target_flag_1
$6 = 1

It indicates that the init_start probe is in dl_main, and that it is called in Thumb mode.

We can use readelf to inspect the symbol table and verify that this is correct:

$ readelf -s libc6-syms/usr/lib/debug/lib/arm-linux-gnueabihf/ld-2.27.so | grep dl_main
    42: 0000172d 10516 FUNC LOCAL DEFAULT 11 dl_main

Note that bit 0 of the target address is set.

If lookup_minimal_symbol_by_pc fails, then we’re basically out of luck and arm_pc_is_thumb will return 0 (indicating that the breakpoint address is in an area that is executing ARM instructions). But this depends on the .symtab ELF section being present, so there is an obvious issue here as this is stripped from the binary in the build (and shipped in a separate debug object).

I then ran gdb in gdb without the dynamic loader symbols installed and set a breakpoint on default_memory_insert_breakpoint:

$ gdb --args gdb --batch --command=~/test.script
(gdb) set debug-file-directory /home/ubuntu/libc6-syms/usr/lib/debug/:/usr/lib/debug/                                                                                                                             
(gdb) break default_memory_insert_breakpoint(gdbarch*, bp_target_info*)                                  
Breakpoint 1 at 0x190c34: file ./gdb/mem-break.c, line 39.                                                                                                                                                         
(gdb) run                                                                                                                                                                                                          
Starting program: /usr/bin/gdb --batch --command=\~/test.script                                                                                                                                                    
Cannot parse expression `.L1207 4@r4'.                                                                                                                                                                             
warning: Probes-based dynamic linker interface failed.                                                   
Reverting to original interface.                                                                         

[Thread debugging using libthread_db enabled]                                                                                                                                                                      
Using host libthread_db library "/lib/arm-linux-gnueabihf/libthread_db.so.1".                                                                                                                                      

Breakpoint 1, default_memory_insert_breakpoint (gdbarch=0x9288c0, bp_tgt=0x922fe8) at ./gdb/mem-break.c:39
39      ./gdb/mem-break.c: No such file or directory.
(gdb) bt
#0  default_memory_insert_breakpoint (gdbarch=0x9288c0, bp_tgt=0x922fe8) at ./gdb/mem-break.c:39         
#1  0x004de3aa in bkpt_insert_location (bl=0x922f90) at ./gdb/breakpoint.c:12525
#2  0x004e8426 in insert_bp_location (bl=bl@entry=0x922f90, tmp_error_stream=tmp_error_stream@entry=0xfffef07c, disabled_breaks=disabled_breaks@entry=0xfffeefec,
    hw_breakpoint_error=hw_breakpoint_error@entry=0xfffeeff0, hw_bp_error_explained_already=hw_bp_error_explained_already@entry=0xfffeeff4) at ./gdb/breakpoint.c:2553
#3  0x004e9556 in insert_breakpoint_locations () at ./gdb/breakpoint.c:2977
#4  update_global_location_list (insert_mode=insert_mode@entry=UGLL_MAY_INSERT) at ./gdb/breakpoint.c:12177
#5  0x004ea0a0 in update_global_location_list_nothrow (insert_mode=UGLL_MAY_INSERT) at ./gdb/breakpoint.c:12215
#6  0x004ea484 in create_solib_event_breakpoint_1 (insert_mode=UGLL_MAY_INSERT, address=address@entry=4152138308, gdbarch=gdbarch@entry=0x0) at ./gdb/breakpoint.c:7555
#7  create_solib_event_breakpoint (gdbarch=gdbarch@entry=0x9288c0, address=address@entry=4152138308) at ./gdb/breakpoint.c:7562
#8  0x004497bc in svr4_create_probe_breakpoints (objfile=0x933238, probes=0xfffef148, gdbarch=0x9288c0) at ./gdb/solib-svr4.c:2089
#9  svr4_create_solib_event_breakpoints (gdbarch=0x9288c0, address=<optimized out>) at ./gdb/solib-svr4.c:2173
#10 0x00449c5c in enable_break (from_tty=<optimized out>, info=<optimized out>) at ./gdb/solib-svr4.c:2465
#11 svr4_solib_create_inferior_hook (from_tty=<optimized out>) at ./gdb/solib-svr4.c:3057
#12 0x0056bba6 in post_create_inferior (target=0x801084 <current_target>, from_tty=from_tty@entry=0) at ./gdb/infcmd.c:469
#13 0x0056c736 in run_command_1 (args=<optimized out>, from_tty=0, run_how=RUN_NORMAL) at ./gdb/infcmd.c:665
#14 0x00465334 in cmd_func (cmd=<optimized out>, args=<optimized out>, from_tty=<optimized out>) at ./gdb/cli/cli-decode.c:1886
#15 0x006062a6 in execute_command (p=<optimized out>, p@entry=0x874758 "run", from_tty=0) at ./gdb/top.c:630
#16 0x00548760 in command_handler (command=0x874758 "run") at ./gdb/event-top.c:583
#17 0x00606a66 in read_command_file (stream=stream@entry=0x875da8) at ./gdb/top.c:424
#18 0x004684e2 in script_from_file (stream=stream@entry=0x875da8, file=file@entry=0xfffef7d0 "~/test.script") at ./gdb/cli/cli-script.c:1592
#19 0x004639bc in source_script_from_stream (file_to_open=0xfffef7d0 "~/test.script", file=0xfffef7d0 "~/test.script", stream=0x875da8) at ./gdb/cli/cli-cmds.c:568
#20 source_script_with_search (file=0xfffef7d0 "~/test.script", from_tty=<optimized out>, search_path=<optimized out>) at ./gdb/cli/cli-cmds.c:604
#21 0x0058821a in catch_command_errors (command=0x463a89 <source_script(char const*, int)>, arg=0xfffef7d0 "~/test.script", from_tty=0) at ./gdb/main.c:379
#22 0x00588ea0 in captured_main_1 (context=<optimized out>) at ./gdb/main.c:1125
#23 captured_main (data=<optimized out>) at ./gdb/main.c:1147
#24 gdb_main (args=<optimized out>) at ./gdb/main.c:1173
#25 0x004343ac in main (argc=<optimized out>, argv=<optimized out>) at ./gdb/gdb.c:32
(gdb) p/x bp_tgt->placed_address
$1 = 0xf77c9a44
(gdb) p bp_tgt->kind
$2 = 4

Sure enough, default_memory_insert_breakpoint is called this time with kind == 4 (ARM_BP_KIND_ARM) which seems to be incorrect. Setting a breakpoint in arm_pc_is_thumb again, we can verify that the reason for this is that the call to lookup_minimal_symbol_by_pc fails:

(gdb) break ./gdb/arm-tdep.c:439                                                                                                                                                                                   
Breakpoint 1 at 0x3ec54: file ./gdb/arm-tdep.c, line 439.                                                                                                                                                          
(gdb) run
Starting program: /usr/bin/gdb --command=\~/test.script                                                  
Cannot parse expression `.L1207 4@r4'.                                                                   
warning: Probes-based dynamic linker interface failed.                                                   
Reverting to original interface.                    

[Thread debugging using libthread_db enabled]       
Using host libthread_db library "/lib/arm-linux-gnueabihf/libthread_db.so.1".
GNU gdb (Ubuntu 8.1-0ubuntu3) 8.1.0.20180409-git
Copyright (C) 2018 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.  Type "show copying"
and "show warranty" for details.
This GDB was configured as "arm-linux-gnueabihf".
Type "show configuration" for configuration details.
For bug reporting instructions, please see:
<http://www.gnu.org/software/gdb/bugs/>.
Find the GDB manual and other documentation resources online at:
<http://www.gnu.org/software/gdb/documentation/>.
For help, type "help".
Type "apropos word" to search for commands related to "word".

Breakpoint 1, arm_pc_is_thumb (gdbarch=0x928fb0, memaddr=4152138308) at ./gdb/arm-tdep.c:439
439     ./gdb/arm-tdep.c: No such file or directory.
(gdb) p/x memaddr
$1 = 0xf77c9a44
(gdb) p sym.minsym
$2 = (minimal_symbol *) 0x0

This results in arm_pc_is_thumb returning 0 and arm_breakpoint_kind_from_pc returning ARM_BP_KIND_ARM, which results in arm_sw_breakpoint_from_kind returning the wrong breakpoint instruction sequence.

TL;DR

GDB causes a crash in the dynamic loader (ld.so) on armv7 if ld.so has been stripped of its symbol table, because it is unable to correctly determine the appropriate instruction set when inserting probe event breakpoints.

If you’ve got to the end of this (congratulations), then you’re probably going to be disappointed to hear that I’m not sure what the proper fix is – this isn’t really my area of expertise. For the rustc package, I just added a Build-Depends: libc6-dbg [armhf] as a workaround for now, and that might even be the correct fix. But, it’s certainly nicer to understand why it didn’t work in the first place.