Debugging Go Code with LLDB

Updated Jan 4, 2016

This applies to the gc toolchain. Besides this overview you might want to consult the LLDB manual.

Introduction

When you compile and link your Go programs with the gc toolchain on Linux, Mac OS X, FreeBSD or NetBSD, the resulting binaries contain DWARFv3 debugging information that recent versions (>3.7) of the LLDB debugger can use to inspect a live process or a core dump. You will probably need to build LLDB from trunk to get the go support.

Pass the '-w' flag to the linker to omit the debug information (for example, go build -ldflags "-w" prog.go).

The code generated by the gc compiler includes inlining of function invocations and registerization of variables. These optimizations make debugging with lldb harder. To disable them when debugging, pass the flags -gcflags "-N -l" to the go command used to build the code being debugged.

Getting lldb

The latest release lldb (3.7) doesn’t contain the go extensions, so you will need to build it from trunk.

Common Operations

Go Extensions

Expression Parsing

LLDB includes a go expression parser.

(lldb) p x
(lldb) expr *(*int32)(t)
(lldb) help expr

Interfaces

By default, LLDB shows the dynamic type of interface values. This is usually a pointer. Consider func foo(a interface{}) { ... }. If you callfoo(1.0), lldb will treat a as *float64inside foo. You can disable this behavior for a single expression or globally:

(lldb) expr -d no-dynamic-values -- a
(lldb) settings set target.prefer-dynamic-values no-dynamic-values

Data Formatters

LLDB includes data formatters for go strings and slices. See the LLDB docs for custom variable formatting. If you want to extend the builtin formatters, see GoLanguageRuntime.cpp.

Channels and maps are 'reference' types, which lldb treats as pointers to C++-like types hash<int,string>*. Dereferencing will show the internal representation.

Goroutines

LLDB treats Goroutines as threads.

(lldb) thread list
(lldb) bt all
(lldb) thread select 2

Known Issues

Tutorial

In this tutorial we will inspect the binary of the regexp package's unit tests. To build the binary, change to $GOROOT/src/regexp and run go test -gcflags "-N -l" -c. This should produce an executable file named regexp.test.

Getting Started

Launch lldb, debugging regexp.test:

$ lldb regexp.test
(lldb) target create "regexp.test"
Current executable set to 'regexp.test' (x86_64).
(lldb)

Setting breakpoints

Set a breakpoint at the TestFind function:

(lldb) b regexp.TestFind

Sometimes the go compiler prefixes function names with the full path. If you can’t find the simple name, you can try using a function name regex:

(lldb) break set -r regexp.TestFind$
Breakpoint 5: where = regexp.test`_/code/go/src/regexp.TestFind + 37 at find_test.go:149, address = 0x00000000000863a5

Run the program:

(lldb) run --test.run=TestFind
Process 8496 launched: '/code/go/src/regexp/regexp.test' (x86_64)
Process 8496 stopped
* thread #9: tid = 0x0017, 0x00000000000863a5 regexp.test`_/code/go/src/regexp.TestFind(t=0x000000020834a000) + 37 at find_test.go:149, stop reason = breakpoint 2.1 3.1 5.1
    frame #0: 0x00000000000863a5 regexp.test`_/code/go/src/regexp.TestFind(t=0x000000020834a000) + 37 at find_test.go:149
   146         // First the simple cases.
   147
   148         func TestFind(t *testing.T) {
-> 149                 for _, test := range findTests {
   150                         re := MustCompile(test.pat)
   151                         if re.String() != test.pat {
   152                                 t.Errorf("String() = `%s`; should be `%s`", re.String(), test.pat)

Execution has paused at the breakpoint. See which goroutines are running, and what they're doing:

(lldb) thread list
Process 8496 stopped
  thread #1: tid = 0x12201, 0x000000000003c0ab regexp.test`runtime.mach_semaphore_wait + 11 at sys_darwin_amd64.s:412
  thread #2: tid = 0x122fa, 0x000000000003bf7c regexp.test`runtime.usleep + 44 at sys_darwin_amd64.s:290
  thread #4: tid = 0x0001, 0x0000000000015865 regexp.test`runtime.gopark(unlockf=0x00000000000315a0, lock=0x00000002083220b8, reason="chan receive") + 261 at proc.go:131
  thread #5: tid = 0x0002, 0x0000000000015865 regexp.test`runtime.gopark(unlockf=0x00000000000315a0, lock=0x00000000002990d0, reason="force gc (idle)") + 261 at proc.go:131
  thread #6: tid = 0x0003, 0x0000000000015754 regexp.test`runtime.Gosched + 20 at proc.go:114
  thread #7: tid = 0x0004, 0x0000000000015865 regexp.test`runtime.gopark(unlockf=0x00000000000315a0, lock=0x00000000002a07d8, reason="finalizer wait") + 261 at proc.go:131
* thread #9: tid = 0x0017, 0x00000000000863a5 regexp.test`_/code/go/src/regexp.TestFind(t=0x000000020834a000) + 37 at find_test.go:149, stop reason = breakpoint 2.1 3.1 5.1

the one marked with the * is the current goroutine.

Inspecting the Source

Use the "l" or "list" command to inspect source code.

(lldb) l
(lldb) # Hit enter to repeat last command. Here, list the next few lines

Naming

Variable and function names must be qualified with the name of the packages they belong to. The Compile function from the regexp package is known to lldb as 'regexp.Compile'.

Methods must be qualified with the name of their receiver types. For example, the *Regexp type’s String method is known as 'regexp.(*Regexp).String'.

Variables referenced by closures will appear as pointers magically prefixed with '&'.

Inspecting the stack

Look at the stack trace for where we’ve paused the program:

(lldb) bt
* thread #9: tid = 0x0017, 0x00000000000863a5 regexp.test`_/code/go/src/regexp.TestFind(t=0x000000020834a000) + 37 at find_test.go:149, stop reason = breakpoint 2.1 3.1 5.1
  * frame #0: 0x00000000000863a5 regexp.test`_/code/go/src/regexp.TestFind(t=0x000000020834a000) + 37 at find_test.go:149
    frame #1: 0x0000000000056e3f regexp.test`testing.tRunner(t=0x000000000003b671, test=0x000000020834a000) + 191 at testing.go:447
    frame #2: 0x00000000002995a0 regexp.test`/code/go/src/regexp.statictmp_3759 + 96
    frame #3: 0x000000000003b671 regexp.test`runtime.goexit + 1 at asm_amd64.s:2232

The stack frame shows we’re currently executing the regexp.TestFind function, as expected.

The command frame variable lists all variables local to the function and their values, but is a bit dangerous to use, since it will also try to print uninitialized variables. Uninitialized slices may cause lldb to try to print arbitrary large arrays.

The function’s arguments:

(lldb) frame var -l
(*testing.T) t = 0x000000020834a000

When printing the argument, notice that it’s a pointer to a Regexp value.

(lldb) p re
(*_/code/go/src/regexp.Regexp) $3 = 0x000000020834a090
(lldb) p t
(*testing.T) $4 = 0x000000020834a000
(lldb) p *t
(testing.T) $5 = {
  testing.common = {
    mu = {
      w = (state = 0, sema = 0)
      writerSem = 0
      readerSem = 0
      readerCount = 0
      readerWait = 0
    }
    output = (len 0, cap 0) {}
    failed = false
    skipped = false
    finished = false
    start = {
      sec = 63579066045
      nsec = 777400918
      loc = 0x00000000002995a0
    }
    duration = 0
    self = 0x000000020834a000
    signal = 0x0000000208322060
  }
  name = "TestFind"
  startParallel = 0x0000000208322240
}
(lldb) p *t.startParallel
(hchan<bool>) $3 = {
  qcount = 0
  dataqsiz = 0
  buf = 0x0000000208322240
  elemsize = 1
  closed = 0
  elemtype = 0x000000000014eda0
  sendx = 0
  recvx = 0
  recvq = {
    first = 0x0000000000000000
    last = 0x0000000000000000
  }
  sendq = {
    first = 0x0000000000000000
    last = 0x0000000000000000
  }
  lock = (key = 0x0000000000000000)
}

That hchan<bool> is the runtime-internal representation of a channel.

Stepping forward:

(lldb) n # execute next line
(lldb) # enter is repeat
(lldb) # enter is repeat
Process 17917 stopped
* thread #8: tid = 0x0017, 0x000000000008648f regexp.test`_/code/go/src/regexp.TestFind(t=0x000000020834a000) + 271 at find_test.go:151, stop reason = step over
    frame #0: 0x000000000008648f regexp.test`_/code/go/src/regexp.TestFind(t=0x000000020834a000) + 271 at find_test.go:151
   148         func TestFind(t *testing.T) {
   149                 for _, test := range findTests {
   150                         re := MustCompile(test.pat)
-> 151                         if re.String() != test.pat {
   152                                 t.Errorf("String() = `%s`; should be `%s`", re.String(), test.pat)
   153                         }
   154                         result := re.Find([]byte(test.text))
(lldb) p test.pat
(string) $4 = ""
(lldb) p re
(*_/code/go/src/regexp.Regexp) $5 = 0x0000000208354320
(lldb) p *re
(_/code/go/src/regexp.Regexp) $6 = {
  expr = ""
  prog = 0x0000000208ac6090
  onepass = 0x0000000000000000
  prefix = ""
  prefixBytes = (len 0, cap 0) {}
  prefixComplete = true
  prefixRune = 0
  prefixEnd = 0
  cond = 0
  numSubexp = 0
  subexpNames = (len 1, cap 1) {
    [0] = ""
  }
  longest = false
  mu = (state = 0, sema = 0)
  machine = (len 0, cap 0) {}
}
(lldb) p *re.prog
(regexp/syntax.Prog) $7 = {
  Inst = (len 3, cap 4) {
    [0] = {
      Op = 5
      Out = 0
      Arg = 0
      Rune = (len 0, cap 0) {}
    }
    [1] = {
      Op = 6
      Out = 2
      Arg = 0
      Rune = (len 0, cap 0) {}
    }
    [2] = {
      Op = 4
      Out = 0
      Arg = 0
      Rune = (len 0, cap 0) {}
    }
  }
  Start = 1
  NumCap = 2
}

We can step into the Stringfunction call with "s":

(lldb) s
Process 17917 stopped
* thread #8: tid = 0x0017, 0x0000000000067332 regexp.test`_/code/go/src/regexp.(re=0x0000000208354320, ~r0="").String + 18 at regexp.go:104, stop reason = step in
    frame #0: 0x0000000000067332 regexp.test`_/code/go/src/regexp.(re=0x0000000208354320, ~r0="").String + 18 at regexp.go:104
   101
   102         // String returns the source text used to compile the regular expression.
   103         func (re *Regexp) String() string {
-> 104                 return re.expr
   105         }
   106
   107         // Compile parses a regular expression and returns, if successful,

Get a stack trace to see where we are:

(lldb) bt
* thread #8: tid = 0x0017, 0x0000000000067332 regexp.test`_/code/go/src/regexp.(re=0x0000000208354320, ~r0="").String + 18 at regexp.go:104, stop reason = step in
  * frame #0: 0x0000000000067332 regexp.test`_/code/go/src/regexp.(re=0x0000000208354320, ~r0="").String + 18 at regexp.go:104
    frame #1: 0x00000000000864a0 regexp.test`_/code/go/src/regexp.TestFind(t=0x000000020834a000) + 288 at find_test.go:151
    frame #2: 0x0000000000056e3f regexp.test`testing.tRunner(t=0x000000000003b671, test=0x000000020834a000) + 191 at testing.go:447
    frame #3: 0x00000000002995a0 regexp.test`/code/go/src/regexp.statictmp_3759 + 96
    frame #4: 0x000000000003b671 regexp.test`runtime.goexit + 1 at asm_amd64.s:2232