No instrumentation Golang Logging with eBPF (GoSF talk 11/11/20)

GoSF
Nov 11, 2020
No-Instrumentation Golang
Logging with eBPF

ABOUT ME
👋 i’m Zain
@zainasgar
Co-Founder/CEO Pixie Labs &
Adjunct Professor of CS @ Stanford

DEVELOPER PROBLEM
You’re an application developer, and your program is misbehaving.
● No problem. You have logs! Right?
● Uh-oh, not in the spot you need. 😞
We’ve all been there:
● “I just wish I could see the variable x when Foo() is called”

- Delve is one of the main Go debuggers.
- You can also use GDB, but it has less support for Go runtime, data structures, etc.
- Install Delve to get started
go get github.com/go-delve/delve/cmd/dlv
First steps with a debugger
DEBUGGERS

Github Link: https://github.com/pixie-labs/pixie/tree/main/demos/simple-gotracing
Relevant Code:
// computeE computes the approximation of e by running a
fixed number of iterations.
func computeE(iterations int64) float64 {
res := 2.0
fact := 1.0
for i := int64(2); i < iterations; i++ {
fact *= float64(i)
res += 1 / fact
}
return res
}
Let’s look at test application
Use Case
GET /e?iters={iterations}

// computeE computes the approximation of e by running a
fixed number of iterations.
func computeE(iterations int64) float64 {
res := 2.0
fact := 1.0
for i := int64(2); i < iterations; i++ {
fact *= float64(i)
res += 1 / fact
}
return res
}
What if we just want to log the iterations?
Use Case
fmt.Printf("iterations: %dn”, iterations)

Running and using the debugger
% dlv exec app
Type 'help' for list of commands.
(dlv) b app.go:15
Breakpoint 1 set at 0x125ef40 for
main.computeE() ./app.go:15
(dlv) c
Starting server on: :9090
> main.computeE() ./app.go:15 (hits goroutine(34):1 total:1)
(PC: 0x125ef40)
Warning: debugging optimized function
...
14: for i := int64(2); i < iterations; i++ {
=> 15: fact *= float64(i)
16: res += 1 / fact
17: }
...
(dlv) print iterations
100000
(dlv) print i
2
Delve

(dlv) bt
0 0x000000000125ef40 in main.computeE
at ./app.go:15
1 0x000000000125f2b1 in main.main.func1
at ./app.go:39
2 0x0000000001232f24 in
net/http.HandlerFunc.ServeHTTP
at /opt/golang/src/net/http/server.go:2007
3 0x0000000001234dfd in
net/http.(*ServeMux).ServeHTTP
4 0x0000000001235d74 in
net/http.serverHandler.ServeHTTP
5 0x0000000001231f55 in net/http.(*conn).serve
6 0x000000000105ab11 in runtime.goexit
at /opt/golang/src/runtime/asm_amd64.s:1357
(dlv) grs
Goroutine 1 - User:
/opt/golang/src/runtime/netpoll.go:184
internal/poll.runtime_pollWait (0x102aa45)
Goroutine 2 - User: /opt/golang/src/runtime/proc.go:305
runtime.gopark (0x1030360)
* Goroutine 34 - User: ./app.go:15 main.computeE
(0x125ef40) (thread 36175033)
Goroutine 35 - User:
/opt/golang/src/runtime/netpoll.go:184
internal/poll.runtime_pollWait (0x102aa45)
[7 goroutines]
Stack Goroutines
GETTING CONTEXT

RECAP ON DEBUGGERS
● Debuggers provide and easy way to get information about executing code
● No need to add fmt.Print and recompile.
○ Can easily step through code and get information.
● Dlv and GDB can connect to remote binaries to help debug machines running
on servers.
○ Unfortunately this turns out to be harder than expected when running in
environments like Kubernetes .
○ Checkout Solo.io, etc.

CAN WE RUN THIS SAFELY ON PRODUCTION CODE?
● Debuggers can be attached to production binaries by using a remote
attachment.
● Unfortunately, they can mutate, stop execution of the code and lead to
unexpected failures.
○ You probably don’t want this on prod
● What else can we do?

YOUR OPTIONS
Option 1: Add a log to your program, re-compile and re-deploy.
○ This can be simple log statements, or
○ More comprehensive like Open tracing.
Option 2: Debugger
○ GDB
○ Delve
Option 3: Linux tracing utility
○ strace/ftrace
○ LTTng/USDT
Option 4: eBPF 🤔

What is
Credit: https://ebpf.io/
eBPF

What are we going to build?
eBPF

[0] % objdump --syms app|grep computeE
00000000006609a0 g F .text 000000000000004b main.computeE
[0] % objdump -d app | less
00000000006609a0 <main.computeE>:
6609a0: 48 8b 44 24 08 mov 0x8(%rsp),%rax
6609a5: b9 02 00 00 00 mov $0x2,%ecx
6609aa: f2 0f 10 05 16 a6 0f movsd 0xfa616(%rip),%xmm0 #
75afc8 <$f64.3ff0000000000000>
6609b1: 00
6609b2: f2 0f 10 0d 36 a6 0f movsd 0xfa636(%rip),%xmm1 #
75aff0 <$f64.4000000000000000>
Diving into the details
eBPF

Using uprobes
eBPF
...
...
main.computeE
0x6609a0
{
App Binary
main.main
{
0x6609f0
...
...
main.computeE
Uprobe Hook
{
App Binary
main.main
{
0x6609f0
BPF
Program
Perf
Buffer Tracer
Binary

What does the BPF program look like?
● The function is simply invoked
whenever main.computeE is called.
● The registration is done via UProbes
● It attaches to every running version
of the binary
#include <uapi/linux/ptrace.h>
BPF_PERF_OUTPUT(trace);
inline int computeECalled(struct pt_regs *ctx)
{
// The input argument is stored in ax.
long val = ctx->ax;
trace.perf_submit(ctx, &val, sizeof(val));
return 0;
}
eBPF
Github Link: https://github.com/pixie-labs/pixie/tree/main/demos/simple-gotracing/trace_example

● Utilizing tracepoints for dynamic logging allows for easy
instrumentation of production binaries
● The complexities of the Go ABI make it difﬁcult to do. Especially
when you consider: interfaces, channels, etc.
What’s next?
eBPF

eBPF
https://github.com/pixie-labs/pixie
https://github.com/kinvolk/inspektor-gadget
Some related projects
https://github.com/draios/sysdig/wiki/eBPF

eBPF
Dynamic
Tracepoint
(live injection)
func Foo(a, b int) int {
c := a + 2;
if c > threshold {
return Bar(b)
}
return 0
}
fmt.Printf("%v, %v, %v", b, c, threshold)
- Function Argument Tracing → “What are the arguments being passed to Foo(x, y, z)?”
- Function Input-Output Tracing → “What are the arguments and return value of calls to Foo(x, y, z)?”
- Function Latency Proﬁling → “What is the latency (call to return) of calls to Login(username)?”

At, less than 2% CPU overhead and Latency degradation
Less than 2%
overhead at
0% sampling !!

...
...
main.computeE
0x6609a0
{
App Binary
main.main
{
0x6609f0
...
...
main.computeE
Uprobe Hook
{
App Binary
main.main
{
0x6609f0
BPF
Program
Perf
Buffer Tracer
Binary

COMPARING TOOLS
GDB, Delve etc. USDT, LTTng OpenTracing eBPF
Traceable
Code
Instrumented
Application Code
All application
code
Instrumented
Application Code
All application
code
Application
Disruption
High Low Low Low
Distributed
Analysis
No No Multi-Node No
Performance
Impact
High Low Low/Med Low

No instrumentation Golang Logging with eBPF (GoSF talk 11/11/20)

More Related Content

What's hot (20)

Similar to No instrumentation Golang Logging with eBPF (GoSF talk 11/11/20) (20)

Recently uploaded (20)

No instrumentation Golang Logging with eBPF (GoSF talk 11/11/20)