#1354

Incorrect line numbers with the offline symbol resolver

Openariel-miculasopened 5mo ago

I'm instrumenting rust code (using https://docs.rs/tracy-client/latest/tracy_client/ and setting TRACY_SYMBOL_OFFLINE_RESOLVE=1) and I noticed that the line numbers don't seem quite right. For example, a symbol was showing a line number containing the closing } delimiter.

I'm using https://docs.rs/addr2line/latest/addr2line/ and I checked that it returns the same line (containing }) for that specific offset. For the symbol resolution I'm using ./update/build/tracy-update -r <input.tracy> <output.tracy>

Claude suggests subtracting 1 from symbol offset, the reasoning being that the symbol points to the return address so we need to get back to the instruction that performed the call:

diff --git a/update/src/OfflineSymbolResolverAddr2Line.cpp b/update/src/OfflineSymbolResolverAddr2Line.cpp
index e8809243..1c92d2e4 100644
--- a/update/src/OfflineSymbolResolverAddr2Line.cpp
+++ b/update/src/OfflineSymbolResolverAddr2Line.cpp
@@ -97,7 +97,7 @@ public:
             for( ; entryIdx < batchEndIdx; entryIdx++ )
             {
                 const FrameEntry& entry = inputEntryList[entryIdx];
-                ss << " 0x" << std::hex << entry.symbolOffset;
+                ss << " 0x" << std::hex << (entry.symbolOffset - 1);
             }

             std::string resultStr = ExecShellCommand( ss.str().c_str() );

gdb does subtract 1 in get_frame_address_in_block, so it might be a plausible explanation.

Activity

4
wolfpld2mo ago

Very interesting finding. Unlocks a lot to think about. AI analysis below.


1. Debuggers — both subtract 1, and both guard it

gdb (gdb/frame.c, source) — verified earlier, get_frame_address_in_block:

if ((get_frame_type (next_frame) == NORMAL_FRAME || get_frame_type (next_frame) == TAILCALL_FRAME)
    && (get_frame_type (this_frame) == NORMAL_FRAME || ... ))
  return pc - 1;
return pc;

The guard matters: the subtraction happens only for frames that exist because of a call. gdb's own comment (in get_frame_language) states the rule: "we cannot use the frame PC as is, because it usually points to the instruction following the 'call', which is sometimes the first instruction of another function".

lldb is even more explicit — it has a dedicated API pair separating "raw PC" from "PC for symbolication":

  • StackFrame::GetSymbolContext (StackFrame.cpp, L329-332):

    "If this is not frame zero, then we need to subtract 1 from the PC value when doing address lookups since the PC will be on the instruction following the function call instruction..." → Address lookup_addr(GetFrameCodeAddressForSymbolication());

  • StackFrame::GetFrameCodeAddressForSymbolication (L231-254): skips the -1 for m_behaves_like_zeroth_frame (the frame that actually executed the stop — a signal/trap frame), does lookup_addr.Slide(-1) otherwise, and has an explicit special case for "a 'noreturn' function at the end of a section": subtract 1 from the actual load address and recompute the section. That's the boundary case gdb's comment also warns about, handled in production code.
  • RegisterContext::GetPCForSymbolication (RegisterContext.cpp, L129-141): if (!BehavesLikeZerothFrame() && pc != 0) pc--;

Both debuggers display the raw return address in the frame header (#1 0x0000... in caller), and use the adjusted PC only for symbol/line/block resolution. That duality is the key to your last question (§3).

2. Language runtimes

Go (src/runtime/traceback.go, source) — this is the clearest documentation of the convention anywhere in the ecosystem, in a shipping runtime:

// symPC returns the PC that should be used for symbolizing the current frame.
// Specifically, this is the PC of the last instruction executed in this frame.
//
// If this frame did a normal call, then frame.pc is a return PC, so this will
// return frame.pc-1, which points into the CALL instruction. If the frame was
// interrupted by a signal (e.g., profiler, segv, etc) then frame.pc is for the
// trapped instruction, so this returns frame.pc. See issue #34123. Finally,
// frame.pc can be at function entry when the frame is initialized without
// actually running code, like in runtime.mstart, in which case this returns
// frame.pc because that's the best we can do.
func (u *unwinder) symPC() uintptr {
	if u.flags&unwindTrap == 0 && u.frame.pc > u.frame.fn.entry() {
		return u.frame.pc - 1
	}
	return u.frame.pc
}

It's wired directly into the goroutine-dump print path: newInlineUnwinder(f, u.symPC()) → iu.fileLine(uf) → the printed file:line. And the comment in tracebackPCs states the storage convention: "Callers expect the pc buffer to contain return addresses and do the -1 themselves." Note the three guards: normal call → pc-1; trap (profiler/segv) → raw pc; synthetic entry frame → raw pc. Same shape as gdb's frame-type check.

Rust — the backtrace crate (what std::backtrace uses), symbolize/mod.rs:

// IP values from stack frames are typically (always?) the instruction
// *after* the call that's the actual stack trace. Symbolizing this on
// causes the filename/line number to be one ahead and perhaps into
// the void if it's near the end of the function.
//
// This appears to basically always be the case on all platforms, so we always
// subtract one from a resolved ip to resolve it to the previous call
// instruction instead of the instruction being returned to.
fn adjust_ip(a: *mut c_void) -> *mut c_void {
    if a.is_null() { a } else { (a as usize - 1) as *mut c_void }
}

Applied unconditionally to all symbolization (resolve and resolve_frame), with a null check as the underflow guard. The comment's "one ahead and perhaps into the void if it's near the end of the function" is the issue reporter's exact } symptom, independently observed by the Rust project. Note the consequence for the reporter specifically: Rust's own Backtrace (online) would show the correct line, while Tracy's offline path showed the wrong one — the two were inconsistent, and the offline resolver is the one that needs the -1.

3. Crash reporters and profilers — the counter-examples

breakpad (Chromium's crash reporter) — basic_source_line_resolver.cc (source), ResolveFrameInfo:

MemAddr address = frame->instruction - frame->module->base_address();
...
if (functions_.RetrieveNearestRange(address, &func, ...) && address >= function_base && ...)
...
if (func->lines.RetrieveRange(address, &line, ...))

No -1 — breakpad resolves the raw return address, so Chromium crash reports carry the same off-by-one line. This is the real-world proof that the unadjusted behavior is a live defect, not a style preference.

perf — I grepped tools/perf/util/callchain.c and symbol.c (source): no -1 adjustment anywhere. But perf is a sampling profiler, which matters for your conceptual question: the sampled leaf IP is the actual instruction pointer (a true PC — no -1 possible or needed), and return addresses appear only in the callchain frames above it, resolved as-is. perf's report is symbol-centric, which is why the off-by-one stays mostly invisible there (it can surface as a wrong symbol at a function boundary).

Windows dbghelp — the API design itself encodes the convention: SymGetLineFromAddr64 takes an address as-is (no -1), but the inline-expansion API is defined to receive a return address — MS docs: "StartRetAddress: Contains the return address of the start of the current block." So Microsoft's design: raw return address is the input convention for frame queries; compensation is the tool's job. This has a direct implication for Tracy's Windows path (below): the inline APIs must keep getting the raw ptr.

Normative layer: System V AMD64 ABI — call "pushes the address of the next instruction onto the stack" (psABI); DWARF line tables map [row_start, next_row_start) → line for any address — the -1 decision belongs to the consumer, and every serious consumer makes it the same way.

Summary

Projectstored valueline/symbol resolutionoutcome
gdbreturn addresspc-1, normal frames onlycall-site line
lldbreturn addresspc-1 non-zeroth frames; noreturn section-edge handledcall-site line
Go runtimereturn PCsymPC(): pc-1 for calls, raw for traps/entrycall-site line
Rust backtracereturn addressalways pc-1 (adjust_ip)call-site line
Tracy (today, online + offline)return addressas-isnext line after call ← issue 1354
breakpadreturn addressas-isoff-by-one line (defect in prod)
perftrue IP (leaf) / return addr (callchain)as-isleaf exact; callchain symbol can miss at boundaries
Lectem2mo ago

I somehow thought libbacktrace handled this for us internally but I did see something similar happening on a custom platform which could explain some weird symbols being surfaced. will test.

ariel-miculas2mo ago

I tested this on MacOs

Lectem2mo ago

I can confirm that on another platform with a custom walker (not libbacktrace, I would have to test that on another project), adding -1 solves some of my weird resolution issues, but seems to create other issues. Need to investigate a bit more. Never saw the problem on Win32 on same project, so AI analysis is correct in saying DbgHelp handles this internally and compensates.