`perf record -a`: all pre-existing processes show `?? [??]` — synthesized FORK events clobber their mmaps via `initAfterFork()`

Summary

In system-wide recordings (perf record -a -g), hotspot fails to symbolize every process that was already running when the recording started — all their frames show as ?? [??]. Processes spawned during the recording, and single-process recordings (perf record -g -p <pid> or launching the target), symbolize fine.

Root cause (details below): perf record -a synthesizes FORK + COMM + MMAP2 records at time == 0 for every pre-existing process. In perfparser's buffered event processing, all time-0 MMAP2 records are flushed when the first fork event is processed; each subsequent synthesized FORK(child ← parent) then calls PerfSymbolTable::initAfterFork(), which replaces the child's just-registered mmaps with the parent's. Since daemons have ppid 1, every pre-existing process ends up with a copy of systemd's address space, and none of its samples resolve.

Environment

• hotspot 1.6.0 (distro package) and the current AppImage; perfparser master as of 2026-08 shows the same code path.
• Recording taken on an embedded aarch64 target (Linux 6.1.x, perf 6.1), analyzed on x86_64 with --sysroot/--kallsyms/--arch aarch64. The bug is not specific to cross-analysis — it is in the record ordering logic.

Reproduction

1. Start a CPU-busy test program (with symbols) before recording: ./busy & (any -g-built spin loop)
2. perf record -F 99 -a -g -- sleep 10
3. Open in hotspot (or run perfparser --input perf.data --output /dev/null).
4. The pre-existing busy process (and every daemon) shows ?? [??]; anything spawned during the 10 s window resolves normally.

A/B control: record the same process with perf record -g -p $(pgrep busy) — it symbolizes perfectly with identical sysroot/debug files.

Root cause walkthrough

perf record -a synthesizes for each pre-existing process (all at time == 0):

PERF_RECORD_FORK(2707:2707):(1:1)          <- parent = pid 1
PERF_RECORD_COMM: mydaemon:2707/2707
PERF_RECORD_MMAP2 2707/2707: ... /usr/bin/mydaemon
PERF_RECORD_MMAP2 2707/2707: ... /usr/lib/libfoo.so
...

perfparser buffers task events and mmap events separately (m_taskEventsBuffer / mmap buffer) and interleaves them again at flush time in PerfUnwind::flushEventBuffer():

// perfunwind.cpp (flushEventBuffer)
if (taskEventIt->m_type == ThreadStart && taskEventIt->m_pid != taskEventIt->m_payload) {
    forwardMmapBuffer(mmapIt, mmapEnd, taskEventIt->time());   // forwards ALL mmaps with time <= 0
    ...
    symbolTable(childPid)->initAfterFork(symbolTable(parentPid));
}

Because every synthesized record has time == 0, the forwardMmapBuffer() call on the first fork event forwards all synthesized MMAP2 records for all processes. Every subsequent synthesized fork then executes initAfterFork():

// perfsymboltable.cpp
void PerfSymbolTable::initAfterFork(const PerfSymbolTable* parent)
{
    m_elfs.copyDataFrom(&parent->m_elfs);      // <- hard replacement
    m_firstElf = ElfAndFile(parent->m_firstElf.fullPath());
}

// perfelfmap.h
void copyDataFrom(const PerfElfMap *parent)
{
    m_elfs = parent->m_elfs;                   // child's own maps are discarded
    ...
}

So the child's correct, already-forwarded maps are overwritten with the parent's (pid 1 → systemd). All samples of that process then fall outside any known mapping → ?? [??].

Why the working cases work:

• Processes forked during recording: real fork/exec/mmap records have strictly increasing timestamps; initAfterFork() runs before the child's own mmaps arrive, which then correctly override the inherited copy.
• perf record -p <pid>: perf synthesizes only COMM + MMAP2 for the target — no FORK records — so initAfterFork() never runs.

Confirmation

Neutralizing the synthesized forks in the perf.data file (rewriting ppid := pid / ptid := tid for FORK records with time == 0, which makes the m_pid != m_payload guard skip initAfterFork()) fully restores symbolization of all pre-existing processes with an unmodified hotspot — in our recording, the busiest daemon went from 0 resolved frames to complete C++ symbolization incl. inline frames.

Neutralizer used for the test (stdlib-only):

#!/usr/bin/env python3
import struct, sys
src, dst = sys.argv[1], sys.argv[2]
data = bytearray(open(src, 'rb').read())
assert data[0:8] in (b'PERFILE2', b'2ELIFREP')
data_off, data_sz = struct.unpack_from('<QQ', data, 8+8+8+16)
pos, end, fixed = data_off, data_off + data_sz, 0
while pos + 8 <= end:
    rtype, misc, size = struct.unpack_from('<IHH', data, pos)
    if size < 8: break
    if rtype == 7 and pos + 32 <= end:   # PERF_RECORD_FORK
        pid, ppid, tid, ptid, t = struct.unpack_from('<IIIIQ', data, pos+8)
        if t == 0 and ppid != pid:
            struct.pack_into('<IIII', data, pos+8, pid, pid, tid, tid)
            fixed += 1
    pos += size
open(dst, 'wb').write(data)
print(f"neutralized {fixed} synthesized fork records")

Suggested fix directions

1. Don't clobber a non-empty child table: in initAfterFork(), merge or skip when the child already has registered elfs (covers any ordering).
2. Ordering: forward only mmaps with time < taskEvent->time() (strictly less) before handling a fork, so same-timestamp (synthesized) child mmaps are applied after the inherited copy and override it.
3. Skip synthesized forks: a fork record with time == 0 cannot carry meaningful inheritance information — its process's maps are synthesized separately anyway.

Sorry for the AI slop, but I don't have the time (and also knowledge) to reformulate it properly.