I took the debian/sid package of chromium and re-built it in a pretty old environment (a mix of Ubuntu 16 and 18). After some patching: The local built of chromium seems to work, but on the console it crashes with this message:
*** stack smashing detected ***
Crash Reports
Chromium has internal crash reports which it could upload, but for my custom build, this doesn’t make much sense. This URL: chrome://crashes shows the available ones.
They are available in the home directory of the user at ~/.config/chromium/Crash Reports/pending/ and can be copied elsewhere. There are always two files:
3dafacfb-ddb1-4b0b-936a-09741ac5da47.dmp
3dafacfb-ddb1-4b0b-936a-09741ac5da47.meta
To make sense of them, there is an explanation page. Ok, so I checked out the repo and built minidump_stackwalk and dump_syms.
$ minidump_stackwalk 3dafacfb-ddb1-4b0b-936a-09741ac5da47.dmp
[...lots of output...]
2022-11-24 12:32:55: stackwalker.cc:103: INFO: Couldn't load symbols for: libc.so.6|C9EE994AB451E675BF88D7CC48BC87370
2022-11-24 12:32:55: stackwalker.cc:103: INFO: Couldn't load symbols for: /usr/lib/chromium/chromium|966DB22D6096B879BE334E1BA0DCCDAA0
[...]
Crash reason: SIGABRT
Crash address: 0x0
Process uptime: 0 seconds
Thread 0 (crashed)
0 libc.so.6 + 0x35438
rax = 0x0000000000000000 rdx = 0x0000000000000006
rcx = 0x00007f667b24f438 rbx = 0x00000000000001b3
rsi = 0x0000000000000001 rdi = 0x0000000000000001
rbp = 0x00007ffd5eae9cf0 rsp = 0x00007ffd5eae99d8
r8 = 0x3933323936323631 r9 = 0x3736333830323736
r10 = 0x0000000000000008 r11 = 0x0000000000000202
r12 = 0x00000000000001b3 r13 = 0x00007ffd5eae9b68
r14 = 0x00007ffd5eae9b68 r15 = 0x0000000000000001
rip = 0x00007f667b24f438
Found by: given as instruction pointer in context
1 libc.so.6 + 0x3703a
rbp = 0x00007ffd5eae9cf0 rsp = 0x00007ffd5eae99e0
rip = 0x00007f667b25103a
Found by: stack scanning
[...]
A crash, but without debug symbols not very useful.
To get the full debug symbols I rebuilt chromium with “-g2” (which needs a lot of main memory for linking). In debian/rules change symbol_level=0 to symbol_level=2.
After building, the chromium binary is here: out/Release/chrome (6.5GB!).
To extract the debug symbols for libc6 I use the debug package:
$ apt download libc6-dbg
$ aunpack ./libc6-dbg*.deb
$ dump_syms ./usr/lib/debug/.build-id/d8/4a78a23a590014d52c08c51066791797014213.debug /usr/debug/lib > libc.so.6/C9EE994AB451E675BF88D7CC48BC87370/libc.so.6.sym
For chromium:
$ dump_syms ../chromium/out/Release/chrome /usr/debug/lib > chromium/966DB22D6096B879BE334E1BA0DCCDAA0/chromium.sym
Now the trace looks better:
Thread 0 (crashed)
0 libc.so.6!_fini + 0x22964
rax = 0x0000000000000000 rdx = 0x0000000000000006
rcx = 0x00007f667b24f438 rbx = 0x00000000000001b3
rsi = 0x0000000000000001 rdi = 0x0000000000000001
rbp = 0x00007ffd5eae9cf0 rsp = 0x00007ffd5eae99d8
r8 = 0x3933323936323631 r9 = 0x3736333830323736
r10 = 0x0000000000000008 r11 = 0x0000000000000202
r12 = 0x00000000000001b3 r13 = 0x00007ffd5eae9b68
r14 = 0x00007ffd5eae9b68 r15 = 0x0000000000000001
rip = 0x00007f667b24f438
Found by: given as instruction pointer in context
1 libc.so.6!_fini + 0x24566
rbp = 0x00007ffd5eae9cf0 rsp = 0x00007ffd5eae99e0
rip = 0x00007f667b25103a
Found by: stack scanning
2 chromium!base::ScopedClosureRunner::~ScopedClosureRunner() [callback_helpers.cc : 28 + 0x8]
rbp = 0x00007ffd5eae9cf0 rsp = 0x00007ffd5eae9a70
rip = 0x00005618310150f0
Found by: stack scanning
3 libc.so.6!_fini + 0x106878
rbx = 0x0000561837437070 rbp = 0x00007f667b3a9681
rsp = 0x00007ffd5eae9d00 rip = 0x00007f667b33334c
Found by: call frame info
4 libc.so.6!_fini + 0x10681c
rsp = 0x00007ffd5eae9d20 rip = 0x00007f667b3332f0
Found by: stack scanning
5 chromium!content::ContentMainRunnerImpl::Run() [partition_alloc_support.h : 0 + 0xa]
rsp = 0x00007ffd5eae9d30 rip = 0x0000561830b8fb7a
Found by: stack scanning
6 chromium!base::CommandLine::GetSwitchValueASCII[abi:cxx11](base::BasicStringPiece<char, std::char_traits<char> >) const [command_line.cc : 308 + 0x5]
rsp = 0x00007ffd5eae9d60 rip = 0x00005618310161c3
Found by: stack scanning
7 chromium!content::(anonymous namespace)::IsSubprocess() [basic_string.h : 6045 + 0x5]
rsp = 0x00007ffd5eae9dd0 rip = 0x0000561830b8d796
Found by: stack scanning
[...]
Hmm. libc and _fini and some destructor. Didn’t give me much clue yet.
GDB
Now, with the full debug build it should also be possible to run chromium with gdb and see a backtrace. The start script has even support for that: ‘chromium -g’. But then the crashes were gone.
Also, my target machine didn’t have enough disk space for this large binary. Shouldn’t it be possible to have proper
coredumps which I can then look at on a bigger machine? This option helps: --disable-in-process-stack-traces, but
is not enough, --no-sandbox needs also to be used:
$ /usr/lib/chromium/chromium --disable-in-process-stack-traces --no-sandbox
Strip down the binary and scp to target machine:
$ strip -o chrome.stripped chrome
$ scp chrome.stripped target:/usr/lib/chromium/chromium
Now I get a proper coredump (a few immediately on startup) which I can copy off the machine and look at:
$ gdb -c core.chromium.4997.1669046723 ../chromium/out/Release/chrome
Hmm, this didn’t go well: there are several libs in /usr/lib/chromium which are also expected. Ok, cheating a little:
$ mkdir chromium-installed
$ rsync -avP target:/usr/lib/chromium/ chromium-installed/
$ sudo ln -s `pwd`/chromium-installed /usr/lib/chromium # no chromium installed on analysis machine
Now it works:
$ gdb -c core.chromium.4997.1669046723 /usr/lib/chromium/chromium
[...]
Core was generated by `/usr/lib/chromium/chromium --type=utility --utility-sub-type=data_decoder.mojom'.
Program terminated with signal SIGABRT, Aborted.
#0 0x00007f9c18a59438 in __GI_raise (sig=sig@entry=6) at ../sysdeps/unix/sysv/linux/raise.c:54
54 ../sysdeps/unix/sysv/linux/raise.c: No such file or directory.
[Current thread is 1 (Thread 0x7f9c128fdc00 (LWP 4997))]
(gdb) bt
#0 0x00007f9c18a59438 in __GI_raise (sig=sig@entry=6) at ../sysdeps/unix/sysv/linux/raise.c:54
#1 0x00007f9c18a5b03a in __GI_abort () at abort.c:89
#2 0x00007f9c18a9b7fa in __libc_message (do_abort=do_abort@entry=1, fmt=fmt@entry=0x7f9c18bb369f "*** %s ***: %s terminated\n") at ../sysdeps/posix/libc_fatal.c:175
#3 0x00007f9c18b3d34c in __GI___fortify_fail (msg=<optimized out>, msg@entry=0x7f9c18bb3681 "stack smashing detected") at fortify_fail.c:37
#4 0x00007f9c18b3d2f0 in __stack_chk_fail () at stack_chk_fail.c:28
#5 0x000055855cc84b7a in content::ContentMainRunnerImpl::Run() (this=0x55856352c070) at ../../content/common/partition_alloc_support.h:55
#6 0x000055855cc81dc4 in content::RunContentProcess(content::ContentMainParams, content::ContentMainRunner*) (params=..., content_main_runner=0x55856352c070) at ../../content/app/content_main.cc:433
#7 0x000055855cc82844 in content::ContentMain(content::ContentMainParams) (params=...) at ../../content/app/content_main.cc:461
#8 0x00005585593bc265 in ChromeMain(int, char const**) (argc=<optimized out>, argv=0x7ffd16aa17b8) at ../../chrome/app/chrome_main.cc:182
#9 0x00007f9c18a44840 in __libc_start_main (main=0x5585593bc110 <main(int, char const**)>, argc=8, argv=0x7ffd16aa17b8, init=<optimized out>, fini=<optimized out>, rtld_fini=<optimized out>, stack_end=0x7ffd16aa17a8) at ../csu/libc-start.c:291
#10 0x00005585593bc029 in _start ()
This looks more useful. The crash happened in content::ContentMainRunnerImpl::Run(). This was already visible in
the minidump above, but got confused by ~ScopedClosureRunner and thought, this might be the problem.
Code looks like this content/app/content_main_runner_impl.cc:1006:
|
|
Two clues:
- This function has NO_STACK_PROTECTOR set, because the protector must be off here it seems
- The comment hints for a command line option
Regarding the second clue: Google helps of course:
“Adds the command line switch to Chrome that changes the stack canary on fork, which strengthens the stack canary security mitigation, which can currently be brute forced.
This command line switch can be force disabled manually on the browser command line. This is done so in PrepareBrowserCommandLineForTests() as changing the stack canary will cause crashes when the browser terminates, and we don’t need hardened security mitigations for all of our tests.
This only has an effect on Linux and ChromeOS where we fork.”
Running like this:
$ chromium --change-stack-guard-on-fork=disable
and crash is gone. Yeah.
GDB disassemble
But wait; why does it crash in the first place? Shouldn’t this function be not protected?
(gdb) disassemble content::ContentMainRunnerImpl::Run
No type "ContentMainRunnerImpl" within class or namespace "content".
Huh?
(gdb) frame 5
#5 0x000055855cc84b7a in content::ContentMainRunnerImpl::Run (this=0x55856352c070) at ../../content/common/partition_alloc_support.h:55
55 ../../content/common/partition_alloc_support.h: No such file or directory.
(gdb) disassemble
Dump of assembler code for function _ZN7content21ContentMainRunnerImpl3RunEv:
0x000055855cc848a0 <+0>: push %rbp
0x000055855cc848a1 <+1>: mov %rsp,%rbp
0x000055855cc848a4 <+4>: push %r15
0x000055855cc848a6 <+6>: push %r14
0x000055855cc848a8 <+8>: push %rbx
0x000055855cc848a9 <+9>: sub $0xc8,%rsp
0x000055855cc848b0 <+16>: mov %rdi,%r15
0x000055855cc848b3 <+19>: mov %fs:0x28,%rax
[...]
0x000055855cc84b75 <+725>: call 0x558562ba2b20 <__stack_chk_fail@plt>
[...]
According to this guide: “On amd64 magic value is %fs:0x28.”. Yes, check is present here. And at the end a call to stack_chk_fail. This function is protected.
Well, for now the workaround is good enough. TODO: figure out the root cause. Compiler bug? Some mis-configuration in the build system? Not today.