[lld][ELF] linker script lexer #1

hongyu-dev · 2024-07-04T01:36:50Z

The linker script's grammar is not complex but ambiguous due to the lack of the formal specification of the language. What we are trying to do in this and other files in LLD is to make a "reasonable" linker script processor.

Current goal:

create a new lexer that is totally separate from LLD
Hook it up with the existing parser
create unit test for linker script lexer/parser
quality of diagnostic messages
performance improvements

We also should:

start by defining the API
potential design change for parser

For error and warning messages

Using StringRef would be eariser to generate token so plan to change code design.

Decided to have the lexer to generate token as parser requests. Current lexer generates all token in vector<StringRef>. Added more TODO comments here.

This commit contains imcomplete code; this does not introduce a build error but API in this commit is not a working one. This commit is for saving changes.

Added TokenInfo struct for token kind and StringRef val and basic file setup for adding unittests for later.

Added a private function for checking linker script keyword and public functions for accessing TokenInfo variables.

This updates for keeping the commit; the unittests are not working yet

If we have single '.' we return ScriptToken::Dot, but if we have characters and symbols following it we generate token based on first not of "ABCDEFGHIJKLMNOPQRSTUVWXYZabcdefghijklmnopqrstuvwxyz" "0123456789_.$/\\~=+[]*?-!^:"

This patch adds a frame recognizer for Clang's `__builtin_verbose_trap`, which behaves like a `__builtin_trap`, but emits a failure-reason string into debug-info in order for debuggers to display it to a user. The frame recognizer triggers when we encounter a frame with a function name that begins with `__clang_trap_msg`, which is the magic prefix Clang emits into debug-info for verbose traps. Once such frame is encountered we display the frame function name as the `Stop Reason` and display that frame to the user. Example output: ``` (lldb) run warning: a.out was compiled with optimization - stepping may behave oddly; variables may not be available. Process 35942 launched: 'a.out' (arm64) Process 35942 stopped * thread #1, queue = 'com.apple.main-thread', stop reason = Misc.: Function is not implemented frame #1: 0x0000000100003fa4 a.out`main [inlined] Dummy::func(this=<unavailable>) at verbose_trap.cpp:3:5 [opt] 1 struct Dummy { 2 void func() { -> 3 __builtin_verbose_trap("Misc.", "Function is not implemented"); 4 } 5 }; 6 7 int main() { (lldb) bt * thread #1, queue = 'com.apple.main-thread', stop reason = Misc.: Function is not implemented frame #0: 0x0000000100003fa4 a.out`main [inlined] __clang_trap_msg$Misc.$Function is not implemented$ at verbose_trap.cpp:0 [opt] * frame #1: 0x0000000100003fa4 a.out`main [inlined] Dummy::func(this=<unavailable>) at verbose_trap.cpp:3:5 [opt] frame llvm#2: 0x0000000100003fa4 a.out`main at verbose_trap.cpp:8:13 [opt] frame llvm#3: 0x0000000189d518b4 dyld`start + 1988 ```

…linux (llvm#99613) Examples of the output: ARM: ``` # ./a.out AddressSanitizer:DEADLYSIGNAL ================================================================= ==122==ERROR: AddressSanitizer: SEGV on unknown address 0x0000007a (pc 0x76e13ac0 bp 0x7eb7fd00 sp 0x7eb7fcc8 T0) ==122==The signal is caused by a READ memory access. ==122==Hint: address points to the zero page. #0 0x76e13ac0 (/lib/libc.so.6+0x7cac0) #1 0x76dce680 in gsignal (/lib/libc.so.6+0x37680) llvm#2 0x005c2250 (/root/a.out+0x145250) llvm#3 0x76db982c (/lib/libc.so.6+0x2282c) llvm#4 0x76db9918 in __libc_start_main (/lib/libc.so.6+0x22918) ==122==Register values: r0 = 0x00000000 r1 = 0x0000007a r2 = 0x0000000b r3 = 0x76d95020 r4 = 0x0000007a r5 = 0x00000001 r6 = 0x005dcc5c r7 = 0x0000010c r8 = 0x0000000b r9 = 0x76f9ece0 r10 = 0x00000000 r11 = 0x7eb7fd00 r12 = 0x76dce670 sp = 0x7eb7fcc8 lr = 0x76e13ab4 pc = 0x76e13ac0 AddressSanitizer can not provide additional info. SUMMARY: AddressSanitizer: SEGV (/lib/libc.so.6+0x7cac0) ==122==ABORTING ``` AArch64: ``` # ./a.out UndefinedBehaviorSanitizer:DEADLYSIGNAL ==99==ERROR: UndefinedBehaviorSanitizer: SEGV on unknown address 0x000000000063 (pc 0x007fbbbc5860 bp 0x007fcfdcb700 sp 0x007fcfdcb700 T99) ==99==The signal is caused by a UNKNOWN memory access. ==99==Hint: address points to the zero page. #0 0x007fbbbc5860 (/lib64/libc.so.6+0x82860) #1 0x007fbbb81578 (/lib64/libc.so.6+0x3e578) llvm#2 0x00556051152c (/root/a.out+0x3152c) llvm#3 0x007fbbb6e268 (/lib64/libc.so.6+0x2b268) llvm#4 0x007fbbb6e344 (/lib64/libc.so.6+0x2b344) llvm#5 0x0055604e45ec (/root/a.out+0x45ec) ==99==Register values: x0 = 0x0000000000000000 x1 = 0x0000000000000063 x2 = 0x000000000000000b x3 = 0x0000007fbbb41440 x4 = 0x0000007fbbb41580 x5 = 0x3669288942d44cce x6 = 0x0000000000000000 x7 = 0x00000055605110b0 x8 = 0x0000000000000083 x9 = 0x0000000000000000 x10 = 0x0000000000000000 x11 = 0x0000000000000000 x12 = 0x0000007fbbdb3360 x13 = 0x0000000000010000 x14 = 0x0000000000000039 x15 = 0x00000000004113a0 x16 = 0x0000007fbbb81560 x17 = 0x0000005560540138 x18 = 0x000000006474e552 x19 = 0x0000000000000063 x20 = 0x0000000000000001 x21 = 0x000000000000000b x22 = 0x0000005560511510 x23 = 0x0000007fcfdcb918 x24 = 0x0000007fbbdb1b50 x25 = 0x0000000000000000 x26 = 0x0000007fbbdb2000 x27 = 0x000000556053f858 x28 = 0x0000000000000000 fp = 0x0000007fcfdcb700 lr = 0x0000007fbbbc584c sp = 0x0000007fcfdcb700 UndefinedBehaviorSanitizer can not provide additional info. SUMMARY: UndefinedBehaviorSanitizer: SEGV (/lib64/libc.so.6+0x82860) ==99==ABORTING ```

``` UBSan-Standalone-sparc :: TestCases/Misc/Linux/diag-stacktrace.cpp ``` `FAIL`s on 32 and 64-bit Linux/sparc64 (and on Solaris/sparcv9, too: the test isn't Linux-specific at all). With `UBSAN_OPTIONS=fast_unwind_on_fatal=1`, the stack trace shows a duplicate innermost frame: ``` compiler-rt/test/ubsan/TestCases/Misc/Linux/diag-stacktrace.cpp:14:31: runtime error: execution reached the end of a value-returning function without returning a value #0 0x7003a708 in f() compiler-rt/test/ubsan/TestCases/Misc/Linux/diag-stacktrace.cpp:14:35 #1 0x7003a708 in f() compiler-rt/test/ubsan/TestCases/Misc/Linux/diag-stacktrace.cpp:14:35 llvm#2 0x7003a714 in g() compiler-rt/test/ubsan/TestCases/Misc/Linux/diag-stacktrace.cpp:17:38 ``` which isn't seen with `fast_unwind_on_fatal=0`. This turns out to be another fallout from fixing `__builtin_return_address`/`__builtin_extract_return_addr` on SPARC. In `sanitizer_stacktrace_sparc.cpp` (`BufferedStackTrace::UnwindFast`) the `pc` arg is the return address, while `pc1` from the stack frame (`fr_savpc`) is the address of the `call` insn, leading to a double entry for the innermost frame in `trace_buffer[]`. This patch fixes this by moving the adjustment before all uses. Tested on `sparc64-unknown-linux-gnu` and `sparcv9-sun-solaris2.11` (with the `ubsan/TestCases/Misc/Linux` tests enabled).

When compiling for an SVE target we can use INDEX to generate constant fixed-length step vectors, e.g.: ``` uint32x4_t foo() { return (uint32x4_t){0, 1, 2, 3}; } ``` Currently: ``` foo(): adrp x8, .LCPI1_0 ldr q0, [x8, :lo12:.LCPI1_0] ret ``` With INDEX: ``` foo(): index z0.s, #0, #1 ret ``` The logic for this was already in `LowerBUILD_VECTOR`, though it was hidden under a check for `!Subtarget->isNeonAvailable()`. This patch refactors this to enable the corresponding code path unconditionally for constant step vectors (as long as we can use SVE for them).

We've found that basic profiling could help improving/optimizing when developing clang-tidy checks. This PR adds an extra command ``` set enable-profile (true|false) Set whether to enable matcher profiling. ``` which enables profiling queries on each file. Sample output: ``` $ cat test.cql set enable-profile true m binaryOperator(isExpansionInMainFile()) $ cat test.c int test(int i, int j) { return i + j; } $ clang-query --track-memory -f test.cql test.c -- Match #1: {{.*}}/test.c:2:10: note: "root" binds here 2 | return i + j; | ^~~~~ 1 match. ===-------------------------------------------------------------------------=== clang-query matcher profiling ===-------------------------------------------------------------------------=== Total Execution Time: 0.0000 seconds (0.0000 wall clock) ---User Time--- --System Time-- --User+System-- ---Wall Time--- ---Mem--- --- Name --- 0.0000 (100.0%) 0.0000 (100.0%) 0.0000 (100.0%) 0.0000 (100.0%) 224 {{.*}}/test.c 0.0000 (100.0%) 0.0000 (100.0%) 0.0000 (100.0%) 0.0000 (100.0%) 224 Total ```

llvm#115376) …15019)" This reverts commit 9f79615. This is breaking compiler-rt/lib/sanitizer_common/... Author knows about the breakage.

… depobj construct (llvm#114221) A codegen crash is occurring when a depend object was initialized with omp_all_memory in the depobj directive. llvm#114214 The root cause of issue looks to be the improper handling of the dependency list when omp_all_memory was specified. The change introduces the use of OMPTaskDataTy to manage dependencies. The buildDependences function is called to construct the dependency list, and the list is iterated over to emit and store the dependencies. Reduced Test Case : ``` #include <omp.h> int main() { omp_depend_t obj; #pragma omp depobj(obj) depend(inout: omp_all_memory) } ``` ``` #1 0x0000000003de6623 SignalHandler(int) Signals.cpp:0:0 llvm#2 0x00007f8e4a6b990f (/lib64/libpthread.so.0+0x1690f) llvm#3 0x00007f8e4a117d2a raise (/lib64/libc.so.6+0x4ad2a) llvm#4 0x00007f8e4a1193e4 abort (/lib64/libc.so.6+0x4c3e4) llvm#5 0x00007f8e4a10fc69 __assert_fail_base (/lib64/libc.so.6+0x42c69) llvm#6 0x00007f8e4a10fcf1 __assert_fail (/lib64/libc.so.6+0x42cf1) llvm#7 0x0000000004114367 clang::CodeGen::CodeGenFunction::EmitOMPDepobjDirective(clang::OMPDepobjDirective const&) (/opt/cray/pe/cce/18.0.1/cce-clang/x86_64/bin/clang-18+0x4114367) llvm#8 0x00000000040f8fac clang::CodeGen::CodeGenFunction::EmitStmt(clang::Stmt const*, llvm::ArrayRef<clang::Attr const*>) (/opt/cray/pe/cce/18.0.1/cce-clang/x86_64/bin/clang-18+0x40f8fac) llvm#9 0x00000000040ff4fb clang::CodeGen::CodeGenFunction::EmitCompoundStmtWithoutScope(clang::CompoundStmt const&, bool, clang::CodeGen::AggValueSlot) (/opt/cray/pe/cce/18.0.1/cce-clang/x86_64/bin/clang-18+0x40ff4fb) llvm#10 0x00000000041847b2 clang::CodeGen::CodeGenFunction::EmitFunctionBody(clang::Stmt const*) (/opt/cray/pe/cce/18.0.1/cce-clang/x86_64/bin/clang-18+0x41847b2) llvm#11 0x0000000004199e4a clang::CodeGen::CodeGenFunction::GenerateCode(clang::GlobalDecl, llvm::Function*, clang::CodeGen::CGFunctionInfo const&) (/opt/cray/pe/cce/18.0.1/cce-clang/x86_64/bin/clang-18+0x4199e4a) llvm#12 0x00000000041f7b9d clang::CodeGen::CodeGenModule::EmitGlobalFunctionDefinition(clang::GlobalDecl, llvm::GlobalValue*) (/opt/cray/pe/cce/18.0.1/cce-clang/x86_64/bin/clang-18+0x41f7b9d) llvm#13 0x00000000041f16a3 clang::CodeGen::CodeGenModule::EmitGlobalDefinition(clang::GlobalDecl, llvm::GlobalValue*) (/opt/cray/pe/cce/18.0.1/cce-clang/x86_64/bin/clang-18+0x41f16a3) llvm#14 0x00000000041fd954 clang::CodeGen::CodeGenModule::EmitDeferred() (/opt/cray/pe/cce/18.0.1/cce-clang/x86_64/bin/clang-18+0x41fd954) llvm#15 0x0000000004200277 clang::CodeGen::CodeGenModule::Release() (/opt/cray/pe/cce/18.0.1/cce-clang/x86_64/bin/clang-18+0x4200277) llvm#16 0x00000000046b6a49 (anonymous namespace)::CodeGeneratorImpl::HandleTranslationUnit(clang::ASTContext&) ModuleBuilder.cpp:0:0 llvm#17 0x00000000046b4cb6 clang::BackendConsumer::HandleTranslationUnit(clang::ASTContext&) (/opt/cray/pe/cce/18.0.1/cce-clang/x86_64/bin/clang-18+0x46b4cb6) llvm#18 0x0000000006204d5c clang::ParseAST(clang::Sema&, bool, bool) (/opt/cray/pe/cce/18.0.1/cce-clang/x86_64/bin/clang-18+0x6204d5c) llvm#19 0x000000000496b278 clang::FrontendAction::Execute() (/opt/cray/pe/cce/18.0.1/cce-clang/x86_64/bin/clang-18+0x496b278) llvm#20 0x00000000048dd074 clang::CompilerInstance::ExecuteAction(clang::FrontendAction&) (/opt/cray/pe/cce/18.0.1/cce-clang/x86_64/bin/clang-18+0x48dd074) llvm#21 0x0000000004a38092 clang::ExecuteCompilerInvocation(clang::CompilerInstance*) (/opt/cray/pe/cce/18.0.1/cce-clang/x86_64/bin/clang-18+0x4a38092) llvm#22 0x0000000000fd4e9c cc1_main(llvm::ArrayRef<char const*>, char const*, void*) (/opt/cray/pe/cce/18.0.1/cce-clang/x86_64/bin/clang-18+0xfd4e9c) llvm#23 0x0000000000fcca73 ExecuteCC1Tool(llvm::SmallVectorImpl<char const*>&, llvm::ToolContext const&) driver.cpp:0:0 llvm#24 0x0000000000fd140c clang_main(int, char**, llvm::ToolContext const&) (/opt/cray/pe/cce/18.0.1/cce-clang/x86_64/bin/clang-18+0xfd140c) llvm#25 0x0000000000ee2ef3 main (/opt/cray/pe/cce/18.0.1/cce-clang/x86_64/bin/clang-18+0xee2ef3) llvm#26 0x00007f8e4a10224c __libc_start_main (/lib64/libc.so.6+0x3524c) llvm#27 0x0000000000fcaae9 _start /home/abuild/rpmbuild/BUILD/glibc-2.31/csu/../sysdeps/x86_64/start.S:120:0 clang: error: unable to execute command: Aborted ``` --------- Co-authored-by: Chandra Ghale <ghale@pe31.hpc.amslabs.hpecorp.net>

…abort (llvm#117603) Hey guys, I found that Flang's built-in ABORT function is incomplete when I was using it. Compared with gfortran's ABORT (which can both abort and print out a backtrace), flang's ABORT implementation lacks the function of printing out a backtrace. This feature is essential for debugging and understanding the call stack at the failure point. To solve this problem, I completed the "// TODO:" of the abort function, and then implemented an additional built-in function BACKTRACE for flang. After a brief reading of the relevant source code, I used backtrace and backtrace_symbols in "execinfo.h" to quickly implement this. But since I used the above two functions directly, my implementation is slightly different from gfortran's implementation (in the output, the function call stack before main is additionally output, and the function line number is missing). In addition, since I used the above two functions, I did not need to add -g to embed debug information into the ELF file, but needed -rdynamic to ensure that the symbols are added to the dynamic symbol table (so that the function name will be printed out). Here is a comparison of the output between gfortran 's backtrace and my implementation: gfortran's implemention output: ``` #0 0x557eb71f4184 in testfun2_ at /home/hunter/plct/fortran/test.f90:5 #1 0x557eb71f4165 in testfun1_ at /home/hunter/plct/fortran/test.f90:13 llvm#2 0x557eb71f4192 in test_backtrace at /home/hunter/plct/fortran/test.f90:17 llvm#3 0x557eb71f41ce in main at /home/hunter/plct/fortran/test.f90:18 ``` my impelmention output: ``` Backtrace: #0 ./test(_FortranABacktrace+0x32) [0x574f07efcf92] #1 ./test(testfun2_+0x14) [0x574f07efc7b4] llvm#2 ./test(testfun1_+0xd) [0x574f07efc7cd] llvm#3 ./test(_QQmain+0x9) [0x574f07efc7e9] llvm#4 ./test(main+0x12) [0x574f07efc802] llvm#5 /usr/lib/libc.so.6(+0x25e08) [0x76954694fe08] llvm#6 /usr/lib/libc.so.6(__libc_start_main+0x8c) [0x76954694fecc] llvm#7 ./test(_start+0x25) [0x574f07efc6c5] ``` test program is: ``` function testfun2() result(err) implicit none integer :: err err = 1 call backtrace end function testfun2 subroutine testfun1() implicit none integer :: err integer :: testfun2 err = testfun2() end subroutine testfun1 program test_backtrace call testfun1() end program test_backtrace ``` I am well aware of the importance of line numbers, so I am now working on implementing line numbers (by parsing DWARF information) and supporting cross-platform (Windows) support.

hongyu-dev added 29 commits July 4, 2024 01:25

[lld][ELF] Added tokenizer file for linker script

85ea00c

[lld][ELF] update Linkerscript lexer

9c30902

[lld][ELF] added new files for new linker script lexer

98ea972

[lld][ELF] update swtich current char

571c9b4

[lld][ELF] updated token and lexer

ba0cdf3

[lld][ELF] added llvm source manager and SMDiagnostic

252e217

For error and warning messages

[lld][ELF] updated slicing token from char to StringRef

4cabbbb

Using StringRef would be eariser to generate token so plan to change code design.

[lld][ELF] update lexer and tokens

4cafc77

Decided to have the lexer to generate token as parser requests. Current lexer generates all token in vector<StringRef>. Added more TODO comments here.

[lld][ELF] added more switch case for generating tokens

df59067

This commit contains imcomplete code; this does not introduce a build error but API in this commit is not a working one. This commit is for saving changes.

[lld][ELF] Added TokenInfo struct and unittest files

4b0583f

Added TokenInfo struct for token kind and StringRef val and basic file setup for adding unittests for later.

[lld][ELF] Lexer update

2116448

Added a private function for checking linker script keyword and public functions for accessing TokenInfo variables.

[lld][ELF] updated unittest

c6e9ff2

This updates for keeping the commit; the unittests are not working yet

[lld][ELF] update lexer unittest

c5463eb

[lld][ELF] added more unittest cases for linker script lexer

1fde417

[lld][ELF] added * and = cases in new lexer

05c781c

[lld][ELF] change leading undercase case as identifier

200afda

[lld][ELF] NFC update identify to identifier

7077d18

[lld][ELF] added test case for PROVIDE

ef9b874

[lld][ELF] added test case and change "." tokens

5960675

If we have single '.' we return ScriptToken::Dot, but if we have characters and symbols following it we generate token based on first not of "ABCDEFGHIJKLMNOPQRSTUVWXYZabcdefghijklmnopqrstuvwxyz" "0123456789_.$/\\~=+[]*?-!^:"

[lld][ELF] fixed pos error in getDigits

882c59f

[lld][ELF] missing '-' and '+' case

b1a46ae

[lld][ELF] Added test for addr-zero

bcccd0a

[lld][ELF] Add unit test for addr.test

75fa6a5

[lld][ELF] Add unittest for align-empty.test

eecdc0b

[lld][ELF] Add test case for Memory

21b84b8

[lld][ELF] add unittest case from bss-fill.test

42af510

[lld][ELF] Add unit test case for CONSTRUCTORS

e25a2b4

[lld][ELF] Add unit test based on data-commands2.test

b4ec7d7

[lld][ELF] Add unittest for DEFINED

1f9979d

hongyu-dev closed this Jul 22, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

[lld][ELF] linker script lexer #1

[lld][ELF] linker script lexer #1

Uh oh!

hongyu-dev commented Jul 4, 2024

Uh oh!

Uh oh!

[lld][ELF] linker script lexer #1

[lld][ELF] linker script lexer #1

Uh oh!

Conversation

hongyu-dev commented Jul 4, 2024

Uh oh!

Uh oh!