Crash while parsing unsaved file

[asandroq]

Hi,

I have a specific file that is parsed correctly when saved, but crashes irony-server when sent as unsaved content. This is the backtrace:

Program terminated with signal SIGSEGV, Segmentation fault.
#0  0x00007efdb0d6bffb in ?? () from /usr/local/llvm-3.6/lib/libclang.so.3.6
gdb$ bt
#0  0x00007efdb0d6bffb in ?? () from /usr/local/llvm-3.6/lib/libclang.so.3.6
#1  0x00007efdb1bab654 in std::ios_base::Init::~Init (this=<optimized out>, __in_chrg=<optimized out>) at ../../../../../gcc-5.1.0/libstdc++-v3/src/c++98/ios_init.cc:136
#2  0x00007efdb06e65b1 in __run_exit_handlers (status=0x40f5e0, listp=0x7efdb0a62688 <__exit_funcs>, run_list_atexit=0x1) at exit.c:78
#3  0x00007efdb06e6635 in __GI_exit (status=<optimized out>) at exit.c:100
#4  0x00007efdb06cc774 in __libc_start_main (main=0x40a680 <main>, argc=0x4, ubp_av=0x7ffcfbfeb3e8, init=<optimized out>, fini=<optimized out>, rtld_fini=<optimized out>, stack_end=0x7ffcfbfeb3d8) at libc-start.c:258
#5  0x0000000000404639 in _start ()

The problem seems to be in Command.cpp, line 265:

    char nl;
    std::cin.read(&nl, 1);
    if (nl != '\n') {
      std::clog << "error: missing newline for unsaved file content\n";
      return 0;
    }

I have done some debugging and it seems that, for this specific file, when unsaved, the newline is never sent to the server. Or the server somehow loses it. I cannot explain why. Moreover, the server should not crash in this situation. I don't know if the backtrace is a hint as to why the input stream loses the newline.

The file only has ASCII characters. I also tried changing the Elisp code to send another character instead of a newline, but the end result is the same.

[Sarcasm]

Can you try to reduce the content of the file to the bare minimum and send it to me?

I'm pretty sure something doesn't work well with some files with special characters (see #163), I fear that emacs is doing some conversion or such but I don't understand all the implication of such things.

[asandroq]

I have verified this file only has ASCII characters, so I don't think it's the same problem as in #163. I'll try to look into it further.

[Sarcasm]

FYI I believe 2e73722 fixes the segfault.

This doesn't fix the missing newline issue though, it will just make the error less cryptic.

[asandroq]

Thanks for the changes! Indeed I have updated irony-mode today and have not experienced crashes so far. I get these every now and then though:

Company: An error occurred in auto-begin
Process Irony not running
irony process stopped!
Error while checking syntax automatically: (file-error "Writing to process" "bad file descriptor" #<process Irony>

and

error in process filter: irony--process-server-response: End of file during parsing
error in process filter: End of file during parsing

And not just in that same file anymore. BTW, I use irony + company + flycheck.

[Sarcasm]

Could you enable debug M-x toggle-debug-on-error RET and also give the output of the log file in /tmp/irony*.log?

[asandroq]

Turned the flag on but didn't get any extra information from Emacs. The log file has this:

execute: Command{action=Command::Complete, file=..., dir='', line=262, column=1, flags=..., unsavedFiles.count=1, opt=off}
libclang: crash detected in code completion

[asandroq]

I got a backtrace from Emacs. It was an error about removing the compiler string from the compilation command line when setting up the compilation database. I then removed all *.elc files and started Emacs again. So far so good.

[Sarcasm]

Ok. Are you able to reproduce the clang crash on the command line?

$ ~/.emacs.d/irony/bin/irony-server complete /path/to/your/file.c 262 1
-the -command -line -flags -on -this -line

[asandroq]

I do not have any more irony-server crashes. I got another backtrace from Emacs:

Debugger entered--Lisp error: (wrong-type-argument sequencep t)
  mapcar((lambda (cmd) (let ((opt (irony-cdb--remove-compiler-from-flags (car cmd))) (wdir (cdr cmd))) (cons (irony-cdb-json--adjust-compile-options opt file wdir) wdir))) t)
  irony-cdb-libclang--adjust-options-and-remove-compiler("XXXXX.cpp" t)
  (let ((build-dir (file-name-directory db-file)) (file buffer-file-name)) (irony-cdb-libclang--adjust-options-and-remove-compiler file (irony--send-request-sync "get-compile-options" build-dir file)))
  irony-cdb-libclang--server-exact-flags("/home/alex.silva/devel/codebase/compile_commands.json")
  (progn (irony-cdb-libclang--server-exact-flags it))
  (progn (progn (irony-cdb-libclang--server-exact-flags it)))
  (if it (progn (progn (irony-cdb-libclang--server-exact-flags it))))
  (let ((it (irony-cdb-json--locate-db))) (if it (progn (progn (irony-cdb-libclang--server-exact-flags it)))))
  irony-cdb-libclang--get-compile-options()
  (cond ((eql command (quote get-compile-options)) (irony-cdb-libclang--get-compile-options)))
  irony-cdb-libclang(get-compile-options)
  funcall(irony-cdb-libclang get-compile-options)
  (let ((it (funcall compilation-database (quote get-compile-options)))) (if it (progn (progn (throw (quote found) (list compilation-database (caar it) (cdar it)))))))
  (condition-case err (let ((it (funcall compilation-database (quote get-compile-options)))) (if it (progn (progn (throw (quote found) (list compilation-database (caar it) (cdar it))))))) ((debug error) (message "Irony CDB: error in compilation database: %S" err) nil))
  (while --dolist-tail-- (setq compilation-database (car --dolist-tail--)) (condition-case err (let ((it (funcall compilation-database (quote get-compile-options)))) (if it (progn (progn (throw (quote found) (list compilation-database ... ...)))))) ((debug error) (message "Irony CDB: error in compilation database: %S" err) nil)) (setq --dolist-tail-- (cdr --dolist-tail--)))
  (let ((--dolist-tail-- irony-cdb-compilation-databases) compilation-database) (while --dolist-tail-- (setq compilation-database (car --dolist-tail--)) (condition-case err (let ((it (funcall compilation-database (quote get-compile-options)))) (if it (progn (progn (throw ... ...))))) ((debug error) (message "Irony CDB: error in compilation database: %S" err) nil)) (setq --dolist-tail-- (cdr --dolist-tail--))))
  (catch (quote found) (let ((--dolist-tail-- irony-cdb-compilation-databases) compilation-database) (while --dolist-tail-- (setq compilation-database (car --dolist-tail--)) (condition-case err (let ((it (funcall compilation-database ...))) (if it (progn (progn ...)))) ((debug error) (message "Irony CDB: error in compilation database: %S" err) nil)) (setq --dolist-tail-- (cdr --dolist-tail--)))))
  irony-cdb--autodetect-compile-options()
  (let ((it (irony-cdb--autodetect-compile-options))) (if it (progn (progn (setq irony-cdb--compilation-database (nth 0 it)) (irony-cdb--update-compile-options (nth 1 it) (nth 2 it))))))
  irony-cdb-autosetup-compile-options()
  run-hooks(irony-mode-hook irony-mode-on-hook)
  (let ((last-message (current-message))) (setq irony-mode (if (eq arg (quote toggle)) (not irony-mode) (> (prefix-numeric-value arg) 0))) (if irony-mode (irony--mode-enter) (irony--mode-exit)) (run-hooks (quote irony-mode-hook) (if irony-mode (quote irony-mode-on-hook) (quote irony-mode-off-hook))) (if (called-interactively-p (quote any)) (progn nil (if (and (current-message) (not (equal last-message (current-message)))) nil (message "Irony mode %sabled" (if irony-mode "en" "dis"))))))
  irony-mode(1)
  qnective-c++-mode-hook()
  run-hooks(change-major-mode-after-body-hook prog-mode-hook c-mode-common-hook c++-mode-hook c++-mode-hook)
  apply(run-hooks (change-major-mode-after-body-hook prog-mode-hook c-mode-common-hook c++-mode-hook c++-mode-hook))
  run-mode-hooks(c++-mode-hook)
  c++-mode()
  set-auto-mode-0(c++-mode nil)
  set-auto-mode()
  normal-mode(t)
  after-find-file(nil t)
  find-file-noselect-1(#<buffer XXXXX.cpp> "XXXXX.cpp" nil nil "XXXXX.cpp" (4329526 64515))
  find-file-noselect("XXXXX.cpp")
  ad-Advice-compilation-find-file(#[(marker filename directory &rest formats) "�\204��\306�   \n\203��\307\n!\202���\310\211\211\211���'�(�)�*\311�+!\203>�\312\307�+!!�+\313�+!�*B�*\314�+!�+�*\203\203��(\204\203��*@\206P��)�'�\211�\203z��(\204z�\307\315
@�+\"�'\"�\316\f!\205q�\317\f!�(
A\211�\204X��*A\211�*\204C��(\204@�\212\320�,\321\322�-!\323\")�.r\322�-!q\210�-b\210�.\203\352��.�-�/�.�0\250\203\312�\324�.\212�/b\210\325\326�0Z!\210`)\"\210\202\342�\327 @\330\232\203\342�\324�.\212�/b\210\325\326!\210`)\"\210\331�.�/\"*\210)\332\315\333�1�+#�)�+\320\310%\211��2\316\f!\204��\334\335\f\"\210\336 \210\337\340!\210\202<�\341\f!\2037�\316\307�+\f\"\211�!\2047�\334\342�+�2#\210\336 \210\337\340!\210\202<�\317\f!�(,\202\203�\343ed\"\310�3\211�4\203j��4@�3\344�3\345\"\203a�\346�3\345\310#\210�4A\211�4\204M�*�(.�\207" [formats compilation-search-path directory default-directory name fmts ("%s") expand-file-name nil file-name-absolute-p abbreviate-file-name file-name-directory file-name-nondirectory format file-exists-p find-file-noselect t display-buffer marker-buffer (nil (allow-no-window . t)) set-window-start beginning-of-line 1 window-fringes 0 set-window-point read-file-name "Find this %s in (default %s): " message "Cannot find file `%s'" ding sit-for 2 file-directory-p "No `%s' in directory %s" overlays-in overlay-get intangible overlay-put thisdir buffer spec-dir dirs filename pop-up-windows marker w mk compilation-context-lines compilation-error ...] 7 ("/usr/local/emacs/share/emacs/24.5/lisp/progmodes/compile.elc" . 81415)] #<marker at 291 in *ggtags-global*> "XXXXX.cpp" nil)
  apply(ad-Advice-compilation-find-file #[(marker filename directory &rest formats) "�\204��\306� \n\203��\307\n!\202���\310\211\211\211���'�(�)�*\311�+!\203>�\312\307�+!!�+\313�+!�*B�*\314�+!�+�*\203\203��(\204\203��*@\206P��)�'�\211�\203z��(\204z�\307\315
@�+\"�'\"�\316\f!\205q�\317\f!�(
A\211�\204X��*A\211�*\204C��(\204@�\212\320�,\321\322�-!\323\")�.r\322�-!q\210�-b\210�.\203\352��.�-�/�.�0\250\203\312�\324�.\212�/b\210\325\326�0Z!\210`)\"\210\202\342�\327 @\330\232\203\342�\324�.\212�/b\210\325\326!\210`)\"\210\331�.�/\"*\210)\332\315\333�1�+#�)�+\320\310%\211��2\316\f!\204��\334\335\f\"\210\336 \210\337\340!\210\202<�\341\f!\2037�\316\307�+\f\"\211�!\2047�\334\342�+�2#\210\336 \210\337\340!\210\202<�\317\f!�(,\202\203�\343ed\"\310�3\211�4\203j��4@�3\344�3\345\"\203a�\346�3\345\310#\210�4A\211�4\204M�*�(.�\207" [formats compilation-search-path directory default-directory name fmts ("%s") expand-file-name nil file-name-absolute-p abbreviate-file-name file-name-directory file-name-nondirectory format file-exists-p find-file-noselect t display-buffer marker-buffer (nil (allow-no-window . t)) set-window-start beginning-of-line 1 window-fringes 0 set-window-point read-file-name "Find this %s in (default %s): " message "Cannot find file `%s'" ding sit-for 2 file-directory-p "No `%s' in directory %s" overlays-in overlay-get intangible overlay-put thisdir buffer spec-dir dirs filename pop-up-windows marker w mk compilation-context-lines compilation-error ...] 7 ("/usr/local/emacs/share/emacs/24.5/lisp/progmodes/compile.elc" . 81415)] (#<marker at 291 in *ggtags-global*> "XXXXX.cpp" nil))
  compilation-find-file(#<marker at 291 in *ggtags-global*> "XXXXX.cpp" nil)
  apply(compilation-find-file #<marker at 291 in *ggtags-global*> "XXXXX.cpp" nil nil)
  compilation-next-error-function(0 nil)
  next-error-internal()
  compile-goto-error()
  compilation-auto-jump(#<buffer *ggtags-global*> 291)
  apply(compilation-auto-jump (#<buffer *ggtags-global*> 291))
  byte-code("r\301�\302H�\303H\"\210)\301\207" [timer apply 5 6] 4)
  timer-event-handler([t 21963 13103 158688 nil compilation-auto-jump (#<buffer *ggtags-global*> 291) nil 188000])
  input-pending-p(t)
  sit-for(0)
  (if (let* ((val (compilation-next-single-property-change (point-min) (quote compilation-message)))) (if (let* ((pt val)) pt) (let ((pt val)) (compilation-next-single-property-change (save-excursion (goto-char pt) (end-of-line) (point)) (quote compilation-message))) nil)) (and ggtags-navigation-mode (ggtags-global--display-buffer)) (sit-for 0) (ggtags-navigation-mode -1) (ggtags-navigation-mode-cleanup buf 0))
  (cond (ggtags-global-continuation (let ((cont (prog1 ggtags-global-continuation (setq ggtags-global-continuation nil)))) (funcall cont buf how))) ((string-prefix-p "exited abnormally" how) (ggtags-global--display-buffer) (if (save-excursion (goto-char (point-max)) (re-search-backward (quote "^global: \\(?:G\\(?:PATH\\|\\(?:RTAG\\|SYM\\|TAG\\)S\\)\\) not found.$") nil t)) (progn (ggtags-echo "WARNING: Global tag files missing in `%s'" ggtags-project-root) (remhash ggtags-project-root ggtags-projects)))) (ggtags-auto-jump-to-match (if (let* ((val (compilation-next-single-property-change (point-min) (quote compilation-message)))) (if (let* ((pt val)) pt) (let ((pt val)) (compilation-next-single-property-change (save-excursion ... ... ...) (quote compilation-message))) nil)) (and ggtags-navigation-mode (ggtags-global--display-buffer)) (sit-for 0) (ggtags-navigation-mode -1) (ggtags-navigation-mode-cleanup buf 0))))
  ggtags-global-handle-exit(#<buffer *ggtags-global*> "finished\n")
  run-hook-with-args(ggtags-global-handle-exit #<buffer *ggtags-global*> "finished\n")
  compilation-handle-exit(exit 0 "finished\n")
  compilation-sentinel(#<process global> "finished\n")

[Sarcasm]

Could you show some of the content of /home/alex.silva/devel/codebase/compile_commands.json? Was it generated by cmake or some other tools?

@Hylen may be of help too.

It may be nice to test the get-compile-options in the console to see if you have some weird output.

~/.emacs.d/irony/bin/irony-server get-compile-options <build-dir> <file>

Does it happens every single time or is it intermittent?

[Hylen]

It looks as if the call to irony--send-request-sync in irony-cdb-libclang--server-exact-flags returns t. I can't guess why it would do that. I don't think the above call to irony-server would give t, so maybe the callback stack is messed up or something else is wrong on the elisp side.

[asandroq]

@Sarcasm It is intermittent. Testing get-compile-options from the command line works.

[Sarcasm]

Yeah, t looks like it could be a parse answer, this one is triggered by flycheck, so maybe get-compile-options get the answer from a parse instead of get-compile-options.

[Hylen]

@asandroq, can you disable flycheck and see if you still get the error?

[asandroq]

Without flycheck everything seems to work fine (irony, company-irony, company, projectile). Which is a shame, I really like the flycheck integration. :)

It is weird because I don't see the "newline missing at the end" error anymore, although AFAIK the function used by irony.el to send the buffer to the server is the same for both company-irony and flycheck-irony.

UPDATE: Scratch this, I still see problems.

From Emacs:

error in process filter: let: End of file during parsing
error in process filter: End of file during parsing

From the irony log:

libclang: crash detected in code completion

[asandroq]

Core was generated by `/home/alex.silva/.emacs.d/irony/bin/irony-server -i'.
Program terminated with signal SIGABRT, Aborted.
#0  0x00007fe48b238cc9 in __GI_raise (sig=sig@entry=0x6) at ../nptl/sysdeps/unix/sysv/linux/raise.c:56
56  ../nptl/sysdeps/unix/sysv/linux/raise.c: No such file or directory.
gdb$ bt
#0  0x00007fe48b238cc9 in __GI_raise (sig=sig@entry=0x6) at ../nptl/sysdeps/unix/sysv/linux/raise.c:56
#1  0x00007fe48b23c0d8 in __GI_abort () at abort.c:89
#2  0x00007fe48b231b86 in __assert_fail_base (fmt=0x7fe48b382830 "%s%s%s:%u: %s%sAssertion `%s' failed.\n%n", assertion=assertion@entry=0x7fe48d0dad70 "TemplateIds.empty() && \"Still alive TemplateIdAnnotations around?\"", file=file@entry=0x7fe48d0dacf8 "/home/alex.silva/Downloads/llvm/llvm-3.6.2.src/tools/clang/lib/Parse/Parser.cpp", line=line@entry=0x1a1, function=function@entry=0x7fe48d0dc8c0 <clang::Parser::~Parser()::__PRETTY_FUNCTION__> "virtual clang::Parser::~Parser()") at assert.c:92
#3  0x00007fe48b231c32 in __GI___assert_fail (assertion=0x7fe48d0dad70 "TemplateIds.empty() && \"Still alive TemplateIdAnnotations around?\"", file=0x7fe48d0dacf8 "/home/alex.silva/Downloads/llvm/llvm-3.6.2.src/tools/clang/lib/Parse/Parser.cpp", line=0x1a1, function=0x7fe48d0dc8c0 <clang::Parser::~Parser()::__PRETTY_FUNCTION__> "virtual clang::Parser::~Parser()") at assert.c:101
#4  0x00007fe48cb49379 in clang::Parser::~Parser (this=0x7fe482612a90, __in_chrg=<optimized out>) at /home/alex.silva/Downloads/llvm/llvm-3.6.2.src/tools/clang/lib/Parse/Parser.cpp:417
#5  0x00007fe48cb495e8 in clang::Parser::~Parser (this=0x7fe482612a90, __in_chrg=<optimized out>) at /home/alex.silva/Downloads/llvm/llvm-3.6.2.src/tools/clang/lib/Parse/Parser.cpp:418
#6  0x00007fe48cb45c66 in llvm::CrashRecoveryContextDeleteCleanup<clang::Parser>::recoverResources (this=0x7fe48262b080) at /home/alex.silva/Downloads/llvm/llvm-3.6.2.src/include/llvm/Support/CrashRecoveryContext.h:168
#7  0x00007fe48ca478fd in llvm::CrashRecoveryContext::~CrashRecoveryContext (this=0x7ffc754d9160, __in_chrg=<optimized out>) at /home/alex.silva/Downloads/llvm/llvm-3.6.2.src/lib/Support/CrashRecoveryContext.cpp:83
#8  0x00007fe48bf38f90 in clang_codeCompleteAt (TU=0x7fe484000e20, complete_filename=0x13efab8 "/home/alex.silva/devel/codebase/Modules/Qt/Tests/ServerEngineOnlineTest/Source/LoginTest.cpp", complete_line=0x125, complete_column=0x32, unsaved_files=0x13ef820, num_unsaved_files=0x1, options=0x5) at /home/alex.silva/Downloads/llvm/llvm-3.6.2.src/tools/clang/tools/libclang/CIndexCodeCompletion.cpp:843
#9  0x0000000000420be8 in Irony::complete (this=0x7ffc754d9818, file=..., line=0x125, col=0x32, flags=..., unsavedFiles=...) at /home/alex.silva/.emacs.d/elpa/irony-20150810.128/server/src/Irony.cpp:175
#10 0x0000000000427f82 in main (ac=0x2, av=0x7ffc754d99b8) at /home/alex.silva/.emacs.d/elpa/irony-20150810.128/server/src/main.cpp:204

[ghost]

@asandroq that's an assertion failing inside clang, are you using a recent llvm? if you are and you can provide input for repro, that's a reportable bug in clang. It looks like it's catching a memory leak in this case.

[Hylen]

I think I've observed the original error with the missing newline lately at work. I don't have much time to debug while working however, and I can't paste any logs or code. What's weird is I only see it at work, and not at home.

In the log file, I'm seeing the message

error: missing newline for unsaved file content

It seems to happen for about every file I open and edit. After removing the "return 0", I got the error

error: invalid command specified: arse

which makes it seem as if the size of the buffer gets calculated wrong. I will investigate more closely when I have time.

Since this doesn't seem to be the error here anymore, should I open a new issue for it?

[Sarcasm]

I guess @rochg opened an issue for this today, see #251.
It seems that both of you came to a similar diagnostic:

Instead of the newline character it was seeing the first character of the next request e.g. a 'p' to start a parse request

Let's continue the discussion on this in issue #251