Make irony-mode work with Tramp

[sethfowler]

I love irony-mode. It works much better than any of the alternatives for me. One problem I've found, though, is that it doesn't seem to work correctly with Tramp. I get some sort of fall-back completion instead of irony-mode when editing remote C++ files.

Can this be fixed? I'm guessing this has something to do with either irony-server or libclang itself being unable to open the remote files. Would it be feasible to run irony-server remotely and communicate with it via TCP? Looks to me like irony-server just communicates via stdin/stdout right now, and it may not be hard to extend it to support IO over a socket as well, but I'm not sure what other implications there are.

[sethfowler]

Thinking about this a little more, this might be very simple. The exact same irony-server executable may work fine; the elisp code would just have to notice that it was looking at a file via Tramp and run irony-server on the remote server, redirecting stdin and stdout over the ssh connection.

[Sarcasm]

I would be interested in such functionality too.

Right now the "snapshot" of a file is saved in the '/tmp/' directory for irony-server to find. Issue #57 propose to just send the unsaved file content to the irony-server process as a part of the request. This should be cheaper than creating a file on disk, saving it's content, ....and in this case I think it can make this issue easier to solve since it will limit dependencies on the file system.

I don't know how to capture the stdin/stdout of a SSH process but hopefully it's not too difficult.

[Sarcasm]

Closing this.

[vibrys]

Hi.

I find Irony module very useful in everyday programming. Its auto completion is fast and precise. Additionally, appending return value as last segment of complete choice is very nice.

The thing I'm missing with it is tramp support, which I find outstanding comparing to all other IDEs. As the problem has been raised twice in the past (in this thread and in #217), I'd like to share my experience with adapting tramp to be used by some emacs module. Maybe someone will find this information useful enough in case of adapting tramp to be used with irony as well.

Tramp allows for handling files/processes that reside on some 'remote site'. 'remote site' means some user@host we have ssh access to, but it also means su/sudo user being on the same machine, some smb share etc. IMHO: from irony perspective, ssh and su/sudo (to root or any other user) are most interesting ones.

Communication with server using tcp socket does not seem to be good choice:

1. su/sudo tramp connection does not require ssh. it doesn't even require network to be enabled.
2. ssh uses TCP, but imagine the case where the only inbound TCP port Your admin keeps open is ssh (22).

Tramp is part of emacs ?core?. This implies that crucial functions used to communicate with external processes have their tramp aware counterparts, which communicate with processes residing on 'remote site'. Example: emacs communicates with sudo::irony_server. What is more, most of these functions are unified, so You do not need to have two versions of code, first version for non-tramp communication and second for tramp one.

Here is the list of functions and its tramp counterparts. Roughly, it is enough to replace one with another.

| local only                  | uniform (both local and remote). |
|-----------------------------+----------------------------------|
| call-process                | process-file                     |
| start-process               | start-file-process               |
| start-process-shell-command | start-file-process-shell-command |
| call-process-region         | ttramp-call-process-region       |
| process-send-region         | process-send-region.             |
| process-send-string         | process-send-string              |
| kill-process                | see below                        |

ttramp-call-process-region is not part of emacs ?core?, so here's the source code:

#+BEGIN_SRC emacs-lisp
(defun ttramp-call-process-region
    (start end program &optional delete buffer display &rest args)
  "Use Tramp to handle `call-process-region'.
Fixes a bug in `tramp-handle-call-process-region'.
Function based on org-babel-tramp-handle-call-process-region"
  (if (and (featurep 'tramp)
           (file-remote-p default-directory))
      (let ((tmpfile (tramp-compat-make-temp-file "")))
        (write-region start end tmpfile)
        (when delete (delete-region start end))
        (unwind-protect
            ;;  (apply 'call-process program tmpfile buffer display args)
            ;; bug in tramp
            (apply 'process-file program tmpfile buffer display args)
          (delete-file tmpfile)))
    (apply 'call-process-region
           start end program delete buffer display args)))
#+END_SRC

Tramp is aimed to be an abstraction layer to work with files and communicate processes regardless they are local or remote.

There are some small pitfalls but I believe they are not that hard to overcome (see below).

For the sake of completeness let's give example path to local file and to remote one:

/home/user/file.txt - local.
/scp:user@host:/home/user/file.txt - remote.

That would be good if irony server does not have the notion if request comes from 'local' emacs or 'remote' one. That would be even better if no changes were required to irony server at all.

To achieve that, elisp code must take care to convert remote paths to local ones when passing them as arguments to remote irony server.

example: visiting the /scp:user@host:/home/user/project/module/file.cpp will issue irony--send-request function with request="get-compile-options" and args=("/scp:user@host:/home/user/project/" "/scp:user@host:/home/user/project/module/file.cpp").

Before irony--send-request will send the request to remote irony server it has to convert remote paths to local ones.

It's as easy as:

#+BEGIN_SRC emacs-lisp
(setq args (mapcar 'untrampify args))
#+END_SRC

where:

#+BEGIN_SRC emacs-lisp
(defun untrampify (location)
  "Gets path segment from tramp path. For non-tramp location just return
it non-modified."
  (if (tramp-tramp-file-p location)
      (tramp-file-name-localname
       (tramp-dissect-file-name location))
    location))
#+END_SRC

so, after that:

args=("/home/user/project/" "/home/user/project/module/file.cpp")

Now it's server part to process the request and produce results. If result contains some paths, these will be returned to remote emacs as local paths (we want server does not have the notion whether emacs is local or remote). Thus when elisp code receives the result, it has to convert such paths back into remote ones. Here's the function:

#+BEGIN_SRC emacs-lisp
(defun trampify (absolute-location)
  "if absolute-location is remote, then return it.
Otherwise if default-directory is tramp one, then use it to convert
absolute-location to remote. Local paths for local default-directory
are untouched."
  (if (or (not (tramp-tramp-file-p default-directory))
          (tramp-tramp-file-p absolute-location))
      absolute-location
    (let ((location-vec (tramp-dissect-file-name default-directory)))
      (aset location-vec 3 absolute-location)
      (apply 'tramp-make-tramp-file-name (append location-vec nil)))))
#+END_SRC

The magic behind this function is default-directory buffer local variable. It keeps remote path for remote files and local path for local ones.

When for example You are visiting /scp:user@host:/home/user/project/module/file.cpp, then (trampify "/some/local/path") => /scp:user@host:/some/local/path. '/scp:user@host:' will be taken from default-directory.

Sometimes one really needs to know if current file is local or remote one. Here's how to get it:

#+BEGIN_SRC emacs-lisp
(tramp-tramp-file-p default-directory)
#+END_SRC

The dirty thing is about killing the remote process. The following is the rough example how it could be done, but it does not mean that there is some much better solution. Anyway I believe that the following code might be usable.

#+BEGIN_SRC emacs-lisp
(defun irony-stop-server ()
  (interactive)
  (when (process-live-p irony-server-process)
    ;; local process will be killed here.
    (kill-process irony-server-process))

  (when (process-live-p irony-server-process)
    ;; kill above failed. It's likely the remote process.
    (when (get-buffer irony-server-buffer-name)
      (with-current-buffer irony-server-buffer-name
        (when (tramp-tramp-file-p default-directory)
          ;; process runs on some remote host
          ;; We need to kill it within that context.
          (let ((result
                 (process-file "kill" nil nil nil
                               (int-to-string (process-id irony-server-process)))))
            (unless (= result 0)
              ;; the kill above did not do. Let's send KILL
              (process-file "kill" nil nil nil "-9"
                            (int-to-string (process-id irony-server-process)))))))))
  (when (get-buffer irony-server-buffer-name)
    (kill-buffer irony-server-buffer-name)))
#+END_SRC

@Sarcasm mentions:

Right now the "snapshot" of a file is saved in the '/tmp/' directory for
irony-server to find. Issue #57 propose to just send the unsaved file content to
the irony-server process as a part of the request.

This should be cheaper than creating a file on disk, saving it's content,
....and in this case I think it can make this issue easier to solve since it
will limit dependencies on the file system.

I don't know how to capture the stdin/stdout of a SSH process but hopefully it's not too difficult.

If irony server will be started with start-file-process-shell-command instead of start-process-shell-command, then already used process-send-string should handle both the local and remote (ssh/su/sudo) cases. There is also process-send-region which might be useful in this case (all in all what is going to be sent is modified buffer contents). As I understand it would imply not using temporary "snapshot" file solution at all.

I hope the information above will be a bit helpful in case someone will try to adapt tramp into irony.

Thank You very much for irony.

regards,
Mat

[Sarcasm]

Just curious, did you get irony working with tramp by doing the modification you propose?
Would it be possible to make a pull request? Even if it's only a draft it can be a good starting point.

This is a functionality I would be very interested in having if there is no limitation.
And I would be happy to test it regularly when adding new features.

You summarized well what needs to be done.
There might be some difficult parts I guess, where if the request returns a path if might need to be supported in the frontend that the path is not local.

One "feature" I'm wondering about, is whether irony-install-server could have a remote counterpart or not. That is only a nice to have.

You made the hard work, it would be super nice if you (and I, I can help) can push it further so it makes it into irony-mode. I would use the feature at work from time to time I think. :)

[vibrys]

Just curious, did you get irony working with tramp by doing the modification you propose?

I didn't do any modifications to the code.

This is a functionality I would be very interested in having if there is no limitation.

irony also uses flycheck. As far as I know flycheck does not work with tramp.
I'm not aware about any other limitations.

There might be some difficult parts I guess, where if the request returns a
path if might need to be supported in the front end that the path is not local.

For the path returned from irony server it is enough to call (trampify path).
I would not expect serious (if any) problems with it.

One "feature" I'm wondering about, is whether irony-install-server could have a remote counterpart or not.

IMHO yes. The only extra step would be to copy (using tramp) irony server sources to some remote directory before compilation.
Then "remote" make (issued from emacs) should do the job.

Maybe it is worth dividing the task into sub-steps. I included questions.

1. figure out communication between emacs and irony.

   is it the one function or many functions? Which ones?

2. figure out how to start remote irony server for proove_of_concept/development phase.

   assumption for remote projects case: one must compile server by hand? one must start it by hand? then exiting the emacs would not have to kill anything.

3. apply tramp to elisp code.

   • change all the communicating with server functions mentioned in the left column of table above into their tramp aware counterparts from right column of the table.

     are there many of them? Maybe step 1 will handle all of these places.

   • call (untrampify path) for all the paths that are just being passed as parameters to irony server. call (trampify path)for paths returned within the result from irony server.

4. let's make testing and proove the concept.

5. do graceful server compilation on remote site, just as it is being done now for local site.

6. do graceful killing of all irony server instances that were involved by currently running emacs session.

What do You think about it?

Luckily all the steps must work with local irony server as well so maybe they could be committed one by one without having to wait by the end of step 6.

Step 1 seems to me to be most difficult. It should be discussed before the subsequent steps will be started.

I'm going to be busy till the end of January 2017, so it will be a bit difficult to me to touch the code by this time, but I'm willing to help as much as I can.

regards,
Mat

[Sarcasm]

1. figure out communication between emacs and irony.

is it the one function or many functions? Which ones?

2 functions:

1. irony--send-request
2. irony--send-parse-request

irony-server is a bit like an interactive shell. It reads request from stdin and output response to stdout.

2. figure out how to start remote irony server for proove_of_concept/development phase.

assumption for remote projects case: one must compile server by hand? one must start it by hand? then exiting the emacs would not have to kill anything.

Hopefully, starting irony-server is as simple as replacing start-process-shell-command by start-file-process-shell-command in irony--start-server-process.

Also irony--start-server-process may have to be taught to associate one irony-server per host.

irony-server can be compiled by hand for starter. I think, stopping the process can be automatic when we kill the tramp buffer/session, I think it will be transparent.

3. apply tramp to elisp code.

• change all the communicating with server functions mentioned in the left column of table above into their tramp aware counterparts from right column of the table.
• are there many of them? Maybe step 1 will handle all of these places.

call (untrampify path) for all the paths that are just being passed as parameters to irony server. call (trampify path)for paths returned within the result from irony server.

Changing communication should be reasonably easy, 2 functions do the actual work. They can probably call one common function that would do the right thing for tramp.

The trampification may be more error prone but there is not that many request to handle in irony-server. Completion, diagnostics and compile options.

There is irony-cdb, the compilation database part of irony-mode may need some work too.

4. let's make testing and proove the concept.
5. do graceful server compilation on remote site, just as it is being done now for local site.
6. do graceful killing of all irony server instances that were involved by currently running emacs session.

What do You think about it?

Sounds great.

Luckily all the steps must work with local irony server as well so maybe they could be committed one by one without having to wait by the end of step 6.

Agreed, or it's okay to work on this in a branch and commit when all is working. Either way is fine for me but I may prefer to push only when at least something is working.

Step 1 seems to me to be most difficult. It should be discussed before the subsequent steps will be started.

To me step 1 is relatively easy. Since irony-server communicates with stdout/stdin, we just have to find a way to communicate with a remote process instead of local but I'm am under the impression that tramp handles this kind of things nicely.

[vibrys]

irony-server is a bit like an interactive shell. It reads request from stdin and output response to stdout.

the contents of modified buffer to be completed as well? I'm asking because, I found the following above:

Right now the "snapshot" of a file is saved in the '/tmp/' directory for irony-server to find.

I just looked at irony-cdb. To see what filesystem aware functions are used there (I hope I didn't overlook anything):

1. locate-dominating-file
2. expand-file-name
3. file-name-directory
4. file-name-as-directory

Good news is that all of them work with remote files.

[Sarcasm]

Irony used to store things in /tmp, now it sends to stdin.

Oh great, if locate-dominating-file works with tramp.
There is also some functions like file-exists-p.

[vibrys]

Irony used to store things in /tmp, now it sends to stdin.

good news. This was the reason why I classified step 1 as most difficult.

There is also some functions like file-exists-p.

works with tramp.

[root42]

I am very interested in this as well. I am still looking for a solution to have Emacs C++ code completion on my remote C++ projects. I couldn't get either auto-complete-clang nor company-clang to work remotely, and of course flycheck was never meant to really work on remote files, which is sad.

[plumenator]

Any progress on this?

[hisnawi]

Bump

[hisnawi]

Is there a way to only disable irony when accessing remote files? It gives too many messages viewing errors. So I want to disable it until irony and tramp can play nice together.

[Sarcasm]

I think you can enable irony-mode only under the following condition:

(unless (and buffer-file-name (file-remote-p buffer-file-name))
    (irony-mode 1))

[hisnawi]

Thanks for the response. I have this setup, so I guess your suggestion wont work for me?

  (use-package irony
    :ensure t
    :defer t
    :diminish irony-mode
    :preface
    (defun my/irony-mode-hook ()
      (define-key irony-mode-map [remap completion-at-point]
        'irony-completion-at-point-async)
      (define-key irony-mode-map [remap complete-symbol]
        'irony-completion-at-point-async)
      (irony-cdb-autosetup-compile-options))
    :init
    (add-hook 'c++-mode-hook #'irony-mode)
    (add-hook 'c-mode-hook #'irony-mode)
    (add-hook 'objc-mode-hook #'irony-mode)
    (add-hook 'irony-mode-hook #'my/irony-mode-hook)
)

[Sarcasm]

I did not give full code because I wanted to give you the opportunity to learn some elisp.

But here you go, you can use this config:

(use-package irony
  :ensure t
  :diminish irony-mode
  :preface
  (defun my/irony-mode ()
    (unless (and buffer-file-name (file-remote-p buffer-file-name))
      (irony-mode 1)
      (irony-cdb-autosetup-compile-options)))
  :hook ((c-mode . my/irony-mode)
         (c++-mode . my/irony-mode)
         (objc-mode . my/irony-mode)
         (c++-mode . my/irony-mode)))