Modifying the command search mechanism under Windows

[hyphenrf]

Hello all, I have a new use-case and I'd like feedback on the way I attempted to tackle it.

I've been using the Windows WSL feature on a work machine, and I installed es there as my user shell. One of the things I miss in this environment is being able to type a command somecmd and if it's not found but somecmd.exe exists, that one gets automatically resolved. I vaguely remember this being available under Cygwin, and of course it is for powershell.

In any case, I've successfully overridden %whatis to give me the desired behavior:

fn %whatis cmd {
   catch @ e {
     catch @{ throw $e }{ $&whatis $cmd.exe }
   }{
     $&whatis $cmd
   }
}

And although whatis uses the new %whatis definition, it appears this is not deep enough for the interactive loop's command resolution mechanism.

My question is, which function should I edit to set up a custom command name resolution working?

[jpco]

The quick answer: %pathsearch is the function you want to override. I'm not so familiar with WSL so there might be some gotchas I don't know about, but a somewhat quick-and-dirty override might look like:

let (ps = $fn-%pathsearch)
fn %pathsearch name {
	catch @ e {
		catch @ e2 {
			throw $e
		} {
			$ps $name^.exe
		}
	} {
		$ps $name
	}
}

There's a little nuance with this that you might want to figure out for yourself. For example, if you have a path of /bin /usr/bin, and your system has both a /bin/foo.exe and a /usr/bin/foo, which of those two binaries do you want returned for foo? That said, this should be an okay start.

---

More in-depth technical discussion follows.

In es, every command is one of three things:

1. Special syntactic forms (primitives, assignment, binders like local and for, lambdas and thunks, etc.)
2. Function names
3. Binaries

Anything in category (1) is handled specially according to what it is in particular. Everything else is passed to %whatis.

So the function whatis calls %whatis which is defined based on $&whatis. Internally, the shell uses code mostly-equivalent to $&whatis. When the shell is given a command name, it:

• looks up a variable fn-$name and if one exists, the definition is evaluated
• otherwise, if $name is an absolute path (precisely, if {~ $name (/* ./* ../*)},) that binary is executed
• otherwise, $name is passed to %pathsearch, which either returns something or throws; if it returns something, that return value is evaluated. (Did you know? %pathsearch can return stuff other than absolute paths! :)

This is all entirely implementable in es, except for one critical problem, illustrated by the following example:

; let (fn explode {echo BOOM}) {
  explode
  whatis explode
}
BOOM
explode: No such file or directory

Why does actually calling the function work, but whatis doesn't know about it? The problem is lexical binding.

The shell doesn't actually use $&whatis or %whatis directly. When the shell evaluates commands as above, the fn-$name lookup is performed in the current lexical context. But the real whatis is an already-closed-over lambda defined in its own lexical context -- it actually doesn't "know" about the definition of explode at all.

(Asymmetrically, the default definition of %whatis doesn't have this problem; if the whatis explode in the above example is changed to echo <={%whatis explode}, things seem to work. This is because fn-%whatis = $&whatis, which doesn't create any closures. If you change it to the otherwise equivalent fn-%whatis = @ {$&whatis $*}, then %whatis "breaks". Because of that, re-defining %whatis via lambdas would break stuff too.)

There is a possible way to resolve all of this, a somewhat-awful hack called $&keeplexicalbinding, which I describe, along with how it would be used for %whatis, in #92. That primitive, or something similar, would allow %whatis to be defined in pure es and the shell to call %whatis for its internal evaluation, which would allow everything to be nicely unified. But I'm not in any big rush to get $&keeplexicalbinding into the shell. Also, a pure-es %whatis is pretty hairy to edit since very core things like if and %seq are themselves functions, which would rely on %whatis to be resolved, so even in that case there would still likely be a %pathsearch hook, and that would still be the recommended function to edit :)

[hyphenrf]

Thanks for the help and the great in-depth explanation! I can't vouch for the benefit of moving name resolution out of c and into es, but I can understand the desire to minimize the c core. I've read that discussion post and there doesn't seem to be an argument for it beyond putting the idea out there. Is it really a case of diminishing returns, or do you foresee it being of great benefit if done this way?

There's a little nuance with this that you might want to figure out for yourself. For example, if you have a path of /bin /usr/bin, and your system has both a /bin/foo.exe and a /usr/bin/foo, which of those two binaries do you want returned for foo?

I think any user would expect the resolution to follow the environment layers from inner to outer; that is, the guest environment takes precedence over the host environment. I believe that's what other environments which allow this do, or at least I remember it being that way.
To answer your question more directly, there's a strict ordering here, rather than overriding pathsearch to be insensitive to a .exe extension and in effect unifying the executables under /bin and /usr/bin in your example, we override it to only look for .exe after exhausting a search for the extensionless form. You can always add .exe to foo to ensure you're looking for the right executable, and as long as access fails if and only if it exhausts the path given to it, you can be sure no such foo exists when you retry with foo.exe

I suppose added nuance here would be ensuring you don't do this when the call is explicitly foo.exe, lest you end up searching for a supposed foo.exe.exe
I'll make sure to address that in the second iteration :)

[jpco]

Is it really a case of diminishing returns, or do you foresee it being of great benefit if done this way?

It's been a long time since I first implemented this so I can't recall exactly what was in my heart at the time, but I think I was initially motivated by minimalism. It was part of a larger effort to identify and minimize calls from C into es, and rename them from $fn-%blah to $'es:blah'. I missed a couple spots because I didn't understand the codebase, or C, as well as I do now, but I nearly got everything down to just three hooks: es:whatis (discussed here), es:set (used to dispatch settor functions), and es:main (see #79).

This minimalism is nice, but when it has tradeoffs like the introduction of a primitive as semantically ugly (and powerful) as $&keeplexicalbinding, it's hard to argue for what's essentially just an aesthetic benefit. It's a more meaningful improvement to have perfect consistency between the output of whatis, the return value of %whatis, and the built-in behavior of the shell -- but I still don't know if that's enough of a benefit to justify the change. I also don't have a great idea of the performance impact of an es-written %whatis, but it seems potentially large, given %whatis needs to run a lot.

I think any user would expect the resolution to follow the environment layers from inner to outer; that is, the guest environment takes precedence over the host environment. I believe that's what other environments which allow this do, or at least I remember it being that way.

Yeah, I think this makes most sense.

I suppose added nuance here would be ensuring you don't do this when the call is explicitly foo.exe, lest you end up searching for a supposed foo.exe.exe

Oh, good call, I didn't even think of that. If you wanted, you could even prompt the user in the "fallback" cases, like

; printit Hello world!
Command 'printit' not found; run '/usr/bin/printit.exe'? (y/n): y
Hello world!