Frecency using fre and fd

[elanmed]

Hey all, I was inspired by this reddit comment which suggested using the fre command to implement a frecency picker in fzf-lua, and I decided to give it a try.

This code does a few things:

• Use vim.system to run the fre --sorted command. vim.system returns a newline-delimited string (when passing text=true), which I convert to a table and iterate over. I use coroutines while iterating so I can open the fzf lua UI immediately and iteratively add each item to the list - based on the great fzf-lua wiki. I noticed that fre files can have duplicates, so I filter those out.
• fre --sorted only populates files that have been explicitly tracked with fre - not every file in the cwd - so I also populate the UI with fd by calling vim.system. By calling run_fd as an on_exit callback, I can ensure that the fre files will always appear at the top of the UI
• I wrap the default enter action to call fre --add with the selected file

local function fre_and_all_files(opts)
  opts = opts or {}
  local cwd = opts.cwd or vim.fn.getcwd()

  local contents = function(fzf_cb)
    local seen = {}

    --- @param on_exit function
    local function run_fre(on_exit)
      vim.system({ "fre", "--sorted", }, {
        text = true,
        stdout = function(err, data)
          if err then return vim.notify(err, vim.log.levels.ERROR) end
          if type(data) ~= "string" then return end
          local files = vim.split(data, "\n")

          coroutine.wrap(function()
            local co = coroutine.running()
            for _, abs_file in ipairs(files) do
              -- `fre` can have files from any directory
              if not vim.startswith(abs_file, cwd) then goto continue end
              -- `fre` can have duplicates, need to check `seen`
              if seen[abs_file] then goto continue end
              seen[abs_file] = true

              local rel_file = vim.fs.relpath(cwd, abs_file)
              local entry = fzf_lua.make_entry.file(rel_file, opts)
              fzf_cb(entry, function()
                coroutine.resume(co)
              end)
              coroutine.yield()

              ::continue::
            end
          end)()
        end,
      }, function()
        on_exit()
      end)
    end

    local function run_fd()
      local fd_cmd = { "fd", "--absolute-path", "--type", "f", "--base-directory", cwd, }

      vim.system(fd_cmd, {
          text = true,
          stdout = function(err, data)
            if err then return vim.notify(err, vim.log.levels.ERROR) end
            if type(data) ~= "string" then return end
            local files = vim.split(data, "\n")

            coroutine.wrap(function()
              local co = coroutine.running()
              for _, abs_file in ipairs(files) do
                -- `fd` only searches in the cwd, no need to filter out other directories
                -- `fd` has no duplicates, no need to add to `seen`
                if seen[abs_file] then goto continue end

                local rel_file = vim.fs.relpath(cwd, abs_file)
                local entry = fzf_lua.make_entry.file(rel_file, opts)
                fzf_cb(entry, function()
                  coroutine.resume(co)
                end)
                coroutine.yield()

                ::continue::
              end
            end)()
          end,
        },
        function()
          fzf_cb(nil)
        end)
    end

    run_fre(run_fd)
  end

  local wrapped_enter = function(action)
    return function(selected, action_opts)
      for _, sel in ipairs(selected) do
        -- based on https://github.com/ibhagwan/fzf-lua/blob/main/lua/fzf-lua/actions.lua#L147
        local file = fzf_lua.path.entry_to_file(sel, action_opts, action_opts._uri)
        vim.system { "fre", "--add", file.path, }
      end

      return action(selected, action_opts)
    end
  end

  local actions = vim.tbl_extend("force", fzf_lua.defaults.actions.files, {
    enter = wrapped_enter(fzf_lua.defaults.actions.files.enter),
  })
  local default_opts = { actions = actions, }
  local fzf_exec_opts = vim.tbl_extend("force", default_opts, opts)
  fzf_lua.fzf_exec(contents, fzf_exec_opts)
end

@ibhagwan if this is the wrong forum for these kinds of posts let me know 😅

[elanmed]

An alternate version of contents using plenary.job lets you skip the loop and coroutines, since on_stdout is called with every line:

  local contents = function(fzf_cb)
    local seen = {}

    --- @param on_exit function
    local function run_fre(on_exit)
      Job:new {
        command = "fre",
        args = { "--sorted", },
        on_stdout = function(err, abs_file)
          if err then return vim.notify(err, vim.log.levels.ERROR) end
          -- `fre` can have files from any directory
          if not vim.startswith(abs_file, cwd) then return end
          -- `fre` can have duplicates, need to check `seen`
          if seen[abs_file] then return end
          seen[abs_file] = true

          local rel_file = vim.fs.relpath(cwd, abs_file)
          local entry = fzf_lua.make_entry.file(rel_file, opts)
          fzf_cb(entry)
        end,
        on_exit = on_exit,
      }:start()
    end

    local function run_fd()
      local fd_args = { "--hidden", "--absolute-path", "--type", "f", "--base-directory", cwd, }

      Job:new {
        command = "fd",
        args = fd_args,
        on_stdout = function(err, abs_file)
          if err then return vim.notify(err, vim.log.levels.ERROR) end
          -- `fd` only searches in the cwd, no need to filter out
          -- `fd` has no duplicates, no need to add to `seen`
          if seen[abs_file] then return end

          local rel_file = vim.fs.relpath(cwd, abs_file)
          local entry = fzf_lua.make_entry.file(rel_file, opts)
          fzf_cb(entry)
        end,
        on_exit = function()
          fzf_cb(nil)
        end,
      }:start()
    end

    run_fre(run_fd)
  end

[ibhagwan]

One thing I noticed when experimenting with multiprocess=false, if fn_transform returns nil, sometimes other items returned by fn_transform won't render. I'm able to reproduce with this:

Is that intentional, is there a new way to filter out items?

Not intentional, returning nil should filter out the current item only, I’ll check.

[ibhagwan]

#2185, somehow fell through in the api CI :)

https://github.com/ibhagwan/fzf-lua/blob/main/tests/api_spec.lua

[elanmed]

Ah cool thanks for making the issue, I should have started with that 😅

[ibhagwan]

Should work with latest commit @elanmed

[elanmed]

Thank you! fzf-lua-frecency.nvim works as expected with multiprocess=false, thanks for the quick turnaround 🙏

[elanmed]

I ended up making my own fzf-lua frecency plugin that implements that same algorithm as fre 😅

https://github.com/elanmed/fzf-lua-frecency.nvim

[ibhagwan]

I ended up making my own fzf-lua frecency plugin that implements that same algorithm as fre 😅

https://github.com/elanmed/fzf-lua-frecency.nvim

Wow, so nice @elanmed! Glanced at the code trying to understand what you used for db, from what I’ve seen it’s just a plain text of sorted files?

When I finish work on my big refactor PR I’ll test your plugin and if share my thoughts on imporvements if such arise 😊

Taking a look at the default options might be worth adding --no-sort similar to what we do for the zoxide picker? This way if you input the entries highest to lowest it won’t mess up the order and highest score will appear first:

fzf-lua/lua/fzf-lua/defaults.lua

Lines 1238 to 1245 in f972ad7

	fzf_opts = {
	["--no-multi"] = true,
	["--delimiter"] = "[\t]",
	["--tabstop"] = "4",
	["--tiebreak"] = "end,index",
	["--nth"] = "2..",
	["--no-sort"] = true, -- sort by score
	},

Thank you so much for sharing.

[elanmed]

Wow, so nice @elanmed! Glanced at the code trying to understand what you used for db, from what I’ve seen it’s just a plain text of sorted files?

Thanks! There's two "databases": one is a lua table that I store in mpack with the format { [cwd] = { [filename] = [date_at_score_one] } } where date_at_score_one is the date that the frecency score decays to 1 - the frecency algorithm suggested storing this value.

When a file is selected from the picker, I loop through the mpack db and calculate the current scores, sort the files by score, and output that list to a plain txt file. The txt file is what I read from when frecency() is called (along with fd).

Since I calculate the scores when selecting a file, if you invoke frecency() after not selecting a file for a while, you might have slightly stale scores. But I think it's worth it to be able to off-load all the calculations away from frecency() and keep that function fast. (The orderings are still accurate even if the scores are stale, and when displaying the score in the picker, it's calculated when invoking frecency() )

I'm not totally sold on mpack, but I read online that it stores smaller files that are faster to parse than json (which is the next section after :h vim.mpack 😅). I was thinking about using sqlite.lua, but it seemed a bit overkill for my use case - especially since I do all the performance-heavy actions when you select a file.

When I finish work on my big refactor PR I’ll test your plugin and if share my thoughts on imporvements if such arise 😊

Thank you! I'm very open to feedback/contributions

Taking a look at the default options might be worth adding --no-sort similar to what we do for the zoxide picker?

Ah good catch, I'll do that 🙏

[ibhagwan]

I'm not totally sold on mpack, but I read online that it stores smaller files that are faster to parse than json (which is the next section after :h vim.mpack 😅). I was thinking about using sqlite.lua, but it seemed a bit overkill for my use case - especially since I do all the performance-heavy actions when you select a file.

I'm not sure as I haven't tested performance at all in this regard, FWIW fzf-lua uses vim.json.decode for the awesome_colorschemes picker db:
https://github.com/ibhagwan/fzf-lua/blob/main/lua/fzf-lua/data/colorschemes.json