A Collection of "Computer Use" Tools

[twashing]

Is there anywhere captured, a repository or collection of "Computer Use" tools?

• Mouse movement
• Mouse clicks
• Screenshot
• Screen Recording
• Recording Audio/Video
• etc

I have Screenshot and Screen Recording functions that I've wrapped in tools, that I can share. But it would be nice to have a standard or full suite of functions.

(require 'cl-lib)

(defun screenshot (&optional emacs-frame-only)
  "Take a screenshot and save it to a file.
   With prefix argument ARG, capture the current active Emacs frame only.
   Otherwise, capture the entire screen."
  (interactive "P")
  (let* ((default-directory (expand-file-name "~/Pictures/"))
         (timestamp (format-time-string "%Y%m%d-%H%M%S"))
         (file-name (concat "screenshot-" timestamp ".png"))
         (full-path (expand-file-name file-name))
         (command
          (cond
           ((eq system-type 'darwin)
            (if emacs-frame-only
                (format "screencapture -l$(osascript -e 'tell app \"Emacs\" to id of window 1') %s"
                        (shell-quote-argument full-path))
              (format "screencapture %s"
                      (shell-quote-argument full-path))))
           ((eq system-type 'gnu/linux)
            (if emacs-frame-only
                (format "ffmpeg -y -f x11grab -i 0x%s -vframes 1 %s"
                        (frame-parameter nil 'outer-window-id)
                        (shell-quote-argument full-path))
              (format "ffmpeg -y -f x11grab -i :0 -vframes 1 %s"
                      (shell-quote-argument full-path))))
           (t (error "No suitable screenshot command found for this system")))))
    (make-directory (file-name-directory full-path) t)
    (if (zerop (shell-command command))
        (progn
          (message "Screenshot saved to %s" full-path)
          (when (y-or-n-p "Open screenshot? ")
            (find-file full-path)))
      (error "Failed to take screenshot"))))


;; Usage
;; `screenshot` can i. take a shot of desktop screen or ii. shot of an active Emacs window
;; (screenshot)
;; (screenshot t)

(defun screencapture (screen-id &optional duration)
  "Record the screen with SCREEN-ID for DURATION seconds using ffmpeg’s avfoundation input.
   SCREEN-ID is the device index shown when running:
   ffmpeg -f avfoundation -list_devices true -i \"\"

   By default, DURATION is 5 seconds.  The resulting file is placed in ~/Pictures/ with a timestamp.
  On non-macOS systems, this function currently signals an error."
  (interactive "nEnter screen device index: \nnDuration in seconds (default 5): ")
  (unless duration
    (setq duration 5))

  (let* ((default-directory (expand-file-name "~/Pictures/"))
         (timestamp (format-time-string "%Y%m%d-%H%M%S"))
         (file-name (concat "screencapture-" timestamp ".mp4"))
         (full-path (expand-file-name file-name))
         ;; Construct the ffmpeg command for avfoundation
         (command (format "ffmpeg -y -f avfoundation -i %s:0 -t %s %s"
                          screen-id
                          duration
                          (shell-quote-argument full-path))))
    (make-directory (file-name-directory full-path) t)
    (if (zerop (shell-command command))
        (progn
          (message "Screen recording saved to %s" full-path))
      (error "Failed to record from screen ID %s" screen-id))))

;; Usage

;; Record the desktop from screen 3 for 10 seconds
;; (screencapture 3 10)
;;
;; Record the desktop from screen 2 for 5 seconds
;; (screencapture 2 5)

;; # ffmpeg can list out the available desktop screen devices
;; ffmpeg -f avfoundation -list_devices true -i ""
;; ...
;; [AVFoundation indev @ 0x12a804080] AVFoundation video devices:
;; [AVFoundation indev @ 0x12a804080] [0] LG UltraFine Display Camera
;; [AVFoundation indev @ 0x12a804080] [1] FaceTime HD Camera
;; [AVFoundation indev @ 0x12a804080] [2] Capture screen 0
;; [AVFoundation indev @ 0x12a804080] [3] Capture screen 1
;; [AVFoundation indev @ 0x12a804080] AVFoundation audio devices:
;; [AVFoundation indev @ 0x12a804080] [0] LG UltraFine Display Audio
;; [AVFoundation indev @ 0x12a804080] [1] MacBook Pro Microphone
;; [AVFoundation indev @ 0x12a804080] [2] Bose QC45

[karthink]

@twashing Thank you for sharing. The tools are being collected at the llm-tool-collection repo. I encourage you to raise a PR there with what you've written.

It's a separate repo because the tool format is the same for all LLM clients for Emacs, and we are trying to use a common shared repository of tools.

It's pretty bare right now but there are many people working on tools for different things -- John Wiegley is writing some tools for filesystem access, some gptel users have written tools for modifying Emacs buffers, and so on. Eventually I hope to see them all in the llm-tool-collection repo.

[twashing]

@karthink, No problem. And thanks for the info. I'll create a PR.

Although I don't fully grok the pattern for llm-tool-collection-deftool (ref). It doesn't use gptel-make-tool.
But some of the llm-tool-collection-get-* helper functions do (ex).

Who's the author? And what are the code patterns and PR etiquette? Should I submit something like so?

(gptel-make-tool :name "screenshot"
                 :description "Take a screenshot and save it to a file."

                 :function #'screenshot
                 :args (list '(:name "emacs-frame-only"
                               :type "string"
                               :description "Parameter for emacs-frame-only"))
                 :category "desktop")

(gptel-make-tool :name "screencapture"
                 :description "Record the screen with SCREEN-ID for DURATION seconds using ffmpeg’s avfoundation input."

                 :function #'screencapture
                 :args (list '(:name "screen-id"
                               :type "string"
                               :description "Parameter for screen-id")
                             '(:name "duration"
                               :type "string"
                               :description "Parameter for duration"))
                 :category "desktop")