Support for Arbitrary Hashes and Checksums

[scholarsmate]

It would be advantageous for the Daffodil Data Editor extension for Visual Studio Code to support selecting an arbitrary byte range from a source file and sending it to an external utility for computing hashes (e.g., MD5, SHA-2) or checksums (e.g. SUM, CRC-16, CRC-32). The computed result should be displayed within the UI and optionally inserted at a specified offset within the source file.

To enable this, a standardized interface can be defined whereby the selected byte range is streamed to the external tool via standard input, and the computed result is read from standard output (e.g., using md5sum, sha512sum). If a tool does not conform to this I/O convention, users may implement a wrapper script to adapt its behavior accordingly.

Workflow

1. User selects a segment of data from the source file in VSCode.

2. User selects the external tool to send the segment to, then clicks "Go" to dispatch the task.

3. Child process is spawned in Node.js to run the external utility, the selected segment is piped into it, and the results are collected from stdout and stderr.

4a. stdout is then parsed by the extension in VSCode and displayed to the user.

4b. stderr is routed to the logs.

5. The user reviews the displayed results in VSCode and either cancels the display, copies the contents to the clipboard, or inserts/overwrites the data at an offset in the source file.

Implementation Ideas

1. User Workflow Integration

• Enable users to select a byte range via the UI (possibly use the data profiler)
• Provide a right-click or command-palette action labeled e.g., "Compute Hash/Checksum."
• Prompt the user to select from a list of registered external tools (e.g., md5sum, sha512sum, crc32, custom).

2. Tool Configuration
   Allow users to register external tools in the extension’s settings, specifying:

• Unique name to use as the registered external tool
• Path to executable, script, or adapter
• Optional arguments
• Optional regex to parse the output (e.g., first token on the first line)
• Timeout threshold

3. Execution
   Validate tool path, permissions, and modes.
   Execute external tools in a subprocess with:

• Input piped from memory
• stdout captured (e.g., child_process.spawn in Node.js), stderr could be captured and written to the log file to help debug problems

4. Output Handling

• Parse the tool’s output using the configurable regex from the registration settings.
• Display the result in a dedicated panel or floating UI component.
• Offer a “Copy to Clipboard” button and an “Insert at Offset…” dialog that guides the user to insert (or overwrite) the result at a valid offset inside the source file.

5. Extensibility and Error Handling
   Provide fallback behavior and clear error reporting if:

• The tool fails to execute
• Output is not parseable
• Input range exceeds a configurable size threshold
• Execution time has exceeded the configured timeout threshold

6. Testing and Validation
   Include unit tests for:

• Input/output piping logic
• External tool invocation and timeout handling
• Output parsing and insertion correctness
• Provide integration tests with commonly used tools (e.g., md5sum, sha512sum, cksum).

This design ensures flexibility while maintaining a consistent user experience and aligning with standard UNIX-like tool conventions.

Example JSON Schema

{
  "$schema": "http://json-schema.org/draft-07/schema#",
  "title": "External Tool Configuration Schema",
  "type": "object",
  "patternProperties": {
    "^[a-zA-Z0-9_-]+$": {
      "type": "object",
      "required": ["path"],
      "properties": {
        "path": {
          "type": "string",
          "description": "Absolute path to the executable, script, or adapter"
        },
        "args": {
          "type": "array",
          "items": {
            "type": "string"
          },
          "description": "Optional arguments to pass to the tool"
        },
        "parseRegex": {
          "type": "string",
          "description": "Regular expression to extract the hash/checksum result from stdout"
        },
        "timeoutMs": {
          "type": "integer",
          "minimum": 100,
          "description": "Execution timeout in milliseconds"
        }
      },
      "additionalProperties": false
    }
  },
  "additionalProperties": false,
  "description": "Mapping of user-defined tool names to configuration objects"
}

Example JSON Configuration

JSON like the following could be integrated in a contributes.configuration block in package.json:

{
  "md5sum": {
    "path": "/usr/bin/md5sum",
    "args": [],
    "parseRegex": "^([a-fA-F0-9]{32})",
    "timeoutMs": 3000
  },
  "sha512sum": {
    "path": "/usr/bin/sha512sum",
    "args": [],
    "parseRegex": "^([a-fA-F0-9]{128})",
    "timeoutMs": 3000
  },
  "crc32": {
    "path": "/usr/local/bin/crc32-adapter.sh",
    "args": [],
    "parseRegex": "^CRC32:\\s*([a-fA-F0-9]+)",
    "timeoutMs": 2000
  }
}

[JeremyYao]

Some thoughts: I like the usage of being able to connect external hashing utilities. My go-to thought for implementing hashing would be to use a library like https://www.npmjs.com/package/jshashes?activeTab=readme, and it seems like it supports common hashing algorithms. I'm guessing that it's better to use external hashing utilities due to the fact there may be certain algorithms that aren't supported by the library?