Major Refactor and New Features: Modularization, Enhanced CLI, and Logging (#20)

* added whitelist and blacklist

* added unit test

* update rules

* fail

* unit tests

* strong unit tests

* update main

* compilation

* update code

* extracted filter

* filter test implemented

* glob approach

* filter in progress

* filter test pass

* test filter good

* split into modules

* include ok

* logger

* docstring

* fixed the exclude pattern empty bug

* unit test done

* git test

* code cleaning

* README Updated

* cleanup

* debug cleaned

* template support

* improve api

* test template

* git test

* clean the code

* handling error properly

* cleaned

* documentation

* semver

* ver 2.0.0

---------

Co-authored-by: SChattot <stephane.chattot@heig-vd.ch>

Author: ODAncona

Cargo.toml

@@ -1,7 +1,7 @@
 [package]
 name = "code2prompt"
-version = "1.1.0"
-authors = ["Mufeed VH <mufeed@lyminal.space>"]
+version = "2.0.0"
+authors = ["Mufeed VH <mufeed@lyminal.space>","Olivier D'Ancona <olivier.dancona@master.hes-so.ch>"]
 description = "A command-line (CLI) tool to generate an LLM prompt from codebases of any size, fast."
 keywords = ["code", "prompt", "llm", "gpt", "ai"]
 categories = ["command-line-utilities", "development-tools"]
@@ -17,6 +17,7 @@ edition = "2021"
 name = "code2prompt"
 test = false
 bench = false
+path = "src/main.rs"
 
 [dependencies]
 clap = { version = "4.0", features = ["derive"] }
@@ -32,6 +33,10 @@ anyhow = "1.0.80"
 inquire = "0.7.1"
 regex = "1.10.3"
 git2 = { version = "0.18.2", default_features = false, features = [ "https", "vendored-libgit2", "vendored-openssl" ] }
+glob = "0.3.1"
+once_cell = "1.19.0"
+log = "0.4"
+env_logger = "0.11.3"
 arboard = "3.4.0"
 
 [profile.release]
@@ -44,3 +49,9 @@ section = "utility"
 assets = [
     ["target/release/code2prompt", "/usr/bin/", "755"],
 ]
+
+[dev-dependencies]
+tempfile = "3.3"
+assert_cmd = "2.0"
+predicates = "2.0"
+env_logger = "0.11.3"

README.md

@@ -1,19 +1,4 @@
-# `code2prompt`
-
-A CLI tool to convert your codebase into a single LLM prompt with source tree, prompt templating, and token counting.
-
----
-
-You can run this tool on the entire directory and it would generate a well-formatted Markdown prompt detailing the source tree structure, and all the code. You can then upload this document to either GPT or Claude models with higher context windows and ask it to:
-
-- Rewrite the code to another language.
-- Find bugs/security vulnerabilities.
-- Document the code.
-- Implement new features.
-
-You can customize the prompt template to achieve any of the desired use cases. It essentially traverses a codebase and creates a prompt with all source files combined. In short, it automates copy-pasting multiple source files into your prompt and formatting them along with letting you know how many tokens your code consumes. 
-
-> I initially wrote this for personal use to utilize Claude 3.0's 200K context window and it has proven to be pretty useful so I decided to open-source it!
+# code2prompt
 
 [![crates.io](https://img.shields.io/crates/v/code2prompt.svg#cache1)](https://crates.io/crates/code2prompt)
 [![LICENSE](https://img.shields.io/github/license/mufeedvh/code2prompt.svg#cache1)](https://github.com/mufeedvh/code2prompt/blob/master/LICENSE)
@@ -22,125 +7,137 @@ You can customize the prompt template to achieve any of the desired use cases. I
   <a href="https://github.com/mufeedvh/code2prompt"><img src=".assets/code2prompt-screenshot.png" alt="code2prompt"></a>
 </h1>
 
+`code2prompt` is a command-line tool (CLI) that converts your codebase into a single LLM prompt with a source tree, prompt templating, and token counting.
+
 ## Table of Contents
 
-* [Features](#features)
-* [Installation](#installation)
-* [Usage](#usage)
-* [Templates](#templates)
-* [User Defined Variables](#user-defined-variables)
-* [Build From Source](#build-from-source)
-* [Contribution](#contribution)
-* [License](#license)
-* [Support The Author](#liked-the-project)
+- [Features](#features)
+- [Installation](#installation)
+- [Usage](#usage)
+- [Templates](#templates)
+- [User Defined Variables](#user-defined-variables)
+- [Tokenizers](#tokenizers)
+- [Build From Source](#build-from-source)
+- [Contribution](#contribution)
+- [License](#license)
+- [Support The Author](#support-the-author)
 
 ## Features
 
+You can run this tool on the entire directory and it would generate a well-formatted Markdown prompt detailing the source tree structure, and all the code. You can then upload this document to either GPT or Claude models with higher context windows and ask it to:
+
 - Quickly generate LLM prompts from codebases of any size.
 - Customize prompt generation with Handlebars templates. (See the [default template](src/default_template.hbs))
-- Follows `.gitignore`.
-- Filter and exclude files by extension.
-- Display the token count of the generated prompt. (See [Tokenizers](#Tokenizers) for more details)
-- Optionally include the Git diff output (staged files) in the generated prompt.
-- Copy the generated prompt to the clipboard on generation.
+- Respects `.gitignore`.
+- Filter and exclude files using glob patterns.
+- Display the token count of the generated prompt. (See [Tokenizers](#tokenizers) for more details)
+- Optionally include Git diff output (staged files) in the generated prompt.
+- Automatically copy the generated prompt to the clipboard.
 - Save the generated prompt to an output file.
 - Exclude files and folders by name or path.
 - Add line numbers to source code blocks.
 
+You can customize the prompt template to achieve any of the desired use cases. It essentially traverses a codebase and creates a prompt with all source files combined. In short, it automates copy-pasting multiple source files into your prompt and formatting them along with letting you know how many tokens your code consumes.
+
 ## Installation
 
+### Latest Release
+
 Download the latest binary for your OS from [Releases](https://github.com/mufeedvh/code2prompt/releases) OR install with `cargo`:
 
-```
+```sh
 cargo install code2prompt
 ```
 
 For unpublished builds:
 
-```
+```sh
 cargo install --git https://github.com/mufeedvh/code2prompt
 ```
 
-## Usage
-
-
-Generate a prompt from a codebase directory:
+### Prerequisites
 
-```
-code2prompt path/to/codebase
-```
+For building `code2prompt` from source, you need to have these tools installed:
 
-Use a custom Handlebars template file:
+- [Git](https://git-scm.org/downloads)
+- [Rust](https://rust-lang.org/tools/install)
+- Cargo (Automatically installed when installing Rust)
 
-```
-code2prompt path/to/codebase -t path/to/template.hbs
+```sh
+git clone https://github.com/mufeedvh/code2prompt.git
+cd code2prompt/
+cargo build --release
 ```
 
-Filter files by extension:
+The first command clones the `code2prompt` repository to your local machine. The next two commands change into the `code2prompt` directory and build it in release mode.
 
-```
-code2prompt path/to/codebase -f rs,toml
-```
+## Usage
 
-Exclude files by extension:
+Generate a prompt from a codebase directory:
 
-```  
-code2prompt path/to/codebase -e txt,md
+```sh
+code2prompt path/to/codebase
 ```
 
-Exclude files by name:
+Use a custom Handlebars template file:
 
-```
-code2prompt path/to/codebase --exclude-files "file1.txt,file2.txt"
+```sh
+code2prompt path/to/codebase -t path/to/template.hbs
 ```
 
-Exclude files by folder/directory path:
+Filter files using glob patterns:
 
-```
-code2prompt path/to/codebase --exclude-folders "tests,docs"
+```sh
+code2prompt path/to/codebase --include="*.rs,*.toml"
 ```
 
-Use relative paths instead of absolute paths:
+Exclude files using glob patterns:
 
-```
-code2prompt path/to/codebase --relative-paths
+```sh
+code2prompt path/to/codebase --exclude="*.txt,*.md"
 ```
 
-Display token count of the generated prompt:
+Display the token count of the generated prompt:
 
-```
+```sh
 code2prompt path/to/codebase --tokens
 ```
 
-Specify tokenizer for token count:
+Specify a tokenizer for token count:
 
+```sh
+code2prompt path/to/codebase --tokens --encoding=p50k
 ```
-code2prompt path/to/codebase --tokens --encoding p50k
-```
-
-Supported tokenizers: `cl100k`, `p50k`, `p50k_edit`, `r50k_base`.
 
+Supported tokenizers: `cl100k`, `p50k`, `p50k_edit`, `r50k_bas`.
 > [!NOTE]  
-> See [Tokenizers](#Tokenizers) for more details.
+> See [Tokenizers](#tokenizers) for more details.
 
 Save the generated prompt to an output file:
 
-```
-code2prompt path/to/codebase -o output.txt  
+```sh
+code2prompt path/to/codebase --output=output.txt
 ```
 
-Generate git commit message (for staged files):
+Generate a Git commit message (for staged files):
 
-```
-code2prompt path/to/codebase --diff -t "templates/write-git-commit.hbs"    
+```sh
+code2prompt path/to/codebase --diff -t templates/write-git-commit.hbs
 ```
 
 Add line numbers to source code blocks:
 
-```
+```sh
 code2prompt path/to/codebase --line-number
 ```
 
+- Rewrite the code to another language.
+- Find bugs/security vulnerabilities.
+- Document the code.
+- Implement new features.
+
+> I initially wrote this for personal use to utilize Claude 3.0's 200K context window and it has proven to be pretty useful so I decided to open-source it!
+
 ## Templates
 
 `code2prompt` comes with a set of built-in templates for common use cases. You can find them in the [`templates`](templates) directory.
@@ -153,7 +150,7 @@ Use this template to generate prompts for documenting the code. It will add docu
 
 Use this template to generate prompts for finding potential security vulnerabilities in the codebase. It will look for common security issues and provide recommendations on how to fix or mitigate them.
 
-### [`clean-up-code.hbs`](templates/clean-up-code.hbs) 
+### [`clean-up-code.hbs`](templates/clean-up-code.hbs)
 
 Use this template to generate prompts for cleaning up and improving the code quality. It will look for opportunities to improve readability, adherence to best practices, efficiency, error handling, and more.
 
@@ -175,7 +172,7 @@ Use this template to generate prompts for improving the performance of the codeb
 
 You can use these templates by passing the `-t` flag followed by the path to the template file. For example:
 
-```
+```sh
 code2prompt path/to/codebase -t templates/document-the-code.hbs
 ```
 
@@ -206,24 +203,6 @@ For more context on the different tokenizers, see the [OpenAI Cookbook](https://
 
 `code2prompt` makes it easy to generate prompts for LLMs from your codebase. It traverses the directory, builds a tree structure, and collects information about each file. You can customize the prompt generation using Handlebars templates. The generated prompt is automatically copied to your clipboard and can also be saved to an output file. `code2prompt` helps streamline the process of creating LLM prompts for code analysis, generation, and other tasks.
 
-## Build From Source
-
-### Prerequisites
-
-For building `code2prompt` from source, you need to have these tools installed:
-
-* [Git](https://git-scm.org/downloads)
-* [Rust](https://rust-lang.org/tools/install) 
-* Cargo (Automatically installed when installing Rust)
-
-```
-$ git clone https://github.com/mufeedvh/code2prompt.git
-$ cd code2prompt/
-$ cargo build --release
-```
-
-The first command clones the `code2prompt` repository to your local machine. The next two commands change into the `code2prompt` directory and build it in release mode.
-
 ## Contribution
 
 Ways to contribute:
@@ -240,4 +219,4 @@ Licensed under the MIT License, see <a href="https://github.com/mufeedvh/code2pr
 
 ## Liked the project?
 
-If you liked the project and found it useful, please give it a :star: and consider supporting the author!
+If you liked the project and found it useful, please give it a :star: and consider supporting the authors!

src/filter.rs

@@ -0,0 +1,63 @@
+//! This module contains the logic for filtering files based on include and exclude patterns.
+
+use colored::*;
+use glob::Pattern;
+use log::{debug, error};
+use std::fs;
+use std::path::Path;
+
+/// Determines whether a file should be included based on include and exclude patterns.
+///
+/// # Arguments
+///
+/// * `path` - The path to the file to be checked.
+/// * `include_patterns` - A slice of strings representing the include patterns.
+/// * `exclude_patterns` - A slice of strings representing the exclude patterns.
+/// * `include_priority` - A boolean indicating whether to give priority to include patterns if both include and exclude patterns match.
+///
+/// # Returns
+///
+/// * `bool` - `true` if the file should be included, `false` otherwise.
+pub fn should_include_file(
+    path: &Path,
+    include_patterns: &[String],
+    exclude_patterns: &[String],
+    include_priority: bool,
+) -> bool {
+    // ~~~ Clean path ~~~
+    let canonical_path = match fs::canonicalize(path) {
+        Ok(path) => path,
+        Err(e) => {
+            error!("Failed to canonicalize path: {}", e);
+            return false;
+        }
+    };
+    let path_str = canonical_path.to_str().unwrap();
+
+    // ~~~ Check glob patterns ~~~
+    let included = include_patterns
+        .iter()
+        .any(|pattern| Pattern::new(pattern).unwrap().matches(path_str));
+    let excluded = exclude_patterns
+        .iter()
+        .any(|pattern| Pattern::new(pattern).unwrap().matches(path_str));
+
+    // ~~~ Decision ~~~
+    let result = match (included, excluded) {
+        (true, true) => include_priority, // If both include and exclude patterns match, use the include_priority flag
+        (true, false) => true,            // If the path is included and not excluded, include it
+        (false, true) => false,           // If the path is excluded, exclude it
+        (false, false) => include_patterns.is_empty(), // If no include patterns are provided, include everything
+    };
+
+    debug!(
+        "Checking path: {:?}, {}: {}, {}: {}, decision: {}",
+        path_str,
+        "included".bold().green(),
+        included,
+        "excluded".bold().red(),
+        excluded,
+        result
+    );
+    result
+}