Move sections to the proposals folder

elifarley · elifarley · commit b73614f4d978 · 2024-11-16T13:06:06.000-03:00
diff --git a/README.md b/README.md
@@ -14,8 +14,6 @@
 - [Supported Languages](#supported-languages)
 - [How can CEDARScript be used](#how-can-cedarscript-be-used)
 - [Examples](#examples)
-- [Planned Features](#planned-features)
-- [Future Enhancements](#future-enhancements)
 - [Proposals](#proposals)
 - [Related](#related)
 
@@ -230,207 +228,8 @@ UPDATE FILE "app/main.py" REPLACE FUNCTION "calculate_total" WITH ED '''
 ''';
 ```
 
-
-
 There are [many more examples](test/corpus) to look at...
 
-# Planned Features
-
-## Onboarding Capabilities
-
-This capability is designed to help developers, AI assistants, and other tools quickly gain a comprehensive understanding of a project's structure, conventions, and context.
-
-### Key Onboarding Features
-<details>
-1. **Convention Discovery**:
-   CEDARScript can automatically extract coding conventions from designated files like `CONVENTIONS.md`:
-
-   ```sql
-   SELECT CONVENTIONS
-   FROM ONBOARDING;
-   ```
-
-2. **Context Retrieval**:
-   Quickly access project context from files like `.context.md` or `.contextdocs.md`:
-
-   ```sql
-   SELECT CONTEXT
-   FROM ONBOARDING;
-   ```
-
-3. **Comprehensive Project Overview**:
-   Gather all essential project information in one query:
-
-   ```sql
-   SELECT *
-   FROM ONBOARDING;
-   ```
-</details>
-
-# Future Work
-
-## Future Enhancements
-
-<details>
-<summary>Ideas to explore:</summary>
-
-- Automatic generation of project structure visualizations
-- Integration with version control history for context-aware onboarding
-- Customizable onboarding queries for specific project needs
-
-
-1. [Tree-Sitter query language](https://cycode.com/blog/tips-for-using-tree-sitter-queries/) integration, which could open up many possibilities;
-2. [Comby](https://github.com/comby-tools/comby) notation for an alternate syntax to express refactorings on code or data formats;
-3. Create a browser extension that allows web-chat interfaces of Large Language Models to tackle larger file changes;
-4. Select a model to fine-tune so that it natively understands `CEDARScript`;
-5. Provide language extensions that will improve how LLMs interact with other resource types;
-6. Explore using it as an **LLM-Tool Interface**;
-
-## Tree-Sitter Query Language Integration
-
-This could open up many possibilities, like:
-
-### Advanced Code Analysis: provide statistics about functions in the project
-
-```sql
-QUERY LANGUAGE 'tree-sitter'
-FROM PROJECT
-PATTERN '''
-(function_definition
-  name: (identifier) @func_name
-  parameters: (parameters) @params
-  body: (block 
-    (return_statement) @return_stmt))
-'''
-WITH ANALYSIS
-  COUNT @func_name AS "Total Functions"
-  AVERAGE (LENGTH @params) AS "Avg Parameters"
-  PERCENTAGE (IS_PRESENT @return_stmt) AS "Functions with Return";
-```
-
-### Concisely modify all methods
-
-Find all classes and their methods in Python files, then insert a print statement after each method definition:
-
-```sql
-QUERY LANGUAGE 'tree-sitter'
-FROM PROJECT
-PATTERN '''
-(class_definition
-  name: (identifier) @class_name
-  body: (block
-    (function_definition
-      name: (identifier) @method_name)))
-'''
-WITH ACTIONS
-  INSERT AFTER @method_name
-  CONTENT '''
-@0:    print("Method called:", @method_name)
-''';
-```
-
-### Cross-language refactoring: replace all calls to "deprecated_function" across Python, JavaScript, and TypeScript files.
-
-```sql
-QUERY LANGUAGE 'tree-sitter'
-FROM PROJECT
-LANGUAGES ["python", "javascript", "typescript"]
-PATTERN '''
-(call_expression
-  function: (identifier) @func_name
-  (#eq? @func_name "deprecated_function"))
-'''
-WITH ACTIONS
-  REPLACE @func_name
-  WITH CONTENT "new_function";
-```
-
-### Custom Linting Rules:
-We can define project-specific linting rules using Tree-sitter queries:
-
-```sql
-QUERY LANGUAGE 'tree-sitter'
-FROM PROJECT
-PATTERN '''
-(import_statement
-  (dotted_name) @import_name
-  (#match? @import_name "^(os|sys)$"))
-'''
-WITH LINT
-  SEVERITY "WARNING"
-  MESSAGE "Direct import of system modules discouraged. Use custom wrappers instead.";
-```
-
-## Comby Notation
-
-To replace 'failUnlessEqual' with 'assertEqual':
-```sql
-UPDATE PROJECT
-REAFCTOR LANGUAGE "comby" 
-WITH PATTERN '''
-comby 'failUnlessEqual(:[a],:[b])' 'assertEqual(:[a],:[b])' example.py
-'''
-```
-
-## CEDARScript Browser Extension for LLM Web Interfaces
-
-<details>
-<summary>As Large Language Models (LLMs) become increasingly accessible through web-based chat interfaces, there's a growing need to enhance their ability to handle larger codebases and complex file changes. We propose developing a browser extension that leverages CEDARScript to bridge this gap.</summary>   
-
-- **Seamless Integration**: The extension would integrate with popular LLM web interfaces (e.g., ChatGPT, Claude, Gemini) by leveraging [llm-context.py](https://github.com/cyberchitta/llm-context.py), allowing users to work with larger files and codebases directly within these platforms.
-
-- **CEDARScript Translation**: The changes proposed by the LLM would be concisely expressed as `CEDARScript` commands, enabling more efficient token usage.
-
-- **Local File System Access**: The extension could securely access the user's local file system, allowing for direct manipulation of code files based on `CEDARScript` instructions generated by the LLM.
-
-- **Diff Visualization**: Changes proposed by the LLM would be presented as standard diffs _or_ as `CEDARScript` code, allowing users to review and approve modifications before applying them to their codebase.
-
-- **Context Preservation**: The extension would maintain context across chat sessions, enabling long-running refactoring tasks that span multiple interactions.
-
-This browser extension would expand the capabilities of web-based LLM interfaces, allowing developers to leverage these powerful AI tools for more substantial code modification and analysis tasks. By using CEDARScript as an intermediary language, the extension would ensure efficient and accurate communication between the user, the LLM, and the local codebase.
-
-</details>
-
-## Fine-tuning a Model for Native CEDARScript Understanding
-
-<details>
-   
-<summary>This initiative could enhance the efficiency and effectiveness of AI-assisted code analysis and transformation.</summary>
-
-### Why Fine-tune?
-
-1. **Improved Accuracy**: A fine-tuned model will have a deeper understanding of CEDARScript syntax and semantics, leading to more accurate code analysis and generation.
-2. **Efficiency**: Native understanding of CEDARScript will reduce the need for extensive prompting.
-3. **Consistency**: A model trained specifically on CEDARScript will produce more consistent and idiomatic output, adhering closely to the language's conventions and best practices.
-4. **Extended Capabilities**: Fine-tuning could enable the model to perform more complex CEDARScript operations and understand nuanced aspects of the language that general-purpose models might miss.
-
-### Approach
-
-1. **Model Selection**: We will evaluate various state-of-the-art language models to determine the most suitable base model for fine-tuning. Factors such as model size, pre-training data, and architectural features will be considered.
-2. **Dataset Creation**: A comprehensive dataset of CEDARScript examples, covering a wide range of use cases and complexities, will be created. This dataset will include both CEDARScript commands and their corresponding natural language descriptions or intentions.
-3. **Fine-tuning Process**: The selected model will undergo fine-tuning using the created dataset. We'll experiment with different fine-tuning techniques, depending on the resources available and the desired outcome.
-4. **Evaluation**: The fine-tuned model will be rigorously tested on a held-out test set to assess its performance in understanding and generating CEDARScript. Metrics such as accuracy, fluency, and task completion will be used.
-5. **Iterative Improvement**: Based on the evaluation results, we'll iteratively refine the fine-tuning process, potentially adjusting the dataset, fine-tuning parameters, or even the base model selection.
-
-</details>
-
-## LLM-Tool Interface
-
-<details>
-   
-<summary>As Large Language Models continue to evolve and find applications in various real-world scenarios, there's a growing need for standardized ways for LLMs to interact with external tools and APIs. We envision `CEDARScript` as a potential solution to this challenge.</summary>
-
-- **Standardized Tool Interaction**: `CEDARScript` could serve as an intermediary language between LLMs and various tools, providing a consistent, SQL-like syntax for expressing tool usage intentions.
-- **Tool-Agnostic Commands**: By defining a set of generic commands that map to common tool functionalities, `CEDARScript` could simplify the process of integrating new tools and APIs.
-- **Complex Tool Pipelines**: The language's SQL-like structure could allow for easy chaining of multiple tool operations, enabling more complex workflows.
-- **Abstraction of API Complexity**: CEDARScript could hide the underlying complexity of diverse tool APIs behind a simpler, unified interface.
-
-This approach could potentially enhance LLMs' ability to leverage external tools and capabilities, making it easier to deploy them in diverse real-world applications. Future work could explore the feasibility and implementation of this concept, aiming to create a more seamless integration between LLMs and the tools they use to interact with the world.
-
-</details>
-
-</details>
-
 # Proposals
 See [current proposals](proposals/)
 
diff --git a/proposals/LLM Tool Interface/README.md b/proposals/LLM Tool Interface/README.md
@@ -0,0 +1,13 @@
+# LLM-Tool Interface
+
+As Large Language Models continue to evolve and find applications in various real-world scenarios, there's a growing need for standardized ways for LLMs to interact with external tools and APIs. We envision `CEDARScript` as a potential solution to this challenge.
+
+- **Standardized Tool Interaction**: `CEDARScript` could serve as an intermediary language between LLMs and various tools, providing a consistent, SQL-like syntax for expressing tool usage intentions.
+- **Tool-Agnostic Commands**: By defining a set of generic commands that map to common tool functionalities, `CEDARScript` could simplify the process of integrating new tools and APIs.
+- **Complex Tool Pipelines**: The language's SQL-like structure could allow for easy chaining of multiple tool operations, enabling more complex workflows.
+- **Abstraction of API Complexity**: CEDARScript could hide the underlying complexity of diverse tool APIs behind a simpler, unified interface.
+
+This approach could potentially enhance LLMs' ability to leverage external tools and capabilities, 
+making it easier to deploy them in diverse real-world applications. Future work could explore the feasibility and 
+implementation of this concept, aiming to create a more seamless integration between LLMs and the tools they use
+to interact with the world.
diff --git a/proposals/README.md b/proposals/README.md
@@ -0,0 +1,35 @@
+# Proposals
+
+## Current Proposals
+1. [Onboarding Capabilities](onboarding-features)
+2. [Browser Extension](browser-extension)
+3. [Capture External Command Output](capture-external-command-output)
+4. [LLM Tool Interface](LLM%20Tool%20Interface)
+5. [Model Fine-Tuning](model-fine-tuning)
+6. [Tree-Sitter Query Language Integration](tree-sitter-query-language)
+
+## Ideas to explore
+
+- Automatic generation of project structure visualizations
+- Integration with version control history for context-aware onboarding
+- Customizable onboarding queries for specific project needs
+
+
+1. [Tree-Sitter query language](https://cycode.com/blog/tips-for-using-tree-sitter-queries/) integration, which could open up many possibilities;
+2. [Comby](https://github.com/comby-tools/comby) notation for an alternate syntax to express refactorings on code or data formats;
+3. Create a browser extension that allows web-chat interfaces of Large Language Models to tackle larger file changes;
+4. Select a model to fine-tune so that it natively understands `CEDARScript`;
+5. Provide language extensions that will improve how LLMs interact with other resource types;
+6. Explore using it as an **LLM-Tool Interface**;
+
+## Comby Notation
+
+To replace 'failUnlessEqual' with 'assertEqual':
+```sql
+UPDATE PROJECT
+REAFCTOR LANGUAGE "comby" 
+WITH PATTERN '''
+comby 'failUnlessEqual(:[a],:[b])' 'assertEqual(:[a],:[b])' example.py
+'''
+```
+
diff --git a/proposals/browser-extension/README.md b/proposals/browser-extension/README.md
@@ -0,0 +1,17 @@
+# CEDARScript Browser Extension for LLM Web Interfaces
+
+As Large Language Models (LLMs) become increasingly accessible through web-based chat interfaces, 
+there's a growing need to enhance their ability to handle larger codebases and complex file changes. 
+We propose developing a browser extension that leverages CEDARScript to bridge this gap.
+
+- **Seamless Integration**: The extension would integrate with popular LLM web interfaces (e.g., ChatGPT, Claude, Gemini) by leveraging [llm-context.py](https://github.com/cyberchitta/llm-context.py), allowing users to work with larger files and codebases directly within these platforms.
+
+- **CEDARScript Translation**: The changes proposed by the LLM would be concisely expressed as `CEDARScript` commands, enabling more efficient token usage.
+
+- **Local File System Access**: The extension could securely access the user's local file system, allowing for direct manipulation of code files based on `CEDARScript` instructions generated by the LLM.
+
+- **Diff Visualization**: Changes proposed by the LLM would be presented as standard diffs _or_ as `CEDARScript` code, allowing users to review and approve modifications before applying them to their codebase.
+
+- **Context Preservation**: The extension would maintain context across chat sessions, enabling long-running refactoring tasks that span multiple interactions.
+
+This browser extension would expand the capabilities of web-based LLM interfaces, allowing developers to leverage these powerful AI tools for more substantial code modification and analysis tasks. By using CEDARScript as an intermediary language, the extension would ensure efficient and accurate communication between the user, the LLM, and the local codebase.
diff --git a/proposals/model-fine-tuning/README.md b/proposals/model-fine-tuning/README.md
@@ -0,0 +1,19 @@
+# Fine-tuning a Model for Native CEDARScript Understanding
+
+This initiative could enhance the efficiency and effectiveness of AI-assisted code analysis and transformation.
+
+## Why Fine-tune?
+
+1. **Improved Accuracy**: A fine-tuned model will have a deeper understanding of CEDARScript syntax and semantics, leading to more accurate code analysis and generation.
+2. **Efficiency**: Native understanding of CEDARScript will reduce the need for extensive prompting.
+3. **Consistency**: A model trained specifically on CEDARScript will produce more consistent and idiomatic output, adhering closely to the language's conventions and best practices.
+4. **Extended Capabilities**: Fine-tuning could enable the model to perform more complex CEDARScript operations and understand nuanced aspects of the language that general-purpose models might miss.
+
+## Approach
+
+1. **Model Selection**: We will evaluate various state-of-the-art language models to determine the most suitable base model for fine-tuning. Factors such as model size, pre-training data, and architectural features will be considered.
+2. **Dataset Creation**: A comprehensive dataset of CEDARScript examples, covering a wide range of use cases and complexities, will be created. This dataset will include both CEDARScript commands and their corresponding natural language descriptions or intentions.
+3. **Fine-tuning Process**: The selected model will undergo fine-tuning using the created dataset. We'll experiment with different fine-tuning techniques, depending on the resources available and the desired outcome.
+4. **Evaluation**: The fine-tuned model will be rigorously tested on a held-out test set to assess its performance in understanding and generating CEDARScript. Metrics such as accuracy, fluency, and task completion will be used.
+5. **Iterative Improvement**: Based on the evaluation results, we'll iteratively refine the fine-tuning process, potentially adjusting the dataset, fine-tuning parameters, or even the base model selection.
+
diff --git a/proposals/onboarding-features/README.md b/proposals/onboarding-features/README.md
@@ -0,0 +1,31 @@
+# Onboarding Capabilities
+
+This capability is designed to help developers, AI assistants, and other tools quickly gain a comprehensive understanding
+of a project's structure, conventions, and context.
+
+## Key Onboarding Features
+
+1. **Convention Discovery**:
+   CEDARScript can automatically extract coding conventions from designated files like `CONVENTIONS.md`:
+
+   ```sql
+   SELECT CONVENTIONS
+   FROM ONBOARDING;
+   ```
+
+2. **Context Retrieval**:
+   Quickly access project context from files like `.context.md` or `.contextdocs.md`:
+
+   ```sql
+   SELECT CONTEXT
+   FROM ONBOARDING;
+   ```
+
+3. **Comprehensive Project Overview**:
+   Gather all essential project information in one query:
+
+   ```sql
+   SELECT *
+   FROM ONBOARDING;
+   ```
+µ
diff --git a/proposals/tree-sitter-query-language/README.md b/proposals/tree-sitter-query-language/README.md
