v0.5 Devdocs: Dockerization implementation and more enhancements in UI. (#29)

Here's what's new:

• 🐳 Complete Docker Overhaul - One command now handles everything from
setup to monitoring Now you can launch the entire DevDocs ecosystem with
a single command and focus on documentation, not setup

• 🖥️ True Cross-Platform Support - Carefully crafted scripts for both
Unix and Windows Enjoy the same reliable experience whether you're on
macOS, Linux, or Windows

• 📊 Enhanced Logging & Monitoring - Docker-native logging and
comprehensive health checks Easily troubleshoot issues with real-time
log streaming and container-specific access

• 📚 Improved Documentation - New "Scripts and Their Purpose" section and
platform-specific guides Find the right tool for the job without digging
through code

• 🖼️ Better Project Organization - Dedicated assets directory and
standardized script locations Navigate a cleaner repository structure
that's easier to maintain

• 🔧 Technical Improvements - On-demand MCP model and improved Windows
permission handling Experience more efficient resource usage and fewer
permission-related errors on Windows

❤️ Thanks to all the contributors and collaborators that helped make
this release happen!

Author: Shubham-Khichi

.env.template

@@ -0,0 +1,18 @@
+# Crawl4AI API Token (optional)
+# If set, this token will be used for authentication with the Crawl4AI service
+# A default demo key is provided for testing, but you may want to change this for production
+CRAWL4AI_API_TOKEN=devdocs-demo-key
+
+# Resource limits for Crawl4AI container
+# Adjust these values based on your system resources
+MAX_CONCURRENT_TASKS=5
+
+# MCP Host (used by containers to communicate with the MCP server on the host)
+# By default, this is set to host.docker.internal in the docker-compose.yml file
+# For macOS and Windows: Leave this commented out, it will work automatically
+# For Linux: If you have connectivity issues, uncomment and set to your host's IP address:
+# MCP_HOST=192.168.1.x  # Replace with your actual host IP
+
+# Uncomment and set these if you want to use LLM features in Crawl4AI
+# OPENAI_API_KEY=
+# ANTHROPIC_API_KEY=

Dockerfile.backend

@@ -0,0 +1 @@
+docker/dockerfiles/Dockerfile.backend

Dockerfile.frontend

@@ -0,0 +1 @@
+docker/dockerfiles/Dockerfile.frontend

Dockerfile.mcp

@@ -0,0 +1 @@
+docker/dockerfiles/Dockerfile.mcp

README.md

@@ -9,12 +9,13 @@
   </p>
 
   <p align="center">
-    <a href="#perfect-for">Perfect For</a> •
-    <a href="#features">Features</a> •
-    <a href="#why-devdocs">Why DevDocs</a> •
-    <a href="#getting-started">Getting Started</a> •
-    <a href="#pricing-comparison">Compare to FireCrawl</a> •
-    <a href="#join-our-community">Discord</a>
+    <a href="#-perfect-for">Perfect For</a> •
+    <a href="#-features">Features</a> •
+    <a href="#-why-devdocs">Why DevDocs</a> •
+    <a href="#-getting-started">Getting Started</a> •
+    <a href="#-scripts-and-their-purpose">Scripts</a> •
+    <a href="#-pricing-comparison">Compare to FireCrawl</a> •
+    <a href="#-join-our-community">Discord</a>
   </p>
 </div>
 
@@ -90,72 +91,125 @@ DevDocs brings documentation to you. Point it at any tech documentation URL, and
 
 ## 🚀 Getting Started
 
-We made using DevDocs extremely easy, no complex documentation to learn DevDocs. (UI is very intuitive and self learning)
+DevDocs is designed to be easy to use with Docker, requiring minimal setup for new users.
 
-<ins>Mac & Linux(WSL) Users</ins>
+### Prerequisites
 
+- [Docker](https://docs.docker.com/get-docker/) and [Docker Compose](https://docs.docker.com/compose/install/) installed on your system
+- Git for cloning the repository
+
+### Quick Start with Docker (Recommended)
+
+For Mac/Linux users:
 ```bash
 # Clone the repository
 git clone https://github.com/cyberagiinc/DevDocs.git
 
-# Install all good stuff
+# Navigate to the project directory
 cd DevDocs
-./fast-markdown-mcp/setup.sh
-
-# Run Devdocs, (next time just run ./start.sh as all requirements are already installed)
-./start.sh
 
-#logs are located under /logs for 
-backend.log 
-frontend.log 
-mcp.log
+# Start all services using Docker
+./docker-start.sh
 ```
 
-Visit `http://localhost:3001` and start scraping and discovering documents!
+For Windows users:
+```cmd
+# Clone the repository
+git clone https://github.com/cyberagiinc/DevDocs.git
 
-<ins>Windows Users</ins>
+# Navigate to the project directory
+cd DevDocs
 
-#### Option 1: Using Batch Scripts (Command Prompt)
+# Start all services using Docker
+docker-start.bat
+```
 
-1. Run the setup script:
-   ```
-   fast-markdown-mcp\setup.bat
+> **Note for Windows Users**: If you encounter permission issues, you may need to run the script as administrator or manually set permissions on the logs, storage, and crawl_results directories. The script uses the `icacls` command to set permissions, which might require elevated privileges on some Windows systems.
+>
+> **Manually Setting Permissions on Windows**:
+>
+> If you need to manually set permissions, you can do so using either the Windows GUI or command line:
+>
+> **Using Windows Explorer**:
+> 1. Right-click on each directory (logs, storage, crawl_results)
+> 2. Select "Properties"
+> 3. Go to the "Security" tab
+> 4. Click "Edit" to change permissions
+> 5. Click "Add" to add users/groups
+> 6. Type "Everyone" and click "Check Names"
+> 7. Click "OK"
+> 8. Select "Everyone" in the list
+> 9. Check "Full control" under "Allow"
+> 10. Click "Apply" and "OK"
+>
+> **Using Command Prompt (as Administrator)**:
+> ```cmd
+> icacls logs /grant Everyone:F /T
+> icacls storage /grant Everyone:F /T
+> icacls crawl_results /grant Everyone:F /T
+> ```
+
+This single command will:
+1. Create all necessary directories
+2. Set appropriate permissions
+3. Build and start all Docker containers
+4. Monitor the services to ensure they're running properly
+
+### Accessing DevDocs
+
+Once the services are running:
+- Frontend UI: http://localhost:3001
+- Backend API: http://localhost:24125
+- Crawl4AI Service: http://localhost:11235
+
+### Logs and Monitoring
+
+When using Docker, logs can be accessed :
+
+1. **Container Logs** (recommended for debugging):
+   ```bash
+   # View logs from a specific container
+   docker logs devdocs-frontend
+   docker logs devdocs-backend
+   docker logs devdocs-mcp
+   docker logs devdocs-crawl4ai
+   
+   # Follow logs in real-time
+   docker logs -f devdocs-backend
    ```
 
-2. Start all services:
-   ```
-   start.bat
-   ```
+To stop all services, press `Ctrl+C` in the terminal where docker-start is running.
 
-#### Option 2: Using PowerShell Script
+## 📜 Scripts and Their Purpose
 
-1. First, run the setup batch script:
-   ```
-   fast-markdown-mcp\setup.bat
-   ```
+DevDocs includes various utility scripts to help with development, testing, and maintenance. Here's a quick reference:
 
-2. Then start all services using PowerShell:
-   ```powershell
-   .\start.ps1
-   ```
+### Startup Scripts
+- `start.sh` / `start.bat` / `start.ps1` - Start all services (frontend, backend, MCP) for local development.
+- `docker-start.sh` / `docker-start.bat` - Start all services using Docker containers.
 
-   Note: If you encounter execution policy restrictions, you may need to run:
-   ```powershell
-   Set-ExecutionPolicy -ExecutionPolicy Bypass -Scope Process
-   ```
+### MCP Server Scripts
+- `check_mcp_health.sh` - Verify the MCP server's health and configuration status.
+- `restart_and_test_mcp.sh` - Restart Docker containers with updated MCP configuration and test connectivity.
 
-#### Option 3: Using Windows Subsystem for Linux (WSL)
+### Crawl4AI Scripts
+- `check_crawl4ai.sh` - Check the status and health of the Crawl4AI service.
+- `debug_crawl4ai.sh` - Run Crawl4AI in debug mode with verbose logging for troubleshooting.
+- `test_crawl4ai.py` - Run tests against the Crawl4AI service to verify functionality.
+- `test_from_container.sh` - Test the Crawl4AI service from within a Docker container.
 
-If you prefer a Linux-like environment on Windows:
+### Utility Scripts
+- `view_result.sh` - Display crawl results in a formatted view.
+- `find_empty_folders.sh` - Identify empty directories in the project structure.
+- `analyze_empty_folders.sh` - Analyze empty folders and categorize them by risk level.
+- `verify_reorganization.sh` - Verify that code reorganization was successful.
 
-1. Install WSL by following the [official instructions](https://docs.microsoft.com/en-us/windows/wsl/install)
-2. Open a WSL terminal
-3. Navigate to your project directory
-4. Run the original Linux scripts:
-   ```bash
-   ./fast-markdown-mcp/setup.sh
-   ./start.sh
-   ```
+These scripts are organized in the following directories:
+- Root directory: Main scripts for common operations
+- `scripts/general/`: General utility scripts
+- `scripts/docker/`: Docker-specific scripts
+- `scripts/mcp/`: MCP server management scripts
+- `scripts/test/`: Testing and verification scripts
 
 ## 🌍 Built for Developers, by Developers
 
@@ -226,7 +280,7 @@ Final Output Construction: The final answer should be organized, directly addres
 
 ## 📝 Technology Partners
 
-<img src="image-6.png" width="200" height="100"> <img src="image-7.png" width="250" height="100"> <img src="image-8.png" width="300" height="100">
+<img src="assets/image-6.png" width="200" height="100"> <img src="assets/image-7.png" width="250" height="100"> <img src="assets/image-8.png" width="300" height="100">
 
 ## Star History
 

app/api/all-files/route.ts

@@ -0,0 +1,132 @@
+import { NextResponse } from 'next/server'
+import fs from 'fs/promises'
+import path from 'path'
+
+const STORAGE_DIR = path.join(process.cwd(), 'storage/markdown')
+
+export async function GET(request: Request) {
+  try {
+    // Only get .md files
+    const files = await fs.readdir(STORAGE_DIR)
+    const mdFiles = files.filter(f => f.endsWith('.md'))
+    const jsonFiles = files.filter(f => f.endsWith('.json'))
+    
+    // Get disk files
+    const diskFileDetails = await Promise.all(
+      mdFiles.map(async (filename) => {
+        const mdPath = path.join(STORAGE_DIR, filename)
+        const jsonPath = path.join(STORAGE_DIR, filename.replace('.md', '.json'))
+        const stats = await fs.stat(mdPath)
+        const content = await fs.readFile(mdPath, 'utf-8')
+        
+        // Check if this is a consolidated file by examining the JSON metadata
+        let isConsolidated = false
+        let pagesCount = 0
+        let rootUrl = ''
+        
+        if (jsonFiles.includes(filename.replace('.md', '.json'))) {
+          try {
+            const jsonContent = await fs.readFile(jsonPath, 'utf-8')
+            const metadata = JSON.parse(jsonContent)
+            
+            // If the metadata has a "pages" array, it's a consolidated file
+            if (metadata.pages && Array.isArray(metadata.pages)) {
+              isConsolidated = true
+              pagesCount = metadata.pages.length
+              rootUrl = metadata.root_url || ''
+            }
+          } catch (e) {
+            console.error(`Error reading JSON metadata for ${filename}:`, e)
+          }
+        } else {
+          // Create JSON file if it doesn't exist
+          const jsonContent = JSON.stringify({
+            content,
+            metadata: {
+              wordCount: content.split(/\s+/).length,
+              charCount: content.length,
+              timestamp: stats.mtime
+            }
+          }, null, 2)
+          await fs.writeFile(jsonPath, jsonContent, 'utf-8')
+        }
+        
+        // Extract sections to count how many pages are included
+        if (!pagesCount && isConsolidated) {
+          // Count sections that start with "## " and have a URL: line after them
+          const sectionMatches = content.match(/## .+\nURL: .+/g)
+          pagesCount = sectionMatches ? sectionMatches.length : 0
+        }
+        
+        return {
+          name: filename.replace('.md', ''),
+          jsonPath,
+          markdownPath: mdPath,
+          timestamp: stats.mtime,
+          size: stats.size,
+          wordCount: content.split(/\s+/).length,
+          charCount: content.length,
+          isConsolidated,
+          pagesCount: isConsolidated ? pagesCount : 1,
+          rootUrl: rootUrl || '',
+          isInMemory: false
+        }
+      })
+    )
+    
+    // Define interface for in-memory file
+    interface MemoryFile {
+      name: string;
+      path: string;
+      timestamp: string;
+      size: number;
+      wordCount: number;
+      charCount: number;
+      isInMemory: boolean;
+      isJson: boolean;
+      metadata?: any;
+    }
+    
+    // Get in-memory files from the backend
+    let memoryFiles = []
+    try {
+      const memoryResponse = await fetch('http://localhost:24125/api/memory-files')
+      if (memoryResponse.ok) {
+        const memoryData = await memoryResponse.json()
+        if (memoryData.success && Array.isArray(memoryData.files)) {
+          // Convert in-memory files to the same format as disk files
+          memoryFiles = memoryData.files
+            .filter((file: MemoryFile) => !file.isJson) // Only include markdown files
+            .map((file: MemoryFile) => ({
+              name: file.name,
+              jsonPath: file.path.replace('.md', '.json'),
+              markdownPath: file.path,
+              timestamp: new Date(file.timestamp),
+              size: file.size,
+              wordCount: file.wordCount,
+              charCount: file.charCount,
+              isConsolidated: false,
+              pagesCount: 1,
+              rootUrl: '',
+              isInMemory: true
+            }))
+        }
+      }
+    } catch (e) {
+      console.error('Error fetching in-memory files:', e)
+    }
+    
+    // Combine disk and memory files - return ALL files, not just consolidated ones
+    const allFiles = [...diskFileDetails, ...memoryFiles]
+    
+    return NextResponse.json({
+      success: true,
+      files: allFiles
+    })
+  } catch (error) {
+    return NextResponse.json(
+      { success: false, error: error instanceof Error ? error.message : 'Failed to load files' },
+      { status: 500 }
+    )
+  }
+}