🛡️ TLD-Queried Web Application CMS-Based Vulnerability Assessment Automation Framework

📚 Project Overview

This project is a complete, automated vulnerability assessment pipeline for web applications, filtered by user-specified TLD (Top-Level Domain) and SLD (Second-Level Domain). It queries Common Crawl for target domains, filters CMS-based applications, runs automated vulnerability scans, and uploads the results to DefectDojo for analysis and visualization.

🔍 Powered by Streamlit, Nmap, WPScan, ZAP, SSLyze, Nuclei, MongoDB, and more!

---

🎯 Key Features

• ✅ TLD + SLD Based Discovery using Common Crawl
• ✅ CMS Detection (WordPress, Drupal, Joomla, etc.)
• ✅ Parallel Scanning via Nmap, WPScan, ZAP, SSLyze, Nuclei, Nikto, Amass
• ✅ Live Logging & Progress Bar in Streamlit UI
• ✅ DefectDojo Integration — Central vulnerability dashboard
• ✅ MongoDB Storage for scan logs and traceability
• ✅ Dockerized Deployment for easy local/CI/CD setup

---

🛠️ Tools & Frameworks Used

Tool	Purpose
Streamlit	Elegant UI and user input interface
Common Crawl	Source of global web data for discovery
WhatWeb	CMS fingerprinting
Nmap	Port, service & script-based vulnerability scanning
WPScan	WordPress-focused vulnerability scanning
OWASP ZAP	Active and passive security analysis
Nuclei	Template-based vulnerability scanner (CVEs, panels, SSL issues, etc.)
Nikto	Web server misconfiguration and known issue scanner
SSLyze	Deep SSL/TLS vulnerability scanning
Amass	Subdomain enumeration
MongoDB	Backend storage for scans and statuses
DefectDojo	Central vulnerability management and visualization

---

📦 Project Architecture

\[Streamlit UI] ───> \[Common Crawl URL Extractor]
│
▼
\[CMS Detector (WhatWeb)]
│
┌──────┴──────┐
▼             ▼
\[CMS URLs]   \[Non-CMS URLs]
│             │
▼             ▼
\[Vulnerability Scans: Nmap, WPScan, ZAP, Nuclei, etc.]
│
▼
\[Upload JSON/XML Results to DefectDojo + Store in MongoDB]
│
▼
\[Streamlit + DefectDojo Results Viewer]

---

📁 Directory Structure

.
├── full\_app.py          # Main Streamlit dashboard
├── VA_Automation.py      # VAPT core script
├── Input.py                 # Entry point for CLI-based scanning
├── scan\_results/        # All tool outputs saved here
├── bash\_scripts/        # Bash pipeline for domain extraction
├── Dockerfile            # Docker setup for full app
├── requirements.txt      # All dependencies
└── README.md             # This file

---

🧪 How to Run

🔧 Prerequisites

• Python 3.8+
• MongoDB (local or Docker)
• DefectDojo running (Docker recommended)
• Tools installed:

sudo apt install nmap whatweb nikto sslyze nuclei wpscan zaproxy amass -y

🔨 Install Python requirements

pip install -r requirements.txt

🚀 Run the app

streamlit run Defectdojo_Pipeline.py

The app runs on http://localhost:8501 by default.

---

🗃️ MongoDB Usage

All scanned URLs and their status are logged into a MongoDB collection vapt-db.urls. To inspect:

mongosh
> use vapt-db
> db.urls.find().pretty()

---

📈 DefectDojo Integration

Scan results (ZAP, Nmap JSON, SSLyze, etc.) are uploaded to DefectDojo via API. You'll need:

• DefectDojo host URL
• API token
• Product ID / Engagement ID (created programmatically)

---

🧵 Future Enhancements

• Email Alerts on critical vulnerabilities
• PDF export of scan reports
• Authenticated scans (login forms)
• CVSS filtering & AI recommendation engine
• Integration with GitHub Projects / Jira

---

💡 Real-World Applications

• 🌐 TLD-wide passive and active reconnaissance for national CERTs
• 🏢 Enterprise red teaming for CMS-heavy websites
• 🛡️ University-level infosec research labs
• 🧪 Cybersecurity training and automation projects

---

📜 License

MIT License. Free to use, modify, and deploy.

---

🙌 Acknowledgements

Special thanks to:

• Common Crawl Team
• OWASP Foundation
• DefectDojo contributors
• Community developers of open-source scanning tools