Skip to content

Home ‐ URL Extractor

Jump to bottom
YogSec edited this page Mar 1, 2025 · 1 revision

📚 URL Extractor Wiki — Designed By YogSec

🌐 Problem Statement

💻 The Modern Researcher's Struggle

In the field of bug bounty hunting, penetration testing, and open-source intelligence (OSINT), URLs act like the lifeline of the web. Security researchers often need to extract every single URL from source code, configurations, or documentation files. These URLs can be:

  • Internal API endpoints 📡
  • External third-party services 🌐
  • Sensitive files hosted accidentally 🔓
  • Redirect points or vulnerable links 🚨
  • Hidden backups or debug endpoints 💣

😖 The Manual Pain

Traditionally, researchers do one of the following:

  • Manually scan files, wasting hours ⏳
  • Use grep with fragile regex patterns
  • Use heavy tools that are overkill for simple extraction

❗ The more files you have, the slower and more error-prone these methods become.

💡 The Solution — URL Extractor by YogSec

🔧 What is URL Extractor?

This tool is a simple but powerful Bash script designed to:

✅ Automatically extract URLs from any file — HTML, JS, CSS, JSON, XML, Python, PHP, etc.
✅ Process an entire folder recursively, not just single files.
✅ Run with concurrency for faster folder processing.
✅ Show extracted URLs directly in the terminal or save them to a file for further analysis.

🛠️ How It Works

URL Extractor scans files using a reliable URL regex pattern that catches:

  • http:// and https:// links
  • Full domain links (with or without www)
  • Subdomains
  • URLs inside scripts, comments, or hidden in code

🚀 Key Features

Feature Description
Multiple File Types Supports txt, json, xml, js, py, php, css, and more
Single File Mode Scan a single file with -u
Folder Mode (Concurrent) Scan all files in a folder with -l
Save Results Use -s to save URLs to a file
Terminal Output URLs also shown live in terminal
Beautiful Header Professional tool banner for YogSec
Help & Version Quick info with -h and -v

🧰 Pro Tips

✅ Combine with cat to feed from multiple files:

cat file1.js file2.html | ./url_extractor.sh -u -

✅ Pipe to tools like httpx for quick probing:

./url_extractor.sh -l /path/to/source | httpx

✅ Chain with grep to search for sensitive patterns:

./url_extractor.sh -l /path/to/folder | grep -i "admin"

👨‍💻 Developed By

🧑‍💻 Abhinav Singwal — Founder of YogSec
✉️ Contact me: abhinavsingwal@gmail.com

⭐️ Contribute & Support

If you love this tool, please star the repo ⭐️ and contribute by:

  • Suggesting new features
  • Improving regex patterns
  • Reporting bugs

🔗 Why YogSec Tools?

✔️ Built for Bug Hunters
✔️ Lightweight & Fast
✔️ No dependencies
✔️ Works out-of-the-box

Let me know if you want me to generate this file (WIKI.md) for you as a ready-to-upload file. Would you also like a CONTRIBUTING.md or LICENSE? 😊

Clone this wiki locally