- Introduction
- Features
- Antivirus Warning
- Version History
- Installation
- Usage
- Contributing
- Version Checking
- License
- malScraper is a modular tool that streamlines the process of scraping and managing lists of Payload Domains, IOC's & C2 IPs from various feeds. It simplifies the task of blacklisting for security and threat intelligence purposes.
- Modular Design: Easily extendable with additional modules.
- Version Checking: Verify if you are using the latest version on startup.
- Reopen Functionality: Ability to reopen a previously composed report.
- Download Progress Bars: Clean, color-coded progress bars for all downloads.
- Clear File Save Feedback: After each download, the full path of the saved file is displayed.
- Payload Report Obfuscation/Zip: Choose to obfuscate or zip the payload report before it is written to disk, to help avoid antivirus false positives.
Some reports generated by malScraper (especially PayloadReport.txt) may be flagged or quarantined by antivirus software (such as Windows Defender) because they contain known malware indicators. These files are for research and defensive use only.
- You will be prompted to obfuscate or zip the payload report to help avoid false positives.
- If you need to retain these files, consider adding an exclusion for the report directory, or use the obfuscate/zip options.
Complete rewrite in Python, making it fully cross-platform compatible with Windows, macOS, and Linux environments. Maintains all the key functionality of the original bash script while adding new features, improving performance, and enhancing the user experience.
- Now works seamlessly on Windows, macOS, and Linux
- Platform-specific optimizations for file paths and system operations
- No external dependencies beyond Python and the requests library
- Improved ASCII art header with pyfiglet support
- Consistent color-coded output across all platforms
- Better formatted menus and command outputs
- Progress bars for downloads
- Clear file save locations after each download
- User prompt for obfuscating/zipping payload report
- Tab Completion: Auto-complete commands by pressing TAB (requires readline/pyreadline)
- Detailed version checking with GitHub API integration
- Visual download progress indicators
- Automatic backup of previous version before updating
- Clean restart process after updates
- Display of release notes within the application
- Graceful exit handling (Ctrl+C, Ctrl+D)
- Comprehensive exception management
- Network connectivity error handling
- File operation error recovery
- Faster data processing
- More efficient memory usage
- Improved file I/O operations
- Python 3.6 or higher
- Requests library (
pip install requests) - prompt_toolkit for tab completion (
pip install prompt_toolkit) - Optional: pyfiglet for enhanced ASCII art (
pip install pyfiglet) - Optional: pyreadline for tab completion on Windows (
pip install pyreadline)
python -m pip install requests
python -m pip install prompt_toolkit # For tab completion
python -m pip install pyfiglet # Optional
python -m pip install pyreadline # Optional, for additional readline support
python malScraper.pypip3 install requests
pip3 install prompt_toolkit # For tab completion
pip3 install pyfiglet # Optional
python3 malScraper.pyUsers of previous bash-based versions can simply download the new Python script and run it. The application will maintain the same folder structure and file naming conventions.
- Some terminal emulators may not fully support color codes
- Character encoding issues may occur in certain environments with emoji display
- Antivirus software may flag or quarantine reports containing known malware indicators (see Antivirus Warning)
- 1.4.6 - Tab Completion & Enhanced UX
- Added robust tab completion for all commands (press TAB to auto-complete)
- Uses prompt_toolkit for reliable tab completion across all platforms
- Works on macOS, Linux, and Windows without additional dependencies
- Enhanced help menu with tab completion tips
- Improved command discovery and user experience
- 1.4.5 - Download Experience & Antivirus Handling
- Cleaned up download output: only the download name, progress bar, and a single success/failure line per file.
- After each successful download, the full path of the file is displayed.
- Payload report handling prints both .txt and .zip paths as appropriate.
- Added payload line count to the summary.
- Improved error handling and user feedback throughout the download process.
- Added antivirus warning and obfuscate/zip options for payload report.
- 1.3 - Initial Python Conversion
- The tool has been converted to Python for improved functionality.
- 1.2 - Version Checking
- malScraper now verifies if you are using the latest version on startup.
- 1.1 - Reopen Functionality
- Added the ability to reopen a previously composed report.
- 1.0 - Initial Implementation
- Core functionality was implemented in the first release.
- Clone the repository.
git clone https://github.com/rynmon/malScraper
- Navigate to the project directory.
cd malScraper
- Install dependencies.
pip install -r requirements.txt
- To use malScraper, follow these steps:
- Run the tool.
python malScraper.py
- Follow the on-screen instructions to configure and use the tool.
- After each scan, you will see a summary and the save locations for all reports.
- For the payload report, you will be prompted to obfuscate or zip the file before it is written to disk.
- Contributions are welcome! If you find a bug or have an enhancement in mind, feel free to open an issue or submit a pull request.
- malScraper will automatically check for updates on startup. Make sure you are using the latest version to benefit from new features and improvements.
- This project is licensed under the MIT License.