Skip to content

v1.0.6 (Media Collection)
Compare
Choose a tag to compare
@jaypyles jaypyles released this 10 May 20:15
· 72 commits to master since this release
v1.0.6
8cd3059
This commit was created on GitHub.com and signed with GitHub’s verified signature.
GPG key ID: B5690EEEBB952194
Verified
Learn about vigilant mode.

✨ Add Media Collection to Scraping Pipeline

Summary

This PR introduces the collect_media function, which enhances scraping capabilities by automatically detecting and downloading various types of media assets from a web page using a Selenium-controlled browser session.

🔧 Features

Supported Media Types:

  • Images (<img>)
  • Videos (<video>)
  • Audio files (<audio>)
  • PDFs (<a href="*.pdf">)
  • Documents (.doc, .docx, .txt, .rtf)
  • Presentations (.ppt, .pptx)
  • Spreadsheets (.xls, .xlsx, .csv)

Functionality:

  • Uses CSS selectors to find elements containing media links.
  • Downloads each valid media file (HTTP/HTTPS only).
  • Saves all assets to a structured media/ directory, grouped by media type.
  • Writes a download_summary.txt with the original URLs and their local file paths.

Error Handling:

  • Skips failed downloads and logs the error.
  • Generates fallback filenames when none are detected in the URL.
Assets 2
Loading