Skip to content

@scrapinghub Scrapinghub

Change the repository type filter

All

    Repositories list

    183 repositories

    • scrapinghub-stack-scrapy

      Public
      Software stack with latest Scrapy and updated deps
      Dockerfile
      •
      BSD 3-Clause "New" or "Revised" License
      •20•63•2•2•Updated Jun 20, 2025Jun 20, 2025

    • shub-workflow

      Public
      Python
      •
      BSD 3-Clause "New" or "Revised" License
      •15•15•2•2•Updated Jun 13, 2025Jun 13, 2025

    • scrapy-frontera

      Public
      More flexible and featured Frontera scheduler for Scrapy
      Python
      •
      BSD 3-Clause "New" or "Revised" License
      •5•37•2•1•Updated Jun 6, 2025Jun 6, 2025

    • hcf-backend

      Public
      Crawl Frontier HCF backend
      Python
      •
      BSD 3-Clause "New" or "Revised" License
      •5•8•2•1•Updated Jun 6, 2025Jun 6, 2025

    • frontera

      Public
      A scalable frontier for web crawlers
      Python
      •
      BSD 3-Clause "New" or "Revised" License
      •217•1.3k•78•17•Updated Jun 6, 2025Jun 6, 2025

    • web-poet

      Public
      Web scraping Page Objects core library
      pythonweb-scrapinghacktoberfestpage-objects
      Python
      •
      BSD 3-Clause "New" or "Revised" License
      •15•101•15•13•Updated Jun 6, 2025Jun 6, 2025

    • scrapinghub-entrypoint-scrapy

      Public
      Scrapy entrypoint for Scrapinghub job runner
      Python
      •
      BSD 3-Clause "New" or "Revised" License
      •16•26•7•1•Updated Jun 5, 2025Jun 5, 2025

    • dateparser

      Public
      python parser for human readable dates
      hacktoberfest
      Python
      •
      BSD 3-Clause "New" or "Revised" License
      •480•2.7k•298•53•Updated May 29, 2025May 29, 2025

    • scrapy-poet

      Public
      Page Object pattern for Scrapy
      Python
      •
      BSD 3-Clause "New" or "Revised" License
      •28•123•11•4•Updated May 26, 2025May 26, 2025

    • andi

      Public
      Library for annotation-based dependency injection
      Python
      •
      BSD 3-Clause "New" or "Revised" License
      •6•22•4•1•Updated May 14, 2025May 14, 2025

    • shub

      Public
      Scrapinghub Command Line Client
      Python
      •
      BSD 3-Clause "New" or "Revised" License
      •81•133•47•14•Updated Apr 21, 2025Apr 21, 2025

    • spidermon

      Public
      Scrapy Extension for monitoring spiders execution.
      testingmonitoringscrapingcrawlingspidershacktoberfestmonitoring-toolscrapinghub
      Python
      •
      BSD 3-Clause "New" or "Revised" License
      •101•542•42•3•Updated Apr 11, 2025Apr 11, 2025

    • extruct

      Public
      Extract embedded metadata from HTML markup
      microformatssemantic-webrdfajson-ldmicrodataopengraphhacktoberfest
      Python
      •
      BSD 3-Clause "New" or "Revised" License
      •118•919•39•14•Updated Mar 24, 2025Mar 24, 2025

    • python-scrapinghub

      Public
      A client interface for Scrapinghub's API
      Python
      •
      BSD 3-Clause "New" or "Revised" License
      •61•208•25•4•Updated Feb 21, 2025Feb 21, 2025

    • price-parser

      Public
      Extract price amount and currency symbol from a raw text string
      hacktoberfest
      Python
      •
      BSD 3-Clause "New" or "Revised" License
      •50•330•17•9•Updated Feb 13, 2025Feb 13, 2025

    • social-app-django

      Public
      Python Social Auth - Application - Django
      Python
      •
      BSD 3-Clause "New" or "Revised" License
      •385•2•0•1•Updated Nov 18, 2024Nov 18, 2024

    • Formasaurus

      Public
      Formasaurus tells you the type of an HTML form and its fields using machine learning
      HTML
      •48•7•1•0•Updated Nov 7, 2024Nov 7, 2024

    • number-parser

      Public
      Parse numbers written in natural language
      hacktoberfest
      Python
      •
      BSD 3-Clause "New" or "Revised" License
      •25•117•13•6•Updated Oct 23, 2024Oct 23, 2024

    • python-crfsuite

      Public
      A python binding for crfsuite
      data-sciencecrfcrfsuite
      Python
      •
      MIT License
      •222•774•45•3•Updated Oct 1, 2024Oct 1, 2024

    • streamparse

      Public
      streamparse lets you run Python code against real-time streams of data. Integrates with Apache Storm.
      Python
      •
      Apache License 2.0
      •218•2•0•1•Updated Sep 20, 2024Sep 20, 2024

    • splash

      Public
      Lightweight, scriptable browser as a service with an HTTP API
      Python
      •
      BSD 3-Clause "New" or "Revised" License
      •516•4.2k•374•26•Updated Aug 2, 2024Aug 2, 2024

    • pgcontents

      Public
      A Postgres-backed ContentsManager implementation for IPython
      Python
      •
      Apache License 2.0
      •86•2•0•1•Updated Jul 18, 2024Jul 18, 2024

    • shublang

      Public
      Pluggable DSL that uses pipes to perform a series of linear transformations to extract data
      Python
      •
      BSD 3-Clause "New" or "Revised" License
      •8•16•23•6•Updated Jul 9, 2024Jul 9, 2024

    • woodpecker

      Public
      An opinionated fork of the Drone CI system
      Go
      •
      Other
      •430•0•0•5•Updated Jul 7, 2024Jul 7, 2024

    • varanus

      Public
      A command line spider monitoring tool
      spidermonitoringpython36
      Python
      •7•8•2•2•Updated Jul 6, 2024Jul 6, 2024

    • scrapyrt

      Public
      HTTP API for Scrapy spiders
      pythoncrawlerscrapercrawlingtwistedscrapywebcrawlerhacktoberfestwebcrawlinghacktoberfest2021
      Python
      •
      BSD 3-Clause "New" or "Revised" License
      •161•860•24•6•Updated Jun 28, 2024Jun 28, 2024

    • portia

      Public
      Visual scraping for Scrapy
      Python
      •
      BSD 3-Clause "New" or "Revised" License
      •1.4k•9.4k•111•19•Updated Jun 26, 2024Jun 26, 2024

    • sklearn-crfsuite

      Public
      scikit-learn inspired API for CRFsuite
      Python
      •214•2•0•0•Updated Jun 18, 2024Jun 18, 2024

    • webstruct-demo

      Public
      HTTP demo for https://github.com/scrapinghub/webstruct
      Python
      •
      MIT License
      •2•4•0•3•Updated Jun 17, 2024Jun 17, 2024

    • autologin

      Public
      A project to attempt to automatically login to a website given a single seed
      Python
      •
      Apache License 2.0
      •43•11•0•2•Updated Jun 17, 2024Jun 17, 2024