Ruby · Web Crawling

Web Crawling

• sparklemotion/mechanize on GitHub, opens in a new tab

  4.4Kstars

  Mechanize is a ruby library that makes automated web interaction easy.

  Ruby·MIT License·steady·3 months ago

• propublica/upton on GitHub, opens in a new tab

  1.6Kstars

  A batteries-included framework for easy web-scraping.

  HTML·MIT License·dormant·8 years ago

• felipecsl/wombat on GitHub, opens in a new tab

  1.4Kstars

  Web scraper with an elegant DSL that parses structured data from web pages.

  Ruby·MIT License·steady·4 months ago

• vifreefly/kimuraframework on GitHub, opens in a new tab

  1.1Kstars

  A modern web scraping framework written in Ruby which works out of box with Headless Chromium/Firefox, PhantomJS, or simple HTTP requests and allows to scrape and interact with JavaScript rendered websites.

  Ruby·MIT License·steady·7 months ago

• jaimeiniesta/metainspector on GitHub, opens in a new tab

  1Kstars

  Ruby gem for web scraping purposes.

  Ruby·MIT License·steady·3 months ago

• postmodern/spidr on GitHub, opens in a new tab

  835stars

  A versatile Ruby web spidering library that can spider a site, multiple domains, certain links or infinitely. Spidr is designed to be fast and easy to use.

  Ruby·MIT License·steady·7 months ago

• gottfrois/link_thumbnailer on GitHub, opens in a new tab

  510stars

  Ruby gem that generates thumbnail images and videos from a given URL. Much like popular social website with link preview.

  Ruby·MIT License·slowing·last year