Class: Heatwave::Crawler::Tiers::Amazon

Inherits:

OxylabsBase

• Object
• OxylabsBase
• Heatwave::Crawler::Tiers::Amazon

Defined in:

app/services/heatwave/crawler/tiers/amazon.rb

Overview

Tier 2a: dedicated Amazon scraping via Oxylabs structured sources
(cheaper and more reliable than generic rendering for amazon.* URLs).
Only applies to Amazon hosts.

Constant Summary

AMAZON_HOST =

/\A(www\.)?amazon\./i

Constants inherited from OxylabsBase

OxylabsBase::OXYLABS_TIMEOUT

Instance Method Summary

• #applies_to?(url) ⇒ Boolean

Methods inherited from OxylabsBase

#attempt, #initialize

Constructor Details

This class inherits a constructor from Heatwave::Crawler::Tiers::OxylabsBase

Instance Method Details

#applies_to?(url) ⇒ Boolean

Returns:

• (Boolean)

# File 'app/services/heatwave/crawler/tiers/amazon.rb', line 11

def applies_to?(url)
  URI.parse(url.to_s.strip).host.to_s.match?(AMAZON_HOST)
rescue URI::InvalidURIError
  false
end