Class: Heatwave::Crawler::Tiers::WebUnblocker

Inherits:
Object
  • Object
show all
Defined in:
app/services/heatwave/crawler/tiers/web_unblocker.rb

Overview

Last-resort tier: Oxylabs Web Unblocker (AI-powered proxy for the
hardest bot-walls, per-GB billing). Only reached after datacenter AND
residential rendering both failed.

Instance Method Summary collapse

Constructor Details

#initialize(policy) ⇒ WebUnblocker

Returns a new instance of WebUnblocker.

Parameters:



10
11
12
 
# File 'app/services/heatwave/crawler/tiers/web_unblocker.rb', line 10

def initialize(policy)
  @policy = policy
end

Instance Method Details

#applies_to?(_url) ⇒ Boolean

The Web Unblocker is transport-generic — any URL qualifies.

Parameters:

  • _url (String)

Returns:

  • (Boolean) 


17
 
# File 'app/services/heatwave/crawler/tiers/web_unblocker.rb', line 17

def applies_to?(_url) = true

#attempt(url) ⇒ Heatwave::Crawler::FetchOutcome

Parameters:

  • url (String)

Returns:



21
22
23
24
25
26
27
28
 
# File 'app/services/heatwave/crawler/tiers/web_unblocker.rb', line 21

def attempt(url)
  result = api.fetch(url, geo_location: policy.residential_geo)
  return Tiers.error_outcome(tier: :web_unblocker, url: url, error: result.error) unless result.success?

  FetchOutcome.new(tier: :web_unblocker, url: url, html: result.html, status: result.status)
rescue StandardError => e
  Tiers.error_outcome(tier: :web_unblocker, url: url, error: e, exception: e)
end