Module: Heatwave::Crawler::Tiers

Defined in:
app/services/heatwave/crawler/tiers.rb,
app/services/heatwave/crawler/tiers/amazon.rb,
app/services/heatwave/crawler/tiers/direct.rb,
app/services/heatwave/crawler/tiers/oxylabs.rb,
app/services/heatwave/crawler/tiers/playwright.rb,
app/services/heatwave/crawler/tiers/oxylabs_base.rb,
app/services/heatwave/crawler/tiers/web_unblocker.rb,
app/services/heatwave/crawler/tiers/playwright/limiter.rb,
app/services/heatwave/crawler/tiers/oxylabs_residential.rb

Overview

Namespace for the crawler's fetch tiers plus helpers shared across them.

Defined Under Namespace

Classes: Amazon, Direct, Oxylabs, OxylabsBase, OxylabsResidential, Playwright, WebUnblocker

Class Method Summary collapse

Class Method Details

.acceptable?(outcome, mode:) ⇒ Boolean

The acceptability gate between tiers. Probe mode accepts any HTTP
answer (a 404 or 403 IS the answer the caller asked for); content modes
require a usable page: success status, non-garbled HTML, no bot
interstitial, non-blocked extracted text.

Parameters:

Returns:

  • (Boolean) 


39
40
41
42
43
44
 
# File 'app/services/heatwave/crawler/tiers.rb', line 39

def acceptable?(outcome, mode:)
  return false if outcome.error
  return outcome.status.present? if mode == :probe

  content_acceptable?(outcome)
end

.content_acceptable?(outcome) ⇒ Boolean

Parameters:

Returns:

  • (Boolean) 


48
49
50
51
52
53
54
55
56
57
 
# File 'app/services/heatwave/crawler/tiers.rb', line 48

def content_acceptable?(outcome)
  html = outcome.html
  return false if html.blank?
  return false unless outcome.status.nil? || (200..299).cover?(outcome.status)
  return false if Heatwave::Crawler::Guards.garbled_content?(html)
  return false if Heatwave::Crawler::Guards.bot_challenge?(html, outcome.status || 200)

  candidate = Heatwave::Crawler::Extract::PlainText.call(html)
  candidate.present? && !Heatwave::Crawler::Guards.blocked_content?(candidate)
end

.error_outcome(tier:, url:, error:, exception: nil) ⇒ Heatwave::Crawler::FetchOutcome

Builds a transport-failure outcome for a tier attempt, normalizing the
exception into the details hash consumers (probes, telemetry) read.

Parameters:

  • tier (Symbol) —

    tier registry name

  • url (String) —

    the URL attempted

  • error (String, Exception) —

    failure message or the exception itself

  • exception (Exception, nil) (defaults to: nil) —

    the rescued exception, when available

Returns:



15
16
17
18
19
20
21
 
# File 'app/services/heatwave/crawler/tiers.rb', line 15

def error_outcome(tier:, url:, error:, exception: nil)
  Heatwave::Crawler::FetchOutcome.new(
    tier: tier, url: url,
    error: error.is_a?(Exception) ? error.message : error,
    details: exception_details(exception)
  )
end

.exception_details(exception) ⇒ Hash

Returns error_class + exception details, or empty when nil.

Parameters:

  • exception (Exception, nil)

Returns:

  • (Hash) —

    error_class + exception details, or empty when nil



25
26
27
28
29
 
# File 'app/services/heatwave/crawler/tiers.rb', line 25

def exception_details(exception)
  return {} unless exception

  { error_class: exception.class.name, exception: exception }
end