Class: Heatwave::Crawler

Inherits:
Object
  • Object
show all
Defined in:
app/services/heatwave/crawler.rb,
app/services/heatwave/crawler/guards.rb,
app/services/heatwave/crawler/policy.rb,
app/services/heatwave/crawler/result.rb,
app/services/heatwave/crawler/extract/seo.rb,
app/services/heatwave/crawler/tiers/amazon.rb,
app/services/heatwave/crawler/tiers/direct.rb,
app/services/heatwave/crawler/fetch_outcome.rb,
app/services/heatwave/crawler/tiers/oxylabs.rb,
app/services/heatwave/crawler/result_enricher.rb,
app/services/heatwave/crawler/tiers/playwright.rb,
app/services/heatwave/crawler/tiers/oxylabs_base.rb,
app/services/heatwave/crawler/tiers/web_unblocker.rb,
app/services/heatwave/crawler/tiers/playwright/limiter.rb,
app/services/heatwave/crawler/tiers/oxylabs_residential.rb

Overview

Unified content-fetch service ("supercrawler") — consolidates the app's
ad-hoc external fetch paths behind one entry point with explicit modes:

:read — slim Markdown main-content extraction (direct fetch only)
:probe — status-only check, no extraction (URL validators, link checks)
:inspect — full headless-browser page load + technical-SEO artifacts
:bypass — gated / bot-protected content (Oxylabs datacenter → residential →
Web Unblocker)
:auto — escalate direct → Amazon preset → headless browser → Oxylabs
render → Oxylabs residential → Web Unblocker

Plan: doc/tasks/202607181233_SUPERCRAWLER.md

Examples:

Read a page as Markdown, escalating past bot-walls

result = Heatwave::Crawler.fetch('https://example.com/article', mode: :auto)
result.content # => "# Article\n..."

Status probe without following redirects

policy = Heatwave::Crawler::Policy.new(http_method: :head, follow_redirects: false)
Heatwave::Crawler.fetch(url, mode: :probe, policy: policy).status # => 301

Defined Under Namespace

Modules: Extract, Guards, Tiers Classes: FetchOutcome, Policy, Result, ResultEnricher

Constant Summary collapse

MODES =

Fetch modes accepted by fetch.

%i[read probe inspect bypass auto].freeze
MODE_TIERS =

Default tier sequence per mode. Tier instances are filtered by
#applies_to? (e.g. the Amazon preset only engages for amazon.* URLs).

{
  read: %i[direct],
  probe: %i[direct],
  inspect: %i[playwright],
  bypass: %i[oxylabs_render oxylabs_residential web_unblocker],
  auto: %i[direct oxylabs_amazon playwright oxylabs_render oxylabs_residential web_unblocker]
}.freeze
TIER_CLASSES =

Tier registry name -> strategy class.

{
  direct: Tiers::Direct,
  oxylabs_amazon: Tiers::Amazon,
  playwright: Tiers::Playwright,
  oxylabs_render: Tiers::Oxylabs,
  oxylabs_residential: Tiers::OxylabsResidential,
  web_unblocker: Tiers::WebUnblocker
}.freeze

Class Method Summary collapse

Instance Method Summary collapse

Constructor Details

#initialize(url, mode, policy) ⇒ Crawler

Returns a new instance of Crawler.

Parameters:

Raises:

  • (ArgumentError) —

    on an unknown mode



59
60
61
62
63
64
65
66
 
# File 'app/services/heatwave/crawler.rb', line 59

def initialize(url, mode, policy)
  raise ArgumentError, "unknown crawl mode #{mode.inspect} (known: #{MODES.join(', ')})" unless MODES.include?(mode)

  @url = url.to_s.strip
  @mode = mode
  @policy = policy
  @enricher = ResultEnricher.new(mode: mode, policy: policy)
end

Class Method Details

.fetch(url, mode: :auto, policy: Policy.new) ⇒ Heatwave::Crawler::Result

Parameters:

  • url (String) —

    absolute http(s) URL

  • mode (Symbol) (defaults to: :auto) —

    :read, :probe, :inspect, :bypass or :auto

  • policy (Heatwave::Crawler::Policy) (defaults to: Policy.new) —

    per-caller configuration

Returns:



52
 
# File 'app/services/heatwave/crawler.rb', line 52

def fetch(url, mode: :auto, policy: Policy.new) = new(url, mode, policy).fetch

Instance Method Details

#fetch ⇒ Heatwave::Crawler::Result

Returns:



69
70
71
72
73
74
75
 
# File 'app/services/heatwave/crawler.rb', line 69

def fetch
  Guards.validate_public_url!(url, allow_private_hosts: policy.allow_private_hosts)
  walk_tiers
rescue ArgumentError => e
  Result.new(url: url, error: e.message,
             details: { error_class: e.class.name, guarded: true })
end