Class: Heatwave::Crawler::Tiers::Direct

Inherits:
Object
  • Object
show all
Defined in:
app/services/heatwave/crawler/tiers/direct.rb

Overview

Tier 1: plain HTTP GET/HEAD with a desktop-Chrome User-Agent and
RFC 9421 Web Bot Auth signing. Cheapest tier — no render, no budget.

Detects +application/pdf+ responses and returns a +fatal+ outcome so
the caller can route to its PDF flow (Sunny reads PDFs via Claude)
instead of feeding binary through the extraction path.

Constant Summary collapse

DEFAULT_USER_AGENT = 
'Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) ' \
'AppleWebKit/537.36 (KHTML, like Gecko) ' \
'Chrome/122.0.0.0 Safari/537.36'
TIMEOUT_RANGE = 
(5..30)
MAX_REDIRECT_HOPS = 
5

Instance Method Summary collapse

Constructor Details

#initialize(policy) ⇒ Direct

Returns a new instance of Direct.

Parameters:



20
21
22
 
# File 'app/services/heatwave/crawler/tiers/direct.rb', line 20

def initialize(policy)
  @policy = policy
end

Instance Method Details

#applies_to?(_url) ⇒ Boolean

Returns:

  • (Boolean) 


24
 
# File 'app/services/heatwave/crawler/tiers/direct.rb', line 24

def applies_to?(_url) = true

#attempt(url) ⇒ Heatwave::Crawler::FetchOutcome

Parameters:

  • url (String)

Returns:



28
29
30
31
32
33
34
35
36
 
# File 'app/services/heatwave/crawler/tiers/direct.rb', line 28

def attempt(url)
  timeout = policy.timeout_seconds.to_i.clamp(TIMEOUT_RANGE)
  response = fetch(url, timeout)
  return pdf_outcome(url, response) if pdf_content_type?(response)

  build_outcome(url, response)
rescue HTTP::Error, SocketError, OpenSSL::SSL::SSLError, Addressable::URI::InvalidURIError => e
  Tiers.error_outcome(tier: :direct, url: url, error: e, exception: e)
end