Class: Retailer::WebUnblockerApi

Inherits:
Object
  • Object
show all
Defined in:
app/services/retailer/web_unblocker_api.rb

Overview

Oxylabs Web Unblocker client — AI-powered proxy for the hardest bot-walls.

Separate product from the Web Scraper API: proxy-style GET through
unblock.oxylabs.io:60000 with header-driven control, separate credentials
(oxylabs_unblocker in credentials), per-GB billing. Used as the
last-resort transport for pages that fault (613) at the Scraper API — see
the WEB_UNBLOCKER_FALLBACK extractor flag and the crawler's web_unblocker
tier.

Examples:

result = Retailer::WebUnblockerApi.new.fetch('https://www.wayfair.com/pdp/x.html')
result.html # => rendered page HTML

Defined Under Namespace

Classes: Result

Constant Summary collapse

PROXY_HOST =

Web Unblocker proxy endpoint (header-driven control).

'unblock.oxylabs.io'
PROXY_PORT =

Web Unblocker proxy port.

60_000
AUTH_REJECTION_STATUSES =

Statuses that mean the proxy refused our credential rather than attempting
the fetch. They cost no budget and signal a configuration fault.

[401, 403, 407].freeze
CREDENTIAL_ALERT_KEY =

Throttle for the dead-credential alert — see #report_credential_failure.

'retailer:web_unblocker:credential_alert'
CREDENTIAL_ALERT_INTERVAL = 
1.hour
DEFAULT_TIMEOUT =

Read timeout in seconds (render waits can be long).

120
US_GEO_LOCATION_POOL =

Pool of US states for geo_location rotation. Diversifies egress geography
per request instead of hammering a target from a single state. Also
referenced by Retailer::Extractors::Wayfair for the same purpose on the
Scraper API.

[
  'Illinois,United States',
  'California,United States',
  'Texas,United States',
  'Florida,United States',
  'New York,United States',
  'Georgia,United States',
  'Colorado,United States',
  'Washington,United States'
].freeze

Instance Method Summary collapse

Constructor Details

#initialize(options = {}) ⇒ WebUnblockerApi

Returns a new instance of WebUnblockerApi.

Parameters:

  • options (Hash) (defaults to: {})

Options Hash (options):

  • :username (String) —

    Web Unblocker API user (default: credentials)

  • :password (String) —

    Web Unblocker API password (default: credentials)

  • :timeout (Integer) —

    read timeout in seconds (default: DEFAULT_TIMEOUT)

  • :logger (Logger) 


66
67
68
69
70
71
 
# File 'app/services/retailer/web_unblocker_api.rb', line 66

def initialize(options = {})
  @username = options[:username] || Heatwave::Configuration.fetch(:oxylabs_unblocker, :username)
  @password = options[:password] || Heatwave::Configuration.fetch(:oxylabs_unblocker, :password)
  @timeout = options[:timeout] || DEFAULT_TIMEOUT
  @logger = options[:logger] || Rails.logger
end

Instance Method Details

#configured? ⇒ Boolean

Whether the unblocker credentials are configured — callers treat a false
here as "fallback unavailable" and must never see an exception instead.

Returns:

  • (Boolean) 


76
 
# File 'app/services/retailer/web_unblocker_api.rb', line 76

def configured? = @username.present? && @password.present?

#fetch(url, geo_location: nil) ⇒ Result

Fetch a URL through the unblocker with JS rendering.

Parameters:

  • url (String) —

    absolute http(s) URL

  • geo_location (String, nil) (defaults to: nil) —

    explicit geo (e.g. 'Canada'); US egress
    rotates across US_GEO_LOCATION_POOL when nil or 'United States'

Returns:



84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
 
# File 'app/services/retailer/web_unblocker_api.rb', line 84

def fetch(url, geo_location: nil)
  return Result.new(success: false, error: 'Web Unblocker not configured') unless configured?

  # Parse/validate first so a malformed URL never consumes daily budget.
  uri = target_uri(url)

  # Shares the daily scrape-spend ceiling with the Scraper API, and bills the
  # same way it does (Retailer::OxylabsApi#submit_job): a pre-flight ceiling
  # check, then record the spend only once the request actually returns
  # something. Incrementing up front charged us for every rejection — with the
  # unblocker credential dead (a 401 on every call, found 2026-08-08) that
  # silently drained the daily budget on Wayfair's fallback path without ever
  # fetching a page. A runaway is still bounded, because the ceiling check
  # counts the spend already recorded.
  Retailer::DailyCostGuard.ensure_within_budget!

  response = http(uri).request(build_request(uri, geo_location))
  body = response.body.to_s
  status = response.code.to_i
  # A rejected credential fetched nothing, so it costs nothing. Anything else
  # — including a 550 faulted target — means the unblocker did go and try.
  Retailer::DailyCostGuard.record! unless auth_rejection?(status)

  if status == 200 && body.present?
    Result.new(success: true, html: body, status: 200)
  else
    report_credential_failure("Web Unblocker returned #{status}") if auth_rejection?(status)
    Result.new(success: false, status: status, error: "Web Unblocker returned #{response.code}")
  end
rescue Retailer::DailyCostGuard::BudgetExceeded => e
  Result.new(success: false, error: e.message)
rescue StandardError => e
  @logger.error "[WebUnblocker] #{e.class}: #{e.message}"
  report_credential_failure(e.message) if auth_rejection_message?(e.message)
  Result.new(success: false, error: "#{e.class}: #{e.message}")
end