Class: Retailer::Extractors::Wayfair

Inherits:
Base
  • Object
show all
Defined in:
app/services/retailer/extractors/wayfair.rb

Overview

Wayfair data extractor.
Uses data-test-id attributes for reliable price extraction.

Wayfair Variant Handling:
When searching by internal SKU (e.g., TCT240-3.7W-749-FS), Wayfair redirects
to the parent product page with URL params like ?redir=SKU&piid=123,456.
The page initially shows the LOWEST variant price, then JavaScript updates
the selection based on URL parameters. We use browser_instructions to wait
for the variant selection to complete before extracting the price.

Constant Summary collapse

RENDER_REQUIRED =

Wayfair pricing is JS-driven (variant selection via URL params runs after
initial page load). browser_instructions below also assume rendering, so
this MUST stay true.

true
FAULT_RETRY_LIMIT =

Wayfair jobs fault at Oxylabs far more than any other retailer (~40% of
submissions some days), so the default 2/day fault-retry budget still lets
transient faults through to recorded failures. Give Wayfair more attempts
before WebhookProcessors::OxylabsProcessor records a failed probe.

5
US_GEO_LOCATION_POOL =

Pool of US states for geo_location rotation. Wayfair faults chronically at
Oxylabs, so US egress is diversified across states per probe instead of
hammering every PDP from a single geography. Lives on
Retailer::WebUnblockerApi (shared with the Web Unblocker transport);
aliased here for existing callers/tests.

Retailer::WebUnblockerApi::US_GEO_LOCATION_POOL
WEB_UNBLOCKER_FALLBACK =

Wayfair PDPs chronically fault at the Scraper API (613 on every payload
variant — verified live 2026-07-27), so a no-content probe retries once
through the Web Unblocker before being recorded as failed (see
Retailer::WebhookResultProcessor#probe_via_unblocker).

true

Class Method Summary collapse

Instance Method Summary collapse

Class Method Details

.browser_instructions ⇒ Array<Hash>

Browser instructions to wait for Wayfair's variant selection to complete.
Wayfair uses JavaScript to update pricing based on URL params (redir, piid).
We wait for the price element to stabilize after redirect/variant selection.
Reference: https://github.com/oxylabs/how-to-scrape-wayfair

Returns:

  • (Array<Hash>) —

    Oxylabs browser instructions



92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
 
# File 'app/services/retailer/extractors/wayfair.rb', line 92

def self.browser_instructions
  [
    # Wait for initial page load and price element to appear
    # Primary selector from data-test-id (most reliable)
    {
      type: 'wait_for_element',
      selector: {
        type: 'css',
        value: '[data-test-id="PriceDisplay"]'
      },
      timeout_s: 10
    },
    # Additional wait for variant selection JavaScript to complete
    # Wayfair's redirect/variant selection takes ~2-5 seconds
    { type: 'wait', wait_time_s: 5 }
  ]
end

.build_payload(url:, geo_location: nil) ⇒ Hash

Build Oxylabs payload for Wayfair product scraping.
Uses the dedicated 'wayfair' source with JS rendering and
browser_instructions to wait for variant-specific pricing to load.
browser_instructions are REQUIRED — without them the dedicated source
returns empty results on variant (piid) PDPs, and with them the extracted
price matches the selected variant exactly (verified live, 2026-07-27).
Reference: https://github.com/oxylabs/how-to-scrape-wayfair

Parameters:

  • url (String) —

    Full product URL

  • geo_location (String, nil) (defaults to: nil) —

    Storefront country; US is rotated per probe

Returns:

  • (Hash) —

    Oxylabs API payload



48
49
50
51
52
53
54
55
56
57
 
# File 'app/services/retailer/extractors/wayfair.rb', line 48

def self.build_payload(url:, geo_location: nil)
  {
    source: 'wayfair',
    url: url,
    render: render_value,
    user_agent_type: 'desktop_safari',
    geo_location: diversify_geo_location(geo_location),
    browser_instructions: browser_instructions
  }.compact
end

.diversify_geo_location(geo_location) ⇒ String

Rotate US egress across the state pool; non-US storefronts (Canada,
Germany) keep their explicit country so the right storefront pricing is
served.

Parameters:

  • geo_location (String, nil)

Returns:

  • (String) 


80
81
82
83
84
 
# File 'app/services/retailer/extractors/wayfair.rb', line 80

def self.diversify_geo_location(geo_location)
  return US_GEO_LOCATION_POOL.sample if geo_location.blank? || geo_location == 'United States'

  geo_location
end

.storefront_geo(catalog) ⇒ String

Storefront country for a Wayfair catalog's geo targeting (Wayfair prices
per country, not by postal code). Shared by the batch payload and the
Web Unblocker probe fallback, which must not scrape wayfair.ca via US
egress.

Parameters:

Returns:

  • (String) 


66
67
68
69
70
71
72
 
# File 'app/services/retailer/extractors/wayfair.rb', line 66

def self.storefront_geo(catalog)
  case catalog.country_iso3
  when 'CAN' then 'Canada'
  when 'DEU' then 'Germany'
  else 'United States'
  end
end

Instance Method Details

#extract(check, content) ⇒ Object 



110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
 
# File 'app/services/retailer/extractors/wayfair.rb', line 110

def extract(check, content)
  return unless valid_html?(content)

  check.scraper_source = source_name
  check.currency = catalog.id == WAYFAIR_CANADA ? 'CAD' : 'USD'

  doc = parse_html(content)

  # The H1 names the SELECTED variant, not just the parent product — e.g.
  # "TempZone Floor Heating Cable 120V, (3.7W/ft.), 30 ft., 1.0A …". Recording
  # it is free (the page is already fetched and parsed) and is the only signal
  # that tells us the `?piid=` resolved to the size we meant: every variant of
  # a parent shares one URL path, so #validate_product_identity's WRM check
  # passes even when the piid is wrong. Stored, not yet enforced — see the
  # note there.
  check.raw_title = doc.at_css('h1')&.text.to_s.squish.presence

  # Check availability
  check.product_available = doc.at_css('[data-test-id="AddToCartButton"]').present? ||
                            content.exclude?('Out of Stock')

  # IMPORTANT: Scope price extraction to the main product pricing section only.
  # Wayfair's "Compare Similar Items" carousel + sponsored ads reuse the same
  # PriceDisplay markup, so anything outside this container risks a wrong price.
  pricing_section = find_main_pricing_section(doc)
  if pricing_section
    # Sale price: data-test-id="StandardPricingPrice-SALE" (when on sale)
    # Primary price: data-test-id="StandardPricingPrice-PRIMARY" (otherwise)
    extract_current_price(check, pricing_section)

    # Original/was price: data-test-id="StandardPricingPrice-PREVIOUS"
    extract_previous_price(check, pricing_section)

    # Fallback: first PriceDisplay WITHIN the main pricing section
    extract_fallback_prices(check, pricing_section) if check.price.blank?
  end

  # Last fallback: JSON-LD schema.org offers (structured, main-product scoped)
  extract_json_ld_price(check, doc) if check.price.blank?
end

#validate_product_identity(check, content, catalog_item) ⇒ Boolean

Wayfair never shows our manufacturer SKU, so the base identity check (which
looks for our SKU/UPC on the page) always fails for it. The stored URL is the
Wayfair Catalog API's canonical PDP, so identity is confirmed by the page
carrying the URL's Wayfair SKU — which also catches a redirect to a different
product. Falls back to the base check when the URL has no Wayfair SKU.

Parameters:

Returns:

  • (Boolean) 


161
162
163
164
165
166
167
168
169
170
171
172
 
# File 'app/services/retailer/extractors/wayfair.rb', line 161

def validate_product_identity(check, content, catalog_item)
  wf_sku = wayfair_sku_from_url(check.url)
  return super if wf_sku.blank?
  # Match case-insensitively: the URL may carry a lowercase SKU (slugged
  # redirect target) while the page prints it upper-cased, or vice versa.
  return true if content.to_s.match?(/#{Regexp.escape(wf_sku)}/i)

  check.status = 'product_mismatch'
  check.error_message = "Wayfair SKU #{wf_sku} (from URL) not found on page"
  Rails.logger.warn "[#{source_name}] Wayfair mismatch for catalog_item #{catalog_item.id}: #{check.error_message}"
  false
end