Class: Retailer::Extractors::Base

Inherits:
Object
  • Object
show all
Includes:
CatalogConstants
Defined in:
app/services/retailer/extractors/base.rb

Overview

Base class for retailer data extractors.
Uses Nokogiri for HTML parsing as recommended by Oxylabs:
https://github.com/oxylabs/webscraping-with-ruby

Examples:

Subclass implementation

class Retailer::Extractors::Amazon < Retailer::Extractors::Base
  def extract(check, content)
    # Amazon-specific extraction logic
  end
end

Constant Summary collapse

RENDER_REQUIRED =

Whether this retailer requires JavaScript rendering for price extraction.
Override in subclasses to opt out (or override the constant value to false).

render: 'html' is roughly 5x more expensive at Oxylabs than non-rendered
requests. Most extractors currently set true to preserve historical
behavior; flipping to false per-retailer should be done one at a time
alongside a manual probe to confirm the page still parses.

Returns:

  • (Boolean) 
true
WEB_UNBLOCKER_FALLBACK =

Whether a failed probe (the page never yielded content at the Scraper
API) retries once through the Oxylabs Web Unblocker before being
recorded. Opt-in per retailer — Wayfair, whose PDPs chronically 613, is
the first. See Retailer::WebhookResultProcessor#probe_via_unblocker.

Returns:

  • (Boolean) 
false
MIN_NORMALIZED_IDENTITY_LENGTH =

Shortest identifier we will match with separators stripped. Punctuation-free
comparison is what lets "TRT120-KIT-OT-3.0x08" match Home Depot's
"TRT120OT-3.0x08", but on a short token it would match almost any page, so
anything below this length is only ever compared verbatim.

6

Constants included from CatalogConstants

CatalogConstants::ALL_MAIN_CATALOG_IDS, CatalogConstants::AMAZON_CATALOG_IDS, CatalogConstants::AMAZON_CA_CATALOG_IDS, CatalogConstants::AMAZON_EU_CATALOG_IDS, CatalogConstants::AMAZON_NA_SELLER_IDS, CatalogConstants::AMAZON_SC_BE_CATALOG_ID, CatalogConstants::AMAZON_SC_CATALOG_IDS, CatalogConstants::AMAZON_SC_CA_CATALOG_ID, CatalogConstants::AMAZON_SC_DE_CATALOG_ID, CatalogConstants::AMAZON_SC_ES_CATALOG_ID, CatalogConstants::AMAZON_SC_FR_CATALOG_ID, CatalogConstants::AMAZON_SC_IT_CATALOG_ID, CatalogConstants::AMAZON_SC_NL_CATALOG_ID, CatalogConstants::AMAZON_SC_PL_CATALOG_ID, CatalogConstants::AMAZON_SC_SE_CATALOG_ID, CatalogConstants::AMAZON_SC_UK_CATALOG_ID, CatalogConstants::AMAZON_SC_US_CATALOG_ID, CatalogConstants::AMAZON_SELLER_IDS, CatalogConstants::AMAZON_US_CATALOG_IDS, CatalogConstants::AMAZON_VC_CATALOG_IDS, CatalogConstants::AMAZON_VC_CA_CATALOG_ID, CatalogConstants::AMAZON_VC_CA_CATALOG_IDS, CatalogConstants::AMAZON_VC_DIRECT_FULFILLMENT_CATALOG_IDS, CatalogConstants::AMAZON_VC_US_CATALOG_IDS, CatalogConstants::AMAZON_VC_US_WASN4_CATALOG_ID, CatalogConstants::AMAZON_VC_US_WAX7V_CATALOG_ID, CatalogConstants::AMAZON_VC_WAT0F_CA_CATALOG_ID, CatalogConstants::AMAZON_VC_WAT4D_CA_CATALOG_ID, CatalogConstants::AMAZON_VENDOR_CODE_TO_CATALOG_ID, CatalogConstants::BESTBUY_CANADA, CatalogConstants::BUILD_COM, CatalogConstants::CANADIAN_TIRE, CatalogConstants::CA_CATALOG_ID, CatalogConstants::COSTCO_CANADA, CatalogConstants::COSTCO_CATALOGS, CatalogConstants::COSTCO_USA, CatalogConstants::EU_CATALOG_ID, CatalogConstants::HOME_DEPOT_CANADA, CatalogConstants::HOME_DEPOT_CATALOGS, CatalogConstants::HOME_DEPOT_USA, CatalogConstants::HOUZZ, CatalogConstants::LOCALE_TO_CATALOG, CatalogConstants::LOWES_CANADA, CatalogConstants::LOWES_USA, CatalogConstants::RONA_CANADA, CatalogConstants::US_CATALOG_ID, CatalogConstants::WALMART_CATALOGS, CatalogConstants::WALMART_SELLER_CANADA, CatalogConstants::WALMART_SELLER_USA, CatalogConstants::WAYFAIR_CANADA, CatalogConstants::WAYFAIR_CATALOGS, CatalogConstants::WAYFAIR_GERMANY, CatalogConstants::WAYFAIR_USA

Instance Attribute Summary collapse

Class Method Summary collapse

Instance Method Summary collapse

Methods included from CatalogConstants

amazon_catalog?, amazon_seller_catalog?, costco_catalog?, home_depot_catalog?, walmart_catalog?, wayfair_catalog?

Constructor Details

#initialize(catalog) ⇒ Base

Returns a new instance of Base.



49
50
51
 
# File 'app/services/retailer/extractors/base.rb', line 49

def initialize(catalog)
  @catalog = catalog
end

Instance Attribute Details

#catalog ⇒ Object (readonly)

Returns the value of attribute catalog.



47
48
49
 
# File 'app/services/retailer/extractors/base.rb', line 47

def catalog
  @catalog
end

#discovered_url ⇒ String? (readonly)

Returns a discovered direct URL if found during extraction.
Used to capture and store canonical URLs for future direct access.
Override in subclasses that can discover URLs (e.g., from search results).

Returns:

  • (String, nil) 


71
72
73
 
# File 'app/services/retailer/extractors/base.rb', line 71

def discovered_url
  @discovered_url
end

Class Method Details

.render_value ⇒ String?

Returns the Oxylabs render payload value for this extractor.

Returns:



43
44
45
 
# File 'app/services/retailer/extractors/base.rb', line 43

def self.render_value
  self::RENDER_REQUIRED ? 'html' : nil
end

Instance Method Details

#catalog_base_url ⇒ String? (protected)

Get base URL for this catalog's retailer
Override in subclasses for specific domains

Returns:

  • (String, nil) 


131
132
133
 
# File 'app/services/retailer/extractors/base.rb', line 131

def catalog_base_url
  nil
end

#check_availability(html, unavailable_phrases = []) ⇒ Boolean (protected)

Check if page indicates product is available

Parameters:

  • html (String) —

    HTML content

  • unavailable_phrases (Array<String>) (defaults to: []) —

    Phrases indicating unavailability

Returns:

  • (Boolean) 


222
223
224
225
226
227
 
# File 'app/services/retailer/extractors/base.rb', line 222

def check_availability(html, unavailable_phrases = [])
  default_phrases = ['Out of Stock', 'Sold Out', 'Currently Unavailable', 'Not Available']
  phrases = unavailable_phrases + default_phrases

  phrases.none? { |phrase| html.include?(phrase) }
end

#collect_product_identifiers(catalog_item) ⇒ Array<String>

Collect all product identifiers that can be used to validate the page

Parameters:

Returns:

  • (Array<String>) —

    List of identifiers (SKU, UPC, third party number, etc.)



267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
 
# File 'app/services/retailer/extractors/base.rb', line 267

def collect_product_identifiers(catalog_item)
  identifiers = []

  # Our internal SKU
  identifiers << catalog_item.sku

  # UPC from the parent item
  identifiers << catalog_item.store_item&.item&.upc

  # Third party number (retailer's part number)
  identifiers << catalog_item.third_party_part_number

  # Third party SKU (retailer-assigned / our marketplace SKU)
  identifiers << catalog_item.third_party_sku

  # Variant selector — the Wayfair piid that pins the exact size on a shared PDP
  identifiers << catalog_item.third_party_sku_variant_id

  # Parent SKU (e.g., WRM1245 for Wayfair variants)
  # This is used in search URLs and should appear on the page
  identifiers << catalog_item.parent_sku

  identifiers.compact.compact_blank.uniq
end

#content_declares?(content, identifier) ⇒ Boolean

Whether the PAGE itself claims one of our identifiers.

Deliberately never looks at check.url. The URL is a string we supplied, and
most retailers put our SKU in their canonical slug — Home Depot's
/p/WarmlyYours-Elements-4-Bar-...-TW-E4PCP/203917548, rona's
...-tws5-ibz06kh-332109624 — so matching against it made this check
tautological: it could not fail, and a wrong-product landing (a retired id
re-pointed, a redirect) would have had its price accepted. That is the same
failure PR #1480 removed from the search fallback, except it sat in the
shared base class and so silently covered every extractor.

What anchors it instead is third_party_part_number — the RETAILER's own id
for the product, which their page necessarily carries. Verified across both
Home Depot storefronts: present in content for 8 of 8 sampled items, while
our own SKU was absent from 3 of 4 US pages because Home Depot's listing
requirements forced SKU adjustments on our side.

Parameters:

  • content (String, Hash) —

    page body

  • identifier (String, nil)

Returns:

  • (Boolean) 


312
313
314
315
 
# File 'app/services/retailer/extractors/base.rb', line 312

def content_declares?(content, identifier)
  pattern = identity_pattern(identifier)
  pattern.present? && content.to_s.match?(pattern)
end

#extract(check, content) ⇒ void

This method returns an undefined value.

Extract data from content and populate the check record

Parameters:

Raises:

  • (NotImplementedError) 


57
58
59
 
# File 'app/services/retailer/extractors/base.rb', line 57

def extract(check, content)
  raise NotImplementedError, 'Subclasses must implement #extract'
end

#extract_canonical_url(doc) ⇒ String? (protected)

Extract canonical URL from page head or og:url meta tag

Parameters:

  • doc (Nokogiri::HTML::Document) —

    Parsed HTML document

Returns:

  • (String, nil) 


78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
 
# File 'app/services/retailer/extractors/base.rb', line 78

def extract_canonical_url(doc)
  # Try rel="canonical" first (most reliable)
  canonical_el = doc.at_css('link[rel="canonical"]')
  if canonical_el
    url = canonical_el['href']
    return url if url.present? && url.start_with?('http')
  end

  # Try og:url meta tag
  og_url = doc.at_css('meta[property="og:url"]')
  if og_url
    url = og_url['content']
    return url if url.present? && url.start_with?('http')
  end

  nil
end

#extract_json_ld_price(check, doc) ⇒ Object (protected)

Extract price from JSON-LD structured data (schema.org)
Most reliable method across retailers

Parameters:

  • check (CatalogItemRetailerProbe) —

    The probe record to update

  • doc (Nokogiri::HTML::Document) —

    Parsed HTML document



153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
 
# File 'app/services/retailer/extractors/base.rb', line 153

def extract_json_ld_price(check, doc)
  doc.css('script[type="application/ld+json"]').each do |script|
    data = JSON.parse(script.text)

    # Handle @graph structure
    data = data['@graph'].find { |item| item['offers'] } || data if data['@graph'].is_a?(Array)

    offers = data['offers']
    next unless offers

    # Handle array of offers
    offers = offers.first if offers.is_a?(Array)

    price = offers['price']
    check.price = extract_numeric_price(price) if price

    # Also try to get regular/high price
    high_price = offers['highPrice'] || data['highPrice']
    if high_price && check.price.present?
      regular = extract_numeric_price(high_price)
      check.regular_price = regular if regular && regular > check.price
    end

    break if check.price.present?
  rescue JSON::ParserError
    next
  end
end

#extract_numeric_price(text) ⇒ Float? (protected)

Extract numeric price from text

Parameters:

  • text (String, Numeric) —

    Price text or number

Returns:

  • (Float, nil) 


185
186
187
188
189
190
191
192
 
# File 'app/services/retailer/extractors/base.rb', line 185

def extract_numeric_price(text)
  price_val = if text.is_a?(Numeric)
                text.to_f
              else
                text.to_s.delete('^0-9.').to_f
              end
  price_val if valid_price?(price_val)
end

#extract_price_from_selectors(check, doc, selectors) ⇒ Object (protected)

Extract price from common CSS selectors

Parameters:

  • check (CatalogItemRetailerProbe) —

    The probe record to update

  • doc (Nokogiri::HTML::Document) —

    Parsed HTML document

  • selectors (Array<String>) —

    CSS selectors to try



205
206
207
208
209
210
211
212
213
214
215
216
 
# File 'app/services/retailer/extractors/base.rb', line 205

def extract_price_from_selectors(check, doc, selectors)
  selectors.each do |selector|
    el = doc.at_css(selector)
    next unless el

    price_val = extract_numeric_price(el['content'] || el.text)
    if valid_price?(price_val)
      check.price = price_val
      break
    end
  end
end

#extract_product_link_from_search(doc, selectors) ⇒ String? (protected)

Extract product link from search results

Parameters:

  • doc (Nokogiri::HTML::Document) —

    Parsed HTML document

  • selectors (Array<String>) —

    CSS selectors for product links

Returns:

  • (String, nil) 


100
101
102
103
104
105
106
107
108
109
110
111
112
 
# File 'app/services/retailer/extractors/base.rb', line 100

def extract_product_link_from_search(doc, selectors)
  selectors.each do |selector|
    link = doc.at_css(selector)
    next unless link

    href = link['href']
    next if href.blank?

    # Make absolute URL if relative
    return make_absolute_url(href) if href.present?
  end
  nil
end

#extract_title(doc) ⇒ String? (protected)

Extract title from common selectors

Parameters:

  • doc (Nokogiri::HTML::Document) —

    Parsed HTML document

Returns:

  • (String, nil) 


232
233
234
235
 
# File 'app/services/retailer/extractors/base.rb', line 232

def extract_title(doc)
  title_el = doc.at_css('h1') || doc.at_css('[data-testid="product-title"]')
  title_el&.text&.strip&.truncate(255)
end

#identity_pattern(identifier) ⇒ Regexp?

An identifier as a regex that tolerates separator differences but is fenced
at both ends, so a part number can never validate a DIFFERENT product that
merely starts the same way — "SS-01" must not match a page selling "SS-01X".

Retailers reformat part numbers (Home Depot lists TRT120-1-5x24 for our
TRT120-1.5x24), so the alphanumeric runs are joined by "any separators".
That tolerance is only granted to identifiers long enough to stay specific;
a short one is matched verbatim.

A NUMERIC identifier may be preceded by zero-PADDING, because barcodes
arrive padded and our UPC 881308082307 has to keep matching a page's GTIN-14
00881308082307. The padding is fenced too ((?<![a-z0-9])0*) rather than
just allowing one leading zero: a bare "not 1-9" lookbehind would let
123456 match the tail of A0123456, since it only inspects a single
character. Requiring a real token boundary before the zeros accepts
"SKU-00123456" and rejects embedded runs.

Everything else gets the strict fence: zero-padding is meaningless for an
alphanumeric part number, so allowing it there would only widen the match.

Parameters:

  • identifier (String, nil)

Returns:

  • (Regexp, nil) 


339
340
341
342
343
344
345
346
347
348
349
350
 
# File 'app/services/retailer/extractors/base.rb', line 339

def identity_pattern(identifier)
  runs = identifier.to_s.downcase.scan(/[a-z0-9]+/)
  return nil if runs.empty?

  core = if runs.join.length >= MIN_NORMALIZED_IDENTITY_LENGTH
           runs.map { |run| Regexp.escape(run) }.join('[^a-z0-9]*')
         else
           Regexp.escape(identifier.to_s.downcase.strip)
         end
  lead = runs.join.match?(/\A\d+\z/) ? '(?<![a-z0-9])0*' : '(?<![a-z0-9])'
  /#{lead}#{core}(?![a-z0-9])/i
end

#make_absolute_url(href) ⇒ String (protected)

Convert relative URL to absolute

Parameters:

  • href (String) —

    Relative or absolute URL

Returns:

  • (String) 


117
118
119
120
121
122
123
124
125
126
 
# File 'app/services/retailer/extractors/base.rb', line 117

def make_absolute_url(href)
  return href if href.start_with?('http')

  base_url = catalog_base_url
  return href unless base_url

  URI.join(base_url, href).to_s
rescue URI::InvalidURIError
  href
end

#parse_html(html) ⇒ Nokogiri::HTML::Document (protected)

Parse HTML content with Nokogiri

Parameters:

  • html (String) —

    HTML content

Returns:

  • (Nokogiri::HTML::Document) 


138
139
140
 
# File 'app/services/retailer/extractors/base.rb', line 138

def parse_html(html)
  Nokogiri::HTML(html)
end

#source_name ⇒ String

Identifier for this extractor (used in check.scraper_source)

Returns:

  • (String) 


63
64
65
 
# File 'app/services/retailer/extractors/base.rb', line 63

def source_name
  self.class.name.demodulize.underscore
end

#valid_html?(content) ⇒ Boolean (protected)

Validate that content is HTML string

Parameters:

  • content (Object)

Returns:

  • (Boolean) 


145
146
147
 
# File 'app/services/retailer/extractors/base.rb', line 145

def valid_html?(content)
  content.is_a?(String) && content.present?
end

#valid_price?(price) ⇒ Boolean (protected)

Validate that a price is reasonable

Parameters:

  • price (Float) —

    Price value

Returns:

  • (Boolean) 


197
198
199
 
# File 'app/services/retailer/extractors/base.rb', line 197

def valid_price?(price)
  price.present? && price > 1 && price < 100_000
end

#validate_product_identity(check, content, catalog_item) ⇒ Boolean

Validate that the scraped page actually contains our product identifiers.
This prevents false positives where a retailer redirects to a different product.

Parameters:

  • check (CatalogItemRetailerProbe) —

    The probe record to update

  • content (String) —

    HTML content (or URL for URL-based validation)

  • catalog_item (CatalogItem) —

    The catalog item being checked

Returns:

  • (Boolean) —

    true if validation passed, false if product mismatch detected



249
250
251
252
253
254
255
256
257
258
259
260
261
262
 
# File 'app/services/retailer/extractors/base.rb', line 249

def validate_product_identity(check, content, catalog_item)
  identifiers = collect_product_identifiers(catalog_item)
  return true if identifiers.empty? # Skip validation if no identifiers available

  found = identifiers.any? { |identifier| content_declares?(content, identifier) }

  unless found
    check.status = 'product_mismatch'
    check.error_message = "Product identity validation failed: none of our identifiers (#{identifiers.compact.join(', ')}) found on page"
    Rails.logger.warn "[#{source_name}] Product mismatch for catalog_item #{catalog_item.id}: #{check.error_message}"
  end

  found
end