Class: Retailer::DailyComplianceReport

Inherits:
Object
  • Object
show all
Defined in:
app/services/retailer/daily_compliance_report.rb

Overview

Per-retailer compliance digest emailed daily after the retailer probe runs
(see RetailerComplianceReportWorker + the cron in
config/sidekiq_production_schedule.yml).

Scope: catalogs the probe checks (external_price_check_enabled = true), the
Amazon Seller catalogs (US / Canada / Europe), and any catalog with an open
listing_issues row. One row per catalog, grouped by region (US / Canada /
Europe), each column a count of active catalog items in a problem state,
sourced from:

  • view_product_catalogs — map_violation, sale_price_in_effect,
    price_diverging, product_stock_status (our inventory)
  • listing_issues — open marketplace-reported listing problems
    (Amazon suppression / SP-API issues, Walmart unpublished reasons, …);
    this is the column that covers the (unprobed) Amazon catalogs.
  • the latest CatalogItemRetailerProbe per item (last 14 days) — status:
    'not_found'/'product_mismatch' (scrape failures), 'failed' (unreachable
    online), product_available = FALSE (out of stock online). Amazon
    catalogs are not Oxylabs-probed, so the probe columns read 0 for them.

MAP-violation counts depend on view_product_catalogs resolving MSRP from each
catalog's tree root (fixed in view v54) — before that, all CA/EU catalogs
reported zero.

Constant Summary collapse

TZ =

America/Chicago — the server/report timezone.

'America/Chicago'
PROBE_LOOKBACK_DAYS =

Only consider probes from this lookback window as the item's current state.
Shared with CatalogItemRetailerProbe's latest-probe scopes (which back the
deep links below) so the counts and the linked lists use one window.

CatalogItemRetailerProbe::CURRENT_STATE_LOOKBACK_DAYS
REGIONS =

Root catalog id => region label / display order. Roots: US=1, CA=2, EU=125.

{
  1   => { label: 'United States', order: 0 },
  2   => { label: 'Canada',        order: 1 },
  125 => { label: 'Europe',        order: 2 }
}.freeze
OTHER_REGION =

Fallback bucket for any catalog not rooted at a known region.

{ label: 'Other', order: 99 }.freeze
COUNT_COLUMNS =

Numeric metric columns, in display order. Two distinct "things are wrong
with the listing" signals are kept separate on purpose:

  • listing_issues — the marketplace reports a problem with our listing
    (Amazon suppression / SP-API issues, Walmart unpublished reasons, …),
    sourced from the listing_issues table (ListingIssues::Sync). Also where
    items we've stopped probing land: not_confirmable (human opt-out) and
    probe_stale (no probe in a week).
  • scrape_failures — our Oxylabs probe loaded the page but couldn't read
    a valid product/price (not_found / product_mismatch). A scraper-health
    metric, not a retailer listing defect.

The three probe columns count ACTIVELY PROBED items only (see the
latest_probe CTE). An opted-out item has left the daily run, so its last
failure is frozen history, not a current measurement — counting it made
Ferguson read "236 Scrape Failures" while zero of its live items were
failing, double-reporting the same 236 items already sitting in
listing_issues as not_confirmable. Keeping the columns disjoint is what
makes them add up: abandoned/never-probed → listing_issues, currently
failing → scrape_failures.

%i[
  active_public
  map_violations
  promotions
  price_diverging
  out_of_stock_active
  listing_issues
  scrape_failures
  unreachable_online
  out_of_stock_online
].freeze
SNAPSHOT_COLUMNS =

The metrics persisted daily to catalog_data_points (see
RetailerComplianceReportWorker) = COUNT_COLUMNS plus problem_items, the
count of distinct active items in ANY problem state, which backs the
/retailers health gauge. problem_items is deliberately OUTSIDE
COUNT_COLUMNS so the email table, totals, and deep links stay unchanged;
it rides along in each row hash purely for the snapshot + dashboard.

(COUNT_COLUMNS + %i[problem_items]).freeze
CLICKABLE_QUERY_PARAMS =

Every metric here maps to a ProductCatalogSearch query so its email count
links straight to the filtered CRM list. View-derived columns use Ransack
attribute predicates; the probe-derived columns (scrape failures /
unreachable / out of stock online) use the latest-probe Ransack scopes on
ViewProductCatalog, which reuse this report's lookback window so the linked
list reproduces the count. Each lambda takes a catalog id and returns the
ProductCatalogSearch query_params. (listing_issues links to the dedicated
dashboard instead — see .listing_issues_url.)

{
  active_public: ->(cid) { { catalog_id_in: [cid], catalog_item_state_in: ['active'] } },
  map_violations: lambda { |cid|
    { catalog_id_in: [cid], catalog_item_state_in: ['active'], map_violation_eq: true }
  },
  promotions: lambda { |cid|
    { catalog_id_in: [cid], catalog_item_state_in: ['active'], sale_price_in_effect_eq: true }
  },
  price_diverging: lambda { |cid|
    { catalog_id_in: [cid], catalog_item_state_in: ['active'], price_diverging_eq: true }
  },
  out_of_stock_active: lambda { |cid|
    { catalog_id_in: [cid], catalog_item_state_in: ['active'], product_stock_status_eq: 'OutOfStock' }
  },
  # listing_issues links to the CRM Listing Issues dashboard (review +
  # resolve), not a ProductCatalogSearch — handled in the email view, so it
  # is intentionally absent here.
  scrape_failures: lambda { |cid|
    { catalog_id_in: [cid], catalog_item_state_in: ['active'], latest_probe_scrape_failure: true }
  },
  unreachable_online: lambda { |cid|
    { catalog_id_in: [cid], catalog_item_state_in: ['active'], latest_probe_unreachable_online: true }
  },
  out_of_stock_online: lambda { |cid|
    { catalog_id_in: [cid], catalog_item_state_in: ['active'], latest_probe_out_of_stock_online: true }
  }
}.freeze
COLUMN_BACKED_ISSUE_CODES =

Probe-derived issue codes that already have a column of their own, so
counting them under Listing Issues too reports one problem twice.

Both exist only for items we are actively probing — since 2026-08-08
ListingIssues::RetailerProbeAdapter hands every opted-out item to
not_confirmable instead — so the Scrape Failures and Out of Stock Online
columns represent them completely and excluding them here loses nothing.
not_confirmable and probe_stale have no column of their own and stay
counted, which is why Ferguson keeps all 239 of its issues.

The exclusion is additionally gated on the item being actively probed,
rather than on the code alone. Rows written before that adapter change
still carry not_buyable on opted-out items, and a code-only rule drops
those from the tile the moment this deploys while the columns — which also
exclude skipped items — don't pick them up either: Ferguson silently went
239 → 3 until the next Sync re-coded the rows. Gating on skip_url_checks
makes the count right before AND after that backfill, so deploy order
stops mattering.

The same list rides along on the Listing Issues deep link as
exclude_codes, and the controller applies the identical skip gate, so the
tile and the list it opens still agree.

%w[not_buyable out_of_stock_mismatch].freeze

Instance Attribute Summary collapse

Class Method Summary collapse

Instance Method Summary collapse

Constructor Details

#initialize(generated_at: nil, only_catalog_id: nil, scope: :compliance, rows: nil) ⇒ DailyComplianceReport

Returns a new instance of DailyComplianceReport.

Parameters:

  • generated_at (ActiveSupport::TimeWithZone, nil) (defaults to: nil)
  • only_catalog_id (Integer, nil) (defaults to: nil) —

    when set, restricts the report to a
    single catalog (the per-retailer dashboard) instead of the full scope

  • scope (Symbol) (defaults to: :compliance) —

    which catalogs to cover:

    • :compliance (default, the email) — only catalogs carrying a compliance
      signal: externally price-checked OR an Amazon Seller catalog OR with an
      open listing issue.
    • :dashboard (the dashboard index) — catalogs flagged
      display_in_retailer_dashboard (the integrated retailers we manage,
      seeded from that discovery and editable per-catalog in the catalog
      form). Surfaces EDI/Mirakl marketplaces the email skips.
  • rows (Array<Hash>, nil) (defaults to: nil) —

    pre-decorated rows to reuse instead of
    running the SQL — lets the index serve a short-TTL cached result.
    Symbol-keyed defensively so a JSON cache coder round-trip is safe.



170
171
172
173
174
175
 
# File 'app/services/retailer/daily_compliance_report.rb', line 170

def initialize(generated_at: nil, only_catalog_id: nil, scope: :compliance, rows: nil)
  @generated_at = generated_at || ActiveSupport::TimeZone[TZ].now
  @only_catalog_id = only_catalog_id&.to_i
  @scope = scope
  @rows = rows&.map(&:symbolize_keys)
end

Instance Attribute Details

#generated_at ⇒ ActiveSupport::TimeWithZone (readonly)

Returns when the report was generated.

Returns:

  • (ActiveSupport::TimeWithZone) —

    when the report was generated



154
155
156
 
# File 'app/services/retailer/daily_compliance_report.rb', line 154

def generated_at
  @generated_at
end

Class Method Details

.for_catalog(catalog_id) ⇒ Retailer::DailyComplianceReport

Report scoped to one catalog, for the /retailers/:id dashboard.

Parameters:

  • catalog_id (Integer)

Returns:



192
193
194
 
# File 'app/services/retailer/daily_compliance_report.rb', line 192

def self.for_catalog(catalog_id)
  new(only_catalog_id: catalog_id)
end

.for_dashboard ⇒ Retailer::DailyComplianceReport

Every retailer flagged for the /retailers dashboard index.

Returns:



185
186
187
 
# File 'app/services/retailer/daily_compliance_report.rb', line 185

def self.for_dashboard
  new(scope: :dashboard)
end

.for_scheduled_run ⇒ Retailer::DailyComplianceReport

Entry point for the scheduled worker (the daily email).

Returns:



179
180
181
 
# File 'app/services/retailer/daily_compliance_report.rb', line 179

def self.for_scheduled_run
  new
end

.listing_issues_url(catalog_id, host) ⇒ String

CRM Listing Issues dashboard URL for a catalog's open issues — the
"Listing Issues" column links here (review + mark-fixed) rather than to a
ProductCatalogSearch list.

Parameters:

  • catalog_id (Integer)
  • host (String) —

    CRM host

Returns:

  • (String) 


148
149
150
151
 
# File 'app/services/retailer/daily_compliance_report.rb', line 148

def self.listing_issues_url(catalog_id, host)
  query = { catalog_id:, status: 'open', exclude_codes: COLUMN_BACKED_ISSUE_CODES }.to_query
  "https://#{host}/listing_issues?#{query}"
end

.query_params_for(metric, catalog_id) ⇒ Hash?

ProductCatalogSearch query_params for a clickable metric, or nil.

Parameters:

  • metric (Symbol)
  • catalog_id (Integer)

Returns:

  • (Hash, nil) 


200
201
202
 
# File 'app/services/retailer/daily_compliance_report.rb', line 200

def self.query_params_for(metric, catalog_id)
  CLICKABLE_QUERY_PARAMS[metric]&.call(catalog_id)
end

Instance Method Details

#empty? ⇒ Boolean

(Explicit method rather than delegate :empty?, to: :rows — YARD's DSL
handler crashes on predicate delegates during the docs build.)

Returns:

  • (Boolean) —

    true when no catalogs were found at all



248
 
# File 'app/services/retailer/daily_compliance_report.rb', line 248

def empty? = rows.empty? # rubocop:disable Rails/Delegate -- YARD's DSL handler crashes on predicate delegates

#flagged_rows ⇒ Array<Hash>

Retailers carrying at least one MAP violation or marketplace listing issue
— the rows worth eyeballing first.

Returns:

  • (Array<Hash>) 


240
241
242
 
# File 'app/services/retailer/daily_compliance_report.rb', line 240

def flagged_rows
  rows.select { |r| r[:map_violations].to_i.positive? || r[:listing_issues].to_i.positive? }
end

#rows ⇒ Array<Hash>

One hash per probed/Amazon catalog with its compliance counts and region.

Returns:

  • (Array<Hash>) 


206
207
208
209
210
211
212
213
 
# File 'app/services/retailer/daily_compliance_report.rb', line 206

def rows
  @rows ||= ActiveRecord::Base
            .connection
            .select_all(report_sql)
            .to_a
            .map(&:symbolize_keys)
            .map { |r| decorate(r) }
end

#rows_by_region ⇒ Array<Array(String, Array<Hash>)>

Rows grouped by region label, in display order (US, Canada, Europe, …).

Returns:

  • (Array<Array(String, Array<Hash>)>) —

    [region_label, rows] pairs



217
218
219
220
 
# File 'app/services/retailer/daily_compliance_report.rb', line 217

def rows_by_region
  rows.group_by { |r| r[:region_label] }
      .sort_by { |label, _| region_order_for_label(label) }
end

#totals ⇒ Hash

Column-wise totals across every retailer.

Returns:

  • (Hash) 


224
225
226
227
228
 
# File 'app/services/retailer/daily_compliance_report.rb', line 224

def totals
  @totals ||= COUNT_COLUMNS.index_with do |col|
    rows.sum { |r| r[col].to_i }
  end
end

#totals_for(region_rows) ⇒ Hash

Column-wise totals for one region's rows.

Parameters:

  • region_rows (Array<Hash>) —

    decorated report rows for one region

Returns:

  • (Hash) 


233
234
235
 
# File 'app/services/retailer/daily_compliance_report.rb', line 233

def totals_for(region_rows)
  COUNT_COLUMNS.index_with { |col| region_rows.sum { |r| r[col].to_i } }
end