Amazon SP-API Notifications (SQS + EventBridge)

Real-time push notifications from Amazon SP-API, delivered into the webhook
pipeline instead of waiting for cron pulls. Live today: feed completion,
report completion, and listing-issue changes. Staged next (subscriptions
managed as code, see below): orders, account health, the remaining
listings/product-type family, FBA inventory, and buy-box offer changes —
keep the pull backstops at full cadence until each phase actually lands.

Companion doc: Amazon SP-API Integration. Design history,
rollout phases, and the RES convergence plan:
doc/tasks/202607152125_AMAZON_SPAPI_SQS_NOTIFICATIONS.md.

Architecture

One SQS queue is the single consumer surface. SQS-workflow notification types
deliver straight to it; EventBridge-workflow types (the listings family,
product-type changes, branded content) arrive via a partner event bus whose
one rule targets the same queue.

SP-API ──(SQS destination)────────────────────────► SQS: heatwave-sp-api-notifications ──► DLQ (after 5 receives)
SP-API ──(EventBridge destination)──► partner bus ──► rule ──┘
                                                              │
                                     AmazonSqsNotificationPollerWorker (cron: every minute, ~50s long-poll drain)
                                                              │  normalize envelope + key casing
                                                              │  WebhookLog.ingest!  (dedup on notificationId)
                                                              │  delete SQS message AFTER commit
                                                              ▼
                                              webhook_logs (provider: amazon_sp_api)
                                                              │  WebhookProcessorWorker
                                                              ▼
                                         WebhookProcessors::AmazonSpApiProcessor
                                          ├─ feed_processing_finished   → EdiStatusFlowWorker (immediate feed-result sweep)
                                          ├─ listings_item_issues_change→ ListingIssueRefreshWorker (marketplace-scoped item)
                                          ├─ report_processing_finished → noted
                                          └─ anything else              → noted (observability first)

Why this shape (vs EventBridge-only, per-type queues, Shoryuken, or the
aws-activejob-sqs gem): Heatwave runs on bare metal, so SQS's pull model
needs no public endpoint and no AWS compute; notificationType is in every
payload so routing is one hash lookup; and WebhookLog already provides the
reliability layer (dedup, retry state machine, reprocess UI). The full
decision log is in the task doc.

Delivery semantics

At-least-once, unordered (standard queue; FIFO unsupported by SP-API).
Dedup happens at ingest on (provider, external_id) where external_id is
the SP-API notificationId (falling back to the EventBridge event id). A
redelivered message — including while the first row is still ready — is
dropped and its SQS copy deleted.
Delete only after commit: the poller deletes a message from SQS only
after WebhookLog.ingest! durably commits. Processing retries live in the
WebhookLog state machine (exponential backoff 5 min → 24 h, 5 attempts →
exception + AppSignal), not in SQS redelivery. A message that cannot
even be ingested (poison payload) stays on the queue and lands in the DLQ
after 5 receives.
Envelope + casing normalization at ingest: EventBridge deliveries are
unwrapped from detail, and PascalCase notification families (the listings
2023-12-13 payloads use NotificationType/Payload/Sku) are
canonicalized to camelCase, so webhook_logs.data and the processor only
ever see one shape.
Deploy-safe: the poller no-ops with a log line until the
amazon_notifications credentials profile exists.

Reading a row's origin

Every webhook_logs row with provider amazon_sp_api was written by
AmazonSqsNotificationPollerWorker — nothing else writes that provider. From
there, a row's origin decomposes into fields that are all already stored:

Question	Where the answer lives
Which delivery channel?	Derived from the type: the six EventBridge-workflow categories (`listings_item_*`, `product_type_definitions_change`, `branded_item_content_change`, `item_product_type_change`) rode partner bus → rule → queue; every other category was a direct SQS delivery. Amazon fixes the workflow per type, so this is unambiguous.
Which subscription produced it?	`data.notificationMetadata.subscriptionId` (compare with `amazon_notifications:status`).
Which SP-API application?	`data.notificationMetadata.applicationId`.
When did Amazon emit it?	`data.notificationMetadata.publishTime`.
Delivery latency?	`publishTime` → the row's `created_at` (ingest). Surfaced as Amazon → ingest on the monitor's Recent-notifications table; steady-state is seconds-to-~1 min (poller cadence).
Dedup identity?	`external_id` = `notificationId` (EventBridge event id as fallback).

The monitor page renders channel + latency per recent row (Amazon ids in the
log-link tooltip). For anything older, the stored provenance
(notificationMetadata, external_id) is on the row's Webhook Log page —
channel and latency are derived, not stored: re-derive channel from the
category (EventBridge-workflow list above) and latency from
publishTime → the row's created_at.

Components

Piece	Where
Poller (1-min cron drain)	`app/workers/amazon_sqs_notification_poller_worker.rb`
Processor (category → handler)	`app/services/webhook_processors/amazon_sp_api_processor.rb`
Landing table + dedup + retries	`WebhookLog` (`provider: 'amazon_sp_api'`) — see Webhook Log System
Subscription management	`lib/tasks/amazon_notifications.rake` (`status` / `reconcile` / `unsubscribe[TYPE]`)
AWS infra provisioning	`script/aws_spapi_notifications_setup.sh` (queue + DLQ + policy, idempotent, `--verify`)
Credentials	`Heatwave::Configuration.fetch(:amazon_notifications)` → `queue_url`, `region`, `aws_access_key_id`, `aws_secret_access_key` (minimal IAM user: receive/delete/get-attributes on the one queue). Nested under `production:` in credentials so dev/ad-hoc runs never drain the live queue; the poller also guards on `queue_url` presence

AWS infrastructure (account 328657824100, us-east-1)

Resource	Value
Queue	`heatwave-sp-api-notifications` (14-day retention)
DLQ	`heatwave-sp-api-notifications-dlq` (redrive after 5 receives)
Queue policy	SP-API principal `437568002678` SendMessage + `events.amazonaws.com` SendMessage conditioned on the rule ARN
SP-API destinations	SQS `65d4113d-…` and EventBridge `f4c2551d-…` (destination config is application-level)
Partner event source / bus	`aws.partner/sellingpartnerapi.amazon.com/328657824100/amzn1.sp.solution.1ce0b323-…`, rule `heatwave-sp-api-notifications-to-sqs` → the queue

Re-provisioning is scripted; reruns of the setup script preserve the
EventBridge policy statement, and --verify round-trips a nonce-marked
message without ever touching live notifications.

Subscriptions

DESIRED_SUBSCRIPTIONS in the rake task is the single source of truth —
each entry is (type, destination, payloadVersion, optional filter). Live:

Type	Destination	Notes
`FEED_PROCESSING_FINISHED`	SQS	triggers immediate feed-result processing
`REPORT_PROCESSING_FINISHED`	SQS	recorded; report flows poll their own results (high frequency: ~100/day)
`LISTINGS_ITEM_ISSUES_CHANGE`	EventBridge	payloadVersion `2023-12-13`; targeted per-SKU issue refresh

Staged next (commented in the task): ORDER_CHANGE,
ACCOUNT_STATUS_CHANGED, the remaining listings/product-type family,
FBA_INVENTORY_AVAILABILITY_CHANGES, and ANY_OFFER_CHANGED with 30-minute
aggregation. Rollout order and the pull-cadence demotions that come with each
phase are in the task doc.

Managing subscriptions

# what's live vs desired (NA by default; pass a profile for EU)
mise exec -- bin/rails amazon_notifications:status

# create anything missing from DESIRED_SUBSCRIPTIONS
mise exec -- bin/rails amazon_notifications:reconcile

# remove one type
mise exec -- bin/rails "amazon_notifications:unsubscribe[ORDER_CHANGE]"

Production: kamal app exec --reuse "bin/rails amazon_notifications:status".

Filters are server-side and immutable: a subscription carries either a
CEL filterExpression or a legacy eventFilter (marketplace ids +
aggregation windows of 5/10/30/60 min). Changing a filter means
unsubscribe → reconcile; cross-subscription duplicate deliveries during a
parallel run carry distinct notificationIds, but handlers are idempotent
targeted re-pulls, so overlap is harmless. EventBridge-workflow types that
lack CEL support are filtered at the EventBridge rule pattern instead.

Monitoring

CRM → IT menu → Amazon SP-API Monitor (/amazon_notifications) is the
health dashboard, backed by Amazon::NotificationsHealth. One verdict from
three signal groups:

Poller liveness — the worker writes a Rails.cache heartbeat after
every completed drain cycle; stale (>10 min) or missing is a hard failure.
Queue depths — live GetQueueAttributes on the queue and its DLQ
(the runtime IAM policy covers both). Any DLQ message is a hard failure
(poison payload); a large main-queue backlog is a warning.
Pipeline state — webhook_logs counts by state (exception = hard
failure, retry = warning), last received/processed, 7-day category
breakdown, and recent exceptions with links to their reprocess pages.

Dev/test show ⚪ unconfigured (credentials are production-nested) while
still displaying pipeline stats.

Runbook

Add a notification type: add the entry to DESIRED_SUBSCRIPTIONS
(type, destination, payloadVersion, filter), run reconcile
(createSubscription rejects invalid pairings, so it self-verifies), add a
handler branch in AmazonSpApiProcessor — unmapped types safely land as
noted until then.
Too noisy: prefer server-side tuning — aggregation for offer churn, CEL
for payload-level conditions — over client-side drops. Order of leverage is
documented in the task doc's "Tuning & filtering" section.
Triage: CRM → Webhook Logs, provider amazon_sp_api (states, payload
search, per-row reprocess). exception rows page through AppSignal.
DLQ has messages: inspect via
aws sqs receive-message --queue-url …-dlq; a poison message means the
body couldn't even be parsed/ingested — fix, then redrive
(start-message-move-task) or let it expire (14 days).
Queue empty ≠ broken: overnight Chicago hours are genuinely quiet (the
hourly feed crons are differential and skip no-change runs). Force a proof
any time by requesting a small report — REPORT_PROCESSING_FINISHED
arrives seconds after it completes.
unknown category rows: a notification family with an unexpected key
shape — the graceful fallback. Inspect data, extend normalize /
CATEGORIES if a genuinely new shape appears.
catalog item not found (listings issues): the SKU/ASIN didn't resolve
in the marketplace's catalogs — check AmazonMarketplace#catalogs linkage
and the per-marketplace third_party_part_number semantics (US = seller
SKU, CA = ASIN; documented in the integration doc).
Unknown marketplace ids intentionally do not fall back to unscoped
matching.

Verification history

2026-07-16: SQS path proven end-to-end (~1 s publish→receive latency) via a
forced report request; EventBridge path + the full Rails pipeline proven by
draining 95 real production messages, including three live listings-issue
events resolving to catalog items on both the US and CA conventions.
The live LISTINGS_ITEM_ISSUES_CHANGE subscription was migrated from
payloadVersion 1.0 to 2023-12-13 the same day (Amazon deprecation).