Application Monitoring (AppSignal)

Overview

Heatwave uses AppSignal for comprehensive application monitoring:

Feature Description
Error Tracking Exception capture, alerting, and error grouping
APM Request performance monitoring and slow query detection
Host Metrics CPU, memory, disk usage monitoring
Anomaly Detection Automatic alert triggers for unusual patterns
Deploy Markers Correlate deploys with errors/performance changes
Frontend Errors JavaScript error capture with sourcemaps

Dashboard: https://appsignal.com/warmlyyours

File Locations

Backend (Ruby)

File Purpose
config/appsignal.rb Main AppSignal configuration
app/services/error_reporting.rb Unified error reporting abstraction
app/concerns/models/validation_error_reporting.rb Model validation error reporting
app/concerns/workers/error_reporting_concern.rb Worker/job error reporting helpers
lib/sidekiq/appsignal_context_middleware.rb Sidekiq context enrichment
config/initializers/sidekiq.rb Sidekiq middleware registration
config/application.rb Sets config.x.revision from APP_REVISION (drives deploy markers)

Frontend (JavaScript)

File Purpose
client/js/common/error_handling.js AppSignal JS initialization and ErrorReporter
client/js/www/www.bundler.js WWW bundle entry (calls errorHandlingInit)
client/js/www/core.js Core JS entry (calls errorHandlingInit)
client/js/crm/crm.index.js CRM bundle entry (calls errorHandlingInit)
client/js/common/page_config.js Exports appsignalFrontendKey from the #page-config JSON
app/views/globals/_page_config.html.erb Renders the per-env frontend key into #page-config (runtime)

Deployment

File Purpose
bin/deploy Kamal deploy wrapper; daemonizes the AppSignal sourcemap upload post-deploy
config/appsignal.rb Sets config.revision (the gem emits the deploy marker on revision change)
script/upload_sourcemaps_to_appsignal.rb Uploads frontend sourcemaps to AppSignal's private API

Configuration Details

Backend Configuration (config/appsignal.rb)

Key settings:

  • Development disabled - Prevents noise during local dev
  • Minimal ignore list - Most errors tracked for visibility
  • Rich context - All request headers, params, and session data captured
  • Full instrumentation - ViewComponent, Redis, HTTP, GVL, host metrics

Enabled Integrations

Integration Config Description
ViewComponent config.view_component.instrumentation_enabled Component render performance
Redis instrument_redis = true Cache and Sidekiq visibility
Net::HTTP instrument_net_http = true Outgoing HTTP requests
HTTP.rb instrument_http_rb = true HTTP gem requests
Host Metrics enable_host_metrics = true CPU, memory, disk, network
Minutely Probes enable_minutely_probes = true Metrics collection
Allocation Tracking enable_allocation_tracking = true Memory allocation insights
GVL Timer enable_gvl_global_timer = true Ruby threading insights
GVL Threads enable_gvl_waiting_threads = true Thread waiting analysis
Rails Error Reporter enable_rails_error_reporter = true Rails 7+ integration
Sidekiq Errors sidekiq_report_errors = :all All errors including retries
ActiveJob Errors activejob_report_errors = :all All errors including retries

ViewComponent setup (config/application.rb):

config.view_component.instrumentation_enabled = true
config.view_component.use_deprecated_instrumentation_name = false

See: AppSignal ViewComponent Integration

Captured Request Headers

All headers captured for debugging context:

Category Headers
Standard Accept, Accept-Language, Accept-Encoding, Host, Referer, User-Agent, Origin
Cloudflare CF-Connecting-IP, CF-IPCountry, CF-Ray, CF-Visitor, True-Client-IP
Forwarding X-Forwarded-For, X-Forwarded-Host, X-Forwarded-Proto, X-Real-IP, X-Request-ID
Client Hints Sec-CH-UA, Sec-CH-UA-Mobile, Sec-CH-UA-Platform
Fetch Metadata Sec-Fetch-Dest, Sec-Fetch-Mode, Sec-Fetch-Site
Request Content-Type, Content-Length, Path-Info, Query-String, Request-Method

Filtered Data

Type Filtered Items
Parameters Rails default filter list (password, token, etc.)
Session _csrf_token, session_id, warden.user.account.key

Frontend Configuration (client/js/common/error_handling.js)

The frontend uses @appsignal/javascript with these features:

Feature Implementation
Null Adapter Development mode uses no-op adapter to prevent errors
Runtime Key Per-env frontend key from the #page-config JSON (config.x → page_config.js)
Namespace Detection Uses window.subdomain to set www/crm/api namespace
User Context Automatically includes current_user data when available
Window Error Handling Wraps global error handlers to report errors
Test Function window.testAppSignal() for manual testing

Frontend Plugins

We use AppSignal's official plugins for enhanced error context:

Plugin Description
plugin-breadcrumbs-console Captures console.log/warn/error calls as breadcrumbs
plugin-breadcrumbs-network Captures fetch and XMLHttpRequest calls as breadcrumbs
plugin-window-events Catches unhandled errors and promise rejections

Breadcrumbs show what happened before an error occurred, making debugging much easier:

  • Console logs leading up to the error
  • API calls made (URL, status, timing)
  • User interactions that triggered the error

Frontend key injection flow (runtime, per-env):

  1. config/application.rb fetches the per-env front_end_key from credentials once at boot into config.x.appsignal_frontend_key.
  2. app/views/globals/_page_config.html.erb renders it into the <script id="page-config"> JSON.
  3. client/js/common/page_config.js exports it as appsignalFrontendKey; error_handling.js reads it and inits AppSignal (null adapter when absent).

Injected at runtime, not build time, so the shared prod/staging Docker image serves each
env its own key and rotating it needs no rebuild. null in dev → null adapter. (Fixed in
PR #1168 — the key was previously baked at webpack build time but never set, so frontend
tracking was silently dark in prod/staging.)

Credentials

Stored in Rails credentials (config/credentials.yml.enc):

appsignal:
  production:
    push_api_key: 'backend-api-key'
    front_end_key: 'frontend-public-key'
  staging:
    push_api_key: 'backend-api-key'
    front_end_key: 'frontend-public-key'

Access via:

Heatwave::Configuration.fetch(:appsignal, :push_api_key)
Heatwave::Configuration.fetch(:appsignal, :front_end_key)

Error Reporting API

Namespace Structure

We use 9 clean namespaces combining source and severity.

Configure notification settings per namespace in AppSignal dashboard:

Source Severity Namespace Notification
Rails (www/crm/api) Error web First in Deploy
Rails (www/crm/api) Warning web_warning First Occurrence
Rails (www/crm/api) Info web_info Never Notify
JavaScript Error frontend First in Deploy
JavaScript Warning frontend_warning First Occurrence
JavaScript Info frontend_info Never Notify
Sidekiq/ActiveJob Error background First in Deploy
Sidekiq/ActiveJob Warning background_warning First Occurrence
Sidekiq/ActiveJob Info background_info Never Notify

Setup: In AppSignal dashboard, go to App Settings → Notification Defaults and configure each namespace.

See: AppSignal Namespaces Guide

Backend (Ruby) - ErrorReporting Service

The ErrorReporting class provides a unified, provider-agnostic interface:

# From a controller
ErrorReporting.from_controller(self).error(exception)      # → web
ErrorReporting.from_controller(self).warning(exception)    # → web_warning
ErrorReporting.from_controller(self).informational(ex)     # → web_info

# Direct error reporting
ErrorReporting.error(exception)                            # → web
ErrorReporting.error(exception, { user_id: 123 })          # → web with context

# Warning
ErrorReporting.warning("Unusual condition", { value: x })  # → web_warning

# Critical (goes to web with severity: critical tag)
ErrorReporting.critical(exception, { urgent: true })

# Informational (goes to web_info - configure "Never Notify")
# Use for: bot blocks, 404s, CSRF failures, record not found
ErrorReporting.informational("Bot blocked", { reason: 'bot_detection' })

# Info/debug (logs only, NOT sent to AppSignal)
ErrorReporting.info("Process completed", { count: 100 })

# From a worker (uses background namespace family)
ErrorReporting.error(exception, source: :background)           # → background
ErrorReporting.warning(exception, source: :background)         # → background_warning
ErrorReporting.informational(msg, source: :background)         # → background_info

# Scoped context (thread-safe, applies to all errors in block)
ErrorReporting.scoped({ order_id: order.id }) do
  process_order(order)  # Any errors include the context
end

Context automatically extracted by from_controller:

  • Request URL, method, path, format
  • User agent, remote IP, referer
  • Filtered params
  • Current user/account ID and email
  • Subdomain (www, crm, api) stored as tag for filtering

Frontend (JavaScript) - window.ErrorReporter

// Report an error (goes to 'frontend' namespace)
window.ErrorReporter.error(error, { extra: "data" });

// Report warning (goes to 'frontend_warning' namespace)
window.ErrorReporter.warning(error, { context: "info" });

// Report critical (goes to 'frontend' namespace with severity: critical)
window.ErrorReporter.critical(error);

// Report informational (goes to 'frontend_info' namespace)
window.ErrorReporter.informational("Expected event", { reason: "x" });

// Info/debug (console only, NOT sent to AppSignal)
window.ErrorReporter.info("Info message");
window.ErrorReporter.debug("Debug message");

// Test function (also works via URL: ?test_appsignal=1)
window.testAppSignal();

Sidekiq Integration

Automatic Context Middleware

lib/sidekiq/appsignal_context_middleware.rb automatically enriches all Sidekiq jobs with:

Tag Description
job_class Worker class name
job_id Sidekiq JID
queue Queue name
retry_count Number of retries
batch_id Sidekiq Pro batch ID (if applicable)
job_args Sanitized job arguments

Registration (config/initializers/sidekiq.rb):

config.server_middleware do |chain|
  chain.add Sidekiq::AppsignalContextMiddleware
end

Manual Error Reporting in Workers

For batch processing where you want to continue after errors:

class MyWorker
  include Sidekiq::Job
  include Workers::ErrorReportingConcern

  def perform(ids)
    ids.each do |id|
      process_item(id)
    rescue StandardError => e
      report_worker_error(e, item_id: id)
      # Continue with next item
    end
  end
end

Model Validation Errors

For tracking validation failures:

class Order < ApplicationRecord
  include Models::ValidationErrorReporting
  
  after_validation :report_validation_errors, if: :should_report?
  
  private
  
  def should_report?
    errors.present? && some_condition?
  end
end

Deploy Markers

Deploys go through Kamal (bin/deploy), not Capistrano. Markers are
emitted by the AppSignal Ruby gem itself from config.revision, which is set
in config/appsignal.rb to the app's revision:

# config/appsignal.rb
config.revision = Heatwave::Application.config.x.revision

config.x.revision is the full git SHA, baked into the production image at
build time as the APP_REVISION env layer (Dockerfile ARG GIT_REVISION ←
config/deploy.yml builder args) and read back in config/application.rb. When
a new container boots reporting a changed revision, the gem records a deploy
marker showing:

  • Git revision (SHA)
  • Environment
  • Deploy time

Use markers to correlate:

  • Error spikes after deploys
  • Performance changes
  • New errors introduced

Sourcemaps

Frontend sourcemaps are uploaded privately to AppSignal (not served publicly):

  1. Webpack generates hidden sourcemaps (devtool: 'hidden-source-map')
    • Creates .map files but no //# sourceMappingURL comment in JS files
    • This keeps your source code private from end users
  2. Deploy script uploads sourcemaps to AppSignal's private API
    • Uses script/upload_sourcemaps_to_appsignal.rb
    • Tied to git revision for matching errors to correct source versions
  3. Sourcemaps are deleted from build directory after upload
    • They never reach the production servers
  4. AppSignal uses private maps to resolve stack traces

Security Benefits

  • Source code is not exposed to end users
  • Attackers cannot reverse-engineer minified code
  • Stack traces still show original source locations

Manual Upload (if needed)

If automatic upload fails, you can manually upload:

APPSIGNAL_PUSH_API_KEY=xxx \
ENVIRONMENT=production \
REVISION=$(git rev-parse HEAD) \
BUILD_DIR=public/javascripts/webpack-prod \
bundle exec ruby script/upload_sourcemaps_to_appsignal.rb

Set DRY_RUN=true to preview without uploading.

Environments

Environment AppSignal Active Frontend Key
production ✅ production key
staging ✅ staging key
development ❌ (null adapter) N/A
test ❌ N/A

AI Assistant Access

The standalone AppSignal MCP server has been retired. For AI-assistant access
to AppSignal — listing exception/anomaly incidents, fetching samples, querying
logs and metrics, checking deploy markers, adding notes, closing incidents —
use the appsignal skill, which
calls AppSignal's MCP HTTP API directly via curl.

Error Handling in Controllers

ApplicationController Integration

# Rescue handlers (non-local requests only)
rescue_from StandardError, with: :render_500
rescue_from ActiveRecord::RecordNotFound, with: :render_404
rescue_from ActionController::InvalidAuthenticityToken, with: :render_invalid_authenticity_token

def render_500(exception = nil)
  # Reports as ERROR with full request context
  ErrorReporting.from_controller(self).error(exception) if exception
  # ... render error page
end

def render_404(exception = nil)
  # Reports as INFORMATIONAL - expected for old links/bots/typos
  if exception.is_a?(ActiveRecord::RecordNotFound) || exception.is_a?(ActionController::RoutingError)
    ErrorReporting.from_controller(self).informational(exception, reason: 'not_found')
  end
  # ... render 404 page
end

def render_invalid_authenticity_token(exception = nil)
  # Reports as INFORMATIONAL - expected for expired sessions/bots
  ErrorReporting.from_controller(self).informational(exception, reason: 'csrf_token_invalid')
  reset_session
  # ... redirect to login
end

def prevent_bots
  return unless bot_request?
  # Reports as INFORMATIONAL - expected behavior, not an error
  ErrorReporting.from_controller(self).informational("Bot blocked: #{request.user_agent}", reason: 'bot_detection')
  head :forbidden
end

Namespace Classification Guide

Error Type Namespace Notification Reason
InvalidCrossOriginRequest Ignored N/A CORS preflight, browser behavior
RecordNotFound web_info Never Notify 404s - expected for old links/bots
RoutingError web_info Never Notify 404s - expected for old links/bots
InvalidAuthenticityToken web_warning First Occurrence Expected for expired sessions/bots
Bot detection blocks web_info Never Notify Expected security behavior
Turnstile validation failures web_info Never Notify Expected for bots/JS disabled
All other errors web/frontend/background First in Deploy Full tracking

Alerts & Notifications

Configure in AppSignal dashboard:

  • Error rate thresholds
  • Performance anomalies
  • Host metric alerts

Notification channels:

  • Email
  • Slack
  • PagerDuty
  • Webhooks

Troubleshooting

Errors not appearing in AppSignal

  1. Check environment is production or staging
  2. Verify API key in credentials: Heatwave::Configuration.fetch(:appsignal, :push_api_key)
  3. Check Appsignal.active? returns true
  4. Review log/production.log for AppSignal errors

Frontend errors not tracking

Frontend AppSignal is on in production/staging (per-env key injected at runtime via
#page-config — see the Frontend key injection flow above). In dev the null adapter is
expected. If errors aren't tracking in prod/staging, work through:

  1. Verify window.env is not 'development'
  2. Check browser console for AppSignal init messages
  3. Run window.testAppSignal() to test
  4. Check Network tab for requests to appsignal-endpoint.net (204 = success)
  5. Verify the per-env key is present in the #page-config JSON (View Source → <script id="page-config"> → appsignalFrontendKey)

Development mode

Development uses a null adapter - this is intentional:

[AppSignal] Development mode - using null adapter

Errors log to console but don't send to AppSignal.

Checking adapter status

// In browser console:
console.log('env:', window.env);
console.log('_isNullAdapter:', window.appsignal?._isNullAdapter);
console.log('ErrorReporter.enabled:', window.ErrorReporter?.enabled);
window.testAppSignal();  // Should show "✅ Test error sent!"

Deployment errors

The app runs as Puma in Docker behind Thruster, deployed via Kamal. If a
deploy fails to boot the new container:

  1. Check kamal app logs (and log/production.log) for the boot error
  2. bin/deploy aborts before swapping traffic, so the previous release keeps serving
  3. Fix the error and re-run bin/deploy

Related Documentation