Vector Log Shipping to AppSignal

This guide documents how to set up Vector to ship application and database logs to AppSignal.

Overview

Vector is a high-performance observability data pipeline that collects logs from multiple sources and ships them to AppSignal for centralized monitoring.

Architecture

┌─────────────────────────────────────────────────────────────┐
│              chi-vultr-heatwave-util1 (App Server)          │
├─────────────────────────────────────────────────────────────┤
│                                                             │
│  ┌─────────────────┐     ┌─────────────────┐               │
│  │ Sidekiq Workers │────▶│                 │               │
│  └─────────────────┘     │ production.log  │───┐           │
│  ┌─────────────────┐     │                 │   │           │
│  │   Rails App     │────▶│                 │   ▼           │
│  └─────────────────┘     └─────────────────┘ ┌────────┐    │
│                                              │ Vector │────┼──▶ AppSignal
│                                              └────────┘    │    (sidekiq/rails)
└─────────────────────────────────────────────────────────────┘

┌─────────────────────────────────────────────────────────────┐
│              chi-vultr-heatwave-db4 (Database Server)       │
├─────────────────────────────────────────────────────────────┤
│                                                             │
│  ┌─────────────────┐     ┌─────────────────┐  ┌────────┐   │
│  │  PostgreSQL 16  │────▶│ postgresql.log  │─▶│ Vector │───┼──▶ AppSignal
│  └─────────────────┘     └─────────────────┘  └────────┘   │    (postgresql)
│                                                             │
└─────────────────────────────────────────────────────────────┘

Configuration Files

Configuration files are stored per-server in the codebase:

config/deploy/templates/
├── chi-vultr-heatwave-util1/    # Application server
│   ├── vector.yaml              # Vector configuration
│   ├── vector.env.example       # Environment template
│   └── vector.service           # Systemd service (reference)
│
└── chi-vultr-heatwave-db4/      # Database server
    ├── vector.yaml              # Vector configuration
    ├── vector.env.example       # Environment template
    └── vector.service           # Systemd service (reference)

Log Groups in AppSignal

Server	Log Group	Description
util1	sidekiq	Sidekiq worker logs (detected from production.log)
util1	rails	Rails application logs
db4	postgresql	PostgreSQL errors, deadlocks, and warnings

---

Prerequisites

1. AppSignal account with Logging enabled
2. AppSignal Push API Key from AppSignal Settings → Push & Deploy
3. SSH access to both servers

---

Installation

Step 1: Create AppSignal Log Sources

In AppSignal:

1. Go to Logging → Sources → Add log source
2. Create these sources:

Name	Type	Format
sidekiq	vector	logfmt or json
rails	vector	logfmt or json
postgresql	vector	logfmt or json

---

Application Server (chi-vultr-heatwave-util1)

1. Install Vector via APT

ssh deploy@chi-vultr-heatwave-util1

# Add Vector repository
curl -1sLf 'https://repositories.timber.io/public/vector/cfg/setup/bash.deb.sh' | sudo bash

# Install Vector
sudo apt install vector

2. Copy Configuration

# Copy from local machine or after deployment
sudo cp /var/www/heatwave/current/config/deploy/templates/chi-vultr-heatwave-util1/vector.yaml /etc/vector/vector.yaml

3. Create Environment File

sudo tee /etc/vector/vector.env << 'EOF'
APPSIGNAL_PUSH_API_KEY=your-push-api-key-here
HOSTNAME=chi-vultr-heatwave-util1
EOF

sudo chmod 600 /etc/vector/vector.env

4. Configure Systemd Override

The APT package includes a default service, but we need to add environment file support:

sudo mkdir -p /etc/systemd/system/vector.service.d

sudo tee /etc/systemd/system/vector.service.d/override.conf << 'EOF'
[Service]
EnvironmentFile=/etc/vector/vector.env
EOF

5. Fix Permissions

Vector needs write access to its data directory for checkpointing:

sudo chown -R vector:vector /var/lib/vector
sudo chmod 755 /var/lib/vector

6. Enable and Start

sudo systemctl daemon-reload
sudo systemctl enable vector
sudo systemctl start vector
sudo systemctl status vector

---

Database Server (chi-vultr-heatwave-db4)

1. Install Vector via APT

ssh root@chi-vultr-heatwave-db4

# Add Vector repository
curl -1sLf 'https://repositories.timber.io/public/vector/cfg/setup/bash.deb.sh' | sudo bash

# Install Vector
sudo apt install vector

2. Copy Configuration

# Copy the database-specific config
sudo cp /path/to/vector.yaml /etc/vector/vector.yaml

Or copy from local:

# From your local machine:
scp config/deploy/templates/chi-vultr-heatwave-db4/vector.yaml root@chi-vultr-heatwave-db4:/etc/vector/vector.yaml

3. Create Environment File

sudo tee /etc/vector/vector.env << 'EOF'
APPSIGNAL_PUSH_API_KEY=your-push-api-key-here
EOF

sudo chmod 600 /etc/vector/vector.env

4. Configure Systemd Override

sudo mkdir -p /etc/systemd/system/vector.service.d

sudo tee /etc/systemd/system/vector.service.d/override.conf << 'EOF'
[Service]
EnvironmentFile=/etc/vector/vector.env
EOF

5. Fix Permissions

sudo chown -R vector:vector /var/lib/vector
sudo chmod 755 /var/lib/vector

6. Enable and Start

sudo systemctl daemon-reload
sudo systemctl enable vector
sudo systemctl start vector
sudo systemctl status vector

---

Verification

Check Vector Status

sudo systemctl status vector

View Vector Logs

sudo journalctl -u vector -f

Check for Errors

sudo journalctl -u vector --since "5 minutes ago" | grep -i error

Test Configuration

Before starting the service, you can validate manually:

sudo bash -c 'set -a && source /etc/vector/vector.env && set +a && /usr/bin/vector'

Press Ctrl+C to stop after confirming it starts successfully.

Verify in AppSignal

1. Go to Logging → All logs
2. Filter by group: sidekiq, rails, or postgresql
3. Filter by hostname to see logs from specific servers
4. Logs should appear within 1-2 minutes

---

PostgreSQL Log Filtering

The database Vector config filters logs to only ship:

Category	Examples
Errors/Warnings	All ERROR, WARNING, FATAL levels
Deadlocks	deadlock detected, detected deadlock while waiting
Lock Issues	lock timeout, still waiting for, canceling statement due to lock
Timeouts	canceling statement due to statement timeout
Connections	terminating connection, could not connect, connection reset
Constraints	violates, duplicate key
Resources	out of memory, temporary file
Replication	replication slot, WAL segment

This reduces noise from routine LOG-level messages while capturing all debugging-relevant events.

---

Troubleshooting

Vector Not Starting

Check configuration syntax:

sudo bash -c 'set -a && source /etc/vector/vector.env && set +a && vector validate /etc/vector/vector.yaml'

Permission Denied on Checkpoints

# Fix /var/lib/vector permissions
sudo chown -R vector:vector /var/lib/vector
sudo chmod 755 /var/lib/vector
sudo systemctl restart vector

Environment Variable Not Loading

The EnvironmentFile directive does NOT support export:

# WRONG - will be ignored:
export APPSIGNAL_PUSH_API_KEY=xxx

# CORRECT:
APPSIGNAL_PUSH_API_KEY=xxx

401 Unauthorized from AppSignal

1. Verify API key is correct in /etc/vector/vector.env
2. Test manually with curl:

curl -v -X POST "https://appsignal-endpoint.net/logs/json" \
  -H "Authorization: Bearer YOUR_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{"message":"test","level":"info"}'

Should return 200 OK.

413 Payload Too Large

The Vector config includes batching limits to prevent this:

batch:
  max_bytes: 400000  # Stay under AppSignal's 500KB limit
  timeout_secs: 5

If you see this error, clear old checkpoint data:

sudo rm -rf /var/lib/vector/*
sudo systemctl restart vector

No Logs Appearing

1. Check file permissions: Vector needs read access to log files
2. Check network: Ensure outbound HTTPS to appsignal-endpoint.net
3. Check log group: Verify AppSignal source names match .group in config

---

Maintenance

Update Vector

sudo apt update
sudo apt upgrade vector
sudo systemctl restart vector

View Vector Version

vector --version

Clear Checkpoints (Re-process Logs)

If you need to re-ship logs:

sudo systemctl stop vector
sudo rm -rf /var/lib/vector/*
sudo systemctl start vector

Warning: This will re-process logs from the current file position.

---

Related Documentation

• AppSignal Logging Configuration
• AppSignal Vector Platform
• Vector Documentation
• Vector VRL Reference
• AppSignal Monitoring