Rails 8 Drops Redis: How PostgreSQL's SKIP LOCKED Powers SolidQueue

In late 2024, Rails 8 removed Redis from its default stack — a move that challenges the long-standing consensus that specialized middleware outperforms general-purpose databases for queues, caching, and pub/sub. The replacement libraries — SolidQueue, SolidCache, and SolidCable — all run on the application's existing relational database (PostgreSQL, MySQL, or SQLite).

37signals, Rails' creator and maintainer, processes 20 million background jobs daily on MySQL with no Redis. They report the system is simpler and performance is fully adequate after removing Redis.

The Hidden Costs of Specialized Middleware

Adopting Redis introduces ongoing operational overhead: separate server maintenance, version upgrades, security patches, persistence strategy decisions (RDB vs AOF), memory limits and eviction policies, network and firewall configuration, high-availability clustering, dual backup strategies, and cross-storage debugging between SQL and Redis commands. Additional gems like sidekiq-cron for scheduled jobs add further dependencies and failure points.

Each extra component brings configuration, monitoring, backup, and failure-mode burdens that persist throughout the system's lifetime, consuming attention and amplifying troubleshooting difficulty.

How PostgreSQL Replaces Redis: SKIP LOCKED

The key enabler is PostgreSQL 9.5's FOR UPDATE SKIP LOCKED clause (2016). It allows concurrent workers to fetch distinct rows without lock contention:

SELECT * FROM solid_queue_ready_executions
WHERE queue_name = 'default'
ORDER BY priority DESC, job_id ASC
LIMIT 1
FOR UPDATE SKIP LOCKED;

An idle worker runs this query; PostgreSQL returns the highest-priority unlocked job. Multiple workers execute concurrently, each receiving a different job — no locking, waiting, or blocking. This solves the concurrency bottleneck that previously doomed database-backed queues.

SolidQueue Architecture

SolidQueue uses three tables: solid_queue_jobs — stores all job metadata (name, class, timestamps), retained permanently for auditability. solid_queue_scheduled_executions — holds future-scheduled jobs. solid_queue_ready_executions — holds jobs ready for immediate execution; workers claim from here.

High-frequency inserts/deletes are handled by PostgreSQL's MVCC and autovacuum without special tuning.

Independent processes coordinate via the database:

Workers poll ready table at configurable intervals (as low as 0.1s for high-priority queues).

Scheduler polls scheduled table each second, moving due jobs to ready.

Cron manager enqueues recurring jobs per YAML schedule.

Supervisor monitors heartbeats and restarts crashed children.

Each process touches different tables with different polling intervals; ACID transactions handle all coordination — no external lock service, distributed coordinator, or ZooKeeper needed.

Built-in Features That Cost Extra in Sidekiq

Concurrency limiting (per-user, per-resource) is native in SolidQueue, whereas Sidekiq Enterprise ($1,699/year) charges for it:

class ProcessUserOnboardingJob < ApplicationJob
  limits_concurrency to: 1,
    key: ->(user) { user.id },
    duration: 15.minutes
  def perform(user)
    # ...
  end
end

Implemented via solid_queue_semaphores (tracks limits) and solid_queue_blocked_executions (holds waiting jobs). Completion releases the semaphore; scheduler activates the next waiter — all database-native.

Cron scheduling requires no sidekiq-cron. A YAML file defines jobs:

production:
  cleanup_old_sessions:
    class: CleanupSessionsJob
    schedule: every day at 2am
    queue: maintenance
  send_daily_digest:
    class: DailyDigestJob
    schedule: every day at 9am
    queue: mailers

The scheduler writes the next trigger time on each run; crashes/restarts don't affect determinism — "every day at 9am" always resolves to today's 9am. This crash-resistant design borrows from the GoodJob project.

Monitoring and Debugging

Mission Control Jobs (free, open-source) mounts with one route line, providing real-time queue status, failed-job stack traces, bulk retry/discard, cron timeline, and throughput charts.

All data lives in the database. Debugging failed jobs uses plain SQL:

SELECT j.queue_name, COUNT(*) as failed_count
FROM solid_queue_failed_executions fe
JOIN solid_queue_jobs j ON j.id = fe.job_id
WHERE fe.created_at > NOW() - INTERVAL '1 hour'
GROUP BY j.queue_name;

No context-switching to external tools or query languages — just SQL the team already knows.

Performance Boundaries

Historical reference: Shopify (2013) handled 833 req/s with 53 servers and 1,172 workers — Redis was necessary at that scale.

37signals' current reference: 20M jobs/day ≈ 230 jobs/s on MySQL, no Redis.

Using Nate Berkopec's "1000 RPM formula" (instances = req/s × avg response time): a typical app at 100 RPM (1.67 req/s) with 200ms response uses only 0.083 instances — 8% of one instance's capacity.

Rough thresholds:

<100 jobs/s or latency tolerance >100ms: relational database is ample.

100–1,000 jobs/s : benchmark both; let data decide.

>1,000 jobs/s sustained or latency <1ms: Redis remains the right choice.

The "Middleware Tax"

The article coins "middleware tax" — not a cloud bill, but the lifelong hidden levy of every extra configuration, backup drill, cross-storage debug session, and "why don't the numbers match?" late-night call. The tax rate seems low but compounds continuously, and most payers don't realize they're paying.

Industry "best practices" originate from top-tier companies (e.g., Shopify 2013) and become de facto standards. Few stop to ask whether their own load matches that benchmark. Social proof creates implicit insurance: choosing the standard stack avoids justification; choosing a non-standard stack incurs recurring defense costs in every design review. These forces push adoption far beyond technical necessity.

Meanwhile, databases have evolved: PostgreSQL 9.5 added SKIP LOCKED, JSONB, full-text search, window functions. Today's PostgreSQL does what was thought to require specialized tools a decade ago, yet mental models haven't caught up.

Rails 8's decision exposes a neglected truth: the applicability of industry best practices is far narrower than their actual adoption. Projects outside that envelope have been paying for capabilities they never needed — not just in money, but in every config item, every cross-storage debug, every untested backup plan.