Database job queue with SKIP LOCKED for multiple workers

Backend Live Coding November 13, 2026 python

SKIP LOCKED makes each worker pick different rows without waiting.

python 
 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
# ======================================================================
# Q: How do you implement a database-backed job queue that works with many
# workers at the same time?
#
# SKIP LOCKED makes each worker pick different rows without waiting. The same
# pattern is used in the outbox relay.
# ======================================================================

with transaction.atomic():
    jobs = list(Job.objects.select_for_update(skip_locked=True)
                .filter(status='queued').order_by('id')[:10])
    for job in jobs:
        job.status = 'running'
    Job.objects.bulk_update(jobs, ['status'])
# processing happens after the commit

Related Posts

Python shallow copy vs deep copy
Python shallow copy vs deep copy
Python closures and late binding in lambdas
Python closures and late binding in lambdas
Storing heartbeats from 250,000 screens with Redis write coalescing
Storing heartbeats from 250,000 screens with Redis write coalescing
Writing a retry decorator with exponential backoff in Python
Writing a retry decorator with exponential backoff in Python
SQL: what a CTE is and how to find duplicate records
SQL: what a CTE is and how to find duplicate records
Threads vs processes vs asyncio: when to use each
Threads vs processes vs asyncio: when to use each