Running Celery workers on ECS Fargate with RDS and SQS, and what happens on deploy

# ======================================================================
# Q: CTO: "We run on ECS Fargate with RDS and SQS. How would you set up the
# Celery workers and what happens during a deploy?"
#
# You: "Separate ECS services for web, workers and beat, using the same image.
# Beat runs as a single task, otherwise scheduled jobs run twice. SQS is the
# broker, with a visibility timeout longer than the slowest task and a DLQ with
# a redrive policy. The result backend is not SQS, but Redis or the database, if
# needed. On deploy, ECS sends SIGTERM and Celery does a warm shutdown: it
# finishes the current task. Since we use acks_late and idempotent tasks,
# anything that gets cut off safely reruns. Workers autoscale based on queue
# length in CloudWatch. Migrations run as a one-off task before the new release
# and are always backwards compatible."
# ======================================================================

CELERY_BROKER_URL = 'sqs://'          # credentials from the ECS task role
CELERY_BROKER_TRANSPORT_OPTIONS = {
    'region': 'eu-central-1',
    'visibility_timeout': 3600,        # > the slowest task
    'predefined_queues': {
        'default': {'url': 'https://sqs.eu-central-1.amazonaws.com/<account>/default'},
    },
}
CELERY_TASK_ACKS_LATE = True
CELERY_WORKER_PREFETCH_MULTIPLIER = 1  # fair distribution across long tasks