System design: a notification service (email, SMS, push) for 1 million users

# ======================================================================
# Q: CTO: "Design a notification service (email, SMS, push) for 1 million
# users."
#
# You, with a structure: 1) Requirements (volume, latency, user preferences,
# opt-out/GDPR). 2) An API that accepts notification requests (idempotency key).
# 3) Outbox and queue (RabbitMQ/SQS or Kafka), separate workers per channel. 4)
# Templates and localization. 5) Provider rate limits, retries with backoff,
# dead-letter queue. 6) Status stored per message, delivery webhooks. 7) Metrics
# (sent, failed, latency) and alerts. 8) Horizontal scaling with more workers.
# Priorities with separate queues, so that an OTP does not wait behind 1 million
# marketing messages.
# ======================================================================

CELERY_TASK_ROUTES = {
    'notifications.tasks.send_otp':       {'queue': 'critical'},
    'notifications.tasks.send_receipt':   {'queue': 'default'},
    'notifications.tasks.send_marketing': {'queue': 'bulk'},
}

@shared_task(rate_limit='50/s', autoretry_for=(ProviderError,),
             retry_backoff=True, max_retries=6)
def send_marketing(notification_id): ...

# celery -A proj worker -Q critical -c 8
# celery -A proj worker -Q default,bulk -c 4