Exporting 10 million rows without exhausting memory (yield_per, iterator)

Backend Live Coding November 13, 2026 python

The export runs in a background job, writes to S3, and the user gets a link when it finishes.

python
 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
# ======================================================================
# Q: How do you read 10 million rows from the database for an export without
# running out of memory?
#
# The export runs in a background job, writes to S3, and the user gets a link
# when it finishes. Never inside an HTTP request.
# ======================================================================

# SQLAlchemy: server-side cursor + yield_per
stmt = select(Event).where(Event.tenant_id == tid).order_by(Event.id)
for partition in session.execute(stmt.execution_options(yield_per=2000)).scalars().partitions():
    write_rows(partition)

# Django equivalent
for e in Event.objects.filter(tenant_id=tid).iterator(chunk_size=2000):
    write_row(e)