The export runs in a background job, writes to S3, and the user gets a link when it finishes.
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 | # ====================================================================== # Q: How do you read 10 million rows from the database for an export without # running out of memory? # # The export runs in a background job, writes to S3, and the user gets a link # when it finishes. Never inside an HTTP request. # ====================================================================== # SQLAlchemy: server-side cursor + yield_per stmt = select(Event).where(Event.tenant_id == tid).order_by(Event.id) for partition in session.execute(stmt.execution_options(yield_per=2000)).scalars().partitions(): write_rows(partition) # Django equivalent for e in Event.objects.filter(tenant_id=tid).iterator(chunk_size=2000): write_row(e) |