The Export That Ate Memory

The situation

The monthly creator export report reads the entire export into a list, builds three more lists from it, and only then starts summarising. On a big month it runs the workstation out of memory before printing a single line.

What you'll learn

• PCAP 5.1-5.5: list/dict comprehensions, lambdas, map/filter, closures and late binding, generators and lazy evaluation, and text I/O concepts (modes, encoding, cleanup, errors).
• map and filter with lambdas, binary file modes, bytes versus bytearray, errno, a custom exception's args, and closures with nonlocal.

Who you work with

• 

  Amara Okafor

  ML and Data Engineer

• 

  Kabelo Mokoena

  Platform and DevOps Engineer

Scenes

1. 1.Read it all, then choke
2. 2.List or generator?
3. 3.Stream the export
4. 4.Corrupted output on empty input
5. 5.Comprehensions and I/O check

What you leave with

You streamed the supplied export through a generator, counted during iteration and used a file context manager for cleanup on normal and exceptional block exit. Initialising the footer state handles an empty export. Total memory still depends on row size, retained state and I/O buffers.