Generators, yield And Streaming

When you use ChatGPT the answer appears a few words at a time rather than all at once. That is streaming, and underneath it is a generator.

Generators are also how you handle files too large to fit in memory, and how frameworks hand you events from a running agent. One idea, three uses.

The idea in one picture

Think of a waiter.

A normal function is a waiter who disappears into the kitchen and comes back with the entire meal on one enormous tray.

A generator is a waiter who brings one dish, waits, brings the next when you are ready, and remembers exactly where they were in between.

Same food. Very different experience — and the second one you can start eating immediately.

One word of difference

def get_all():
    return [1, 2, 3]

def get_one_at_a_time():
    yield 1
    yield 2
    yield 3

return ends the function and hands back a value.

yield hands back a value and pauses, keeping everything exactly where it was. Next time you ask, it carries on from that line.

print(get_all())
print(get_one_at_a_time())

[1, 2, 3]
<generator object get_one_at_a_time at 0x000001A2...>

The second one is not a list. It is a thing that will produce values when asked.

Ask it with a for loop:

for n in get_one_at_a_time():
    print(n)

1
2
3

Nothing runs until you ask

This surprises everyone once, so let us watch it happen.

def counter():
    print("  ...body started")
    yield 1
    print("  ...between yields")
    yield 2

gen = counter()
print("generator built - notice nothing printed yet")

print("first  ->", next(gen))
print("second ->", next(gen))

generator built - notice nothing printed yet
  ...body started
first  -> 1
  ...between yields
second -> 2

Calling counter() did not run the body. It built a generator that is ready to run. The code only moves forward when something asks for the next value.

This is called lazy evaluation, and it is exactly what makes streaming possible: the source can produce values slowly, and your loop simply waits.

Why it matters for memory

def read_lines(path):
    with open(path, encoding="utf-8") as f:
        for line in f:
            yield line.strip()

for line in read_lines("huge.log"):
    if "ERROR" in line:
        print(line)

This works on a file of any size, because only one line is in memory at a time.

The eager version, lines = f.readlines(), loads the whole file and falls over on a large one.

In agent work this matters when chunking documents for retrieval, or walking a long log to build a test set.

Streaming a model response

Here is the shape you will actually meet:

for chunk in llm.stream("Explain retries in one paragraph"):
    print(chunk.content, end="", flush=True)

llm.stream(...) returns a generator. Each turn of the loop is a piece of the answer as it arrives.

Two details that are easy to miss:

• end="" stops print adding a newline after every chunk
• flush=True pushes it to the screen immediately instead of buffering

Forget flush=True and streaming looks broken — nothing appears until the very end.

When to stream, and when not to

Use stream when a person is watching. Use invoke when the result feeds the next step of your program.

Sitting in the middle of a stream

Because a generator both consumes and produces, you can wrap one:

def with_logging(stream):
    total = 0
    for chunk in stream:
        total += len(chunk)
        yield chunk
    print(f"\n[streamed {total} characters]")

The caller still sees a stream, arriving piece by piece. You have added behaviour without buffering the whole response.

This is how token counting and cost tracking get bolted onto streaming responses.

A generator is used up

gen = get_one_at_a_time()
print("first pass :", list(gen))
print("second pass:", list(gen))

first pass : [1, 2, 3]
second pass: []

Once exhausted it is finished. There is no rewind.

This causes a genuinely puzzling bug: you consume a stream to log it, then try to use it again and get nothing at all, with no error. If you need the values twice, capture them once:

items = list(gen)

Generator expressions

One bracket different from a list comprehension:

squares_list = [n * n for n in range(1_000_000)]    # a million numbers, in memory
squares_gen  = (n * n for n in range(1_000_000))    # a recipe, nothing computed yet

Square brackets build the list. Round brackets build a generator.

When you only intend to loop once, prefer round:

print(sum(n * n for n in range(10)))

285

Async generators

When the source is a network call, the loop gains a keyword:

async for chunk in llm.astream("Explain retries"):
    print(chunk.content, end="", flush=True)

Same idea, with waiting allowed between pieces. Note the a in astream — frameworks name the async version of a method with an a in front.

Coming from Java or C#

C# has this almost exactly: yield return, and IEnumerable<T> is lazy in the same way. If that is your background, you already know this page.

Java's Stream and Iterator are lazy too, but there is no yield keyword — you write an Iterator or use Stream.generate, both far more ceremony for the same result.

Common mistakes

• Reusing an exhausted generator and getting an empty result with no error.
• Calling len() on one. There is no length — it does not know how many values remain.
• Building a list when you only loop once. Memory cost, no benefit.
• Forgetting flush=True while streaming, so streaming looks broken.
• Expecting the body to run on call. It runs on the first request for a value.

Practise this

• Write a generator yielding 1 to 5 and print them with a for loop.
• Add a print before the first yield and prove it does not run until you ask.
• Consume a generator twice and observe the empty second pass.
• Turn a list comprehension into a generator expression and use it with sum().
• Write with_logging and wrap a generator with it.