PHP Generators: yield, yield from and Memory Use

A generator is a function that contains yield. Calling it does not run the body; it returns a Generator object that foreach can loop over. Each yield $value hands one value to the loop and pauses the function until the loop asks for the next one, so values are produced one at a time instead of being collected in an array first.

<?php
function countTo(int $n): Generator
{
    for ($i = 1; $i <= $n; $i++) {
        echo "(making $i) ";
        yield $i;
    }
}

foreach (countTo(3) as $number) {
    echo "got $number\n";
}

The output alternates between the generator and the loop: each value is made only when the loop asks for it.

Generators save memory

range(1, 1000000) builds an array of a million integers before the loop starts. A generator that yields the same numbers never holds more than one of them. Run it and compare the memory each version needs:

<?php
function numbers(int $max): Generator
{
    for ($i = 1; $i <= $max; $i++) {
        yield $i;
    }
}

$start = memory_get_usage();
$sum = 0;
foreach (range(1, 1_000_000) as $n) {
    $sum += $n;
}
$arrayPeak = memory_get_peak_usage() - $start;

memory_reset_peak_usage();
$start = memory_get_usage();
$sum2 = 0;
foreach (numbers(1_000_000) as $n) {
    $sum2 += $n;
}
$genPeak = memory_get_peak_usage() - $start;

printf("array:     sum %d, extra memory %.1f MB\n", $sum, $arrayPeak / 1048576);
printf("generator: sum %d, extra memory %.1f MB\n", $sum2, $genPeak / 1048576);

Both loops print the same sum, 500000500000, but the array version reports extra memory 16.0 MB and the generator extra memory 0.0 MB. The gap grows with the data. The same idea is what makes generators useful for large files, database results and API pages: process one item, drop it, take the next.

Yield keys and values

yield $key => $value sets the key that foreach ($gen as $key => $value) receives. Without a key, a generator numbers its values from 0, like a list. A common use is reading a file line by line with line numbers:

<?php
file_put_contents('log.txt', "boot ok\nERROR disk full\nuser login\nERROR timeout\n");

function readLines(string $path): Generator
{
    $handle = fopen($path, 'r');
    $lineNo = 0;
    while (($line = fgets($handle)) !== false) {
        $lineNo++;
        yield $lineNo => rtrim($line, "\n");
    }
    fclose($handle);
}

foreach (readLines('log.txt') as $no => $line) {
    if (str_starts_with($line, 'ERROR')) {
        echo "line $no: $line\n";
    }
}

Only one line is in memory at a time, so the same function handles a 2 GB log file. fopen() and fgets() cover the file functions used here.

Infinite sequences and lazy pipelines

A generator can loop forever, because it only runs as far as the consumer asks. Stop with break, or wrap it in another generator that takes the first few values. Chaining small generators gives a pipeline where every step is lazy:

<?php
function fibonacci(): Generator
{
    [$a, $b] = [0, 1];
    while (true) {
        yield $a;
        [$a, $b] = [$b, $a + $b];
    }
}

function take(iterable $items, int $n): Generator
{
    if ($n <= 0) {
        return;
    }
    foreach ($items as $item) {
        yield $item;
        if (--$n === 0) {
            return;
        }
    }
}

function filter(iterable $items, callable $keep): Generator
{
    foreach ($items as $item) {
        if ($keep($item)) {
            yield $item;
        }
    }
}

echo implode(' ', iterator_to_array(take(fibonacci(), 10), false)), "\n";

$evenFibs = take(filter(fibonacci(), fn($x) => $x % 2 === 0), 6);
echo implode(' ', iterator_to_array($evenFibs, false)), "\n";

fibonacci() never ends on its own, yet the program finishes: take() stops asking after the requested count, and nothing beyond that is ever computed.

yield from: delegate to another generator

yield from passes through every value of an array, an iterator or another generator. It is how you split a generator into smaller ones, or walk a nested structure recursively:

<?php
function flatten(array $items): Generator
{
    foreach ($items as $item) {
        if (is_array($item)) {
            yield from flatten($item);   // recurse
        } else {
            yield $item;
        }
    }
}

$nested = [1, [2, 3, [4, 5]], 6, [[7]]];
foreach (flatten($nested) as $v) {
    echo $v, ' ';
}
echo "\n";

// Careful: keys from yield from can repeat
function letters(): Generator
{
    yield from ['a', 'b'];
    yield from ['c', 'd'];
}
print_r(iterator_to_array(letters()));         // keys 0,1,0,1: values overwritten
print_r(iterator_to_array(letters(), false));  // a plain list

yield from keeps the inner keys, so both arrays yield keys 0 and 1. iterator_to_array() then overwrites a and b. Pass false as the second argument whenever only the values matter.

send() and getReturn()

A generator can also receive values. $x = yield; pauses and waits; $gen->send($value) resumes the generator with $value as the result of that yield. And a generator can return a final value, which you read with getReturn() once it has finished.

<?php
function runningAverage(): Generator
{
    $total = 0;
    $count = 0;
    $average = null;
    while (true) {
        $value = yield $average;   // send() puts a value here
        if ($value === null) {
            return $count;         // final result
        }
        $total += $value;
        $count++;
        $average = $total / $count;
    }
}

$avg = runningAverage();
$avg->current();                   // run to the first yield

foreach ([10, 20, 60] as $n) {
    echo "after $n: ", $avg->send($n), "\n";
}

$avg->send(null);                  // ends the loop
echo "values seen: ", $avg->getReturn(), "\n";

Most code never needs send(); reading lazily with foreach is the everyday use. It is the building block behind coroutine libraries.

A generator can only be looped once

Generators run forward and cannot be rewound. Once a foreach has consumed one, a second loop over the same object throws:

<?php
// This throws an Exception on purpose: the generator is already finished
function colors(): Generator
{
    yield 'red';
    yield 'green';
}

$gen = colors();
foreach ($gen as $c) {
    echo $c, "\n";
}
foreach ($gen as $c) {   // second pass
    echo $c, "\n";
}

The fix is to call the generator function again for each pass, foreach (colors() as $c), or to collect the values once with iterator_to_array(colors(), false) when you really need them twice. If you find yourself doing that often, a plain array was the right choice for that data.