Node.js Streams and Buffers Interview Questions

A Buffer is a fixed-length chunk of raw bytes (a Uint8Array subclass) for binary data. Strings are text, buffers are bytes, and an encoding converts between them.

const buf = Buffer.from('Hi Node', 'utf8');

console.log(buf.length);
console.log(buf.toString('hex'));
console.log(buf.toString('base64'));
console.log(Buffer.from('SGkgTm9kZQ==', 'base64').toString('utf8'));
console.log(buf[0], String.fromCharCode(buf[0]));

new Buffer() is deprecated; use Buffer.from or Buffer.alloc.

length counts UTF-16 code units; a buffer counts bytes, and UTF-8 é takes two. It matters for Content-Length headers and byte limits.

Each element is one unsigned byte: values are truncated, then wrapped modulo 256, so -1 becomes 255 and 3.9 becomes 3, with no error. buf.writeUInt8(-1) throws a RangeError instead.

subarray returns a view over the same memory, so writing through part changes original; Buffer.from(buffer) copies. A tiny subarray also keeps the whole large buffer alive.

Buffer.alloc(n) returns n zeroed bytes. Buffer.allocUnsafe(n) skips the zeroing, so it is faster, but may hold old data, including secrets from other requests. Use it only when you overwrite every byte at once.

A stream processes data piece by piece instead of loading it all into memory. The four types: Readable (fs.createReadStream, an HTTP request), Writable (an HTTP response), Duplex (a TCP socket) and Transform, a Duplex that changes the data (zlib.createGzip()).

const { Readable, Transform } = require('node:stream');
const { pipeline } = require('node:stream/promises');

let calls = 0;
const upper = new Transform({
  transform(chunk, encoding, callback) {
    calls++;
    callback(null, chunk.toString().toUpperCase());
  },
});

async function main() {
  const out = [];
  await pipeline(
    Readable.from(['stream ', 'data ', 'in chunks']),
    upper,
    async function (source) {
      for await (const chunk of source) out.push(chunk.toString());
    },
  );
  console.log(out.join(''));
  console.log('transform calls:', calls);
}
main();

Readable streams are async iterables, so for await is the simplest way to consume one.

Memory and time to first byte. readFile loads the whole file first, so a 2 GB file needs 2 GB of RAM per download. A stream sends 64 KiB chunks as they are read, so memory stays flat.

const fs = require('node:fs');
const http = require('node:http');
const { pipeline } = require('node:stream/promises');

http.createServer(async (req, res) => {
  res.setHeader('Content-Type', 'video/mp4');
  try {
    await pipeline(fs.createReadStream('movie.mp4'), res);
  } catch (err) {
    console.error('download failed:', err.code);
  }
}).listen(3000);

pipeline also closes the file stream if the client disconnects.

Backpressure is a producer outrunning its consumer. A writable returns false from write() once its buffer passes highWaterMark (64 KiB by default in Node 22 and later); the producer should then wait for 'drain'.

function writeLines(stream, lines, done) {
  let i = 0;
  function writeSome() {
    while (i < lines.length) {
      const ok = stream.write(lines[i++] + '\n');
      if (!ok) return stream.once('drain', writeSome); // buffer full, wait
    }
    done();
  }
  writeSome();
}

pipe() and pipeline() do this for you. The bug to spot: a write() loop that ignores the return value runs out of memory in production.

Both connect streams with backpressure, but pipe() does not forward errors: a failed source leaves the destination open, and an unhandled 'error' crashes the process. pipeline() destroys every stream when one fails and gives you one place to catch.

// leaks the file stream on failure, and a gzip error crashes the process
source.pipe(zlib.createGzip()).pipe(fs.createWriteStream('out.gz'));

// one try/catch, everything cleaned up
await pipeline(source, zlib.createGzip(), fs.createWriteStream('out.gz'));

Use readline on a read stream: it is an async iterable and holds only the current chunk.

const fs = require('node:fs');
const readline = require('node:readline');

fs.writeFileSync('access.log', 'GET /home 200\nGET /cart 500\nPOST /pay 200\n');

async function countErrors(file) {
  const rl = readline.createInterface({
    input: fs.createReadStream(file),
    crlfDelay: Infinity, // treat \r\n as a single line break
  });
  let lines = 0;
  let errors = 0;
  for await (const line of rl) {
    lines++;
    if (line.endsWith(' 500')) errors++;
  }
  return { lines, errors };
}

countErrors('access.log').then(({ lines, errors }) => {
  console.log(`${errors} of ${lines} requests failed`);
  fs.unlinkSync('access.log');
});

It reads a 10 GB log in kilobytes of memory, unlike readFile plus split('\n').