Streaming and large files

parser.stream(readable) takes any Readable and returns an async iterator of records. Lines are assembled from the chunks that arrive, so a chunk boundary in the middle of a line breaks nothing.

import { createReadStream } from 'node:fs';
for await (const rec of parser.stream(createReadStream('big.log'))) {
  // rec is already parsed
}

Backpressure#

While you handle a record inside for await, reading from the source is paused. You never call pause() and resume() by hand. If handling is slow, for example writing to a database, the buffer does not grow.

Compressed files and stdin#

Sverlo does not decompress data itself. Put a decompressor into the chain:

import { createGunzip } from 'node:zlib';
const src = createReadStream('access.log.gz').pipe(createGunzip());
for await (const rec of parser.stream(src)) { /* ... */ }

For standard input pass process.stdin.

Buffer settings#

Stopping#

break inside the loop closes the source. To stop reading on a timer pass a signal:

const ctl = new AbortController();
setTimeout(() => ctl.abort(), 5000);
for await (const rec of parser.stream(src, { signal: ctl.signal })) { /* ... */ }

On abort the iterator ends without an exception and parser.stats.aborted becomes true.