# Node.js Streams: Processing 10MB CSVs Without Crashing
By Govind Bajaj · 2025

Tags: nodejs, streams, csv, performance, backend
fs.readFile on a 10MB CSV consumed 847MB of RAM and crashed the container. Streams processed the same file in 18MB. Here's the code.
The admin dashboard for Sans Herbals has a bulk product import feature.

A merchant uploaded a 10MB CSV with 50,000 product rows. The server called `fs.readFile`, parsed the entire file into memory, then ran `csv-parse` on the resulting string. Memory usage spiked to 847MB. The container's memory limit was 512MB. The process crashed.

The fix was Node.js streams. Same file, same processing logic, 18MB peak memory usage. The difference between reading a file and *streaming* a file.

## Why readFile Fails at Scale

`fs.readFile` loads the entire file into memory as a single buffer. For a 10MB file, that's 10MB for the buffer, plus the parsed representation, plus intermediate objects. The garbage collector can't free anything until the entire operation completes.

```js
// This loads 10MB into memory, then parses it all at once
const fs = require('fs')
const { parse } = require('csv-parse/sync')

const file = fs.readFileSync('./products.csv', 'utf8')
const records = parse(file, { columns: true }) // another 50MB for parsed data
// Now process all 50,000 records simultaneously
```

With large files, this pattern inevitably hits memory limits. The solution is processing data in chunks, not all at once.

## Streams: Process One Chunk at a Time

Node.js streams read data in small chunks (typically 64KB) and process each chunk as it arrives. Memory usage stays flat regardless of file size.

```js
const fs = require('fs')
const { parse } = require('csv-parse')

// Stream: process one row at a time, constant memory
fs.createReadStream('./products.csv')
  .pipe(parse({ columns: true }))
  .on('data', (row) => {
    // Process one row (one product)
    console.log(row.name, row.price)
  })
  .on('end', () => {
    console.log('Done')
  })
  .on('error', (err) => {
    console.error('Error:', err.message)
  })
```

The `data` event fires once per CSV row. At any given moment, only one row is in memory. The stream backpressures if processing slows down — it stops reading from the file until the consumer is ready.

## Adding Async Processing with Transform Streams

For real-world imports, you need to validate rows, write to the database, and handle errors per row. Transform streams are the clean way to do this:

```js
const { Transform } = require('stream')
const { pipeline } = require('stream/promises')

// Transform stream: validate and save each product
const productImporter = new Transform({
  objectMode: true,
  transform: async (row, encoding, callback) => {
    try {
      // Validate
      if (!row.name || !row.price) {
        callback(null, { status: 'skipped', reason: 'Missing name or price', row })
        return
      }
      
      // Save to database
      const product = await Product.create({
        name: row.name,
        price: parseFloat(row.price),
        category: row.category,
        inventory: parseInt(row.inventory) || 0
      })
      
      callback(null, { status: 'success', id: product._id })
    } catch (err) {
      callback(null, { status: 'error', reason: err.message, row })
    }
  }
})

// Pipeline: readable → transform → writable
async function importProducts(filePath) {
  const results = []
  
  await pipeline(
    fs.createReadStream(filePath),
    parse({ columns: true }),
    productImporter,
    new Writable({
      objectMode: true,
      write: (result, encoding, callback) => {
        results.push(result)
        callback()
      }
    })
  )
  
  return results
}
```

Each row is validated, saved, and the result collected — one at a time. The database receives one insert per event loop tick, not 50,000 at once.

## Handling Backpressure

Backpressure is the most important stream concept for production. If your database write is slower than your file read, data buffers in memory. Without backpressure handling, you get the same memory problem you're trying to avoid.

The `pipeline()` function handles backpressure automatically. If the transform stream is busy, it pauses the readable stream. Use `pipeline` instead of manual `.pipe()` chains — it also cleans up properly on errors.

## Progress Tracking for Large Files

For a 50,000 row import, you need progress tracking. A simple counter in the transform stream works:

```js
const { Transform } = require('stream')

function createProgressTracker(totalRows, onProgress) {
  let processed = 0
  let success = 0
  let failed = 0
  
  return new Transform({
    objectMode: true,
    transform: async (row, encoding, callback) => {
      processed++
      
      try {
        await processRow(row)
        success++
      } catch (err) {
        failed++
      }
      
      // Report progress every 1000 rows
      if (processed % 1000 === 0) {
        onProgress({ processed, total: totalRows, success, failed })
      }
      
      callback()
    }
  })
}
```

For the Sans Herbals bulk import, I emit progress events via WebSocket so the admin sees a live progress bar.

## Takeaways

- `fs.readFile` loads the entire file into memory — it crashes on large files
- Streams process data in chunks, keeping memory usage flat regardless of file size
- Use `pipeline()` instead of manual `.pipe()` — it handles backpressure and cleanup
- Transform streams are the clean pattern for per-row async processing
- Backpressure handling prevents memory buildup when the consumer is slower than the producer
- Add progress tracking for any operation over 1000 rows — silent long-running operations confuse users