Streaming Large File Uploads to S3 with Node.js and Busboy: How to avoid Memory Leaks

16 min read 3,255 words PookieTech Team
Streaming Large File Uploads to S3 with Node.js and Busboy: How to avoid Memory Leaks

Streaming Large File Uploads to S3 with Node.js and Busboy: Avoiding Memory Leaks

You've seen it: a user uploads a large video, a hefty dataset, or a high-resolution image, and suddenly your Node.js server's memory usage spikes, then crashes. Or worse, it becomes unresponsive, leading to a cascade of service disruptions. The culprit? Buffering large files entirely in memory before processing or uploading them. This approach, while simple for smaller files, is a ticking time bomb for large file operations, especially when concurrent uploads are involved.

Consider a scenario where your service handles 10 concurrent 1GB file uploads. If each upload buffers the entire file in memory, your server instantly needs 10GB of available RAM just for the upload process, in addition to its usual operational memory. This is unsustainable, impractical, and a direct path to an Out-Of-Memory (OOM) error or severe performance degradation. For a production environment, this is simply not an option.

The solution lies in streaming. Instead of waiting for the entire file to arrive and sit in memory, we process it piece by piece, as it streams in from the client. This allows us to pipe the incoming data directly to its destination—in this case, Amazon S3—without ever holding the full file in our application's memory. This article details a robust strategy using Node.js, Busboy, and the AWS SDK to stream large multipart/form-data uploads directly to S3, with a sharp focus on preventing memory leaks and ensuring application stability.

The Pitfalls of Traditional File Uploads

Many Node.js file upload libraries, by default, buffer the entire file. Let's look at common patterns and why they fail for large files:

Buffering to Memory

Libraries like Multer, when configured with `memoryStorage`, will store the entire file in a Buffer object in RAM. This is convenient for small files where you might want to process the file content directly without disk I/O, but it's a non-starter for anything beyond a few megabytes.


const express = require('express');
const multer = require('multer');
const AWS = require('aws-sdk'); // Assume configured

const app = express();
const upload = multer({ storage: multer.memoryStorage() });
const s3 = new AWS.S3();

app.post('/upload-memory-unsafe', upload.single('file'), async (req, res) => {
    if (!req.file) {
        return res.status(400).send('No file uploaded.');
    }

    console.log(`Received file: ${req.file.originalname}, size: ${req.file.size / (1024 * 1024)} MB`);
    // At this point, req.file.buffer holds the entire file in memory.
    // For a 1GB file, this is 1GB of RAM consumed by this single request.

    const params = {
        Bucket: 'your-s3-bucket',
        Key: `uploads/${Date.now()}-${req.file.originalname}`,
        Body: req.file.buffer, // The entire file buffer
        ContentType: req.file.mimetype,
    };

    try {
        await s3.upload(params).promise();
        console.log('File uploaded to S3.');
        res.status(200).send('File uploaded successfully!');
    } catch (error) {
        console.error('S3 upload error:', error);
        res.status(500).send('Failed to upload file to S3.');
    }
});

// app.listen(3000, () => console.log('Server running on port 3000'));

This code is concise, but for a 500MB file, `req.file.buffer` will consume 500MB of your server's RAM. Ten concurrent uploads? 5GB RAM gone. This pattern is suitable only for very small files, typically under 10-20MB, depending on your server's resources.

Buffering to Disk First

Another common approach is to save the incoming file to a temporary location on disk and then read it from there to upload to S3. While this avoids memory spikes, it introduces other bottlenecks and potential issues:

  • Disk I/O Overhead: Writing a large file to disk and then immediately reading it back is inefficient. It doubles the I/O operations and adds latency.
  • Disk Space Requirements: Your server needs sufficient free disk space to temporarily store all concurrent large uploads. A burst of 10x 1GB uploads requires 10GB of temporary disk space.
  • Cleanup Logic: You need robust logic to clean up temporary files, especially in error scenarios or server crashes, to prevent disk exhaustion.
  • Performance: Disk I/O can be a bottleneck, particularly with slower storage or high concurrency.

const express = require('express');
const multer = require('multer');
const AWS = require('aws-sdk');
const fs = require('fs');
const path = require('path');

const app = express();
const uploadDir = path.join(__dirname, 'temp-uploads');
fs.mkdirSync(uploadDir, { recursive: true });

const upload = multer({ dest: uploadDir }); // Saves to disk

const s3 = new AWS.S3();

app.post('/upload-disk-inefficient', upload.single('file'), async (req, res) => {
    if (!req.file) {
        return res.status(400).send('No file uploaded.');
    }

    const filePath = req.file.path; // Path to the temporary file on disk
    const originalname = req.file.originalname;
    const mimetype = req.file.mimetype;

    console.log(`Received file on disk: ${originalname}, path: ${filePath}`);

    let fileStream;
    try {
        fileStream = fs.createReadStream(filePath);

        const params = {
            Bucket: 'your-s3-bucket',
            Key: `uploads/${Date.now()}-${originalname}`,
            Body: fileStream, // S3 SDK can accept a stream
            ContentType: mimetype,
        };

        await s3.upload(params).promise();
        console.log('File uploaded to S3.');
        res.status(200).send('File uploaded successfully!');
    } catch (error) {
        console.error('Upload error:', error);
        res.status(500).send('Failed to upload file.');
    } finally {
        // Crucial: Clean up the temporary file
        if (filePath) {
            fs.unlink(filePath, (err) => {
                if (err) console.error(`Error deleting temp file ${filePath}:`, err);
                else console.log(`Deleted temp file: ${filePath}`);
            });
        }
        if (fileStream) {
            fileStream.destroy(); // Ensure stream resources are released
        }
    }
});

// app.listen(3000, () => console.log('Server running on port 3000'));

While this approach avoids memory spikes, it's inefficient. You're waiting for the entire file to be written to disk before you can even *start* reading it for the S3 upload. This introduces unnecessary latency and disk I/O.

The key takeaway: Neither buffering to memory nor buffering to disk first is ideal for large file uploads. We need a way to process the data as it arrives, without intermediary storage.

Streaming with Busboy and AWS S3 ManagedUpload

The optimal solution for large file uploads in Node.js is to stream the incoming request body directly to S3. This requires two main components:

  1. A parser that can handle `multipart/form-data` requests and emit file parts as streams, without buffering the entire request. Busboy is excellent for this.
  2. An S3 client that can accept a Node.js readable stream as the upload body and handle the complexities of multipart uploads to S3 internally. The AWS SDK's `S3.ManagedUpload` (or `S3.upload` with a stream `Body`) is designed for this.

What is Busboy?

Busboy is a Node.js module for parsing incoming `multipart/form-data` and `application/x-www-form-urlencoded` request bodies. Unlike some other parsers, Busboy is event-driven and designed for streaming. It emits events for each form field and file, providing a readable stream for file data as it arrives. This means you get a stream of the file content *before* the entire file has been received by your server.

AWS SDK's ManagedUpload

The AWS SDK for JavaScript (v2 or v3) provides a powerful `S3.ManagedUpload` utility. When you pass a Node.js readable stream as the `Body` parameter to `s3.upload()`, the SDK intelligently handles the upload:

  • It automatically splits the stream into multiple parts.
  • It performs concurrent multipart uploads to S3, significantly speeding up large file transfers.
  • It handles retries for failed parts.
  • It manages memory by buffering only small chunks of the stream at a time, not the entire file.

This combination is a powerful pattern for high-performance, memory-efficient large file uploads.

Setting Up the Environment

First, ensure you have the necessary packages installed:


npm init -y
npm install express busboy aws-sdk dotenv

For AWS credentials, it's best practice to use environment variables or an IAM role for your EC2 instance/container. For local development, a `.env` file is common:


# .env
AWS_ACCESS_KEY_ID=YOUR_ACCESS_KEY
AWS_SECRET_ACCESS_KEY=YOUR_SECRET_KEY
AWS_REGION=your-aws-region # e.g., us-east-1
S3_BUCKET_NAME=your-s3-bucket-name

And load it in your app:


// server.js
require('dotenv').config(); // Load environment variables
const AWS = require('aws-sdk');

AWS.config.update({
    accessKeyId: process.env.AWS_ACCESS_KEY_ID,
    secretAccessKey: process.env.AWS_SECRET_ACCESS_KEY,
    region: process.env.AWS_REGION,
});

const s3 = new AWS.S3();

Core Implementation: Streaming to S3

Let's build a complete Express application that handles streamed uploads.

1. Basic Express Server and Busboy Setup

We'll create an Express route that uses Busboy to parse the incoming `multipart/form-data` request.


// server.js
const express = require('express');
const Busboy = require('busboy');
const AWS = require('aws-sdk');
require('dotenv').config();

// Configure AWS SDK
AWS.config.update({
    accessKeyId: process.env.AWS_ACCESS_KEY_ID,
    secretAccessKey: process.env.AWS_SECRET_ACCESS_KEY,
    region: process.env.AWS_REGION,
});
const s3 = new AWS.S3();
const S3_BUCKET_NAME = process.env.S3_BUCKET_NAME;

const app = express();
const PORT = process.env.PORT || 3000;

app.get('/', (req, res) => {
    res.send(`
        <!DOCTYPE html>
        <html>
        <head>
            <title>Upload Large File</title>
        </head>
        <body>
            <h1>Upload File to S3 (Streaming)</h1>
            <form action="/upload" method="POST" enctype="multipart/form-data">
                <input type="file" name="largeFile" />
                <input type="text" name="description" placeholder="File Description" />
                <button type="submit">Upload</button>
            </form>
            <p>Check server console for progress and S3 upload status.</p>
        </body>
        </html>
    `);
});

app.post('/upload', (req, res) => {
    // Ensure it's a multipart/form-data request
    if (!req.headers['content-type'] || !req.headers['content-type'].startsWith('multipart/form-data')) {
        return res.status(400).send('Content-Type must be multipart/form-data');
    }

    const busboy = Busboy({ headers: req.headers });
    let uploadFinished = false; // Flag to track if S3 upload initiated and completed
    let fileUploadPromise = null; // To hold the promise of the S3 upload
    let fileName = 'unknown'; // To store the file name for logging/response

    // Busboy 'file' event: A file stream has been detected
    busboy.on('file', (fieldname, fileStream, filename, encoding, mimetype) => {
        console.log(`Busboy 'file' event: Fieldname: ${fieldname}, Filename: ${filename}, MimeType: ${mimetype}`);
        fileName = filename; // Store filename

        const s3Key = `uploads/${Date.now()}-${filename}`;
        const params = {
            Bucket: S3_BUCKET_NAME,
            Key: s3Key,
            Body: fileStream, // DIRECTLY pipe the fileStream to S3
            ContentType: mimetype,
        };

        const upload = s3.upload(params, { queueSize: 4, partSize: 10 * 1024 * 1024 }); // Concurrency and part size
        
        // Track upload progress
        upload.on('httpUploadProgress', (progress) => {
            const uploadedMB = (progress.loaded / (1024 * 1024)).toFixed(2);
            const totalMB = (progress.total / (1024 * 1024)).toFixed(2);
            console.log(`Upload progress for ${filename}: ${uploadedMB}MB of ${totalMB}MB (${((progress.loaded / progress.total) * 100).toFixed(2)}%)`);
        });

        fileUploadPromise = upload.promise(); // Store the promise

        fileUploadPromise.then((data) => {
            console.log(`Successfully uploaded ${filename} to S3: ${data.Location}`);
            uploadFinished = true; // Mark S3 upload as completed successfully
        }).catch((err) => {
            console.error(`Error uploading ${filename} to S3:`, err);
            // Crucial: If S3 upload fails, we need to ensure the request stream is consumed
            // or the connection is terminated to prevent hanging.
            // Busboy will continue to read the request body, but we might want to stop it.
            // However, busboy.destroy() might not be ideal here if other fields are expected.
            // The `req.pipe(busboy)` handles consuming the stream.
            // We'll handle the response for the client below.
        });
    });

    // Busboy 'field' event: For non-file fields
    busboy.on('field', (fieldname, val, fieldnameTruncated, valTruncated, encoding, mimetype) => {
        console.log(`Busboy 'field' event: Fieldname: ${fieldname}, Value: ${val}`);
        // You can store these field values if needed
    });

    // Busboy 'finish' event: All parts have been parsed and handled by Busboy
    busboy.on('finish', async () => {
        console.log('Busboy finished parsing request.');
        if (fileUploadPromise) {
            try {
                await fileUploadPromise; // Wait for the S3 upload to complete
                res.status(200).send(`File '${fileName}' uploaded successfully to S3.`);
            } catch (err) {
                res.status(500).send(`Failed to upload file '${fileName}' to S3: ${err.message}`);
            }
        } else {
            // No file was uploaded, or an unexpected scenario
            res.status(400).send('No file found in the upload request.');
        }
    });

    // Handle request errors or client disconnects
    req.on('close', () => {
        if (!uploadFinished && fileUploadPromise) {
            console.warn(`Client disconnected during upload of ${fileName}. Attempting to abort S3 upload.`);
            // S3.ManagedUpload doesn't have a direct 'abort' method on the promise,
            // but if the underlying stream is destroyed, it will eventually fail.
            // The important part is to ensure Node.js resources are released.
            // fileStream.destroy() would be key here, but we directly pipe req.pipe(busboy) to S3.
            // The S3 SDK will handle internal cleanup if the Body stream ends prematurely.
        }
        console.log('Request connection closed.');
    });

    req.on('error', (err) => {
        console.error('Request error:', err);
        busboy.destroy(err); // Destroy busboy to stop processing and release resources
    });

    // Pipe the incoming request stream to Busboy
    req.pipe(busboy);
});

app.listen(PORT, () => {
    console.log(`Server running on port ${PORT}`);
    console.log(`Open http://localhost:${PORT} in your browser to upload files.`);
});

Explanation of the Streaming Logic:

  1. `req.pipe(busboy)`: This is the core of the streaming. Instead of reading the entire request body into memory, we pipe the incoming HTTP request stream directly to Busboy. As chunks of data arrive, Busboy processes them.
  2. `busboy.on('file', ...)`: When Busboy detects a file part, it emits a `'file'` event. The `fileStream` argument provided in this callback is a Node.js readable stream representing the content of that specific file.
  3. `Body: fileStream` for S3: Critically, we pass this `fileStream` directly to `s3.upload()` as the `Body` parameter. The AWS SDK then takes this stream and manages the multipart upload to S3. Your Node.js server never holds the entire file in memory.
  4. `upload.on('httpUploadProgress', ...)`: This event allows you to track the progress of the S3 upload, which is useful for logging or providing feedback to the client (e.g., via WebSockets).
  5. `busboy.on('finish', ...)`: This event fires when Busboy has finished parsing the entire request body. At this point, all file streams have been piped to S3. We then wait for the `fileUploadPromise` to resolve, ensuring the S3 upload is truly complete before responding to the client.

Memory Leak Prevention and Robustness

While streaming inherently reduces memory footprint, proper error handling and resource management are crucial to prevent subtle memory leaks and ensure stability under adverse conditions.

1. Stream Destruction and Error Handling

Streams, like any resource, need to be properly closed or destroyed when errors occur or when they are no longer needed. In our setup:

  • `req.on('error')`: If the incoming HTTP request itself encounters an error, we should destroy Busboy to stop processing and release any associated resources.
  • `req.on('close')`: This event fires when the client connection is closed, regardless of whether the request completed successfully. If an upload is in progress and the client disconnects, the `fileStream` that was piped to S3 will naturally end prematurely. The `S3.ManagedUpload` will detect this and eventually fail. While there isn't a direct `abort()` method on the `ManagedUpload` promise, stopping the incoming data stream (by the client disconnecting) is usually sufficient for the SDK to clean up its internal resources. For Busboy, `req.pipe(busboy)` ensures that if `req` closes, Busboy will also eventually stop receiving data.
  • S3 Upload Errors: If `s3.upload().promise()` rejects, it means the S3 upload failed. We catch this error and respond appropriately to the client. The SDK is designed to clean up its internal resources upon upload failure.

2. Backpressure

Node.js streams inherently handle backpressure. When a writable stream (like the S3 SDK's internal stream buffer) cannot keep up with the data rate from a readable stream (like `fileStream` from Busboy), it signals the readable stream to pause. This prevents the writable stream from buffering too much data in memory. Busboy's `fileStream` respects backpressure from the S3 SDK, and `req.pipe(busboy)` handles backpressure from Busboy. This chain of backpressure propagation is what keeps your memory usage low.

3. Managing Concurrent Uploads

The `S3.ManagedUpload` constructor accepts an options object, including `queueSize` and `partSize`. These parameters allow you to fine-tune the concurrency and chunking behavior for S3 multipart uploads:

  • `queueSize` (default: 4): The maximum number of concurrent part uploads to S3. Increasing this can speed up uploads on high-bandwidth connections but consumes more network resources.
  • `partSize` (default: 5MB): The size of each part for multipart uploads. For very large files (e.g., >100GB), you might increase this to reduce the number of parts and API calls, but the minimum part size is 5MB.

These settings are per-file. Your server's total concurrency for handling multiple incoming `req` streams will be limited by its CPU, memory, and network I/O, but the streaming approach ensures memory scales gracefully.

Comparison Table: Upload Strategies

Let's summarize the trade-offs of different approaches:

Feature Multer (Memory Storage) Multer (Disk Storage) Busboy + S3 Stream
Memory Usage (per file) High (buffers entire file) Low (buffers small chunks during read/write) Very Low (streams directly)
Disk I/O None (direct memory to S3) High (write to disk, then read from disk) None (direct stream to S3)
Latency for Large Files High (wait for full file, then upload) Very High (wait for full write, then wait for full read/upload) Low (upload starts immediately)
Cleanup Logic Minimal (memory managed by GC) Complex (manual temp file deletion, error handling) Minimal (streams handle themselves, S3 SDK cleans up)
Scalability (Concurrent Large Files) Poor (OOM crashes likely) Moderate (disk I/O and space are bottlenecks) Excellent (memory-efficient, leverages S3 multipart)
Complexity Low (simple API) Moderate (needs disk cleanup) Moderate (event-driven, stream handling)
Best Use Case Very small files (<10MB) for quick in-memory processing. Small to medium files (<100MB) where disk buffering is acceptable. Large files (>100MB) or high concurrency where memory efficiency and performance are critical.

Advanced Considerations and Best Practices

File Size Limits

Busboy itself allows you to set limits on file size and number of files. This is a good first line of defense against malicious uploads or accidental large files:


const busboy = Busboy({
    headers: req.headers,
    limits: {
        fileSize: 10 * 1024 * 1024 * 1024, // 10 GB maximum file size
        files: 1, // Only allow one file per request
    }
});

busboy.on('file', (fieldname, fileStream, filename, encoding, mimetype) => {
    // ... S3 upload logic ...
    fileStream.on('limit', () => {
        console.error(`File ${filename} exceeded size limit!`);
        // Destroy the stream to stop processing and prevent further data
        fileStream.destroy();
        // You might want to respond with an error here, but Busboy will eventually emit 'error'
        // or 'finish' with an incomplete upload if the stream is destroyed.
        // It's better to handle the error when the S3 upload promise rejects.
    });
});

Authentication and Authorization

Before you even instantiate Busboy, ensure the user is authenticated and authorized to perform the upload. This prevents unauthorized users from consuming your server resources by initiating large uploads.


app.post('/upload', isAuthenticated, isAuthorizedToUpload, (req, res) => {
    // ... Busboy and S3 logic ...
});

function isAuthenticated(req, res, next) {
    // Implement your authentication logic (e.g., check JWT token)
    if (req.user) { // Assuming req.user is populated by auth middleware
        next();
    } else {
        res.status(401).send('Unauthorized');
    }
}

function isAuthorizedToUpload(req, res, next) {
    // Implement your authorization logic (e.g., check user roles, file type limits)
    if (req.user.canUploadLargeFiles) {
        next();
    } else {
        res.status(403).send('Forbidden: Not authorized to upload files.');
    }
}

Error Response Handling

When an error occurs during the Busboy parsing or S3 upload, it's critical to send a timely and informative error response to the client. This might involve setting a timeout on the request to prevent it from hanging indefinitely if an unexpected error prevents the `busboy.on('finish')` or `fileUploadPromise` from resolving.

Using AWS SDK v3 (@aws-sdk/client-s3)

While the example uses `aws-sdk` (v2), the concept is identical with `@aws-sdk/client-s3` (v3). The main difference is the import and how S3 client is instantiated:


// Using AWS SDK v3
const { S3Client, PutObjectCommand } = require('@aws-sdk/client-s3');

const s3Client = new S3Client({
    region: process.env.AWS_REGION,
    credentials: {
        accessKeyId: process.env.AWS_ACCESS_KEY_ID,
        secretAccessKey: process.env.AWS_SECRET_ACCESS_KEY,
    },
});

// To upload with v3, you'd use PutObjectCommand
Back to Blog
Share