Efficiently Uploading Large Files with S3 Multipart Upload

What is S3 Multipart Upload?

S3 multipart upload is an efficient way to upload large files by dividing them into smaller parts and uploading each part independently. This approach provides several benefits, including improved throughput, quick recovery from network issues, and the ability to pause and resume uploads.

How Does S3 Multipart Upload Work?

The process of S3 multipart upload involves the following steps:

1. Splitting an object into many parts: Divide the file into smaller parts, typically 5MB each.
2. Initiating the multipart upload process: Send a request to S3 to initiate the multipart upload process.
3. Uploading each part: Upload each part of the file using a pre-signed URL.
4. Completing the multipart upload process: Send a request to S3 to complete the multipart upload process.

Benefits of Using S3 Pre-Signed URLs

Using S3 pre-signed URLs provides several benefits, including:

• Security: Pre-signed URLs allow you to grant temporary access to your S3 bucket without exposing your credentials.
• Efficiency: Pre-signed URLs enable parallel uploads, which can significantly improve upload speeds.
• Flexibility: Pre-signed URLs can be used to upload files of any size, making them ideal for large file uploads.

Implementing S3 Multipart Upload with Node.js and React

Implementing S3 multipart upload with Node.js and React involves creating a server-side API to handle the multipart upload process and a client-side utility class to handle file splitting and parallel uploads.

Server-Side API with Node.js

The server-side API consists of three endpoints:

1. Initiate Multipart Upload: Handles the initiation of the multipart upload process.
2. Get Pre-Signed URL: Returns a pre-signed URL for each part of the file.
3. Complete Multipart Upload: Handles the completion of the multipart upload process.

const express = require('express');
const app = express();
const AWS = require('aws-sdk');

const s3 = new AWS.S3({
  region: 'your-region',
  accessKeyId: 'your-access-key-id',
  secretAccessKey: 'your-secret-access-key'
});

app.post('/initiate-multipart-upload', (req, res) => {
  const params = {
    Bucket: 'your-bucket-name',
    Key: 'your-object-key',
    ContentType: 'your-content-type'
  };

  s3.createMultipartUpload(params, (err, data) => {
    if (err) {
      console.log(err);
      res.status(500).send({ message: 'Error initiating multipart upload' });
    } else {
      res.send(data);
    }
  });
});

app.get('/get-pre-signed-url', (req, res) => {
  const params = {
    Bucket: 'your-bucket-name',
    Key: 'your-object-key',
    PartNumber: req.query.partNumber
  };

  s3.getSignedUrl('uploadPart', params, (err, data) => {
    if (err) {
      console.log(err);
      res.status(500).send({ message: 'Error getting pre-signed URL' });
    } else {
      res.send(data);
    }
  });
});

app.post('/complete-multipart-upload', (req, res) => {
  const params = {
    Bucket: 'your-bucket-name',
    Key: 'your-object-key',
    MultipartUpload: {
      Parts: req.body.parts
    }
  };

  s3.completeMultipartUpload(params, (err, data) => {
    if (err) {
      console.log(err);
      res.status(500).send({ message: 'Error completing multipart upload' });
    } else {
      res.send(data);
    }
  });
});

Client-Side Utility Class with React

The client-side utility class handles file splitting and parallel uploads using the pre-signed URLs obtained from the server-side API. The utility class uses Axios to send requests to the server-side API and handle errors.

import axios from 'axios';

class UploadUtility {
  async initiateMultipartUpload(file) {
    const response = await axios.post('/initiate-multipart-upload', {
      fileName: file.name,
      fileSize: file.size
    });

    return response.data;
  }

  async getPreSignedUrl(partNumber) {
    const response = await axios.get(`/get-pre-signed-url?partNumber=${partNumber}`);

    return response.data;
  }

  async completeMultipartUpload(parts) {
    const response = await axios.post('/complete-multipart-upload', {
      parts
    });

    return response.data;
  }

  async uploadFile(file) {
    const uploadId = await this.initiateMultipartUpload(file);

    const parts = [];

    for (let i = 0; i < file.size; i += 5 * 1024 * 1024) {
      const chunk = file.slice(i, Math.min(i + 5 * 1024 * 1024, file.size));
      const partNumber = i / (5 * 1024 * 1024) + 1;

      const preSignedUrl = await this.getPreSignedUrl(partNumber);

      await axios.put(preSignedUrl, chunk, {
        headers: {
          'Content-Type': file.type
        }
      });

      parts.push({
        ETag: '',
        PartNumber: partNumber
      });
    }

    await this.completeMultipartUpload(parts);

    return uploadId;
  }
}

export default UploadUtility;