How To Upload Large Files

Uploading large files is a constant headache for developers. Pains like latency, speed, timeouts and interruptions, especially over mobile devices, are difficult to avoid. However, over the years, it is becoming increasingly important that your application be able to handle large files.

At Filestack, we provide file infrastructure for over 100,000 applications worldwide, handling file uploads, transformations, storage, and delivery. We process millions of files of all sizes and formats every year. At one point, we noticed all file sizes increase about 50% year over year, with one exception: files sized 100MB and above. These large files increased 170% year over year, a sizable increase over their smaller counterparts.

 

This leads us to conclude that file sizes are trending larger. So how can you upload large files on your application?

Choose Your Cloud Storage Service Wisely ?

Not all cloud storage services have the same file size limits. When you choose where to store your files, make sure that their size limits don’t just work for your application now, but that it will continue to work in the long term, as file sizes continue to trend upwards. This is particularly important when handling high resolution images, audio/video files, or other known large file types.

These are the file size limits for popular cloud storage providers:

Cloud Storage Provider	File Size Limit
Amazon S3	5TB
Google Cloud Storage	5TB
Azure	1TB
Rackspace	5GB
Dropbox	20GB

 

Of course, there is more to choosing your cloud storage provider than just file upload size. See our blog post that compares cloud storage providers for more details.

There are also some best practices to keep in mind, no matter what storage services you use:

Tips & Tricks for Uploading Large Files

Store Direct to Cloud Storage

The fewer stops your file has to make en route to its final destination, the faster it will get there. If your file is roaming all over the world, being processed in different locations, it will naturally upload slower. For top speed, upload your file directly to your cloud storage account, whether that is AWS, Azure, Dropbox, or elsewhere.

Use Regional APIs and S3 Buckets

Going direct to cloud storage might still mean you are sending your data halfway around the globe. Make sure your uploads are storing to the nearest bucket of your cloud provider, not traveling transcontinentally. The further your file has to go, the slower the upload will be, and the more chance for errors to occur.

Upload in Chunks

Uploading in chunks breaks apart your larger files into smaller, more manageable pieces and periodically, query uploads API for the upload status. It also ensures that if there is an issue with one chunk, the upload will be able to resume where it left off, instead of starting from scratch. This is imperative for large files, especially when being uploaded from mobile devices where connection might be spotty.

Asynchronous Uploads

Synchronous uploads are error prone and sensitive to network conditions and timeouts. Especially if you store your files somewhere other than S3, there is a large risk of timeouts. Uploading asynchronously allows you to upload files in the background, while your application continues to function.

Filestack’s API for Uploading Large Files

To summarize, to upload large files, you should:

1. Choose the best cloud storage service for your needs (likely Amazon Web Services or Google Cloud Storage).
2. Break large files into smaller chunks
3. Ensure your file takes the shortest path to your cloud storage by relying on a Content Ingestion Network
4. Upload asynchronously to mitigate disruptions to your user experience.

Building this file uploading infrastructure yourself is difficult and time consuming. The simplest way to upload large files is to use an API that manages the process for you, ensuring speed, reliability, and security. Filestack is a file uploading API that you can get going quickly and easily.

Through Filestack’s new uploading API, you can chunk files and upload asynchronously, eliminating the risk of timeouts between client and server. Also, to make it possible for all developers to utilize regional storage without having to build your own extensive infrastructure, we recently released the Filestack Content Ingestion Network (CIN), which anyone can use to increase international file upload speeds up to 10x with the click of a button. Additionally, you can choose to upload files direct to S3 storage, without passing through any other infrastructure, ensuring speed and secure file upload best practices.

Looking for guides on how to use Filestack for your tech stack? Check out these guides on how to handle a PHP file upload, jQuery file upload, or AJAX file upload using Filestack.

Filestack’s team of 20+ dedicated engineers work hard to ensure the speed and safety of your file uploads. We are constantly researching, learning, and improving our file infrastructure to ensure that your application only uses the best. Reach out to us:

Contact Sales

Read More →