Filestack Intelligent Ingestion: Secure & Reliable File Uploads

Filestack’s heritage is pinned deeply to being the world’s best file uploading service. Over the past 5+ years, we’ve had thousands of conversations with customers and prospects about the pains they encounter with file uploading. A common theme: mobile uploads have unique challenges. From maintaining state between cell towers, to ballooning file sizes, to little connectivity in areas with a lack of technological infrastructure, as user-generated content (UGC) creation is at an all time high, so too are the problems our customers face.” While telecom providers work to maintain and expand networks, the truth is, it is expensive and often-times slow. Factors such as population density, capital, and legislation can all create blockers in the process of delivering network access to the masses. With all this in mind, we built Filestack Intelligent Ingestion to solve for all the aforementioned issues. Business does not stop, UGC is not showing any signs of slowing down, and new services are stretching to reach audiences at every edge of the world.

Failed uploads are not only inconvenient to the user, they can also lead to lost business when the product relies on successful delivery of a file each and every time. Dorian Collier, JibJab Head of Product, says “We have thousands of users uploading files on our platform every day. Upload failures can decrease conversion by creating a poor user experience. We’re thrilled to see Filestack tackling this challenge to ensure an uninterrupted experience for our users.”

Goodbye Upload Failures, Hello 99.999% Success Rate

Filestack Intelligent Ingestion (FII) is a breakthrough in uploading technology, expanding our leadership position in the Upload API space. A homegrown service, FII intelligently detects network conditions, device type, and browser, then makes automated decisions on how best chunk a file to ensure it uploads successfully. Chunking uploads is nothing new, many services in use today use chunking on the backend, but they are largely constrained by minimum file sizes (e.g., 5MB recommended by AWS S3) and do not handle network latency spikes well.   Most services, like S3, attempt to resolve this by implementing retry logic, but even those can ultimately fail because they have no intelligence in their retry logic to change chunk sizes during data transfer interruption.

Talk is cheap, so we want to put our money where our mouth is. We ran a series of simulated tests of various file sizes, in varying network conditions, against hosted services and do-it-yourself solutions. Find our results below:

Test A:  Poor Network Quality

This test is intended to simulate rural areas where network performance can be spotty. Mobile networks, faulty hardware, and low bandwidth connections are the most common problems that make this scenario real. In our test, we simulated packet loss at random, ranging between 0 to 60 percent at any given time.

Poor Network Quality

Poor Quality Network Benchmark Data
Poor Quality Network Benchmark Data

Test B: Quality of Service Degradation

This test is intended to simulate network conditions in hotels, trains, coffee shops … anywhere were QoS policies may exist or a high number of users are consuming data and network congestion can cause variance in network performance.

Quality of Service Degradation

Quality of Service Degradation Benchmark Data
Quality of Service Degradation Benchmark Data

Reliability Meets Speed

Filestack Intelligent Ingestion is additive to the Filestack Content Ingestion Network (CIN), which optimizes upload speed for customers in areas that face challenging network conditions, such as mountainous terrain, high network congestion, or limited bandwidth infrastructure. We accomplished this by populating the globe with ingestion points-of-presence, which act like a reverse CDN for uploads. As soon as a file is uploaded into the Filestack CIN, you are returned with a FileLink that you can immediate act on, whether that is performing a file transformation like crop or resize, compressing a file or adding a custom watermark among many other transformations. If your users are in high bandwidth locales, or close to major cloud storage hubs, you may not see significant gains in speed. However, you will see significant speed and reliability gains for users that are further away from these services or have limited or timed access.

What Are You Waiting For?

Signing up for Filestack is free, and integrating Filestack Intelligent ingestion is as simple as adding a few lines of code to your application. Get started with one of our officially supported file upload SDKs today and you’ll never have to worry about your file uploads failing again!

If you have any additional questions about how Filestack Intelligent Ingestion works or could help your business, please reach out to us via email or Twitter!

Read More →