Uploading Multiple Files with Concurrency Limits, Queues and Partial Failure

It’s demo day. The ticket read: “let users upload a whole folder of photos.” Someone selects 40 files. Six of them start uploading. A progress bar climbs smoothly to 90%… and then just sits there, lying. Three files failed somewhere in the middle, silently. The user, unsure if anything worked, selects all 40 files again, and now you have duplicates, wasted bandwidth, and a support ticket.

Uploading multiple files looks like a for-loop problem. It isn’t; it’s a scheduling and failure-recovery problem that happens to run inside a browser tab.

In short: Uploading multiple files reliably requires three things a single-file upload never needs: a concurrency limit (typically 3-6 parallel uploads per browser origin), an upload queue that feeds files through that limit, and per-file failure handling so one failed upload never aborts the batch. Managed services like Filestack implement all three: queuing, parallelism, automatic per-file retries, and per-file progress, through a single picker or SDK call.

This guide walks through what actually happens when someone selects a batch of files, why naive approaches break past a handful of files, how to build a proper queue with bounded concurrency, and how to design for partial failure instead of pretending it won’t happen.

Key Takeaways

• Browsers cap concurrent connections per origin (around 6 on HTTP/1.1), so unbounded Promise.all() calls don’t actually run 50 uploads in parallel; they just queue silently and badly.
• A reliable multi-file upload needs a queue (scheduler) feeding a bounded concurrency pool, not a flat array of promises.
• Each file should be modeled as its own state machine, queued → uploading → succeeded/failed → retrying → dead-lettered, instead of one global “uploading” flag.
• Partial failure is the normal case at scale, not the exception. One failed file should never cancel the other 49.
• Tools like the Filestack file uploader handle queuing, parallelism, retries, and per-file progress by default, which is worth knowing about once you’ve seen how much plumbing the DIY version needs.

Before fixing anything, it helps to see exactly where the naive approaches break, and why “just loop over the files” isn’t the whole story.

What Actually Happens When You Select 50 Files

The two instinctive first attempts both fail in different ways once file counts grow past a handful.

Sequential upload (await in a loop) is the safest-looking option and the slowest. Each file waits for the previous one to fully finish before starting. It’s predictable, but a batch of 50 files at a few seconds each turns into minutes of dead time, and if file #12 throws, the loop can halt the entire remaining batch depending on how the error is handled.

Promise.all() over every file looks like the “parallel” fix, and it’s the one most developers reach for:

// The broken version — looks parallel, isn't safe

async function uploadAll(files) {

const uploads = files.map(file => uploadFile(file));

return Promise.all(uploads); // fails ALL on ANY single rejection

}

This breaks in two ways at once. First, the browser doesn’t actually run 50 simultaneous uploads; more on that below. Second, Promise.all() rejects as soon as any single promise rejects, discarding the results of every upload that had already succeeded. One flaky file on a bad connection can make your code report a full-batch failure when 49 files landed just fine.

If you came here searching for the best methods to upload multiple files at once in a web application, or wondering how to implement multiple file upload in HTML and JavaScript, this is the part worth internalising first: the goal isn’t “more parallelism,” it’s bounded, observable parallelism.

So why doesn’t Promise.all() give you real parallelism in the first place? It comes down to a limit set by the browser itself, not your code.

The Browser’s Connection Limit

Browsers cap the number of concurrent HTTP/1.1 connections to a single origin at roughly six. Call uploadFile() fifty times and the browser doesn’t open fifty sockets; it opens six, and quietly queues the rest at the network layer, invisible to your application code and your progress UI.

HTTP/2’s multiplexing changes the transport detail but not the practical ceiling: a single connection can carry many streams, but the server, your bandwidth, and the client CPU still throttle real throughput. More in-flight requests than the connection can usefully serve just adds contention, including with other requests the page needs to make (auth refreshes, analytics, the next page section loading).

The fix isn’t to fight this limit. It’s to make it explicit in your own code, instead of leaving it to the browser to enforce silently.

Now that we know where the ceiling comes from, the next question is a practical one: what number should your code actually target?

Concurrency Limits: How Many Uploads at Once?

In practice, 3-6 parallel uploads is the sweet spot for most browser-based batch uploads. Pushing concurrency past what the browser’s connection limit allows doesn’t buy you extra throughput; it just adds queuing delay at a layer you can’t see or control and can starve other requests the page needs to make.

The right number also depends on what’s in the batch. Many small files (a folder of thumbnails, a form with a handful of attachments) benefit from higher file-level parallelism, since each individual transfer is quick. A few very large files benefit more from parallelism within a single file, chunking one upload into parallel pieces, a pattern covered in more depth in the companion piece on large file uploads and server memory.

Here’s a compact, dependency-free bounded-concurrency pool, a “worker pull” pattern where a fixed number of workers keep pulling the next file off a queue until it’s empty:

async function uploadWithConcurrency(files, limit = 5) {

const queue = [...files];

const results = [];

async function worker() {

while (queue.length) {

const file = queue.shift();

try {

const res = await uploadFile(file);

results.push({ file, status: "succeeded", res });

} catch (err) {

results.push({ file, status: "failed", err });

}

}

}

// Start `limit` workers, all pulling from the same queue

const workers = Array.from({ length: limit }, () => worker());

await Promise.all(workers);

return results;

}

Fifteen lines, no library, and it already solves both problems from the previous section: uploads are capped at a sane concurrency, and a single failed file just gets recorded rather than aborting everything else.

That’s the mechanics of how many at once. The next question is how work actually flows through that limit, which is where a proper queue comes in.

Building the Upload Queue (State Machine, Not Array)

A concurrency pool tells you how many files can upload at once. A queue tells you what happens to each file before, during, and after that, which matters a lot once things start failing.

The cleanest mental model for a multi-file upload isn’t “an array of files”; it’s a small state machine, one instance per file:

queued → uploading → succeeded

→ failed → retrying → uploading (retry)

→ dead-lettered (after max attempts)

The queue itself is just the scheduler that decides which queued file moves into uploading next, respecting the concurrency limit from the section above.

Wiring This Up in React

If you’re wondering how to support multiple file uploads in React, the short answer is: don’t model progress as a single boolean. A useReducer keyed by file ID, where each file carries its own { status, progress, error }, lets every row in your UI render independently, which matters once failures aren’t all-or-nothing (more on that in the next section).

function uploadReducer(state, action) {

switch (action.type) {

case "START":

return { ...state, [action.id]: { status: "uploading", progress: 0 } };

case "PROGRESS":

return { ...state, [action.id]: { ...state[action.id], progress: action.value } };

case "SUCCESS":

return { ...state, [action.id]: { status: "succeeded", progress: 100 } };

case "FAIL":

return { ...state, [action.id]: { status: "failed", error: action.error } };

default:

return state;

}

}

Cancellation deserves a place in this model too. Wiring each upload to an AbortController lets a user stop an individual file mid-transfer without touching the rest of the batch, and it’s the same mechanism that makes retrying cheap, especially when the underlying transport supports chunked or resumable uploads rather than restarting a file from byte zero. That’s a deeper topic covered in the multi-part and resumable upload patterns in the large-files companion post.

With files properly modeled as individual states, the next piece is deciding what to actually do when one of them lands in failed.

Partial Failure: Designing for “47 of 50 Succeeded”

This is the section that separates a demo from something that survives real users on real networks, because at any real batch size, some files failing is the expected outcome, not an edge case.

The core principle is failure isolation: one file returning a 500 should never abort files #13 through #50. Instead of Promise.all(), collect results the way Promise.allSettled() does, every promise resolves with either a value or a reason, and the batch reports per-file outcomes rather than an all-or-nothing verdict.

A sane retry policy matters just as much as isolating the failure. Retrying immediately and repeatedly just adds load to a server that’s already struggling; exponential backoff with jitter, a capped number of attempts, and a visible “dead-letter” list the user can retry manually is a more honest pattern:

async function uploadWithRetry(file, maxAttempts = 4) {

let attempt = 0;

while (attempt < maxAttempts) {

try {

return await uploadFile(file);

} catch (err) {

if (!isRetryable(err) || attempt === maxAttempts - 1) throw err;

const delay = 2 ** attempt * 500 + Math.random() * 300; // backoff + jitter

await new Promise(r => setTimeout(r, delay));

attempt++;

}

}

}

function isRetryable(err) {

// Network errors, timeouts, 5xx, and 429 are worth retrying.

// 413 (too large) or 415 (unsupported type) are terminal — retrying won't help.

return [0, 429, 500, 502, 503, 504].includes(err.status);

}

Not every failure deserves a retry, which is why isRetryable() matters as much as the backoff timing itself; retrying a 413 or 415 just wastes time and confuses the user.

There’s a server-side half of this too: give each file a client-generated upload ID before the first attempt, and check for that ID on arrival. Without idempotency like this, a retried file can land twice under two different names, a subtle bug that only shows up under real network flakiness, not in local testing.

It’s worth being precise about what “reliability” even means here, since it’s a phrase that gets thrown around loosely. If you’re evaluating which file upload service has the most reliable uptime and upload success rate, the number that matters isn’t raw uptime; it’s the percentage of individual files that complete successfully across a batch, including retries. That’s a function of adaptive retry logic and per-file isolation, not just server availability.

Solid retry logic doesn’t help much if the interface on top of it still shows one spinner for fifty files; the UI needs to reflect the same per-file granularity as the state machine underneath it.

The UX of Many Files: Progress, Errors, and Trust

Two things build trust in a batch upload UI: an aggregate progress number that’s actually honest, and error states that don’t block everything else.

For the aggregate, weight progress by bytes uploaded, not by file count. A batch of one 2GB video and nine 1MB thumbnails is misleading if it reports “1 of 10 files done”, that’s 10% by count but under 1% by actual bytes transferred. Weighting by size keeps the number meaningful.

For errors, the answer to how to improve user experience in file upload UI comes down to three trust signals: visible per-file status (not just a spinner), failures that don’t block the rest of the batch, and a clear, one-click way to retry just what failed, never a prompt to start the whole batch over.

Intake matters too. Supporting drag-and-drop of an image or a whole folder is a common ask, and it’s a bit more involved than a single <input type="file">; the DataTransfer.items API lets you read dropped entries, but traversing dropped directories (as opposed to individual files) requires walking the webkitGetAsEntry() tree recursively, with some inconsistency across browsers worth testing directly.

At this point, the DIY version has grown from “a few lines in a loop” into a queue, a concurrency pool, a retry policy, idempotency handling, and a UI that tracks per-file state, which is exactly the point where it’s worth asking what a managed option actually gives you back.

The Managed Route: Queues, Retries and Progress Out of the Box

Everything built in the sections above – the queue, the concurrency cap, per-file retries and progress – already exists as a drop-in file uploader that treats a 200-file batch as a first-class use case rather than an afterthought.

Concretely, the Filestack picker accepts a multi-file selection or a full drag-and-drop batch, uploads files in parallel within a managed concurrency limit, and reports success or failure per file through separate callbacks, so one failed file in a batch of 200 surfaces its own error without touching the other 199:

import * as filestack from "filestack-js";

const client = filestack.init("YOUR_API_KEY");

client.picker({

maxFiles: 200,

onFileUploadFinished: (file) => {

console.log("Uploaded:", file.filename, file.url);

},

onFileUploadFailed: (file, error) => {

console.warn("Failed, will show a retry option:", file.filename, error);

},

onFileUploadProgress: (file, event) => {

updateProgressBar(file, event.totalPercent);

},

}).open();

That’s roughly ten lines standing in for the concurrency pool, retry wrapper, and reducer built across the sections above. A few specifics worth knowing if you’re comparing this against a custom build: files can go up to 5GB each, Intelligent Ingestion adapts chunk size to the current network conditions to keep success rates high even on flaky mobile connections, and every completed file returns a CDN-backed URL immediately, useful if the next step is generating thumbnails, running a virus scan, or converting a format without a second round trip.

On the backend side, since bytes never pass through your own server, the integration question becomes about registering metadata rather than handling file streams, relevant if you’re figuring out how to integrate a file upload API with Node.js: the server side typically just needs a webhook or callback that records the returned URL and file metadata against the right record in your database.

Two scenarios come up often enough to call out directly. A startup team asking what’s the easiest way to manage hundreds of file uploads is usually dealing with bulk imports, CSVs, product photos, user avatars at signup, where per-file isolation matters more than raw speed. A print company’s IT director looking for the most reliable solution to manage hundreds of file uploads is often dealing with large, high-resolution customer artwork files where a single failed large upload shouldn’t hold up the other 99 orders in the batch.

Whether the DIY queue or a managed uploader is the better fit for a given project comes down to a few concrete variables, which is easier to lay out as a table than a paragraph.

Decision Framework: DIY Queue vs. Managed Uploader

A quick way to sanity-check which side of the build-vs-buy line a given project actually falls on.

Scenario	Batch size	Failure tolerance	Recommended approach
Internal tool, a handful of files at a time	Under 10	Low stakes, manual retry is fine	Sequential or simple Promise.allSettled()
Form attachments, product photo sets	10-50	Users notice failures	Bounded concurrency pool + per-file state
User-generated batch uploads, folder imports	50-500+	Failures must be isolated and retried automatically	Managed uploader (queue, retries, progress included)
Large individual files (video, high-res artwork)	Any count	Network interruption is common	Chunked/resumable transport, ideally paired with a managed picker

The honest cost ledger on the DIY side isn’t just the queue code itself; it’s the retry and backoff logic, resumability for large files, drag-and-drop across browsers, and keeping all of that consistent on mobile. None of those show up in a first prototype; they show up in production, usually during a demo.

Conclusion: Batches Fail Partially, Plan for It

Uploading one file is a request-response problem. Uploading fifty is a scheduling problem with partial failure baked in from the start. Three things make the difference between a batch upload that survives real usage and one that quietly loses files: cap concurrency instead of trusting the browser to sort it out, model each file as its own state rather than one shared flag, and isolate failures so one bad file never takes down the other 49.

If building and maintaining that queue, retry policy, and per-file UI isn’t where the team’s time is best spent, that’s exactly the gap Filestack’s file uploader is built to close, try uploading a real 100-file batch through the Filestack sandbox to see the queuing and per-file retry behaviour firsthand.

FAQ

How many files can I upload at once?

Selection itself is effectively unlimited, but parallelism should be capped, typically 3-6 concurrent uploads, with the rest held in a queue rather than fired all at once.

Why do some of my uploads fail when I upload many files?

It’s usually a mix of the browser’s per-origin connection limit, timeouts on slower connections, and error handling that isn’t isolated per file. Without per-file state tracking, one failure can look like, or actually become, a batch-wide failure.

Does one failed file cancel the whole batch?

It shouldn’t. Proper failure isolation means each file resolves independently; Filestack’s picker surfaces failures through a per-file callback while the rest of the queue keeps running.

Can users upload multiple files from Google Drive or Dropbox in the same batch?

Yes, Filestack’s picker supports pulling files from cloud sources like Google Drive and Dropbox alongside local files in a single session.

Read More →