Launching Profiles Concurrently: Managing API Rate Limits and Resource Caps

Last month I watched a scraping job die at profile 23 of 50. No error in my code. No bug in the target site. Just... frozen. Ctrl+C. Check the logs. ENOMEM: not enough memory. I'd launched 22 Chromium instances, each chewing through 600MB, and my 16GB machine tapped out.

The fix wasn't buying more RAM. (Though I did that too, eventually. We all do.) It was learning to manage concurrency.

If you're automating with antidetect browser profiles — scraping at scale, multi-account management, ad verification workflows, whatever — you've hit some version of this wall. API rate limits. Port exhaustion. RAM ceilings. Five profiles? Fine. Fifty? Weird. Two hundred? Broken.

This tutorial covers the orchestration layer: batching launches, queueing work, handling failures gracefully, and tuning concurrency to match your hardware. By the end, you'll have a pattern that scales from 10 profiles to 500 without crashing your machine or getting rate-limited. (I'm not going to pretend this is elegant. It's plumbing. But plumbing that works.)

Prerequisites

Before we start:

• JustBrowser ($9.99/mo) — REST API access included. The 7-day trial is full access, API and all, so you can build this out before the first charge.
• Node.js 18+ — examples are in JavaScript, but the patterns apply to Python/Go/whatever
• Basic REST API familiarity — you know what a 429 response means
• A machine with at least 16GB RAM — 32GB+ recommended for serious concurrency

We covered basic API integration with Playwright and Puppeteer in an earlier post. This one assumes you can already launch a single profile and connect your automation framework.

Understanding the Constraints

Three things will stop your concurrent profile launches:

1. API Rate Limits

The REST API accepts profile launch requests, but your machine won't survive unlimited requests per second. JustBrowser's local API has no documented request cap, but every launch spawns a full Chromium process, so firing dozens at once will exhaust RAM and CPU long before any HTTP limit matters. Doing that 100 times in 200ms would crash the system.

Other antidetect browsers have similar limits. Multilogin caps at roughly 5/second and GoLogin around 8/second. The exact numbers don't matter — what matters is that you can't Promise.all() 100 launch requests and expect them all to succeed. I've seen people try. It's not pretty.

2. System Resources

Each profile is a full Chromium process. Not lightweight. Expect:

• RAM: 300-800MB per profile depending on page complexity
• CPU: 5-15% per active profile during page loads, ~1% idle
• Ports: Each profile needs several ephemeral ports (WebSocket, debug port, internal Chrome ports)

On a 16GB machine with 8GB free after OS and other apps? You're looking at 10-15 concurrent profiles max. 32GB gets you to 25-35. 64GB gets you to 60-80. These are rough numbers — your mileage varies with what pages you're loading.

Honestly, I wish someone had given me these numbers three years ago. Would've saved me a lot of crashed jobs and angry Slack messages from teammates wondering why the automation server was frozen again.

3. Port Exhaustion

This one's sneaky. Each Chromium instance uses 5-10 ephemeral ports. Your OS has a pool of ~16,000 ephemeral ports by default. Launch 100 profiles, use 100, don't clean them up properly, repeat — suddenly you're out of ports and nothing can connect to anything.

Most people never hit this. But if you're cycling through hundreds of profiles over hours without proper cleanup, you will. Ask me how I know.

Step 1: Batch Your Launches

Don't launch everything at once. Batch.

const BATCH_SIZE = 5;
const DELAY_BETWEEN_BATCHES_MS = 2000;

async function launchProfilesBatched(profileIds) {
  const results = [];

  for (let i = 0; i < profileIds.length; i += BATCH_SIZE) {
    const batch = profileIds.slice(i, i + BATCH_SIZE);

    console.log(`Launching batch ${Math.floor(i / BATCH_SIZE) + 1}: ${batch.join(', ')}`);

    // Launch batch concurrently
    const batchResults = await Promise.allSettled(
      batch.map(id => launchProfile(id))
    );

    results.push(...batchResults);

    // Wait before next batch (unless this is the last one)
    if (i + BATCH_SIZE < profileIds.length) {
      await sleep(DELAY_BETWEEN_BATCHES_MS);
    }
  }

  return results;
}

function sleep(ms) {
  return new Promise(resolve => setTimeout(resolve, ms));
}

Why Promise.allSettled instead of Promise.all? Because one failure shouldn't abort the entire batch. If profile 3 fails while profiles 1, 2, 4, 5 succeed, you want to continue with the successes and handle the failure separately.

Batch size of 5 with 2-second delays keeps you well under rate limits while still launching 150 profiles per minute. Adjust based on your API's tolerance — start conservative and increase until you see 429s.

Yes, it feels slow. No, you shouldn't optimize it yet. Get it working first.

Step 2: Add Retry Logic with Backoff

Launches fail. Networks hiccup. APIs get overwhelmed. Your code needs to handle this without manual intervention.

async function launchProfileWithRetry(profileId, maxRetries = 3) {
  let lastError;

  for (let attempt = 0; attempt < maxRetries; attempt++) {
    try {
      const result = await launchProfile(profileId);
      return result;
    } catch (error) {
      lastError = error;

      if (error.status === 429) {
        // Rate limited — back off exponentially
        const backoffMs = Math.pow(2, attempt) * 1000; // 1s, 2s, 4s
        console.log(`Rate limited on ${profileId}, backing off ${backoffMs}ms`);
        await sleep(backoffMs);
      } else if (error.code === 'ECONNRESET' || error.code === 'ETIMEDOUT') {
        // Network issue — shorter backoff
        await sleep(500);
      } else {
        // Unknown error — don't retry
        throw error;
      }
    }
  }

  throw new Error(`Failed to launch ${profileId} after ${maxRetries} attempts: ${lastError.message}`);
}

Exponential backoff is the key pattern here. If you're rate-limited, waiting 1 second and hammering again just gets you rate-limited again. Waiting 1s, then 2s, then 4s gives the API time to recover.

I learned this the hard way — my first retry implementation used fixed 500ms delays. It worked fine in testing. In production with 200 profiles, I generated thousands of 429s in a row and got my IP temporarily blocked. Don't be me.

Step 3: Implement a Work Queue

Batching handles the launch side. But what about the work each profile does? If 5 profiles finish their tasks before the other 45 even launch, you're wasting resources.

Better: a work queue that keeps N profiles active at all times.

class ProfileWorkQueue {
  constructor(options = {}) {
    this.maxConcurrent = options.maxConcurrent || 10;
    this.profileIds = [...options.profileIds];
    this.activeProfiles = new Map(); // profileId -> { browser, cdpUrl }
    this.pendingWork = [...options.work]; // Array of work items
    this.results = [];
  }

  async run() {
    // Initial fill
    const initialBatch = this.profileIds.splice(0, this.maxConcurrent);
    await this.launchProfiles(initialBatch);

    // Process work until done
    while (this.pendingWork.length > 0 || this.activeProfiles.size > 0) {
      // Find an idle profile
      const idleProfile = this.getIdleProfile();

      if (idleProfile && this.pendingWork.length > 0) {
        const work = this.pendingWork.shift();
        this.processWork(idleProfile, work);
      } else {
        // All profiles busy or no work — wait a bit
        await sleep(100);
      }
    }

    return this.results;
  }

  async processWork(profileId, work) {
    const profile = this.activeProfiles.get(profileId);
    profile.busy = true;

    try {
      const result = await this.executeWork(profile.browser, work);
      this.results.push({ work, result, status: 'success' });
    } catch (error) {
      this.results.push({ work, error: error.message, status: 'failed' });
    } finally {
      profile.busy = false;

      // If queue is low and we have spare profiles, launch more
      if (this.pendingWork.length > this.activeProfiles.size && this.profileIds.length > 0) {
        const nextProfile = this.profileIds.shift();
        await this.launchProfile(nextProfile);
      }
    }
  }

  getIdleProfile() {
    for (const [id, profile] of this.activeProfiles) {
      if (!profile.busy) return id;
    }
    return null;
  }
}

This pattern keeps exactly maxConcurrent profiles active. When one finishes a task, it picks up the next task. When all tasks are done, profiles shut down. No wasted launches, no idle profiles hogging RAM.

The tricky part is tuning maxConcurrent. Too low and you're leaving hardware idle. Too high and you OOM. I usually start at Math.floor(availableRamGB / 0.5) — roughly 500MB per profile — and adjust based on actual usage.

Look, I'm not going to pretend this code is production-ready. It's a pattern. You'll need to adapt it. The point is: keep N profiles active, don't let idle profiles hog RAM, and don't launch new ones faster than the system can handle.

Step 4: Monitor Resources in Real Time

Guessing at concurrency limits is fine for prototyping. In production, monitor actual resource usage and adjust dynamically.

const os = require('os');

function getSystemResources() {
  const totalMem = os.totalmem();
  const freeMem = os.freemem();
  const usedPercent = ((totalMem - freeMem) / totalMem) * 100;

  const cpus = os.cpus();
  const avgLoad = os.loadavg()[0]; // 1-minute load average
  const cpuPercent = (avgLoad / cpus.length) * 100;

  return {
    memoryUsedPercent: usedPercent.toFixed(1),
    cpuLoadPercent: cpuPercent.toFixed(1),
    freeMemoryGB: (freeMem / 1024 / 1024 / 1024).toFixed(2)
  };
}

// In your queue runner
async function shouldLaunchMore() {
  const resources = getSystemResources();

  if (resources.memoryUsedPercent > 80) {
    console.log(`Memory at ${resources.memoryUsedPercent}% — pausing launches`);
    return false;
  }

  if (resources.cpuLoadPercent > 90) {
    console.log(`CPU load at ${resources.cpuLoadPercent}% — pausing launches`);
    return false;
  }

  return true;
}

Now your queue can self-regulate. Memory spiking? Stop launching new profiles until existing ones finish. CPU pegged? Same thing. This prevents the crashes I described at the start — your automation adapts to the machine it's running on.

Is this overkill for a 10-profile job? Absolutely. But you're not reading a tutorial about 10-profile jobs.

For more sophisticated monitoring, hook into your observability stack. If you're using JustAnalytics for your apps, the same pattern of tracking metrics over time applies to automation infrastructure.

Step 5: Graceful Cleanup

Profiles that don't shut down properly leave zombie processes, leaked ports, and eventually a machine that needs a reboot.

async function cleanupProfile(profileId) {
  try {
    // Tell the API to stop the profile
    await fetch(`${API_BASE}/profiles/${profileId}/stop`, {
      method: 'POST',
      headers: { Authorization: `Bearer ${API_TOKEN}` },
      signal: AbortSignal.timeout(5000)
    });
  } catch (error) {
    console.warn(`API cleanup failed for ${profileId}: ${error.message}`);

    // Fallback: kill the process directly if we have the PID
    const profile = activeProfiles.get(profileId);
    if (profile?.pid) {
      try {
        process.kill(profile.pid, 'SIGTERM');
      } catch (e) {
        // Process might already be dead
      }
    }
  }

  activeProfiles.delete(profileId);
}

// Cleanup all profiles on exit
process.on('SIGINT', async () => {
  console.log('Shutting down, cleaning up profiles...');

  const cleanupPromises = [...activeProfiles.keys()].map(id => cleanupProfile(id));
  await Promise.allSettled(cleanupPromises);

  process.exit(0);
});

process.on('uncaughtException', async (error) => {
  console.error('Uncaught exception:', error);

  // Still try to cleanup
  const cleanupPromises = [...activeProfiles.keys()].map(id => cleanupProfile(id));
  await Promise.allSettled(cleanupPromises);

  process.exit(1);
});

The SIGINT handler catches Ctrl+C. The uncaughtException handler catches crashes. Both try to clean up profiles before the process dies. This is essential — I've seen machines with 47 orphaned Chrome processes because someone's script crashed without cleanup.

(That someone was me. More than once.)

Common Errors and Fixes

ENOMEM: not enough memory

Cause: Too many profiles for your RAM.

Fix: Lower maxConcurrent. Monitor memory usage. Consider upgrading RAM or splitting work across multiple machines. 32GB handles ~30 profiles comfortably; 64GB handles ~60.

429 Too Many Requests

Cause: An upstream API — your proxy provider or the target site — is rate-limiting the loop. (The local profile API publishes no cap, so a 429 is coming from somewhere else.)

Fix: Increase delay between batches. Reduce batch size. Add exponential backoff to retries. Start with 5 profiles per batch, 2-second delays.

EADDRINUSE: address already in use

Cause: Port conflict. Previous profile didn't release its debug port.

Fix: Ensure cleanup runs on exit. Use dynamic port assignment instead of fixed ports. Add a startup check that kills orphaned Chrome processes from previous runs. If you're running several automation hosts, check that each one's API port (default 36542, configurable in the app) is not being reused.

Profile launch times out after 30s

Cause: System too loaded to spawn new processes quickly.

Fix: Wait for existing profiles to finish before launching new ones. The queue pattern above handles this automatically. Also check disk I/O — slow disks cause slow process spawns. SSDs are not optional for this work. Spinning rust will make you miserable.

WebSocket connection reset

Cause: Profile crashed or was killed while you held a connection.

Fix: Wrap all browser operations in try/catch. When connection drops, mark that profile as dead in your queue and move its pending work back to the queue. Don't retry on the same profile — launch a fresh one.

Tuning for Your Hardware

Here's a rough concurrency guide based on RAM:

RAM	Max Concurrent Profiles	Notes
8GB	5-8	Tight. Leave headroom for OS.
16GB	12-18	Comfortable for small jobs.
32GB	25-35	Production-ready for mid-scale.
64GB	50-70	Serious automation territory.
128GB+	100+	At this point, network becomes the bottleneck.

These assume headed mode with typical web pages. Headless saves 10-15% RAM. Simple pages (text-heavy, few images) use less. JS-heavy SPAs use more.

If you're running on cloud VMs, memory-optimized instances (AWS r-series, GCP n2-highmem) give better bang-per-buck than compute-optimized. Automation is RAM-bound, not CPU-bound.

My hot take: most people over-spec CPU and under-spec RAM for browser automation. I've watched teams spin up 8-core machines and run out of memory at 12 profiles. Get the RAM right first. Everything else is secondary.

Headless launch is a per-request option ({"headless": true}) on a Windows or Apple Silicon Mac host — there is no Docker or Linux server deployment. The concurrency patterns here apply either way; only your host's RAM ceiling changes.

Frequently Asked Questions

How many antidetect browser profiles can I run concurrently?

It depends on your hardware. Each Chromium profile uses 300-800MB RAM depending on page complexity. A 32GB machine can realistically run 20-30 profiles concurrently with headroom. The API itself doesn't limit concurrency — your RAM and CPU do. Monitor with htop or Task Manager and stop adding profiles when memory hits 80%.

What causes 429 Too Many Requests errors when launching profiles?

Most antidetect browser APIs rate-limit launch requests to prevent resource exhaustion. JustBrowser's API has no documented launch cap, but spawning 50 profiles in a tight loop will run your machine out of memory — batch them. Batch launches with delays between batches — 5 profiles, wait 2 seconds, next 5.

How do I retry failed profile launches without duplicating work?

Track launch state per profile ID. When a launch fails (429, timeout, or crash), add the profile ID back to your pending queue instead of retrying immediately. Use exponential backoff — wait 1s, then 2s, then 4s between retries. After 3-5 failures, log and skip rather than blocking your entire job.

Should I use headless mode to run more profiles?

Headless uses slightly less RAM (maybe 10-15% savings) but is more detectable by anti-bot systems. If you're scraping sites without fingerprint detection, headless saves resources. If you're managing accounts on platforms with real detection, run headed on a desktop machine — the RAM cost is worth avoiding bans.

Next Steps

Once you've got batching and queueing working, the next optimization is distributing across machines. A single 64GB server tops out around 70 concurrent profiles. Two 32GB servers can handle the same load with redundancy.

Fair warning: distributed orchestration is a rabbit hole. I've spent more time debugging cross-machine profile coordination than I'd like to admit. Start with one beefy machine until it's genuinely not enough.

For teams running serious automation infrastructure, consider:

• Load balancing across multiple API instances
• Profile affinity — keeping related work on the same profile to maintain session state
• Monitoring dashboards that track launch success rates, memory trends, and error patterns

If you're building automation for ad verification, pair the profile orchestration with ClickzProtect to ensure the traffic you're analyzing is genuine — their JA4 TLS fingerprinting catches bots that IP blocklists miss. For tracking conversions across your automation runs, JustAnalytics handles attribution without leaking data back to platforms.

And for teams using AI agents to drive browser automation, we covered AI web agents with antidetect profiles — the concurrency patterns here apply directly to agent-driven workflows.

The 100-profile hardware benchmark covers what actual resource usage looks like at scale if you want hard numbers before committing to infrastructure.

---

Try JustBrowser

Native Chromium antidetect browser — not extension-based. Real C++ engine patches at the canvas / WebGL / audio / font / screen layer, so 40+ identity parameters are genuine, not faked. REST API for Playwright, Puppeteer, Selenium. $9.99/month or $99.99/year. 7-day free trial, card required — cancel any time in the seven days and you are not charged. Unlimited profiles.

Get started → · How it differs from Multilogin / GoLogin / AdsPower