Multi-Account Browser + Crawlee: Resilient Node.js Crawlers

A Crawlee run can look fine from the console. The stats table says requestsFinished: 1204, requestsFailed: 3, and the dataset has eleven rows.

Eleven rows out of twelve hundred pages that all returned HTTP 200, rendered a full DOM, fired no errors, and tripped none of Crawlee's retry logic. The crawler reports a great night while the pages were empty shells — intact markup, no data. That is a soft failure, and Step 3 shows how to turn it into a retry.

A second question is who owns browser identity. Crawlee's fingerprint injection is JavaScript, patching a stock Chromium from the outside, after the page context already exists. Canvas rasterization, audio timing and font metrics come from lower layers: the graphics driver, the audio stack and the actual text shaper, not from a navigator property that a script overwrote.

So don't pick between them. Keep Crawlee for what it's excellent at — the request queue, the autoscaling, the retry semantics, the dataset — and hand the browser identity to a multi-account browser engine with native C++ integration over CDP.

That's the whole integration. It's about forty lines, and one of them is the easy one to miss.

What We're Building

A Crawlee PlaywrightCrawler that:

• Leases a JustBrowser profile from a pool instead of launching a local Chromium
• Connects to it over CDP through a custom launcher
• Keeps Crawlee's RequestQueue, Dataset, autoscaling, and retry handling untouched
• Turns Crawlee's own fingerprint layer off, deliberately
• Returns the profile to the pool when the browser retires, without orphaning it

The architecture point worth stating plainly: Crawlee orchestrates, the JustBrowser profile is the identity. Two layers, one job each. Every failure mode below comes from blurring that boundary.

Most scraping stacks don't need a full browser at all. Reaching for one first gives you a slow crawler. Get your request rate, your headers and your retry behaviour right, respect each site's terms and robots rules, and use a browser profile where the pages genuinely need one.

Prerequisites

• Node 20+. Crawlee 3.x and Playwright as a peer dep: npm i crawlee playwright.
• JustBrowser — $9.99/mo for unlimited profiles, or $99.99/year. The REST API and CDP launch are in the 7-day trial too, so the launcher below can be built and tested before the first charge.
• A host that stays up. Every active profile is a real Chromium process, not a tab. Size the host for the number of concurrent profiles and leave headroom.
• Windows x64 or macOS on Apple Silicon for the machine the profiles run on — there's no Linux build, so a bare VPS isn't the host for this. Run the crawler on that same machine: the local API and the cdp_url it returns listen only on 127.0.0.1.
• Proxies. JustBrowser routes them per profile over HTTP, HTTPS, or SOCKS5. Bring your own or buy a proxy plan in the app; none come with the subscription.

Step 1: Turn Crawlee's Fingerprint Layer Off

Start here, because everything else is wasted if you skip it.

const crawler = new PlaywrightCrawler({
  browserPoolOptions: {
    useFingerprints: false,   // <- the line everyone forgets
    maxOpenPagesPerBrowser: 1,
  },
});

Leave useFingerprints on and Crawlee generates a plausible fingerprint, then injects it into every new context. Meanwhile the engine underneath has already set its own values for canvas, WebGL, audio, fonts, screen, timezone — 40+ parameters, patched in C++ before any script runs.

Now a page asks the same question twice and gets two answers.

That's worse than either layer on its own. An unusual-but-internally-consistent browser is just an unusual browser. A browser whose JavaScript-visible navigator disagrees with its own rendering output contradicts itself, and you've lost the point of a profile: one consistent identity that you control. A page doesn't only ask JavaScript, either — CSS @supports probes read the rendering engine directly, and no injected script changes their answers.

One layer owns identity. Pick which one. (It should be the one that isn't a script.)

Step 2: The JustBrowser Launcher

Crawlee's PlaywrightPlugin calls launcher.launch(opts) and expects a Playwright Browser back. It does not care where that browser came from — so we hand it a launcher that connects instead of spawns.

import { PlaywrightCrawler, Dataset } from 'crawlee';
import { chromium } from 'playwright';

const JB = process.env.JB_BASE_URL;      // e.g. http://127.0.0.1:36542
const TOKEN = process.env.JB_API_TOKEN;

async function jb(path, init = {}) {
  const res = await fetch(`${JB}/api/v1${path}`, {
    ...init,
    headers: {
      'Content-Type': 'application/json',
      Authorization: `Bearer ${TOKEN}`,
      ...(init.headers ?? {}),
    },
  });
  if (!res.ok) throw new Error(`JustBrowser ${path} → ${res.status} ${await res.text()}`);
  return res.json();
}

const POOL = ['prof_a1', 'prof_a2', 'prof_a3', 'prof_a4'];
const idle = new Set(POOL);

const jbLauncher = {
  name: () => 'chromium',
  async launch() {
    const profileId = [...idle][0];
    if (!profileId) throw new Error('no idle profile — lower maxConcurrency');
    idle.delete(profileId);

    // set the proxy on the profile (POST /profiles/batch/proxy) rather than at launch
    const { data: { cdp_url: wsEndpoint } } = await jb(`/profiles/${profileId}/start`, {
      method: 'POST',
      body: JSON.stringify({ headless: false }),
    });

    const browser = await chromium.connectOverCDP(wsEndpoint);
    browser.on('disconnected', async () => {
      await jb(`/profiles/${profileId}/stop`, { method: 'POST' }).catch(() => {});
      idle.add(profileId);
    });
    return browser;
  },
};

Two things to flag.

headless: false is on purpose. The API starts profiles headless unless told otherwise, and a headless session has no real window, so its screen and viewport follow headless defaults. Run the profiles headed on a real desktop session; the Playwright and Puppeteer REST API guide covers the details.

And the launcher shim leans on Crawlee's plugin contract rather than a documented public API for swapping browsers. It's stable in 3.x, it's how a lot of people do remote-browser work, but pin your Crawlee version in package.json and don't let a minor bump ride into production unwatched. It's worth saying plainly rather than pretending this is blessed.

Step 3: Wire It Into the Crawler

const crawler = new PlaywrightCrawler({
  launchContext: { launcher: jbLauncher },
  browserPoolOptions: {
    useFingerprints: false,
    maxOpenPagesPerBrowser: 1,
    retireBrowserAfterPageCount: 40,
    closeInactiveBrowserAfterSecs: 300,
  },
  maxConcurrency: POOL.length,        // never exceed the pool
  maxRequestRetries: 3,
  navigationTimeoutSecs: 90,
  sessionPoolOptions: {
    maxPoolSize: POOL.length,
    sessionOptions: { maxUsageCount: 200 },
  },
  async requestHandler({ page, request, enqueueLinks, log }) {
    await page.waitForSelector('.product-card', { timeout: 20_000 });

    const rows = await page.$$eval('.product-card', (cards) =>
      cards.map((c) => ({
        title: c.querySelector('.title')?.textContent?.trim() ?? null,
        price: c.querySelector('.price')?.textContent?.trim() ?? null,
      })),
    );

    if (!rows.length) {
      log.warning(`empty grid at ${request.url} — likely served a shell`);
      throw new Error('empty result set');   // let Crawlee retry on another profile
    }

    await Dataset.pushData(rows);
    await enqueueLinks({ selector: 'a.next-page' });
  },
  failedRequestHandler({ request, log }) {
    log.error(`gave up on ${request.url}`);
  },
});

await crawler.run(['https://example.com/catalog']);

maxConcurrency: POOL.length is not a performance tuning knob here, it's a correctness constraint. Crawlee will happily ask for a seventh browser when you have six profiles, and the start endpoint returns an error for a profile that is already running.

The throw on an empty result set is the fix for the eleven-row run described at the top. A 200 with no data isn't success — it's a soft failure (an error page, a consent wall, a throttled response), and unless you turn it into an exception Crawlee has no way to know. Throwing sends the request back through the queue for another attempt, which fixes the transient cases: a slow render, a consent dialog, a layout that hadn't finished loading. If the empty pages are a throttled response, a retry won't fix it, and neither will spreading the requests across more profiles. Slow down, or ask the site owner for access.

maxUsageCount: 200 is deliberately high. Crawlee's default assumes sessions are cheap — an IP and a cookie jar, discard freely. But a profile with weeks of saved cookies and storage is worth keeping, and rotating it every 50 requests throws that away. The cookie export and import walkthrough covers saving and restoring that state through the local API.

Step 4: Proxies Go On the Profile

Notice the proxy isn't passed to Crawlee's proxyConfiguration, and it isn't passed at start time either — it's set on the profile itself (POST /profiles/batch/proxy) before the crawl runs.

Small detail. Big blast radius.

Set the proxy in Crawlee and Playwright applies it at the context level — which sounds equivalent right up until you remember that the engine's WebRTC protection and its per-profile DNS-over-HTTPS were both configured around the profile's own network path, a path that now disagrees with where the packets are actually going. You get a browser whose network stack and identity stack tell different stories.

Match the exit geo to the profile, too. A São Paulo IP behind a profile set to Europe/Warsaw is an inconsistency, and it's one of the most common self-inflicted problems with profiles: a site that localises by IP serves one market while the page's clock and language say another, and your data mixes the two.

Common Errors and How to Fix Them

connectOverCDP: connect ECONNREFUSED

The browser behind the cdp_url isn't reachable from where the crawler runs. The local API and the cdp_url it returns listen only on 127.0.0.1, so the crawler has to run on the same machine as JustBrowser. Check curl -H "Authorization: Bearer $JB_API_TOKEN" $JB_BASE_URL/api/v1/profiles from that machine first; if it fails, the app isn't running or the port differs (36542 by default, configurable in the app). Don't put the port on a network to work around this.

profile already running

Concurrency exceeded the pool, or a previous crash left a profile up with nobody holding the handle. The disconnected listener in Step 2 covers the clean path. For the unclean one, list profiles at startup and stop anything already running before crawler.run() — three lines, saves an afternoon.

Every profile gets throttled at roughly the same page count

That's request volume, not browser configuration: the site is telling you that your total load is too high. Lower maxConcurrency, add a delay between requests in requestHandler, and keep the total request rate within what the site's terms and robots rules allow. Spreading the same load across more profiles doesn't change what the site is asking. If you need more than it allows, ask the owner for access or an API.

Launch calls piling up

Launch calls are heavier than a normal REST call, and a crawler that retires browsers aggressively will hammer them. Serialize them or back off between starts rather than firing them in a tight loop — same concurrency and backoff patterns that apply to any profile-heavy workload.

Pages return challenges

A challenge page usually means the site doesn't want automated access to that content, or not at your rate. Check its terms and robots rules, slow down, and if the content matters, ask for permission or an official API. Don't try to work around it.

Pages show the wrong market's content

Check consistency: do the profile's timezone, language and proxy location agree with each other? A mismatch means the site localises for one market while the profile reports another.

Next Steps

• Lease profiles by target, not round-robin. One profile per site keeps site-specific cookies and storage together instead of spreading them across everything. Round-robin is what most pool examples show, but per-target leasing is usually the better design.
• Poll profile health. Hit GET /profiles/{id}/fingerprint/validate or GET /profiles/{id}/health between crawls instead of relying on a fixed counter, and schedule the check yourself.
• Re-test after Chromium updates. The engine tracks Chromium; your proxies and profile state drift on their own schedule. Monthly is enough.
• Watch what the crawl costs you. If scraped data feeds a dashboard, JustAnalytics handles the metrics side without reporting back to the platforms you're measuring, and if you're buying traffic next to the scraping infrastructure, ClickzProtect is the traffic-quality half of the same problem.

One caveat to end on: this setup trades Crawlee's zero-config convenience for a moving part you own. The pool can leak and the host can die, and you're now responsible for both. Decide whether that trade is worth it before you build.

Frequently Asked Questions

Does Crawlee's built-in fingerprint generator conflict with a native engine?

Yes, and it's the single most common way this integration goes wrong. Crawlee injects generated fingerprint values through JavaScript at context creation. JustBrowser sets the same surfaces in the C++ engine underneath. Run both and a page reads two different answers for the same question — a canvas hash from the engine, a navigator block from the injection, and no consistency between them. Set browserPoolOptions.useFingerprints to false and let the engine own identity.

Can I use CheerioCrawler instead, or do I need PlaywrightCrawler?

If the target renders server-side, CheerioCrawler is faster and cheaper — no browser at all. If the pages need JavaScript to render their content, plain HTTP requests can't help. Use PlaywrightCrawler with a CDP connection for the pages that need a real browser, and keep Cheerio for the pagination and sitemap crawling that doesn't.

How do I stop Crawlee's session rotation from discarding long-lived profiles?

Raise sessionOptions.maxUsageCount and lower maxPoolSize so sessions live longer and map cleanly onto your profile pool. Crawlee's defaults assume sessions are disposable — a session is just an IP plus a cookie jar to it. A JustBrowser profile with weeks of saved cookies and storage is not disposable. Pin one session to one profile, retire on real error signals rather than usage counters, and never let the pool size exceed the number of profiles you actually have running.

Can I build this on the trial, or do I need to subscribe first?

The trial covers it. Seven days, card taken at checkout, and nothing is held back — the REST API and the CDP launch endpoint are there from the first hour, so you can wire the launcher, run a real crawl, and cancel inside the week if it isn't for you. After that it's one plan: $9.99/mo for unlimited profiles, or $99.99/year. There's no API tier to upgrade into and no profile cap to design around, which removes the usual question of when to switch.

---

Try JustBrowser

A multi-account browser for teams, built on a custom Chromium engine — not extension-based. Real C++ engine patches at the canvas / WebGL / audio / font / screen layer, so each profile's 40+ identity parameters are consistent with each other. Local REST API + CDP for Playwright, Puppeteer, Selenium. $9.99/month or $99.99/year. 7-day free trial, card required — cancel any time in the seven days and you are not charged. Unlimited profiles.

Get started → · How it differs from Multilogin / GoLogin / AdsPower