Browser Pool Orchestration With Airflow: Checkout, Run, Release

Spinning up a fresh antidetect profile for every task is a common way to get a whole batch challenged together. Dozens of "new users" from the same IP block, all with zero browser age, hitting the same endpoints in parallel look like one pattern to a target site, not many separate visitors.

The fix isn't rate limiting. It's architecture. You don't create profiles on demand; that's the usual mistake. You maintain a pool of warmed, aged profiles and check them out like library books. One profile per task. Task finishes, profile goes back on the shelf. Next task grabs a different one. This is how data engineering teams actually run browser automation at scale.

By the end of this tutorial, you'll have an Airflow DAG that checks out profiles from a REST-managed pool, runs Playwright tasks against them, releases them back, and handles failures gracefully. The same pattern works for Prefect, Temporal, or any workflow orchestrator that can hit HTTP endpoints.

Prerequisites

Before we start:

• Apache Airflow 2.6+ — installed and running, with the HTTP provider (pip install apache-airflow-providers-http)
• JustBrowser ($9.99/month or $99.99/year) — the REST API is included in the one plan and in the 7-day trial, so you can build the whole pool before you're charged
• Redis — for distributed profile locking (prevents race conditions)
• Python 3.10+ — for the task code
• Basic Airflow knowledge — you should know what a DAG, operator, and XCom are

You don't need Kubernetes. A single Airflow scheduler with 4-8 workers handles most scraping workloads. Kubernetes scaling is a separate topic — we covered that in the profile scaling guide.

Step 1: Create Your Profile Pool

First, set up the profiles you'll be checking out. These aren't disposable — they're long-lived identities with browsing history.

In JustBrowser, create 10-20 profiles with varied configurations:

• Fingerprints: Different OS identities. Mix OS identities — Windows 11, Windows 10, macOS — with different screen, GPU and font profiles. The browser version tracks the current Chromium release on every profile. Don't make them all identical. JustBrowser's native fingerprint spoofing handles the 40+ identity parameters at the C++ level.
• Proxies: Each profile gets its own dedicated ISP proxy. One IP per profile, always. We covered the proxy-to-profile pinning in detail already.
• Warm-up: Run each profile through JustBrowser's cookie warm-up feature before adding it to the pool. A profile with zero history is a liability.

Export the profile IDs. You'll need them for the pool registry:

# profiles.py
PROFILE_POOL = [
    "profile_a1b2c3d4",
    "profile_e5f6g7h8",
    "profile_i9j0k1l2",
    # ... your 10-20 profiles
]

Why 10-20? Depends on your task volume and run duration. If each task takes 2 minutes and you're running 100 tasks/hour, you need at least 4 profiles (100 tasks × 2 min ÷ 60 min). Double or triple that for headroom and cooldown time. Your mileage may vary.

Step 2: Build the Browser Pool Orchestration Manager

This is the part everyone skips and then regrets. You need a central registry that tracks which profiles are available, which are checked out, and which are cooling down after failures.

# pool_manager.py
import redis
import time
import json
from typing import Optional

class ProfilePoolManager:
    def __init__(self, redis_url: str, profile_ids: list, lock_ttl: int = 1800):
        self.redis = redis.from_url(redis_url)
        self.profile_ids = profile_ids
        self.lock_ttl = lock_ttl  # 30 min default — longer than any task

    def checkout(self, task_id: str) -> Optional[str]:
        """Acquire exclusive lock on an available profile."""
        for profile_id in self.profile_ids:
            lock_key = f"profile_lock:{profile_id}"
            cooldown_key = f"profile_cooldown:{profile_id}"

            # Skip if cooling down
            if self.redis.exists(cooldown_key):
                continue

            # Try to acquire lock (atomic)
            acquired = self.redis.set(
                lock_key,
                json.dumps({"task_id": task_id, "acquired_at": time.time()}),
                nx=True,  # Only set if doesn't exist
                ex=self.lock_ttl
            )

            if acquired:
                return profile_id

        return None  # No profiles available

    def release(self, profile_id: str, failed: bool = False, cooldown_seconds: int = 900):
        """Release profile back to pool. Optional cooldown after failure."""
        lock_key = f"profile_lock:{profile_id}"
        self.redis.delete(lock_key)

        if failed:
            cooldown_key = f"profile_cooldown:{profile_id}"
            self.redis.setex(cooldown_key, cooldown_seconds, "cooling")

    def get_available_count(self) -> int:
        """How many profiles are free right now?"""
        available = 0
        for profile_id in self.profile_ids:
            lock_key = f"profile_lock:{profile_id}"
            cooldown_key = f"profile_cooldown:{profile_id}"
            if not self.redis.exists(lock_key) and not self.redis.exists(cooldown_key):
                available += 1
        return available

The nx=True flag on Redis SET is doing the heavy lifting here. It's atomic — if two workers try to grab the same profile in the same millisecond, only one succeeds. The other loops to the next available profile.

That 30-minute TTL (lock_ttl) is your safety net. Worker dies mid-task without releasing? Lock auto-expires. Tune this to be longer than your longest task but short enough that crashed workers don't block the pool for hours.

Step 3: Create the JustBrowser API Client

JustBrowser's REST API handles profile launch/stop. Wrap it in a client class so your DAG code stays clean:

# justbrowser_client.py
import requests
from typing import Optional

class JustBrowserClient:
    def __init__(self, api_base: str = "http://127.0.0.1:36542/api/v1", api_key: str = None):
        # token from the app's API/AI tab; Authorization: Bearer is required
        self.api_base = api_base
        self.headers = {"Authorization": f"Bearer {api_key}"}

    def launch_profile(self, profile_id: str, headless: bool = False) -> str:
        """Start the profile and return its CDP URL."""
        resp = requests.post(
            f"{self.api_base}/profiles/{profile_id}/start",   # with api_base = http://127.0.0.1:36542/api/v1
            json={"headless": headless},
            headers=self.headers
        )
        resp.raise_for_status()
        return resp.json()["data"]["cdp_url"]

    def stop_profile(self, profile_id: str):
        """Stop a running profile."""
        requests.post(
            f"{self.api_base}/profiles/{profile_id}/stop",
            headers=self.headers
        )

The cdp_url this returns is a standard CDP (Chrome DevTools Protocol) URL. Playwright's connectOverCDP() and Puppeteer's connect({ browserWSEndpoint }) both speak it. We covered the integration details in the Playwright/Puppeteer stealth guide.

(I know what you're thinking — "what if the API is down?" Add retry logic with exponential backoff. Three retries, 2/4/8 second delays. I'm leaving it out here to keep the examples readable, but in production you want it.)

Step 4: Write the Airflow DAG

Here's a complete DAG that scrapes product data from an e-commerce site using pooled profiles:

# dags/browser_pool_scrape.py
from datetime import datetime, timedelta
from airflow import DAG
from airflow.operators.python import PythonOperator
from airflow.exceptions import AirflowSkipException
import asyncio

from profiles import PROFILE_POOL
from pool_manager import ProfilePoolManager
from justbrowser_client import JustBrowserClient

# Config
REDIS_URL = "redis://localhost:6379/0"
JUSTBROWSER_API = "http://127.0.0.1:36542/api/v1"
TARGET_URLS = [
    "https://example-store.com/products/1",
    "https://example-store.com/products/2",
    # ... your target URLs
]

default_args = {
    "owner": "scraping-team",
    "depends_on_past": False,
    "retries": 1,
    "retry_delay": timedelta(minutes=5),
}

def scrape_product(url: str, **context):
    """Checkout profile, scrape URL, release profile."""
    pool = ProfilePoolManager(REDIS_URL, PROFILE_POOL)
    client = JustBrowserClient(JUSTBROWSER_API)

    task_id = context["task_instance"].task_id
    profile_id = None

    try:
        # Checkout
        profile_id = pool.checkout(task_id)
        if not profile_id:
            raise AirflowSkipException("No profiles available — will retry")

        # Launch
        cdp_url = client.launch_profile(profile_id, headless=False)

        # Scrape (using Playwright async)
        result = asyncio.run(_run_scrape(cdp_url, url))

        # Stop browser
        client.stop_profile(profile_id)

        # Release — success
        pool.release(profile_id, failed=False)

        return result

    except Exception as e:
        if profile_id:
            client.stop_profile(profile_id)  # Clean up
            pool.release(profile_id, failed=True, cooldown_seconds=900)  # 15 min cooldown
        raise

async def _run_scrape(cdp_url: str, url: str) -> dict:
    """Actual Playwright scraping logic."""
    from playwright.async_api import async_playwright

    async with async_playwright() as p:
        browser = await p.chromium.connect_over_cdp(cdp_url)
        context = browser.contexts[0]
        page = await context.new_page()

        await page.goto(url, wait_until="networkidle")

        # Your scraping logic here
        title = await page.title()
        # price = await page.locator(".price").text_content()
        # ...

        await page.close()

    return {"url": url, "title": title}

with DAG(
    "browser_pool_scrape",
    default_args=default_args,
    description="Scrape products using pooled antidetect profiles",
    schedule_interval=timedelta(hours=6),
    start_date=datetime(2026, 7, 1),
    catchup=False,
    max_active_tasks=10,  # Limit concurrency to pool size
) as dag:

    tasks = []
    for i, url in enumerate(TARGET_URLS):
        task = PythonOperator(
            task_id=f"scrape_product_{i}",
            python_callable=scrape_product,
            op_kwargs={"url": url},
        )
        tasks.append(task)

Key details worth getting right:

max_active_tasks=10 — This limits concurrent task execution to (roughly) your pool size. If you have 15 profiles but allow 50 concurrent tasks, 35 will skip immediately. Not the end of the world with retries, but wasteful. And annoying to debug when you're staring at logs wondering why half your tasks are skipping.

AirflowSkipException — When no profiles are available, skip rather than fail. Task retries based on your retries config. Gentler than crashing.

Profile stop before release — Always stop the browser before releasing the profile. Otherwise the next task that checks it out might get a stale CDP endpoint.

15-minute cooldown on failure — If a task fails (CAPTCHA, block, crash), the profile may be hitting rate limits or challenges. Cooling it down prevents immediate re-use and respects the site's limits. Tune this based on your targets — strict sites need longer cooldowns, and if a site tells you to stop, stop.

Step 5: Add Failure Handling Hooks

The DAG above handles failures in the task function, but what if the task crashes hard (OOM, worker death) before reaching the except block? Add a callback:

def on_task_failure(context):
    """Emergency profile release on hard failures."""
    pool = ProfilePoolManager(REDIS_URL, PROFILE_POOL)
    task_id = context["task_instance"].task_id

    # Find which profile this task had checked out
    for profile_id in PROFILE_POOL:
        lock_key = f"profile_lock:{profile_id}"
        lock_data = pool.redis.get(lock_key)
        if lock_data:
            import json
            data = json.loads(lock_data)
            if data.get("task_id") == task_id:
                pool.release(profile_id, failed=True, cooldown_seconds=1800)
                break

default_args = {
    # ... existing args
    "on_failure_callback": on_task_failure,
}

This scans the Redis locks to find which profile the dead task held, then releases it with a longer cooldown (30 min instead of 15). Hard crashes are usually worse than graceful failures — something went genuinely wrong, not just a CAPTCHA.

Common Errors and How to Fix Them

Redis connection refused

redis.exceptions.ConnectionError: Error 111 connecting to localhost:6379

Cause: Redis isn't running, or it's on a different host/port.

Fix: Start Redis (redis-server) or update REDIS_URL to point to your actual Redis instance. In Docker: redis://redis:6379/0.

Profile launch returns 404

requests.exceptions.HTTPError: 404 Client Error: Not Found

Cause: Profile ID doesn't exist in JustBrowser, or the API path is wrong.

Fix: Verify profile IDs in JustBrowser's GUI. Check the API base URL — default is http://127.0.0.1:36542/api/v1; the port is configurable in the app's API tab, so confirm it matches.

All profiles stuck in checkout

Cause: Tasks crashed without releasing locks, and TTL hasn't expired yet.

Fix: Wait for TTL expiry, or manually clear locks: redis-cli KEYS "profile_lock:*" | xargs redis-cli DEL. Also add the on_failure_callback hook if you haven't.

Playwright times out connecting to CDP

playwright._impl._errors.Error: Timeout 30000ms exceeded

Cause: Profile launched but browser startup is slow, or the cdp_url is stale.

Fix: Increase Playwright's timeout (timeout=60000). Make sure you're launching the profile fresh — don't reuse a cdp_url from a previous session.

Next Steps

Once this browser pool orchestration DAG is running, you'll probably want to add:

• Monitoring: Track checkout wait times, failure rates by profile, cooldown queue depth. If one profile consistently fails, remove it from the pool. JustAnalytics can help you build dashboards for this without the GDPR compliance headaches.
• Dynamic scaling: Use Airflow's pool feature or a custom sensor to scale max_active_tasks based on available profiles. Overkill for most setups, but nice to have.
• Profile rotation: Periodically retire old profiles and add fresh ones. Even warmed profiles accumulate weird state after hundreds of sessions.

For teams running ad verification pipelines, pair this setup with ClickzProtect to flag fraudulent traffic patterns in the data you're scraping — it detects bot signatures and click fraud in real-time.

If you're orchestrating multiple SaaS products in a pipeline (scraping, email outreach, call tracking), the VDL orchestration patterns post covers how we manage cross-product workflows internally.

The profile-pool pattern scales surprisingly far. At very large pool sizes you start worrying about Redis latency, but for most scraping and automation workloads? This architecture just handles it. Nothing fancy required.

Frequently Asked Questions

Why use a profile pool instead of creating profiles on demand?

Pre-warmed profiles have browsing history, cookies, and established fingerprints that look natural. Fresh profiles trigger more CAPTCHAs and verification prompts. A pool also prevents race conditions — you never want two workers grabbing the same profile simultaneously. The checkout/release pattern guarantees isolation.

How do I handle profile failures in an Airflow DAG?

Use on_failure_callback to release the profile back to the pool even if the task crashes. Mark profiles as "cooling down" after failures so they don't immediately get reassigned. A 15-30 minute cooldown gives rate limits and temporary challenges time to clear, and gives you a chance to look at why the task failed, before the profile runs again.

Can I use Prefect instead of Airflow for browser pool orchestration?

Yes. The pattern is the same — acquire profile at task start, release at end, use a distributed lock (Redis) to prevent double checkout. Prefect's task retries and state handlers map directly to the checkout/release lifecycle. The examples here work with minimal changes in Prefect.

How many browser profiles can one machine handle?

Each headed profile uses 300-600MB RAM. On a 32GB machine, you can realistically run 30-40 concurrent profiles with headroom for the OS and orchestrator. Headless is lighter but more detectable. For larger pools, distribute across multiple machines with a shared Redis lock.

---

Try JustBrowser

Native Chromium antidetect browser — not extension-based. Real C++ engine patches at the canvas / WebGL / audio / font / screen layer, so 40+ identity parameters are genuine, not faked. REST API for Playwright, Puppeteer, Selenium. $9.99/month or $99.99/year. 7-day free trial, card required — cancel any time in the seven days and you are not charged. Unlimited profiles.

Get started → · How it differs from Multilogin / GoLogin / AdsPower