Collecting Public Data at Scale: Robots.txt, Rate Limits and When a Full Browser Profile Helps

Collecting public data from the web is a normal part of price monitoring, market research, ad verification, academic work and plain QA. It goes wrong when volume is treated as the only variable: more nodes, more proxies, more requests per second. At scale the habits matter more than the hardware, and the first of those habits is deciding whether you should be collecting at all.

(If you track how your collection jobs perform, a tool like JustAnalytics can help you spot error-rate spikes early, which is usually a sign you are pushing too hard.)

Start with permission, not with tooling

Before any code runs, answer four questions.

Is there an official route? A public API, a data feed, a partner programme or a downloadable export beats any scraper. It is stable, it is allowed, and it does not break when the page layout changes.

What do the terms say? A site's terms of use can prohibit automated collection even for pages anyone can read. If they do, you need permission, not a cleverer client.

What does robots.txt say? It is not a security control, but it is the site owner's published statement of what automated clients should leave alone. Honour it, including any crawl-delay hints, and check it on every new host.

Is the data yours to process? Public does not mean free of obligations. Personal data carries privacy duties wherever it was published, so decide what you will keep and why before you collect it.

If any answer is "no", stop. The rest of this post assumes you have a legitimate reason and a green light.

Pace yourself like a good guest

At volume, politeness is an engineering requirement.

• Rate limit per host. Set a ceiling on concurrent requests and requests per second for each target, and stay well under what a human team could plausibly generate.
• Back off on errors. A 429 or a run of 5xx responses is the site telling you to slow down. Use exponential backoff and honour any Retry-After header.
• Cache and deduplicate. Do not fetch the same page twice in an hour if the data changes daily. Store ETags and use conditional requests where the site supports them.
• Collect off-peak. Scheduling heavy jobs for quiet hours keeps you out of the way of the site's real users.
• Identify yourself. Where it is appropriate, send a descriptive user agent with a contact address so the owner can reach you instead of guessing.

Many sites sit behind one of the anti-bot vendors that protect much of the open web. If a site starts challenging you, that is the owner's decision about your traffic: treat it as a signal to reduce load or to ask the owner for access. It is not a puzzle to solve with more infrastructure.

When a plain script is enough, and when it is not

A plain HTTP client, or an official API, is the right first choice. It is cheaper, faster and lighter on the target. Reach for a real browser only when you need one:

• The page builds its content with client-side JavaScript and the data is not in the initial HTML.
• The content depends on locale, timezone, language or screen size, and you need to see what a visitor in a given market sees (for example, ad verification or localisation QA).
• You need a logged-in session on an account you are entitled to use, kept separate from your other work.
• You are testing your own site from the outside and want realistic conditions.

For those cases a bare headless script is often the weak link. A headless browser started with defaults renders with software graphics and headless window geometry, so pages that adapt to the display can behave differently from what a person sees. The same goes beyond graphics: hardware APIs such as WebUSB, WebSerial and Web Bluetooth behave differently in headless and headed browsers. A full browser with a consistent identity avoids that class of surprise.

What a full browser profile gives you

JustBrowser is a multi-account browser for teams. Each profile is its own browser, with its own cookies, storage, proxy and identity parameters, so work for different clients, markets or test scenarios stays separate and teammates can share profiles.

It is built on a custom Chromium engine with native C++ integration, not an extension, patched on every upstream Chromium release. The engine applies 40+ identity parameters consistently across canvas, WebGL, audio, fonts, screen, timezone, navigator properties and hardware configuration, so a profile reports one coherent environment instead of a patchwork. Each profile can route through its own HTTP, HTTPS or SOCKS5 proxy, which is how you view a site as a visitor in a given country sees it.

For collection teams the most useful part is the local REST API. You create profiles in the app or through the API, start one, and connect the Playwright, Puppeteer or Selenium code you already run to the CDP endpoint it returns. The API is bound to 127.0.0.1, authenticates with a Bearer token from the app's AI page, and is documented at justbrowser.app/docs.

The flow looks like this:

1. Create a profile and give it its own proxy.
2. Call POST /profiles/{id}/start and read the cdp_url from the response.
3. Connect Playwright or Puppeteer to that WebSocket and use the profile's existing context and first page.
4. Run your collection logic at the pace you set above.
5. End the session with POST /profiles/{id}/stop, since browser.close() only disconnects.

The start call launches headless unless you send {"headless": false}. Headless sessions use headless defaults for screen and viewport, so choose a visible window for any page that depends on real window geometry.

It is more orchestration than a one-line launch call, and it is only worth it when the job needs a full browser. See the Playwright and Puppeteer REST API guide for the integration in detail, and the API docs for each endpoint.

Keep the setup consistent

A few operational points hold up at any volume.

Match the proxy to the profile. A profile set up for a Texas timezone and locale should not exit through a datacenter on another continent. Consistency between proxy location, timezone and language is what keeps pages rendering the way a local visitor would see them.

Use persistent profiles for sessions, fresh ones for stateless jobs. If a job needs a login you are entitled to use, keep it in one dedicated profile so its cookies and storage persist. If it does not, use a clean profile.

Share profiles, not passwords. Team members can run profiles the owner shares with them, with view-only or launch permission, so a colleague can pick up a collection job without anyone passing credentials around.

Log what you collect and why. A short record of the source, purpose and retention period makes audits and deletion requests easy to answer.

Related teams use the same profile isolation for ad verification, checking that ads render correctly in different geos and devices. If you are analysing the traffic you verify, ClickzProtect can help with click-quality questions, and JustEmails covers email on your own domains.

Quick start

1. Check permission. Official API first, then terms, then robots.txt.
2. Set your limits. Per-host concurrency, requests per second, backoff and a contact user agent.
3. Decide if you need a browser. If the data is in the HTML or an API, skip the browser.
4. Start the trial. JustBrowser gives you seven days of full access, REST API included, with a card required and no charge if you cancel inside the seven days.
5. Create a profile with its own proxy, start it through the local API, and attach your Playwright or Puppeteer code to the CDP endpoint.

Frequently Asked Questions

What should I check before collecting public data from a site?

Read the site's terms of use and its robots.txt, look for an official API or data export first, and confirm the data is public and not personal data you have no basis to process. If the site says no, the answer is no. Pace your requests so you never degrade the service for its real users.

When is a full browser profile better than a plain headless script?

When the page only renders correctly in a complete browser: heavy client-side JavaScript, content that depends on locale, timezone or screen size, or pages that need a logged-in session you are entitled to use. A plain HTTP client or an official API is simpler and kinder to the site whenever it is enough.

How much does JustBrowser cost for a data-collection workflow?

JustBrowser is $9.99/month or $99.99/year, with unlimited profiles, REST API access and a 7-day free trial that requires a card. Open-source automation libraries are free; the paid product buys you separate, persistent profiles, per-profile proxies and a local API.

Can I run JustBrowser profiles in headless mode?

Yes. The local API starts a profile headless by default, and you send {"headless": false} when you want a visible window. Headless sessions use headless screen and viewport defaults, so use a visible window when a page depends on real window geometry. JustBrowser ships for Windows x64 and macOS on Apple Silicon, with no Linux build, so you run profiles on a Windows or macOS host and drive them locally.

---

Try JustBrowser

A multi-account browser for teams, built on a custom Chromium engine, not an extension. Real C++ engine patches at the canvas / WebGL / audio / font / screen layer, with 40+ identity parameters rendered by the engine itself. Local REST API + CDP for Playwright, Puppeteer, Selenium. $9.99/month or $99.99/year. 7-day free trial, card required — cancel any time in the seven days and you are not charged. Unlimited profiles.

Get started → · How it differs from Multilogin / GoLogin / AdsPower