Headless browsing with Chrome Headless

4 min read Updated 1 day ago

Headless browsing with Chrome Headless

Chrome Headless is a full Chrome browser that runs without a window. Because it is a real browser it renders JavaScript, waits for network requests and lays out pages exactly the way a visitor's browser would, which makes it useful for three things:

  • Web scraping pages that are built or populated by JavaScript, where a plain HTTP fetch comes back empty.
  • Screenshots of pages, captured server-side and returned to your application.
  • PDF generation from HTML, so you can render invoices, reports and receipts from the same markup your pages use.

You interact with it over HTTP on port 3000. Ploi Cloud runs it for you, so your application never has to install or manage a browser of its own; it just sends requests to the service.

Adding it

  1. Go to your application and add a new service from the Custom services card.
  2. Pick Chrome Headless under Browser.
  3. Choose how much memory it gets. Browsing is memory hungry, so leave a little headroom if you plan to render large pages.
  4. Set a TOKEN in Service configuration or Advanced settings if the service will be reachable from outside your application. Leave it empty if only your app talks to it.

Hit Create service and it starts on its own. Give it a few seconds, then point your code at the address below.

Connecting from your application

The service injects a set of variables into your application's environment so you never have to guess where it lives:

  • CHROME_HEADLESS_URL is the full address, http://HOST:3000. This is the one to use.
  • CHROME_HEADLESS_HOST is the service's hostname on its own.
  • CHROME_HEADLESS_PORT is 3000, if you want to build the address yourself.
  • BROWSERLESS_URL is the same URL under the browserless name. It is an alternative if you already use the browserless library and expect that variable name.

Using it

Chrome Headless speaks the browserless HTTP API on port 3000. The simplest way to try it is to fetch one of its direct endpoints from inside your application:

curl -X POST "$CHROME_HEADLESS_URL/screenshot" \
  -H "Content-Type: application/json" \
  -d '{"url":"https://example.com","options":{"format":"png"}}' \
  --output page.png

The same service can print a page to PDF, or run a scraping session that extracts data from a fully rendered page. The exact request bodies for those live in the browserless documentation, which this service follows; the style of call above is the shape they all take. Where your code runs, use CHROME_HEADLESS_URL as the base and the browserless endpoints will match.

Settings

You can tune the service under Edit resources (opening Service configuration) and in Advanced settings. Changing a setting restarts the service, so anything it is rendering at that moment is dropped.

  • CONNECTION_TIMEOUT is how long a request may take before the service gives up, in milliseconds. The default is 60000.
  • MAX_CONCURRENT_SESSIONS caps how many browser sessions can run at once. The default is 10. If you scrape heavily or parallelize work, raise it to match.
  • TOKEN is the authentication token for API access. It is empty by default, meaning the API accepts requests without a token. Set one if the service is exposed beyond your application.
  • BLOCK_ADS blocks third party ad requests in every browser session. Leave it empty to keep ad blocking on; set it to false to load ads and the scripts they pull in. It only accepts true or false, and clearing it turns ad blocking back on. If you enter anything else you get the error BLOCK_ADS must be true or false.
  • DEFAULT_BLOCK_ADS is the ad blocking default for sessions that do not pass a blockAds option themselves. It also only accepts true or false.