Puppeteer Tutorial for Beginners

Step 1: Install Node.js and Puppeteer

Puppeteer requires Node.js version 18 or higher. Check your current version by running node --version in your terminal. If you need to install or update Node.js, download the latest LTS release from the official Node.js website or use a version manager like nvm (Node Version Manager) that lets you switch between Node.js versions easily.

Once Node.js is ready, create a new project directory and initialize it with npm. Open your terminal and run mkdir puppeteer-tutorial && cd puppeteer-tutorial && npm init -y to create a project folder with a default package.json file.

Install Puppeteer with npm install puppeteer. This command downloads the Puppeteer library and a compatible version of Chrome for Testing, which is a Chrome build specifically designed for automated testing. The download is roughly 170 to 280 megabytes depending on your operating system because it includes a full Chrome binary. This bundled browser ensures your scripts always run against a Chrome version that matches the library's expectations, eliminating version mismatch issues.

If you already have Chrome installed and want to skip the bundled download to save disk space, you can install puppeteer-core instead. This package provides the same API but does not download a browser. You would then specify the path to your existing Chrome installation using the executablePath option when launching the browser. For this tutorial, the full puppeteer package is recommended since it simplifies setup.

Step 2: Write Your First Script

Create a file called first-script.js in your project directory. A minimal Puppeteer script follows a consistent pattern: launch a browser, open a new page, navigate to a URL, perform an action, and close the browser.

Start by requiring the Puppeteer module at the top of your file with const puppeteer = require('puppeteer'). Then create an async function, since nearly every Puppeteer method returns a Promise and needs to be awaited. Inside the function, call await puppeteer.launch() to start a new browser instance. This returns a browser object that represents the running Chrome process.

From the browser, call await browser.newPage() to open a new tab. This returns a page object, which is the primary interface you will use for all interactions. Navigate to a website with await page.goto('https://example.com'). This loads the page and waits for the load event to fire by default.

Take a screenshot with await page.screenshot({ path: 'example.png' }). This captures the current viewport as a PNG image and saves it to your project directory. Finally, call await browser.close() to shut down Chrome and free system resources. Run the script with node first-script.js and check your project directory for the screenshot file.

By default, Puppeteer runs Chrome in headless mode, meaning no browser window appears on screen. If you want to see the browser in action while developing, pass { headless: false } to puppeteer.launch(). This opens a visible Chrome window where you can watch your script interact with pages in real time, which is invaluable for debugging. You can also add slowMo: 250 to the launch options to slow down every Puppeteer operation by 250 milliseconds, making it easier to follow the automation visually.

Step 3: Navigate and Wait for Content

Real-world websites load content dynamically, and your scripts need to wait for that content to appear before interacting with it. Puppeteer provides several wait strategies for different situations.

The page.goto(url, options) method accepts a waitUntil option that controls when navigation is considered complete. The default is 'load', which waits for the page's load event. For single-page applications that fetch data after the initial load, use 'networkidle2' instead, which waits until there are no more than two active network connections for 500 milliseconds. This is the most practical choice for most modern websites because it waits for AJAX requests and lazy-loaded content to finish while not hanging indefinitely on persistent connections like analytics beacons.

After the page loads, you often need to wait for specific elements to appear in the DOM. Use await page.waitForSelector('.my-element') to pause until an element matching the CSS selector exists. This is essential for pages that render content after JavaScript executes, such as search results that appear after an API call completes or product listings that load through infinite scrolling. You can set a timeout on this method to avoid waiting indefinitely if the element never appears.

For more complex waiting conditions, page.waitForFunction() lets you wait until a JavaScript expression returns a truthy value in the browser context. For example, you could wait until a loading spinner disappears, until a specific variable is defined on the window object, or until the number of elements on the page exceeds a threshold. This method evaluates the expression repeatedly until it passes or the timeout expires.

When your script triggers an action that causes navigation, such as clicking a link or submitting a form, use page.waitForNavigation() in combination with the action. The pattern is to start the navigation wait before triggering the action, then await both simultaneously. This prevents a race condition where the navigation completes before your code starts listening for it.

Step 4: Interact with Page Elements

Puppeteer lets you interact with page elements the same way a user would, through clicks, keyboard input, and form manipulation.

To click an element, use await page.click('.button-class') or await page.click('#button-id'). Puppeteer scrolls the element into view if necessary, moves the virtual mouse pointer to its center, and performs a mouse click. For elements that trigger navigation, pair the click with page.waitForNavigation() to ensure you wait for the new page to load before proceeding.

Typing into input fields uses await page.type('#input-id', 'your text here'). This method focuses the input field and types each character individually with a slight delay between keystrokes, simulating natural typing behavior. You can control the typing speed with the delay option. For fields that already contain text, you may need to clear them first by triple-clicking to select all text and then typing the new value.

Dropdown menus (select elements) are handled with await page.select('#dropdown-id', 'option-value'), where the second argument is the value attribute of the option you want to select. For custom dropdown components that do not use native HTML select elements, you need to click the dropdown trigger and then click the desired option, treating it as a sequence of regular click interactions.

Checkboxes and radio buttons respond to page.click() just like any other element. To check whether a checkbox is already checked before toggling it, evaluate its checked property through page.evaluate() first.

For keyboard shortcuts and special keys, use await page.keyboard.press('Enter'), await page.keyboard.down('Shift'), and similar methods. The keyboard API supports all standard key names including modifier keys, function keys, and navigation keys. You can also type directly with page.keyboard.type() for text input that bypasses element targeting.

Step 5: Extract Data from Pages

Data extraction is one of the most common Puppeteer tasks. The primary tool is page.evaluate(), which executes a JavaScript function inside the browser context and returns the result to your Node.js script.

To extract the text content of a specific element, use await page.evaluate(() => document.querySelector('.target-class').textContent). This runs document.querySelector inside the browser, finds the matching element, reads its text content, and returns the string to your script. For multiple elements, use querySelectorAll with Array.from() to convert the NodeList into an array that can be mapped and returned.

You can extract structured data by building objects inside the evaluate function. For example, to scrape a list of products with names and prices, select all product containers, iterate through them, and return an array of objects containing the name and price from each container. The returned data must be JSON-serializable since it crosses the boundary between the browser process and your Node.js process.

For extracting attributes like href values from links or src values from images, access them through the element's properties or use getAttribute(). The page.$eval() method provides a shorthand that combines selector matching with evaluation: await page.$eval('a.link', el => el.href) finds the first matching element and returns its href property.

When extracting data from multiple pages, build a loop that navigates to each page, waits for content, extracts the data, and stores it in an array. For paginated results, find the "next" button, check if it exists, click it, wait for the new results to load, and repeat until you have collected all the data you need or reached your desired limit.

Remember that page.evaluate() runs in a separate context from your Node.js code. Variables defined in your script are not accessible inside the evaluate function unless you pass them as additional arguments. Functions defined outside the evaluate call cannot be referenced inside it. Think of the evaluate function as a self-contained unit that runs inside the browser and communicates with your script only through its return value and arguments.

Step 6: Handle Errors and Clean Up

Robust Puppeteer scripts anticipate failures and handle them gracefully. Browser automation involves network requests, dynamic page content, and external services, all of which can fail unpredictably.

Wrap your automation logic in a try/catch/finally block. The try block contains your main automation code. The catch block handles any errors that occur, logging them for debugging and optionally taking a screenshot of the page state at the time of failure for visual diagnosis. The finally block calls browser.close() to ensure Chrome shuts down even when errors occur, preventing zombie browser processes from consuming system resources.

Set explicit timeouts on operations that might hang. The page.setDefaultTimeout(30000) method sets a 30-second timeout for all Puppeteer operations on that page, including selector waits, navigation, and evaluations. You can also set timeouts on individual operations through their options parameter. Without timeouts, a script waiting for an element that never appears will hang indefinitely.

For operations that might fail transiently, implement retry logic. A simple retry wrapper attempts the operation, catches any error, waits for a brief delay, and tries again up to a maximum number of attempts. This handles situations like network timeouts, temporary server errors, and race conditions where an element takes longer than expected to appear.

Common error types include TimeoutError (when a wait condition is not met within the timeout period), navigation errors (when a page fails to load), and protocol errors (when the browser process crashes or the WebSocket connection drops). Each type requires a different response, from retrying the current operation to relaunching the browser entirely.

As a best practice, always structure your scripts so that browser.close() is guaranteed to execute. Leaked browser processes consume memory and CPU on your system, and if you are running many scripts, unclosed instances can quickly exhaust available resources. Using the try/finally pattern or wrapping your automation in a function that always closes the browser on exit prevents this problem.