How to Collect Public Business Data

Step 1: Identify Your Target Data Points

Before writing any code, define exactly which fields you need from each business record. A typical business data record includes the company name, physical address (street, city, state, postal code, country), phone number, website URL, email address, industry or category, year established, employee count estimate, and owner or key contact name. Not every source will have every field, so prioritize the fields that are most important for your use case.

For lead generation specifically, the most valuable fields are company name, website, industry, employee count, and at least one contact method (phone or email). Company size and industry determine whether a business fits your ideal customer profile, while the contact method enables outreach. Additional fields like year established, number of locations, and annual revenue help with lead scoring and prioritization.

Write your target schema before you start scraping. Having a clear data model prevents the common mistake of collecting inconsistent fields from different sources and then struggling to merge them later. Define field names, data types, and format standards (phone number format, address structure, industry taxonomy) up front.

Step 2: Map Public Data Sources

Different data points come from different sources, and no single source contains everything you need. Build a source map that identifies where each target field is most reliably available.

Business directories are the most common starting point. Google Maps and Google Business Profiles provide company name, address, phone, website, category, hours, and review data for millions of businesses worldwide. Yelp offers similar data with a focus on consumer-facing businesses and adds review content and photos. Yellow Pages and its international equivalents (Yell, Pages Jaunes, Gelbe Seiten) provide phone-focused business listings organized by industry category. Industry-specific directories like Clutch (agencies), G2 (software), Avvo (lawyers), and Healthgrades (medical providers) offer richer data within their verticals, including pricing tiers, specializations, and verified client reviews.

Government registries provide legally filed business information. The SEC's EDGAR database publishes financial filings for public companies in the United States. State-level Secretary of State websites maintain incorporation records, registered agent information, and annual filing statuses. The UK's Companies House, Germany's Handelsregister, and similar registries in other countries publish company formation data, director names, and financial statements. This data is authoritative but often requires more complex scraping due to inconsistent website designs across jurisdictions.

Company websites themselves publish contact information, team pages, office locations, technology stacks (detectable through page source analysis), job openings, press releases, and product or service descriptions. Scraping individual company websites is more labor-intensive than scraping directories, but it yields deeper, more current data for high-priority target accounts.

Professional networks and social platforms like LinkedIn publish company profiles with employee counts, industry classifications, headquarters locations, and individual employee profiles with job titles and tenure. These platforms require browser automation and authenticated sessions to access, and their terms of service restrict automated collection, so this approach requires careful consideration of legal and ethical boundaries.

Step 3: Choose Your Scraping Approach

Match your tool selection to each source's technical requirements. Static HTML pages that load all content in the initial server response can be scraped efficiently with HTTP clients (Python's requests library, Node.js axios) combined with HTML parsers (BeautifulSoup, Cheerio). This approach is fast and resource-efficient because it does not require running a full browser.

JavaScript-rendered pages that load content dynamically through API calls or client-side rendering require browser automation. Playwright is the strongest choice for most lead generation scraping because it supports all major browser engines, handles complex navigation flows, provides reliable wait mechanisms for dynamic content, and includes built-in screenshot and debugging tools. Puppeteer is a solid alternative if you only need Chromium support.

For large-scale directory scraping, consider using pre-built scraping tools rather than writing custom code. Apify's Actor marketplace has production-ready scrapers for Google Maps, Yelp, Yellow Pages, and dozens of other directories. These pre-built tools handle pagination, rate limiting, and anti-bot measures, saving you weeks of development and debugging time. The trade-off is reduced flexibility and a per-run cost.

API-based collection is preferable when available. Google Maps Platform, Yelp Fusion API, and many government open-data portals offer structured data through official APIs with clear usage terms and reliable response formats. API access typically costs more per record than scraping but produces more stable, maintainable data pipelines.

Step 4: Build Your Scraper with Pagination and Error Handling

Regardless of the tool you choose, your scraper needs to handle several common challenges robustly. Pagination is the most basic: directories spread results across many pages, and your scraper must follow pagination links or load-more buttons until all results are collected. Track which pages you have already visited to avoid duplicating data or entering infinite loops.

Rate limiting protects both the target website and your scraper from problems. Most websites will block IP addresses that send too many requests in a short period. Implement delays between requests, typically one to three seconds for most sites, and randomize the intervals slightly to avoid appearing automated. For large-scale scraping, use rotating proxy services that distribute requests across many IP addresses.

Error handling must account for network failures, timeout responses, CAPTCHAs, and page structure changes. Implement retries with exponential backoff for transient errors, log failed URLs for manual review, and validate that extracted data matches expected formats before saving it. A scraper that silently produces garbage data is worse than one that crashes loudly on errors.

Incremental scraping saves time on subsequent runs. Instead of re-scraping your entire source from scratch, track which records you have already collected and only process new or updated listings. Most directories sort by relevance or recency, so you can stop scraping when you start encountering records you already have.

Step 5: Clean and Normalize the Collected Data

Raw scraped data is messy. Company names contain extra whitespace, trailing punctuation, inconsistent capitalization, and legal suffixes (Inc., LLC, Ltd.) in various formats. Addresses mix abbreviated and full state names, sometimes omit postal codes, and occasionally concatenate multiple lines into a single field. Phone numbers appear in every imaginable format.

Build a cleaning pipeline that standardizes every field. For company names, trim whitespace, normalize capitalization, and optionally strip or standardize legal suffixes. For addresses, use a geocoding or address validation API (Google Geocoding, SmartyStreets, Loqate) to parse free-text addresses into structured components and verify they are real locations. For phone numbers, use a library like libphonenumber to parse, validate, and format numbers into a consistent international format.

Industry classification is another area that benefits from normalization. Different directories use different category systems, and even the same directory may list similar businesses under different categories. Map all source-specific categories to a standard taxonomy like NAICS codes, SIC codes, or your own internal classification system.

Step 6: Validate and Deduplicate Records

Deduplication is critical when scraping from multiple sources. The same business may appear in Google Maps, Yelp, Yellow Pages, and several industry directories, each time with slightly different data. Fuzzy matching on company name plus address or company name plus phone number identifies likely duplicates. When duplicates are found, merge the records by keeping the most complete and most recent value for each field.

Validation confirms that collected data is accurate and usable. Email verification services check addresses against SMTP servers without sending a message. Phone validation services confirm that numbers are in service and identify whether they are landline or mobile. Website URL validation checks that domains resolve and are not parked or expired. Running these validations before your data enters any downstream system prevents wasted outreach effort and protects your sender reputation.

Step 7: Store and Maintain the Dataset

Load validated, deduplicated records into your database or CRM. If you are building a custom pipeline, a relational database (PostgreSQL, MySQL) works well for structured business data, with indexes on company name, industry, location, and employee count to support fast filtering. If you are working within a CRM, use the platform's import tools or API to create company and contact records with proper field mapping.

Schedule re-scraping runs at intervals appropriate to each source. Business directories change slowly, so monthly re-scraping is usually sufficient. Job boards and press release sources change daily. Government registries update on filing cycles, typically quarterly or annually. Each re-scraping run should compare new data against existing records and update fields that have changed, flagging records that have disappeared from the source as potentially closed or moved.

Track data provenance for every field in every record: which source provided it, when it was collected, and when it was last verified. This metadata enables you to assess data quality, identify which sources are most reliable, and respond to data subject access requests or deletion requests under GDPR and similar regulations.