Web scraping and lead generation with an AI agent
When to let an AI agent collect data for you, when a plain script is better, and how to build lead lists responsibly.
Agent or script?
| Situation | Better choice |
|---|---|
| One-off research across many different sites | Agent |
| Layouts that change often or need judgment | Agent |
| The same site, thousands of pages, every day | Script (Playwright or an HTTP client) |
| Prototype now, scale later | Agent explores, then writes the script |
The sweet spot is combining them: the agent figures out a site and writes a scraper; the scraper does the heavy lifting on a schedule; the agent fixes it when the site changes.
Building a lead list
- Define the target. "Independent dental clinics in Lisbon with a website" beats "businesses in Portugal".
- Pick public sources. Business directories, maps listings, association member lists, company websites.
- Collect the basics. Name, website, city, category, public business contact details.
- Enrich. The agent visits each website to confirm it's active and note what the business offers.
- Dedupe and clean. Merge duplicates, normalise phone numbers and URLs, drop dead links.
- Export. A CSV or sheet ready for your CRM.
Staying on the right side of the line
- Respect terms and robots rules for each site, and rate-limit so you don't burden anyone's servers.
- Collect business information, not personal data you have no basis to use. Rules such as the GDPR in Europe apply to personal data about individuals.
- Follow anti-spam law (for example CAN-SPAM in the US): honest subject lines, a way to opt out, and honour opt-outs.
- Don't bypass logins or paywalls you aren't entitled to.
Why a dedicated machine helps
Scraping runs are long and bursty. A server with plenty of memory can run several browsers in parallel, keep going overnight, and hold the results — and you can watch the browser live when a site does something unexpected.
Common questions
Should I use an AI agent or a script for web scraping?
Use an agent for messy, changing or one-off sources that need judgment. Use a script for high-volume scraping of stable sites. A good pattern is letting the agent explore a site and write the script.
Is scraping for lead generation legal?
It depends on the source's terms, the type of data and your location. Stick to public business information, respect site terms and rate limits, and follow privacy and anti-spam laws such as the GDPR and CAN-SPAM. When in doubt, get legal advice.
How do I avoid duplicate leads?
Normalise fields like website domain and phone number, then merge records that share them before exporting.
Skip the setup — get a ready-made agent machine
We set up a dedicated 64 GB machine with a live browser, an online terminal and a chat agent (Claude Code-ready — you use your own Claude subscription or API key), on your own domain and secured for you. Access emailed within 24 hours, plus a month of support.
Get set up — $1,000