TL;DR
- Buy a dataset when its covered records, observation dates and fields satisfy most of the job. Crawl when the decisive evidence is missing or must be refreshed on demand.
- Bright Data offers prepared/custom dataset delivery, Diffbot offers structured entities and extraction, and Common Crawl offers an archive that your team must process. These are different levels of finished work.
- Use a hybrid: acquire a baseline, then fetch the missing or stale evidence for a selected subset. Repeatedly crawling every page is rarely the only way to keep account research current.
- Price accepted account records, integration and exception work. A million rows can be poor value if most describe the wrong entities or cannot be used for the intended purpose.
Buy the finished work you need
A web dataset is a retained collection of observations or structured records. A crawler collects pages under your own scope and timing. Neither description says whether the output is ready for account research. Entity resolution, extraction, provenance and permitted use remain separate requirements.
Write the required fields and maximum acceptable age before choosing the acquisition route. A company address may tolerate a slower refresh than a new job posting. A broad snapshot is useful for market mapping but weak evidence for a same-day buying trigger.
Our scraping API guide covers live collection infrastructure. Here the decision is whether to buy already-collected work. Our company data guide covers commercial entity records, which may be a better purchase than raw pages when the desired output is a company profile.
Three dataset routes
| Route | What arrives | Your remaining work | Suitable buyer |
|---|---|---|---|
| Bright Data datasets | Prepared or scoped records/snapshots | Validate schema, identity, freshness and use rights | Team wanting delivery rather than crawler operations |
| Diffbot | Structured graph entities or extracted pages | Match the graph/model to the account contract | Engineering team needing structured web facts |
| Common Crawl | Archive/index access | Select, fetch, parse, extract and resolve | Technical team with a bulk research requirement |
Bright Data
Bright Data's dataset marketplace describes prepared and custom collections with snapshot download delivery. This is relevant when the required source or record class is already represented and buying a filtered subset avoids operating your own collector.
Ask for a sample containing the actual target markets and required fields. Check row identity, source URL, observation date and refresh behaviour. A claim that a dataset is regularly refreshed does not establish that every field in every row was observed yesterday.
Choose it when the offered scope and delivery remove meaningful collection work. Avoid it when the contract supplies a large unrelated universe but no reliable subset for your accounts. Price record units, minimum purchase, recurring refreshes, custom collection and downstream rights from the current dataset-specific proposal; a proxy tariff is not the dataset price. The marketplace currently shows a $250 minimum order and a headline rate of up to $0.0025 per record. Treat that as an entry reference, not a quote for your fields, filters and recurring refresh.
Diffbot
Diffbot describes its Knowledge Graph as structured public-web knowledge and its products distinguish querying that graph from extracting/crawling pages. The purchase can therefore be a structured entity rather than a saved HTML page.
Its pricing page identifies different credit operations, including one credit for page extraction and 25 for exporting a Knowledge Graph entity. Startup lists $299 monthly for 250,000 credits, with overage at $0.001 per credit; Plus lists $899 for one million credits and includes Crawl access. Those units are not equivalent work. A graph entity may save extraction and linking effort, while a live extraction may be necessary for a particular missing field.
Choose it when the graph's entities and properties match the research task. Validate provenance and age on the fields that affect decisions. Avoid treating a graph record as verified simply because its structure is convenient; wrong-entity matches and inferred relationships still need acceptance rules.
Common Crawl
Common Crawl's getting-started guide and URL index explain archive access and indexes pointing to WARC records. This is a collection resource, not a ready-made CRM company database.
It fits a technical team exploring a large historical or bulk web corpus where the necessary pages are present. Use the index before downloading broad archives, then retain crawl identity and capture dates with extracted facts. Missing coverage is a property of that archive selection, not proof that a company has no website or no relevant activity.
Open access reduces acquisition fees but leaves compute, transfer, storage, parsing and interpretation with your team. Check applicable use rights and original content considerations. It is poor value for a small operator who needs a few current account briefs and would build a large data-processing system to obtain them.
Accept coverage and freshness separately
Use a known account sample with ordinary and difficult identities. Measure how many target accounts are present, how many match the correct entity, which mandatory fields exist and whether their dates satisfy the policy. A filled headcount field with unknown observation time may be unsuitable for a growth trigger while remaining usable for broad segmentation.
Keep unavailable, stale and ambiguous fields distinct. Do not merge a dataset's description of a parent with a current crawl of a subsidiary simply because they share a brand. Preserve provider identifiers, domains and entity relationships.
Review licensing before activation. Research, internal CRM use, customer-facing display, redistribution and model training can have different terms. Publicly accessible pages do not automatically confer every commercial downstream right. The contract and source context must support the intended use.
Work the break-even calculation
Assume a 10,000-account research job. A dataset costs an illustrative $800, loading and mapping take six hours at $60 per hour, and 20% of accounts require a $0.15 live refresh. Total is $1,460: $800 + $360 + $300.
A crawl-first alternative fetches five pages per account at an assumed $0.01 per page, costs $100 for extraction/compute and takes 20 hours of setup and exception work at $60 per hour. Total is $1,800. The dataset-plus-refresh route is $340 cheaper under these assumptions, before any differences in usable output.
If the dataset only covers half the accounts and requires substantial manual repair, the apparent saving can disappear. Suppose another eight hours are needed: $480 raises the hybrid total to $1,940. These are hypothetical inputs, not provider prices or test results. Use them to identify the variables that your sample must measure.
If 8,000 accounts satisfy the full acceptance rule under the original hybrid scenario, cost per accepted account is $0.183, rounded. Record the assumed acceptance count clearly. A cheap raw-row rate is not a buying conclusion until matching and freshness pass.
Build a hybrid refresh
Load the baseline with provenance, then select refresh targets by missing field, age and decision importance. A live fetch should update the relevant field and its date, not automatically overwrite every older sourced fact. Keep contradictions for review rather than always preferring the newest text.
Use a search API for discovery where the known domain does not expose the required evidence. Fetch only the chosen pages afterwards. The ABM Account Scoring Template helps define which evidence warrants that extra cost.
Choose a prepared dataset for a covered repeated job, structured graph access for a suitable entity model and an archive for technical bulk research. Retain a live collection path for exceptions. The useful system avoids unnecessary collection while preserving the evidence required to make an account decision.
FAQ
Is a dataset always older than a live crawl?
It is a retained observation, but freshness depends on the field and refresh process. A live crawl can also return cached or outdated source content. Keep observed and source dates and judge them against the specific decision requirement.
Can we use Common Crawl as a company database?
You can derive company information from covered pages, but you must build selection, extraction and entity resolution. The archive is not a guaranteed current profile for every company. Price that engineering work before calling the route free.
Should every account receive a live refresh?
Only if the decision requires it. Prioritise stale, missing and consequential fields. A hybrid can reduce cost while preserving quality, provided the freshness policy is explicit and exceptions do not silently pass.
What belongs in the dataset contract?
Define fields, entity unit, geography, observation dates, refresh frequency, delivery, minimums and permitted downstream uses. Ask how missing and revised records are represented. Those details determine usable output more directly than the advertised row count.
