The practical approach
- Identify exactly who controls the scraping, dataset and AI-training decisions.
- Test any reliance on legitimate interests instead of assuming public data is fair game.
- Build source controls, transparency, data-subject rights and filtering into the data pipeline.
Generative AI models require large quantities of training data. For many developers, one of the quickest ways to obtain that data is to collect text, images, audio and other material from publicly accessible websites. But the fact that content can be viewed online does not place it outside data-protection law.
On 8 July 2026, the European Data Protection Board adopted draft Guidelines 03/2026 on web scraping in the context of generative AI. The guidance addresses how the GDPR applies when private organisations scrape external internet sources or obtain previously scraped datasets for model training or fine-tuning. It remains open for public consultation until 30 October 2026 and may change before final adoption.
The message for AI developers is straightforward: large-scale collection does not remove accountability. Organisations still need to understand what personal data they collect, why they need it, which legal basis applies and how individuals will be protected.
1. Publicly available data is not automatically free to use
One of the most important points in the EDPB guidance is that online availability does not equal consent. A person may publish a photograph, comment, professional profile or forum post for a particular audience and purpose. That does not mean they have agreed to have it collected, stored and used to train a generative AI model.
The EDPB also makes clear that the absence of a robots.txt file does not constitute valid GDPR consent. Whether a website technically permits a crawler to access a page is different from whether the processing of personal data on that page is lawful.
This means organisations need to separate three questions:
- Can the crawler technically access the content?
- Do website terms, technical controls or intellectual-property rules permit its collection?
- Is there a lawful basis under the GDPR for processing any personal data it contains?
Passing the first test does not answer the other two.
2. The GDPR role depends on who makes the decisions
AI training projects frequently involve several parties: the model developer, a scraping provider, a data broker, a cloud platform and potentially a customer commissioning a specialised model. The EDPB says the roles must be determined from the factual relationship, not simply from the labels used in a contract.
Where an AI developer instructs another company to scrape defined sources and categories of data on its behalf, the scraping company may act as a processor while the AI developer generally acts as controller. Where a developer buys an existing dataset from an organisation that independently decided why and how to collect it, the dataset supplier and developer will normally be responsible for their own separate processing activities. Joint controllership may arise when the parties jointly determine the purposes and essential means of the scraping and model development.
A contract saying that a supplier is “GDPR compliant” is therefore not enough. Before data is collected or acquired, organisations should document who selected the websites and data categories, who determined the collection criteria, who can change or stop the crawler, who decides how long the dataset is retained, who decides what the model will be trained to do, and who handles objections, access requests and deletion requests. These decisions establish responsibility more reliably than the commercial description of the service.
3. Legitimate interests is not an automatic AI-training exemption
Consent will often be impractical when personal data is gathered indirectly from large numbers of websites. The EDPB therefore recognises that private organisations frequently consider legitimate interests under Article 6(1)(f) GDPR. However, legitimate interests is not available merely because obtaining consent would be difficult. The controller must satisfy three cumulative conditions.
A legitimate interest
The interest must be lawful, clearly described, real and current rather than vague or speculative. “Developing AI” is unlikely to be sufficiently precise on its own. A stronger description would explain the objective, intended users, commercial or public purpose and whether the model will be used internally or released externally. The EDPB has previously identified examples that may constitute legitimate interests, including developing conversational assistance, detecting fraudulent behaviour and improving cybersecurity threat detection. Each case still requires its own assessment.
Necessity
The organisation must show that processing personal data is necessary for the identified purpose and consider whether the same objective could be achieved through a less intrusive method. That assessment may require narrowing the sources being scraped, excluding unnecessary personal data, using pseudonymised information or replacing some real-world data with synthetic data. A decision to scrape a large section of the internet simply because it may improve a model is not the same as demonstrating necessity.
A balancing test
The organisation must then balance its interest against the rights, freedoms and reasonable expectations of the people whose data is collected. Relevant factors include the sensitivity of the information; whether children or vulnerable people are involved; the type of website and the context in which information was published; whether access was restricted or required a login; whether the website used robots.txt, ai.txt, CAPTCHA or other anti-scraping controls; the scale and indiscriminate nature of the collection; the likelihood that information could be memorised, reproduced or used to infer additional facts about individuals; and how realistically people can object or exercise their GDPR rights.
The greater the possible impact on individuals, the less likely it is that legitimate interests will provide a defensible legal basis.
4. Data minimisation must happen before the crawler starts
A common approach is to scrape broadly and decide what to remove later. The EDPB’s guidance points in the opposite direction. Controllers should assess what information is necessary before collection begins and design the crawler accordingly. The data-minimisation principle does not prohibit the use of large datasets, but it does prohibit collecting personal data that is unnecessary, irrelevant or excessive for the stated purpose.
Practical pre-collection controls may include defining precise source and collection criteria; mapping expected personal-data categories; excluding websites primarily used by children; excluding sensitive directories, forums or subject areas; respecting technical signals that oppose automated collection; filtering formats such as telephone numbers, email addresses, identification numbers and financial details; applying time limits so that unnecessarily old material is not collected; and considering synthetic data before using real personal data.
During and after collection, organisations should continue filtering, deleting, anonymising or pseudonymising information that is not needed. The evidence should show that minimisation was designed into the collection process rather than added only after a privacy review.
5. Transparency remains required at scale
Web scraping creates a difficult transparency problem because the controller may have no direct relationship with the people whose information has been collected. Article 14 GDPR allows an exception from individual notification where providing information is impossible or would require disproportionate effort. But the EDPB warns that organisations should not rely on that exception routinely.
Even where individual notification is genuinely disproportionate, transparency does not disappear. The controller must make the required information publicly available. A dedicated AI-training or web-scraping notice should explain the categories of personal data collected; the purpose and legal basis; whether sources are publicly or privately accessible; the types of websites and sources used; the characteristics of the crawler; the collection dates or periods; how individuals can exercise their rights; and how to object or request exclusion.
The EDPB considers it good practice to provide domain names and URLs in a searchable format where possible. When a developer acquires a previously scraped dataset, it should also explain where the dataset came from and provide information about the original scraping controller. A generic privacy notice stating that information may be used “to improve our services” is unlikely to provide the level of detail needed for a large AI-training dataset.
6. Website restrictions matter to reasonable expectations
The presence or absence of technical restrictions is not itself a GDPR legal basis. It can, however, affect whether individuals would reasonably expect their data to be reused. The EDPB distinguishes between content that a person has deliberately made visible to everyone and information placed behind access restrictions or on platforms that clearly oppose AI scraping.
AI developers should therefore record not only which pages were collected but also whether login or authentication was required; what crawler restrictions existed at the time; whether the platform communicated an AI-training policy; whether the content was made public by the individual or by somebody else; and whether the crawler received information that ordinary users could not see. That history may become important when defending the balancing test later.
7. Accuracy applies to the dataset and the model output
Scraped information may be outdated, duplicated, taken out of context or drawn from unreliable aggregators. The EDPB links these problems directly to the GDPR accuracy principle. Where possible, controllers should use reliable and maintained sources, record when information was collected and validate samples before using them for training.
The accuracy obligation may also extend to the results produced by the model where those outputs constitute personal data. An AI system that confidently reproduces false information about an identifiable person can create a data-protection problem even where the original dataset appeared accurate when collected. A defensible provenance record should capture source domain and URL; collection date; dataset version; relevant crawler configuration; validation and quality checks; exclusions and deletion actions; and the models and training runs that used the data.
8. Special-category data creates a much higher threshold
Large-scale scraping may collect information revealing health, political opinions, racial or ethnic origin, religion, sexual orientation or other special-category data. For intentional collection of such information, an organisation needs both an Article 6 lawful basis and a valid condition under Article 9(2) GDPR. Legitimate interests alone is not sufficient.
The EDPB discusses a narrow possibility for incidental and residual collection, drawing on the Court of Justice’s decision concerning search engines. However, it explicitly warns that this is not a general exemption from Articles 9 and 10. A controller seeking to rely on this reasoning would need a case-specific assessment and strong measures before, during and after collection: precise filters, excluding websites structurally likely to contain sensitive data, deleting special-category data as soon as it is identified, testing resistance to privacy attacks, filtering model outputs and continuously monitoring results.
In practice, organisations should treat incidental collection as a risk to prevent and remediate, not as a convenient legal justification.
9. What good AI-scraping governance looks like
The EDPB does not prescribe one universal technical architecture. It provides a range of measures that controllers should select according to the particular risks. A defensible programme is likely to include a documented purpose and role assessment; a data-source register; a legitimate interests assessment where Article 6(1)(f) is used; a DPIA where the processing is likely to result in high risk; pre-collection source and data-category filters; a public scraping and AI-training notice; a workable objection and opt-out process; processes for deletion, anonymisation and pseudonymisation; controls against model memorisation and regurgitation; dataset and model version records; and regular testing and review.
A 90-day readiness plan
| Weeks 1–4: Map the activity | Identify every internal crawler and externally acquired training dataset; document controller, processor and supplier roles; record the purpose, source categories and expected personal data; identify any current reliance on legitimate interests. |
| Weeks 5–8: Test and reduce exposure | Complete or update the legitimate interests assessment; review whether personal data is genuinely necessary; exclude high-risk websites and unnecessary data categories; add source, date and dataset-version records; test filters for identifiers and special-category information. |
| Weeks 9–12: Build the evidence | Publish or update the AI-training transparency notice; establish objection, exclusion and deletion procedures; record model-level controls against memorisation and disclosure; complete a DPIA where required; create a periodic review and testing schedule. |
Common pitfalls
- “It was public, so we can use it.” Public availability does not remove the need for a lawful basis.
- “The vendor collected it, not us.” Reusing a scraped dataset is itself a processing activity with its own responsibilities.
- “Legitimate interests covers innovation.” The interest must be specific, necessary and balanced against individual rights.
- “We cannot contact everyone, so transparency does not apply.” Public information and appropriate safeguards remain necessary.
- “We will remove sensitive data later.” Filters and exclusions should be designed before collection begins.
- “No robots.txt means consent.” The EDPB expressly rejects that interpretation.
- “Our model does not store personal data.” Memorisation, extraction and identifiable outputs must be tested rather than assumed.
What to do next
Start by mapping the sources and datasets currently used for AI development. Then test whether the purpose, lawful basis, collection criteria and public information would still be defensible if requested by a supervisory authority. The most valuable outcome is not another policy. It is a connected evidence trail showing what was collected, why it was needed, which safeguards were applied and how individuals can exercise meaningful control.
The EDPB’s document remains draft guidance under consultation. Organisations should monitor the final version and update their assessments and controls where the Board’s position changes.
This article is for general information only and does not constitute legal advice. Always seek specific advice for your organisation, processing activities and the jurisdictions in which you operate.
Sources: EDPB Guidelines 03/2026 on web scraping in the context of generative AI (adopted 8 July 2026, public consultation until 30 October 2026); EDPB Opinion 28/2024 on AI models.