Intellectual Property

AI training opt-outs for a company website: limits of crawl controls and legal measures

Where a company does not want articles or images on its website to be used for AI training, the first step is to distinguish what it wants to restrict: the collection of information, its use for training, or its display in search results or AI answers. The scope and effects of directives in robots.txt and similar controls differ between services, and some affect inclusion and display in regular search results.

This guide sets out, for companies operating websites, controls on collection and use, access restrictions, and responses based on terms of use and contracts. It distinguishes opt-out directives from the assessment under copyright law, and explains how to keep records where information has been collected or displayed and what to check when requesting cessation or deletion.

This guide mainly concerns the collection, use, and display of publicly available content. Where circumvention of authentication or a leak of nonpublic information is suspected, please also refer to "Cyber Incident and Data Breach Response" alongside this guide.

Reviewed by Keishi Yoshikawa, Attorney at Law and Patent Attorney (Dai-Ichi Tokyo Bar Association)

Last updated: September 21, 2026

What to check before restricting collection, training, or display by AI

The pages, images, and data concerned, and the scope of the rights held by the company and permissions obtained from third parties
Whether the use to be restricted is collection, training, display in search results, or display in generative AI answers
The providers, services, and crawlers concerned, and the inclusion in regular search results that the company wants to maintain
The extent to which information is currently public, and the settings for robots.txt, meta tags, authentication, and access restrictions
How agreement to the terms of use is obtained, whether there are individual contracts, and how data is provided or sold in practice
Specific examples of collection and display, access logs, and records of the settings and terms in effect at the time

Where the website includes work by external writers or production companies, first check the scope of the rights held by the company. Merely having permission to publish the work does not necessarily entitle the company to bring a claim for injunctive relief based on copyright. Ownership of the rights is checked, and the company coordinates with the rights holder where necessary.

How we can helpSetting out the collection, use and display to be restricted / Examining the legal position of control measures / Considering claims based on copyright / Drafting and reviewing terms of use and individual contracts / Organizing records of collection and display and identifying the recipients of requests / Making requests for cessation, deletion and other measures, and conducting negotiations

Contact Form

Contents

1. Distinguishing the use to be stopped: collection, training, or display in search results
2. Checking the scope and limits of robots.txt, meta tags, and access controls
3. Examining responses based on copyright, terms of use, and contracts
4. Keeping records of collection and display and requesting cessation, deletion, or other measures
How we assist
Frequently asked questions

1. Distinguishing the use to be stopped: collection, training, or display in search results

The three acts are distinct

The following three acts come into question in relation to information on a company's website.

The first is collection and storage. This is where a crawler reads and reproduces a page.

The second is training and analysis. This is where the information collected is analyzed and used to train a model.

The third is generation and display. This is where the company's information appears in search results or in AI answers or summaries.

The provisions that govern these three acts and the measures that can be taken differ. Conclusions such as "crawling is reproduction, so it can be enjoined" or "training has no purpose of enjoyment, so nothing can be done" cannot be reached in advance. The assessment is made after checking the circumstances and purpose of each act.

Deciding the purpose of the restrictions

The measures chosen depend on whether the aim is to stop only use for training or also inclusion in search results. In many cases, the company wants to avoid losing traffic from regular search as a result of strengthening its training opt-out directives. This needs to be addressed beforehand as a business decision.

Directives restricting collection and directives restricting inclusion and display in search results operate at different stages. Directives concerning display may also be reflected in pages already included in search, through processes such as recrawling. On the other hand, changing settings does not necessarily result in deletion from data already collected or from models already trained. What to do about information already collected and how to prevent future collection need to be considered separately.

2. Checking the scope and limits of robots.txt, meta tags, and access controls

The following table sets out technical controls alongside contractual restrictions. The effect of each directive depends on the specifications and settings of the provider concerned.

Measure	Scope and conditions	What can be controlled or agreed	Remaining limits and legal qualifications	Relationship with inclusion and display in regular search results
Controls on collection and use for AI (directives in robots.txt and similar controls)	Identification of the providers, crawlers, and purposes of use covered	Controls on collection and use for training or other purposes, depending on the specifications	The same effects do not necessarily extend to other providers or data already collected	Depends on the directive. Google-Extended does not affect inclusion in Google Search or ranking signals
Robots rules for search crawlers	The crawler concerned supports the directive	Controls on crawling within the specified scope	Does not block access itself. See section 3 for the legal significance of certain opt-out measures	A URL may remain in search results even if crawling is stopped
noindex	A search engine that supports the setting can read it	Controls on indexing and inclusion of the page concerned	Does not impose a blanket prohibition on collection or AI training. This presupposes that the instruction can be read	Excludes the page concerned from search results in search engines that support the setting
nosnippet and similar controls	A provider that supports the directive can read it	Controls on snippets and similar elements in search results according to the directive	Does not stop training or output across all AI services	Also affects descriptions and similar elements in regular search results
Authentication and access restrictions	Server-side restrictions on which users and other entities are permitted access	Technical restrictions on unauthorized access and collection	Onward provision by authorized users and information already collected are separate matters. The legal assessment of circumvention is also made in each case	The body content behind authentication is not normally crawled
Terms of use	Formation of an agreement, incorporation into the contract, and validity of the terms	Setting conditions for purposes of use, onward provision, and related matters	Merely posting the terms does not bind everyone who collects the information. Breach of contract and copyright infringement are separate matters	Posting the terms alone does not change inclusion or display
Individual contracts	Identification of the parties, the data concerned, and the scope of use	Agreement on purposes of use, onward provision, records, handling of deletion, and related matters	Individual contracts do not automatically bind third parties outside them. Securing performance is a separate matter	Where necessary, agreement is also reached on how inclusion in search results will be handled. It does not change automatically

Note: The statements concerning Google are examples based on its specifications and cannot be treated as specifications shared by all providers. Specifications may change, so please check the latest official information when applying the settings.

Three points to avoid confusing what each control covers

Google describes Google-Extended as a token that lets website operators control whether their content is used for the training and grounding of models for Gemini Apps and Vertex AI API for Gemini, and states: "Google-Extended does not impact a site's inclusion in Google Search nor is it used as a ranking signal in Google Search". Directives for AI and controls on inclusion and display in search results are separate matters.

However, this cannot be extended to mean that "there is no effect at all on traffic from search". Some controls, such as noindex and restrictions on search crawlers, affect regular search. Also, AI features displayed on Google Search results pages are not stopped by a Google-Extended opt-out directive. The directives concerning collection and display for search need to be checked separately.

Where a crawler refrains from collection in accordance with robots.txt directives, it may not be able to read instructions such as noindex placed on the page. The robots.txt mechanism does not technically block access and depends on providers that support its directives respecting them, so it operates differently from authentication or access restrictions. When combining measures, check whether the combination works as intended.

The legal significance of these directives is not determined by their names alone. Whether they meet the statutory criteria discussed below is assessed in light of what the directives cover and the actual processing and display.

After setting out the scope of the restrictions sought and the grounds for legal measures, check the specific settings and whether they have taken effect with the person responsible for managing the website or the production company.

3. Examining responses based on copyright, terms of use, and contracts

Use for training: Article 30-4 of the Copyright Act

Article 30-4 of the Copyright Act provides that a work may be used to the extent considered necessary where the purpose is not to personally enjoy or cause another person to enjoy the thoughts or sentiments expressed in the work. Use for information analysis is listed as one such case.

However, the proviso to that Article states that this does not apply where the use would unreasonably prejudice the interests of the copyright owner in light of the nature and purpose of the work and the circumstances of its use.

The first question is whether the use can be said to have no purpose of "enjoyment". The Agency for Cultural Affairs' "General Understanding on AI and Copyright in Japan" states that the Article does not apply where a purpose of information analysis and a purpose of enjoyment coexist. Additional training for the purpose of producing output containing all or part of the creative expression of a particular work, or the creation of a database for the purpose of reproducing expression from the company's articles when generating answers, are among the acts examined from this perspective. The requirements of the main clause of the Article are not met merely because the activity is called "training" or "analysis".

The same document gives, as an example that may fall within the proviso, reproduction or related acts for information analysis of a database work that is sold and that organizes a large volume of information in a form that can readily be used for information analysis. Furthermore, it states that, where technical measures have been taken to prevent reproduction or related acts for AI training, and it can be inferred from facts such as the existence of those measures and the past record that a database work for information analysis containing data from the website is planned for future sale, collecting large amounts of data with a crawler by circumventing those measures may fall within the proviso because it obstructs potential future sales channels for that database work. Instructions in robots.txt and authentication using IDs, passwords, or similar means are given as examples of such technical measures.

The important point here is that this analysis concerns the reproduction of the database work and the obstruction of its potential sales channels. It does not mean that merely posting a statement refusing use for training takes that use outside the limitations on rights. On the other hand, attention is also needed to the fact that the existence of technical measures is itself treated as one factor in deciding whether future sales can be inferred. Nor does it mean that opt-out directives have no legal significance at all.

The cases given in the document are examples and do not exhaust the cases that may fall within the proviso. The reverse conclusion, that the proviso cannot come into question unless there are plans to sell a database work, cannot be drawn either. Also, this is an account of interpretation by a subcommittee of the Cultural Council. It does not guarantee an outcome in an individual case, and the analysis concerning database works cannot be extended unchanged to every article included in them.

Use in search and AI answers: Article 47-5 of the Copyright Act

Where the company's articles are used in search results or AI answers, the application of provisions permitting minor use incidental to search or information analysis is checked separately from use for training.

Article 47-5 of the Copyright Act makes permission unnecessary, subject to certain conditions, for minor use incidental to the provision of search results or information analysis results and related acts. The checks concern whether providing search or analysis results is the primary activity and providing articles or other material is secondary; whether the use is minor in light of factors such as the proportion of the original work, the amount used, and the precision of the display; whether the interests of rights holders are unreasonably prejudiced; and whether the provider meets the statutory criteria. The presence of a short summary or a source link does not, by itself, make the use lawful.

Where URLs are searched and the results provided, directives in robots.txt and similar controls refusing collection for search purposes are also checked. The criteria concerning opt-out directives that follow general practice require that the information concerned not be provided; they are not a system that prohibits AI training generally. They need to be distinguished from directives refusing only training. Also, where reception is restricted through authentication or similar means, whether the person who imposed the restriction has consented comes into question.

The Article may also apply to services that create answers from materials found through search, but this is not determined by the label "AI answers" alone. Reproduction undertaken to prepare answers or similar output and the content shown to users are examined separately. Even where the conditions of the Article are not met, whether there are other grounds for lawful use needs to be checked.

(Legal basis: Article 47-5, paragraphs (1) and (2) of the Copyright Act; Article 7-4 of the Order for Enforcement of the Copyright Act; and Articles 4-4 and 4-5 of the Regulations for Enforcement of the Copyright Act)

Terms of use and individual contracts

Terms of use may state that "use for AI training is prohibited", but merely posting them does not establish a contract with every visitor. What should be checked is whether a contract can be said to have been formed with the other party and whether the prohibition forms part of that contract. Both the way agreement is obtained and the content of the terms need to be checked.

Where the other party can be identified, one option is to enter into an individual contract specifying purposes of use, onward provision, records, handling of deletion, and the handling of breaches. It needs to be understood beforehand that the contract does not automatically bind third parties outside it and that separate means of securing performance need to be put in place.

Also, breach of contract and copyright infringement are separate matters. A breach of the terms does not automatically establish copyright infringement.

4. Keeping records of collection and display and requesting cessation, deletion, or other measures

What to keep on record

A record is kept of the date the settings were applied, the settings at that time, and the version of the terms then in effect. The access logs are also used to check which crawler collected information, when, and to what extent.

Where the company's information is displayed in search results or AI answers, a record is kept of the question, the full answer, the links shown as sources, the service name, and the date and time. The records are then kept so that the version of the company's original text or images at the time can be compared with the specific expression believed to have been used.

The point to note is that the appearance of the company's information in an AI answer and its collection from the company's website are separate facts. Its appearance in an answer alone does not establish that it was collected from the company's website or used to train the model. Collection during search, collection from a site that republished the information, and provision by a third party are also investigated. Records of access and records of display are assessed as separate evidence.

Inquiries and requests to providers

Inquiries to the provider concern which crawler collected the information, for what purposes the information collected is being used, how the provider responds to opt-out directives, and what responses are possible in relation to information already collected and models already trained. The subject matter is then identified, and requests are made for collection to stop, for display to stop, and for deletion.

However, inquiries do not necessarily receive answers, and the company's logs alone do not necessarily identify who collected the information or whether it was used for training.

When making requests, a distinction is drawn between requests for action in accordance with the provider's published specifications and claims based on contract or copyright. The existence of published specifications does not automatically give the company a legal right to demand action. Asking the provider to act voluntarily, regardless of whether it has an obligation to do so, is also an option.

Distinguishing the targets of deletion requests

The targets of deletion requests are data already collected, datasets for future training, displays in search results or elsewhere, and models already trained. Even where copyright infringement is established, the scope of measures that may be granted differs between removal from a dataset and destruction of the model itself. The matter is not left at an explanation that deletion is technically difficult: the request identifies what is sought and in relation to which target.

Issuing an opt-out directive afterward does not automatically remove information already collected or reverse the results of its use for training. After checking when collection took place, whether there were other routes by which the information could have been obtained, and the grounds for the claim, the extent of what can realistically be sought is examined.

This guide explains the position under Japanese law. For collection and use abroad, the applicable law and jurisdiction need to be examined separately according to where the use occurred and the nature of the claim.

Preparing for changes to specifications

Crawler names, the syntax of directives, and the purposes of use covered may change. Applying settings once is not sufficient. A process is needed to check regularly whether the official specifications have changed and to review the settings when they have.

How we assist

We set out the use to be restricted by distinguishing collection, training, and display in search results, and check the scope of the rights held by the company
We set out the legal position of each measure and examine an approach to restrictions (we do not promise to implement the settings, nor do we promise that collection will be blocked with certainty)
We examine the application of Articles 30-4 and 47-5 of the Copyright Act in light of the acts concerned
We draft and review terms of use and individual contracts, including examining how agreement is obtained
We support the organization of records of collection and display and the drafting of inquiries and requests to providers
We support requests for cessation, deletion, or other measures and the examination of subsequent negotiations and claims

Frequently asked questions

Does collection by a crawler disallowed in robots.txt automatically constitute copyright infringement?

Not necessarily. Article 30-4 of the Copyright Act permits, within certain limits, use without the purpose of allowing thoughts or sentiments to be enjoyed, and use for information analysis is one such case. The Agency for Cultural Affairs examines whether the proviso to that Article applies from the perspective of whether potential sales channels for a database work organized in a form usable for information analysis are obstructed. The presence of an opt-out directive is one factor considered, but the analysis does not treat that fact alone as automatically establishing infringement. On the other hand, under Article 47-5 concerning the provision of search results, statements in robots.txt or similar controls refusing collection relate to the conditions for the application of limitations on rights. Which act is in issue therefore needs to be checked separately.

Does opting out through Google-Extended also stop inclusion in regular Google Search results or display in AI Overviews?

No. Google explains that Google-Extended neither affects whether a site is included in Google Search nor serves as a ranking signal. Just as this directive does not stop inclusion in regular search results, this directive alone does not stop AI features displayed on search results pages. Directives concerning collection and display for search need to be checked separately. Specifications may change, so please check the latest official information when applying the settings.

Can a URL remain in search results even after collection has been disallowed in robots.txt?

Yes. Directives that restrict crawling and directives that exclude a page from search results are distinct. Also, stopping collection through robots.txt may prevent the noindex instruction on the page itself from being read. Where the aim is to stop inclusion itself, the options to examine are using a directive that excludes the page while keeping the instruction readable, or restricting access through authentication or similar means.

Can all AI providers be bound by stating "use for AI training is prohibited" in the terms of use?

No. Terms have effect in the relationship with the other party to a contract that has been formed; merely posting them does not establish a contract with every visitor. How agreement is obtained, the content of the terms, and whether the other party can be said to be bound by them are checked in each case. Also, a breach of the terms and copyright infringement are separate matters. Where the other party can be identified, an individual contract specifying purposes of use and responses may be more effective.

If a training opt-out is set afterward, can a company require deletion of data already collected and used for training?

Deletion cannot automatically be required. First, the targets of a deletion request need to be considered separately: data already collected, datasets for future training, displays in search results or elsewhere, and models already trained. Even where copyright infringement is established, the scope of measures that may be granted differs between removal from a dataset and destruction of the model itself. After checking when collection took place, whether there were other routes by which the information could have been obtained, and how it was collected, the process begins with inquiries and requests to the provider. It is also necessary to distinguish the extent to which completion of deletion can be verified from the extent to which it cannot.

Contact

Making an inquiry

Please tell us the outline of the situation, any notice you have received and its date, and any deadline you are working to.