View source for English Wikipedia @ Freddythechick:Controlling search engine indexing

{{Redirect|Wikipedia:INDEXING|pages about indexes (indices) of topics in Wikipedia|Wikipedia:Indexes|the WikiProject|Wikipedia:WikiProject Indexes|directories and indexes|Wikipedia:Directories and indexes}}
{{information page|WP:INDEXING|WP:NOINDEX}}

There are a variety of ways in which Wikipedia attempts to '''control search engine indexing''', commonly termed "noindexing" on Wikipedia. The default behavior is that articles older than 90 days are indexed. All of the methods rely on using the [[noindex]] [[HTML]] [[meta tag]], which tells [[search engine]]s not to index certain pages. Respecting the tag, especially in terms of removing already indexed content, is up to the individual search engine, and in theory the tag may be ignored entirely.

The control methods are:
# Controlling an entire [[Wikipedia:Namespace|namespace]], via [[MediaWiki]] software settings
# Controlling classes of pages, via [[MediaWiki:Robots.txt]] (Wikipedia's [[Robots.txt]] file)
# Controlling individual pages by adding the <code><nowiki>__NOINDEX__</nowiki></code> [[Help:magic word|magic word]] into them, either directly or using the {{tl|NOINDEX}} template, however articles are a special case, see [[#Indexing of articles ("mainspace")]].
# Controlling multiple pages by adding the <code><nowiki>__NOINDEX__</nowiki></code> magic word into standard templates used in certain situations (same caveat as in the third point).

{| class="wikitable" style="margin-left: auto; margin-right: auto; border: none;"
|-
! Namespace !! Status || Indexed !! Can be overridden
|-
| rowspan=3 | (main) || newer than 90 days, unpatrolled || {{no}} || {{no}}
|-
| newer than 90 days, patrolled || {{yes}} || {{yes}}
|-
| older than 90 days || {{yes}} || {{no}}
|-
| rowspan=3 | <code>User:</code> || newer than 90 days, unpatrolled || {{no}} || {{no}}
|-
| newer than 90 days, patrolled || {{no}} || {{yes}}
|-
| older than 90 days || {{no}} || {{yes}}
|-
| <code>User talk:</code> || rowspan=4 | n/a || {{no}} || {{yes}}
|-
| <code>Draft:</code> || {{no}} || {{no}}
|-
| <code>Draft talk:</code> || {{no}} || {{no}}
|-
| All others || {{yes}} || {{yes}}
|}

== Indexing of articles ("mainspace") ==
Articles older than 90 days are automatically indexed.<ref>[[phab:T166852|2017 switch from 30 to 90 days]]</ref> The <code><nowiki>__NOINDEX__</nowiki></code> magic word and the {{tl|NOINDEX}} template do not work on them. Articles younger than 90 days are not indexed, unless they have been [[Wikipedia:New pages patrol|patrolled]] and do not have the <code><nowiki>__NOINDEX__</nowiki></code> magic word or the {{tl|NOINDEX}} template on them (or a template that transcludes the {{tl|NOINDEX}} template, such as the speedy deletion templates).<ref>{{phab|T147544}}</ref><ref>[https://github.com/wikimedia/mediawiki-extensions-PageTriage/blob/24b1f872063e37c5fe59764080d42cb23a9ecf5c/includes/Hooks.php#L295 PageTriage source code]</ref><ref>[https://github.com/wikimedia/operations-mediawiki-config/blob/master/wmf-config/CommonSettings.php#L328 Value of $wgRCMaxAge on WMF wikis]</ref> Note that &action=info will incorrectly state that they are indexed.<ref>See {{phab|T157747}}</ref> Articles that include the {{tl|NOINDEX}} template are listed at [[:Category:Noindexed articles]].

This patrolling may be done automatically by the software, as in the case of articles created by editors with the [[WP:AUTOPATROL|autopatrolled]] user right, or by another editor with the [[Wikipedia:New pages patrol/Reviewers|new page reviewer]] user right (not to be confused with the [[Wikipedia:Reviewing pending changes|pending changes reviewer]] user right).

== Other namespaces and robots.txt ==
=== Namespace control ===
On English Wikipedia the entire <code>User:</code><ref>Decided at [[Wikipedia:Village pump (proposals)/Archive 126#Userpage drafts shown in search engines]], implemented at [[phab:T104797]].</ref> [[Wikipedia:Namespace|namespace]], <code>User talk:</code>, <code>Draft:</code> and <code>Draft talk:</code> namespaces are automatically noindexed via a software setting.<ref>This is [[mw:Manual:$wgNamespaceRobotPolicies|$wgNamespaceRobotPolicies]]. See [https://noc.wikimedia.org/wiki.php?wiki=enwiki Wikimedia's $wgNamespaceRobotPolicies setting for enwiki]</ref>

At the same time, <code><nowiki>__NOINDEX__</nowiki></code> and <code><nowiki>__INDEX__</nowiki></code> are disabled, in addition to [[Wikipedia:Main namespace|article space]], on the [[Wikipedia:Draft namespace|Draft namespace]], and the Draft talk namespace; they have no effect there.<ref>This is controlled by the [[MediaWiki]] software setting [[mw:Manual:$wgExemptFromUserRobotsControl|$wgExemptFromUserRobotsControl]].  On other projects, the exempt namespaces are the same as [[mw:Manual:$wgContentNamespaces|$wgContentNamespaces]], which is set to main space on almost all Wikimedia projects – see [https://phabricator.wikimedia.org/diffusion/MW/browse/master/includes/DefaultSettings.php here] and [http://noc.wikimedia.org/conf/highlight.php?file=InitialiseSettings.php here].</ref>

=== Robots.txt noindexing ===
[[MediaWiki:Robots.txt]] forbids analytic tools from visiting sensitive or potentially sensitive types of pages, primarily in the Wikipedia namespace – for example [[Wikipedia:Deletion process|deletion debates]]. A side effect of not visiting is normally that a page cannot be indexed. Where possible, you should in addition use <code><nowiki>__NOINDEX__</nowiki></code> for those pages.

== NOINDEX magic word ==
=== Individual pages ===
Individual pages can be noindexed by adding the <code><nowiki>__NOINDEX__</nowiki></code> [[Help:Magic words|magic word]] into that page, either directly or using the {{tl|NOINDEX}} template. As explained above, this magic word doesn't work in mainspace (on articles).

Pages with the keyword are listed in [[:Category:Noindexed pages]].<ref>The listing is done by [[MediaWiki]] tracking the keyword. The category name is determined by [[MediaWiki:Noindex-category]].</ref>

=== Standard template noindexing ===
Some standard templates include the <code><nowiki>__NOINDEX__</nowiki></code> keyword, thereby noindexing pages to which the templates are applied. Such templates should be listed in [[:Category:Wikipedia templates which apply NOINDEX]].

==== Biographies of Living Persons talkpage noindexing ====
The templates {{tl|BLP}} and {{tl|BLP others}} include the {{tl|NOINDEX}} parameter. The {{tl|BLP}} template is added automatically by the {{tl|WikiProject Biography}} talkpage template, if given the parameter <code>|living=yes</code>; see the documentation of that template for more details. Pages using these templates are automatically categorised in [[:Category:Biography articles of living people]].

==== Other templates ====
These templates include {{tl|NOINDEX}}:
* {{tl|User sandbox}}
* {{tl|Sockpuppet}}, {{tl|Sockpuppeteer}}, {{tl|Banned user}}, and others
* {{tl|Db-meta}} and {{tl|Deletable file}}, plus the various [[Wikipedia:Criteria for speedy deletion|speedy deletion]] templates built on it
* {{tl|Prod blp}}
See also [[:Category:Wikipedia templates which apply NOINDEX]].
* {{tl|Uw-userspacenoindex}} provides a user warning message for inappropriate use of userspace which required noindexing.

== INDEX magic word ==
=== Individual pages ===
Individual pages can override namespace noindexing by adding the <code><nowiki>__INDEX__</nowiki></code> [[Help:magic word|magic word]] into that page, either directly or using the {{tl|INDEX}} template. Such pages appear in [[:Category:Indexed pages]]. However, INDEX does ''not'' override noindexing via [[MediaWiki:Robots.txt]].<ref>It does override [[mw:Manual:$wgArticleRobotPolicies]], but this is not used on English Wikipedia: [https://noc.wikimedia.org/wiki.php?wiki=enwiki Wikimedia's $wgArticleRobotPolicies setting for enwiki]</ref> As explained above, this magic word doesn't work in mainspace (on articles).

The ability to add the INDEX magic word to user spaces (User:, User talk:) has been restricted by [[Special:AbuseFilter/930|an edit filter]] to [[Wikipedia:User_access_levels#Extended_confirmed_users|extended confirmed]] users following a community discussion.<ref>[[Special:PermaLink/862856598#Prevent_new_users_from_allowing_search_engine_indexing_of_user_pages]]</ref>

== Nofollow HTML attribute ==
{{Main|Meta:nofollow}}

Since 2007, all links to other websites from English Wikipedia have the [[nofollow]] HTML attribute set.<ref>Controlled by [[:mw:Manual:$wgNoFollowLinks|$wgNoFollowLinks]], set to true in [on Wikimedia's settings file for enwiki]</ref> This means that on pages that are indexed by search engines, any links found by a search engine on those pages should not influence the link target's ranking in the search engine's index.

== Past discussions ==
=== Namespace discussions ===
* [[Wikipedia:Requests for comment/User page indexing]] (2009 proposal)
* [[Wikipedia:Search engine indexing]] – 2009 proposal to change the namespace settings for indexing
* [https://en.wikipedia.org/w/index.php?title=Wikipedia:NOINDEX_of_noticeboards&oldid=233157884 Wikipedia:NOINDEX of noticeboards] – Dead/moot proposal to NOINDEX noticeboards (2008)
* [[Wikipedia:Village pump (proposals)/Archive 35#Namespaces in Robot.txt]] – 2008 proposal to noindex several obscure namespaces like "Image talk." Strong majority opposed.
* [[Wikipedia:Village pump (proposals)/Archive 36#Re-enable searches in the user talk space]] – Proposal to re-index user talk pages. Majority opposed.
* [[Wikipedia:Village pump (policy)/Archive 59#NOINDEX of all non-content namespaces]] – Mixed discussion to exclude all non-content namespaces from indexing.
* [[Wikipedia:Village pump (policy)/Archive 62#Where and when to use NOINDEX to remove pages from search engines]] – Proposal to exclude certain pages from indexing.
* [[Wikipedia:Talk pages not indexed by Google]] – A proposal to tell Google not to index the Talk: namespace.
* [[Wikipedia:Requests for comment/NOINDEX]] – Proposal to NOINDEX unpatrolled new articles and articles with specific deletion templates.
* [[Wikipedia:Village pump (proposals)/Archive 126#Userpage drafts shown in search engines]] Noindexed userspace by default
* [[Wikipedia:Village_pump (proposals)/Archive 173#Deindexing talk pages]] – resulted in no consensus

=== Individual template discussions ===
* [[Template talk:Non-free media#Adding NOINDEX]] – Proposal to NOINDEX non-free images. No consensus.
* [[Template talk:WikiProject Biography/Archive 5#Noindex]] – Proposal to NOINDEX BLP talk page template
* [[Template talk:Administrators' noticeboard navbox all]] – NOINDEX on AN archives template

==See also==
* [[wmf:Notices received from search engines]], in particular notifications the Wikimedia Foundation received from Google about [[Right to be forgotten]] deletions of Wikipedia pages from search results in the EU

== Notes ==
{{reflist}}

[[Category:Wikipedia features]]