In the world of WordPress and SEO, some pages should not appear on a search result page, and some even must be totally hidden from search engines. Noindex and Disallow are two crucial tools for doing it.
Noindex controls whether certain pages appear in search, while Disallow tells which parts of your site search engines can check out.
In this article, we’ll break down these concepts, showing you when and how to use them in WordPress.
Table of Contents
- Which Website Sections to Hide and Why?
- Noindex vs. Disallow: What to Choose?
- Disallow: Use Cases and How to Set It Up
- How to Noindex Pages, Posts, and Directories in WordPress
- FAQ
- Bottom Line
Which Website Sections to Hide and Why?
Some pages, posts, or directories are not meant to be “seen” by search engines, and there are two main reasons for this:
- security;
- SEO.
It’s easier with the first point (security) because it’s obvious that the Login page and system or private pages are not meant to be indexed. For most cases, the Disallow directive should be used here.
The other reason – SEO – is broader, and the Noindex directive is helpful in most cases. So, let’s dive deeper into this topic.
Some pages are not meant to appear in a search result page, for example, tag or archive pages, especially if you already have pages with a collection of posts designed the right way. Let’s say you display posts using JetEngine’s Listing Grid and absolutely don’t need to duplicate this content in the archive page, which also may look quite ugly, so users better never find it. Moreover, many SEO specialists consider that any archives must be hidden, even if they are well-designed and displayed to users, but they may cause content duplication.
The same goes for the author’s archives, which are created automatically by WordPress.
Another case is pages or templates generated by various plugins that add their own auxiliary CPTs. For example, galleries or other sections that are inserted into pages using shortcodes, mega menus, JetEngine Listing Items, JetWooBuilder templates, Theme parts, etc.
You can find elements that should be noindexed by using Google Search Console, checking a sitemap, or just manually setting Noindex to certain pages or directories if you understand that they should not appear in SERP.
After setting a Noindex directive, pages may still appear in a search, so it’s better to request re-crawling using the URL inspection tool. Later on, you will be able to see the page indexing report and debug things if needed.
Noindex vs. Disallow: What to Choose?
There are two directives for hiding pages/posts/taxonomies: Disallow and Noindex.
Noindex tells search engines not to index specific URLs so they will not appear in the search results.
Disallow instructs search engines not to crawl certain pages/posts/taxonomies and not to have access to them. However, it doesn’t protect this content from being indexed if it can be accessed through other means, such as external links.
Another very important role the Disallow directive plays is saving a crawling budget. Google robots have a certain limit for crawling your website per day, so closing unimportant website sections will save that budget and let it be used for really valuable pages.
It’s important to use them carefully, understanding what you want to achieve.
NOTE
If you want to apply the Noindex directive to the certain page/post/directory, it should not have the Disallow directive.
Now, the question is when to use each of the directives.
Disallow: Use Cases and How to Set It Up
The primary purpose of Disallow is to control which parts of your website search engine crawlers can access and instruct them not to check specific URLs. But Disallow doesn’t prevent indexing if other links lead to these pages, as I’ve mentioned above.
So, let’s boil it down to the most frequent use cases.
Use cases for Disallow directive:
- duplicating content;
- login page and admin areas;
- sensitive and private content;
- scripts and stylesheets;
- thin content: Thank you page, Search page, paginated pages, etc.;
- also can be tags and archive pages;
- XML sitemaps if they were generated separately and uploaded using Google Console;
- unimportant URLs that don’t add to quality but spend a crawling budget.
How to Disallow pages in WordPress
Disallow is a robots.txt directive. That’s why you must first find your robots.txt page or create it if it doesn’t exist yet.
💡 robots.txt is a file that holds the directives for search engine bots (crawlers) to instruct them on which part of the website to crawl and which not.
To check if the file exists, type yourdomain.com/robots.txt.
If it doesn’t exist, you can create it in the root directory of WordPress. It’s even faster using an SEO plugin. For example, in Yoast SEO, go to Yoast SEO > Tools > File Editor.
Now, write the directive (there can be posts/pages or specific directories/tags):
User-agent: *
Disallow: /my-private-page
Disallow: /my-secret-directory/
The asterisks after “user-agent” means all crawlers. To learn more about their types and how to disallow only a particular one, check this Google article.
Be very careful not to write just the following:
Disallow: /
It will close the whole website from indexing unless this is your goal.
To comment a line in the robots.txt, use the hash symbol (#).
Here are some examples of useful Disallow directives:
User-agent: *
#to hide login pages:
Disallow: /wp-admin/
Disallow: /wp-login.php
#to hide tag, category, and author archives:
Disallow: /tag/
Disallow: /category/
Disallow: /author/
#to hide core and plugin files:
Disallow: /wp-includes/
Disallow: /wp-content/plugins/After writing the directives, you can see whether it works by using the URL Inspection tool in Google Search Console.
How to Noindex Pages, Posts, and Directories in WordPress
Unlike the Disallow directive, Noindex is a meta tag, and it’s located at the beginning of each page (and not in robots.txt). It looks like this:
<meta name="mypage" content="noindex, follow">In WordPress, we can’t edit page HTML code directly, but it’s easy to do using SEO plugins. Let’s see how to do it using Yoast SEO.
If you use JetWooBuilder, you probably don’t want its templates to be indexed. To make it happen, go to Crocoblock > Woo Page Builder and open the template you want to hide. Click “Edit” (not with Elementor), and you will see the Yoast panel either on the right or below the main content area.
Go to the Advanced tab, and you’ll see the label “Allow search engines to show this content in search results?” with a dropdown. Choose “No” there and update the post.
You might also want to set the outcoming links to “Noindex” – in this case, use the next label, “Should search engines follow links on this content?” and choose “No.”
You can do the same to any page, post, or CPT.
To streamline the process, you can noindex certain CPTs or categories/tags in bulk, go to Yoast SEO > Settings, and choose the post or taxonomy you want to hide. Then, switch off the Show {your CPT/taxonomy term} in search results toggle and save changes.
If you use Rankmath, check this comprehensive guide on how to noindex posts using this plugin.
You can find even more cool settings that SEO plugins offer; check the Advanced and other sections in Settings. But it’s important to understand what you are doing and why so the website appearance in SERP won’t have issues.
FAQ
You should apply the Noindex directive to it.
Use SEO plugins that allow you to noindex individual pages.
Disallow is a robots.txt directive that tells search engine bots not to crawl certain pages/posts/taxonomies for multiple reasons: privacy, saving crawling budget, etc. Noindex is a meta tag directive that tells search engines not to show certain pages/posts/taxonomies in SERP.
Bottom Line
Setting the right directives for search engines is crucial for the proper indexing of any website and good positions in SERP. Noindex and Disallow are some of the most important of them and let you hide some parts of the website from crawlers or not show them in search results.
While Disallow is more about efficient search engine indexation, Noindex lets you control what potential site visitors will see and how they can find your website on the Internet without finding themselves on some auxiliary pages that are not able to attract or convert them.
