Skip to content
KoishiAI
ไทย
← Contents

Chapter 2 of 5 · SEO for Thai Websites — Getting Google to Find, Understand and Choose Your Pages

Getting the right pages crawled and indexed: robots.txt, noindex, canonical, sitemaps and internal links

Each tool does a different job, and the wrong one does the opposite of what you want: robots.txt does not keep pages out of Google, noindex only works if Google can read it, canonical is a hint, and links hidden in a collapsed <details> are still followed.

Chapter 1 split Google’s work into three stages. This one covers the first two: getting Google to find the pages you want found, keep the pages you want kept, and not waste its time on the rest.

2.1 Five tools, five jobs

The most common mistake is using the wrong tool — hiding a page with robots.txt, for instance, which does the opposite of what was intended.

You want toUseDo not use
Reduce crawler load on the serverrobots.txt—
Keep a page out of resultsnoindex, a password, or removing the pagerobots.txt
Say which of several URLs is the main onea redirect or rel="canonical"robots.txt, noindex
Tell Google which pages exista sitemap—
Let Google reach a page and understand what it is aboutinternal <a href> linkslinks without an <a href>, such as an onclick alone

2.2 robots.txt: for load, not for hiding

Google writes that robots.txt is used “mainly to avoid overloading your site with requests” and that “it is not a mechanism for keeping a web page out of Google”.

If you block a page in robots.txt but someone links to it, Google can still index the URL, showing the URL and the link text but knowing nothing of the content. You end up with the page in results anyway, described badly — not what you wanted.

Use robots.txt when:

  • the site generates endless URLs, such as internal search pages that change with every filter;
  • there are crawlers you do not want, such as some AI-training crawlers (see the AI crawler checker at the end of this chapter).

To keep a page out of results, use noindex, next.

2.3 noindex: Google has to be able to read it

Two ways to set it:

<meta name="robots" content="noindex">

or the HTTP header X-Robots-Tag: noindex. The header works for HTML pages too, and it is the way to do it for files that have no <head>: Google says it can be used for non-HTML resources such as PDFs, video and image files.

The trap Google warns about: a page with noindex must not be blocked in robots.txt, or Google never sees the noindex and the page can keep appearing in results. Using both “to be safe” is exactly what breaks it.

noindex is also not instant. A page of low importance may wait months before Google fetches it again; you can ask for a recrawl with URL Inspection. If a page must disappear from results quickly, Search Console has a separate Removals tool that hides it for about six months — long enough for the noindex to take effect, but not a replacement for it.

2.4 canonical: saying which page is the real one

The same page is often reachable at several URLs: with and without a trailing /, with a campaign parameter such as ?utm_source=line, over both http and https. Google groups these and picks one representative. You can help with these signals, strongest first:

SignalStrength, per Google
A redirect to the main URLstrong
<link rel="canonical" href="…">strong
Listing the main URL in the sitemapweak

The signals stack, and they are hints: Google may choose another page if the evidence points there.

What Google says to avoid:

  • do not use robots.txt to choose a canonical;
  • do not let different methods name different URLs — the canonical says page A while the sitemap lists page B;
  • noindex is not recommended for choosing the main page within one site.

A reassuring point: Google says your content being reachable at several URLs is “inefficient, but it’s not something that will cause a manual action”. It adds that copying other people’s content is “a different story”. For your own URLs, canonical is housekeeping, not penalty avoidance.

2.5 sitemap: the list of pages you want Google to know

A sitemap is an XML file listing the site’s URLs. The main rules from Google and sitemaps.org:

  • at most 50,000 URLs and 50MB uncompressed per file; split larger ones and list them in a sitemap index;
  • URLs are absolute, starting with https://. sitemaps.org says they must be on the same host as the file; Google also accepts a sitemap for another site if you have verified every site in Search Console, or if that site’s robots.txt points to the sitemap;
  • include only the URLs you want in search results — the canonical ones — not pages with noindex or pages that redirect;
  • Google ignores <priority> and <changefreq>;
  • <lastmod> is used only if it is “consistently and verifiably accurate”; if every URL gets the same date each time the file is generated, Google may ignore it.

That last point is one KoishiAI still owes itself: our sitemap has no lastmod at all, which our own sitemap checker flags. The right fix is each page’s real last-edited date, not the time the file was built.

A sitemap not submitted through Search Console only covers URLs in its own folder: a file at /blog/sitemap.xml applies to pages under /blog/ only.

Google uses links both to find new pages and to understand what they are about. The links it reliably follows are <a> elements with an href.

Google follows theseNot reliably followed
<a href="/en/tools/sitemap-check"><span href="…">
<a href="https://example.com/"><a onclick="goTo('…')"> with no href
<a routerLink="…">

Link text should say what the target is: “sitemap checker” beats “click here”.

The site has many tag pages, and tags with few articles were set to noindex. Yet other pages still linked to those tags, tucked inside collapsed <details> boxes. We assumed that folded away meant out of the picture. But collapsing hides only what the eye sees; the links are all still in the HTML, so Googlebot followed them to pages we had told it not to keep.

Google’s crawl budget guide notes that noindex pages are still fetched and then dropped, “wasting crawling time”. That guide is written for large sites, so we do not claim those links were what held a small site back. But pointing Google at pages we had told it not to keep made no sense at any size. We removed every link to noindex pages as tidying, and added a test so they cannot come back.

To check your own site the same way, read the page’s HTML source, not the screen. Links you cannot see still count.

2.7 Crawl budget: most small sites need not worry

Google’s crawl budget guide is written for sites with more than 1 million unique pages changing weekly, more than 10,000 changing daily, or a large share of URLs stuck at “Discovered - currently not indexed”. If your new pages are crawled the day they are published, Google says you do not need to read it.

For a small site, crawl budget is not the issue; basic tidiness is:

  • do not link to noindex or duplicate pages;
  • return a real 404 for pages that do not exist, not a “not found” page with status 200 (a soft 404), which Google says keeps being crawled;
  • redirect moved URLs to the new page, and update internal links to point straight at the destination.

2.8 After this chapter

  • Open your robots.txt; if any line is there to hide pages, switch those pages to noindex and remove the line.
  • Check that no page with noindex is blocked in robots.txt.
  • Run your sitemap through the checker below: duplicates, relative URLs, or every lastmod the same.
  • Search the HTML source for links to noindex pages or to URLs that redirect.

Sources