robots.txt and llms.txt

FilaRank serves crawl files from Laravel routes, not from files in public/. Edit them under FilaRank SEO → SEO Files.

If public/robots.txt exists, the web server serves that file and the panel editor has no effect. Delete the static file so /robots.txt is the virtual one.

robots.txt

The route is on by default:

'files' => [
    'robots' => [
        'enabled' => true,
    ],
],

The editor ships presets:

Allow all — Allow: / plus a Sitemap: line pointing at /sitemap.xml
Block AI crawlers — common GPTBot / ClaudeBot / Google-Extended rules
Strict — lock down paths you do not want indexed
Custom — free-form text, with revision history

Saving a document that contains Disallow: / asks for confirmation so you cannot deindex the site by accident. Revisions are stored in filarank_robots_revisions. Site-wide snippet limits live in config and merge into the robots meta tag, not into robots.txt:

'robots' => [
    'defaults' => [
        'max-snippet' => null,
        'max-image-preview' => null,
        'max-video-preview' => null,
    ],
],

llms.txt

/llms.txt and /llms-full.txt are generated from scanned models. Unpublished and noindex records are skipped. /llms-full.txt appends each page’s meta description.

'files' => [
    'llms' => [
        'enabled' => true,
        'limit' => 100,
    ],
],

Add a custom block in SEO Files if you want a human-written preamble above the generated page list. Each model still needs getSeoUrl() so the file emits real paths. See Wire SEO fields.

XML sitemap — the Sitemap: line in robots.txt
IndexNow — notify Bing when URLs change
Upgrade to v3

Getting Started

Usage

Analysis

Crawl & indexing

Site health

Research

AI

robots.txt and llms.txt

robots.txt

llms.txt

​robots.txt

​llms.txt

​Related

robots.txt

llms.txt

Related