> ## Documentation Index
> Fetch the complete documentation index at: https://docs.filarank.com/llms.txt
> Use this file to discover all available pages before exploring further.

# robots.txt and llms.txt

> Edit /robots.txt in the panel, generate /llms.txt for AI crawlers, and keep a Sitemap line in sync.

FilaRank serves crawl files from Laravel routes, not from files in `public/`. Edit them under **FilaRank SEO → SEO Files**.

<Warning>
  If `public/robots.txt` exists, the web server serves that file and the panel editor has no effect. Delete the static file so `/robots.txt` is the virtual one.
</Warning>

## robots.txt

The route is on by default:

```php theme={null}
'files' => [
    'robots' => [
        'enabled' => true,
    ],
],
```

The editor ships presets:

* **Allow all** — `Allow: /` plus a `Sitemap:` line pointing at `/sitemap.xml`
* **Block AI crawlers** — common GPTBot / ClaudeBot / Google-Extended rules
* **Strict** — lock down paths you do not want indexed
* **Custom** — free-form text, with revision history

Saving a document that contains `Disallow: /` asks for confirmation so you cannot deindex the site by accident. Revisions are stored in `filarank_robots_revisions`.

Site-wide snippet limits live in config and merge into the robots meta tag, not into `robots.txt`:

```php theme={null}
'robots' => [
    'defaults' => [
        'max-snippet' => null,
        'max-image-preview' => null,
        'max-video-preview' => null,
    ],
],
```

## llms.txt

`/llms.txt` and `/llms-full.txt` are generated from [scanned models](/configuration#scanned-models). Unpublished and `noindex` records are skipped. `/llms-full.txt` appends each page's meta description.

```php theme={null}
'files' => [
    'llms' => [
        'enabled' => true,
        'limit' => 100,
    ],
],
```

Add a custom block in **SEO Files** if you want a human-written preamble above the generated page list.

Each model still needs `getSeoUrl()` so the file emits real paths. See [Wire SEO fields](/usage).

## Related

* [XML sitemap](/sitemap) — the `Sitemap:` line in robots.txt
* [IndexNow](/indexnow) — notify Bing when URLs change
* [Upgrade to v3](/upgrading)


This documentation is built and hosted on [Mintlify](https://mintlify.com), a developer documentation platform.