docs: publish the generated llms.txt instead of a hand-written copy (#8922)

## Why

`docs/llms.txt` is written by hand and was last touched in February, so
the index served at <https://langchain-ai.github.io/langgraph/llms.txt>
had drifted badly:

- 12 pages listed, against the 43 the docs site publishes.
- One entry, `/oss/python/langgraph/why-langgraph`, now 404s.

The deploy workflow only fires on `docs/**` pushes, so nothing brought
the file back in line as the docs site changed. It is served verbatim:
`generate_redirects.py` copies it into `docs/_site` via a `static_files`
list, and `deploy-redirects.yml` uploads that directory to Pages.

The docs site already generates exactly this index, and regenerates it
on every deploy:

```
https://docs.langchain.com/oss/python/langgraph/llms.txt
```

## What changed

`generate_redirects.py` now fetches that index at build time and writes
it into `_site`, so the published file cannot drift from the docs site.
The committed `docs/llms.txt` stays as a fallback only, refreshed here
to current content.

`deploy-redirects.yml` gains a weekly `schedule:` trigger, so docs
changes that never touch this repo still reach the deployed file. The
`permissions:` block is unchanged.

## Fetching remote content in CI

The fetched body is published on a public Pages site, so it is validated
before it is written:

- The URL is a hardcoded module constant, never built from input.
- Both that URL and the post-redirect `response.url` are checked against
an HTTPS-plus-single-host allowlist. `urlopen` follows redirects, so
checking only the request URL would not be enough.
- 30 second timeout, response capped at 1MB, decoded as UTF-8.
- The body must open with a markdown heading and contain a
docs.langchain.com link, which rejects an error page or a truncated
response.

Any failure returns `None` and falls back to the committed copy, so a
docs.langchain.com outage degrades to a stale file rather than a broken
deploy or a published error page.

## Verification

Ran `python docs/generate_redirects.py`: 294 redirect files, `llms.txt`
fetched, 43 entries.

Each rejection path was exercised against a stubbed `urlopen` and all
fall back to the committed file:

| Case | Result |
|---|---|
| Network error | falls back |
| Timeout | falls back |
| Redirect to another host | falls back |
| Body over 1MB | falls back |
| Invalid UTF-8 | falls back |
| HTML error page | falls back |
| Empty body | falls back |
| Valid index | published |

Allowlist rejects `http://docs.langchain.com/...`,
`https://evil.com/...`, and `https://docs.langchain.com.evil.com/...`.

## Note on ordering

The fallback committed here is labelled "LangGraph (Python)", which is
what the docs site will serve once langchain-ai/docs#6032 deploys. Until
then the fetched file reads "Open source (Python)". No action needed; it
self-corrects.

---

Written with Claude Code; I reviewed the diff and ran the verification
above.

Author: lnhsingh

.github/workflows/deploy-redirects.yml

@@ -7,6 +7,10 @@ on:
     paths:
       - 'docs/**'
       - '.github/workflows/deploy-redirects.yml'
+  # llms.txt is fetched from docs.langchain.com at build time, so redeploy on a
+  # schedule to pick up docs changes that never touch this repo.
+  schedule:
+    - cron: '17 6 * * 1'
   workflow_dispatch:
 
 permissions:

docs/generate_redirects.py

@@ -12,13 +12,26 @@
 To add new redirects, simply edit redirects.json and re-run this script.
 """
 
+import http.client
 import json
 import os
+import urllib.error
+import urllib.parse
+import urllib.request
 from pathlib import Path
 
 # Default fallback URL for any path not in the redirect map
 DEFAULT_REDIRECT = "https://docs.langchain.com/oss/python/langgraph/overview"
 
+# The docs site regenerates this index on every deploy, so fetching it here
+# keeps the published llms.txt from drifting. The URL is a hardcoded constant,
+# never built from input, and both it and the post-redirect URL are checked
+# against ALLOWED_LLMS_HOST before anything is read.
+CANONICAL_LLMS_URL = "https://docs.langchain.com/oss/python/langgraph/llms.txt"
+ALLOWED_LLMS_HOST = "docs.langchain.com"
+LLMS_FETCH_TIMEOUT = 30
+LLMS_MAX_BYTES = 1_000_000
+
 HTML_TEMPLATE = """<!doctype html>
 <html lang="en">
 <head>
@@ -75,6 +88,64 @@
 """
 
 
+def is_allowed_llms_url(url):
+    """Return True if url is HTTPS on the one host we accept content from."""
+    parsed = urllib.parse.urlsplit(url)
+    return parsed.scheme == "https" and parsed.hostname == ALLOWED_LLMS_HOST
+
+
+def fetch_canonical_llms_txt():
+    """Return the published LangGraph index, or None if it cannot be used.
+
+    Returning None leaves the caller on the committed docs/llms.txt, so a
+    docs.langchain.com outage degrades to a stale file rather than a broken
+    deploy or a published error page.
+    """
+    if not is_allowed_llms_url(CANONICAL_LLMS_URL):
+        print(f"Refusing to fetch {CANONICAL_LLMS_URL}: host not allowed")
+        return None
+
+    try:
+        with urllib.request.urlopen(  # noqa: S310 - constant, allowlisted URL
+            CANONICAL_LLMS_URL, timeout=LLMS_FETCH_TIMEOUT
+        ) as response:
+            # urlopen follows redirects, so re-check where it actually landed.
+            if not is_allowed_llms_url(response.url):
+                print(f"Refusing {CANONICAL_LLMS_URL}: redirected to {response.url}")
+                return None
+            body = response.read(LLMS_MAX_BYTES + 1)
+    # A connection dropped mid-body raises http.client.IncompleteRead, which
+    # descends from HTTPException rather than OSError, so catching only the
+    # urllib and OS errors would let it escape and fail the whole deploy.
+    except (
+        urllib.error.URLError,
+        http.client.HTTPException,
+        TimeoutError,
+        OSError,
+    ) as exc:
+        print(f"Could not fetch {CANONICAL_LLMS_URL}: {type(exc).__name__}: {exc}")
+        return None
+
+    if len(body) > LLMS_MAX_BYTES:
+        print(f"Refusing {CANONICAL_LLMS_URL}: larger than {LLMS_MAX_BYTES} bytes")
+        return None
+
+    try:
+        text = body.decode("utf-8")
+    except UnicodeDecodeError as exc:
+        print(f"Refusing {CANONICAL_LLMS_URL}: not valid UTF-8: {exc}")
+        return None
+
+    # An index opens with a markdown heading and links to the docs site. A
+    # body that does not is an error page or a truncated response, not content
+    # worth publishing.
+    if not text.startswith("# ") or f"https://{ALLOWED_LLMS_HOST}/" not in text:
+        print(f"Refusing {CANONICAL_LLMS_URL}: does not look like an llms.txt index")
+        return None
+
+    return text
+
+
 def generate_redirects():
     script_dir = Path(__file__).parent
     output_dir = script_dir / "_site"
@@ -126,14 +197,19 @@ def generate_redirects():
     catchall_404.write_text(CATCHALL_404_TEMPLATE.format(default_url=DEFAULT_REDIRECT))
     print(f"Created: {catchall_404}")
 
-    # Copy static files (like llms.txt) that can't be redirected via HTML
-    static_files = ["llms.txt"]
-    for static_file in static_files:
-        src = script_dir / static_file
+    # llms.txt can't be redirected via HTML, so publish the docs site's own
+    # generated index. The committed copy is only a fallback.
+    llms_txt = fetch_canonical_llms_txt()
+    if llms_txt is not None:
+        (output_dir / "llms.txt").write_text(llms_txt)
+        print(f"Fetched: {output_dir / 'llms.txt'} (from {CANONICAL_LLMS_URL})")
+    else:
+        src = script_dir / "llms.txt"
         if src.exists():
-            dst = output_dir / static_file
-            dst.write_text(src.read_text())
-            print(f"Copied: {dst}")
+            (output_dir / "llms.txt").write_text(src.read_text())
+            print(f"Copied: {output_dir / 'llms.txt'} (fallback, may be stale)")
+        else:
+            print("No llms.txt fetched and no committed fallback; skipping")
 
     print(f"\nGenerated {len(redirects)} redirect files in {output_dir}")
 

docs/llms.txt

@@ -1,35 +1,49 @@
-# LangGraph
-
-LangGraph documentation has moved to docs.langchain.com.
-
-## Overview
-
-- [LangGraph Overview](https://docs.langchain.com/oss/python/langgraph/overview): Introduction to LangGraph, a library for building stateful, multi-actor applications with LLMs.
-- [Why LangGraph?](https://docs.langchain.com/oss/python/langgraph/why-langgraph): Motivation for LangGraph and its key features.
-
-## Core Concepts
-
-- [Graph API](https://docs.langchain.com/oss/python/langgraph/graph-api): Learn how to define state, create nodes, and connect them with edges.
-- [Streaming](https://docs.langchain.com/oss/python/langgraph/streaming): Stream outputs from your graph for better UX.
-- [Persistence](https://docs.langchain.com/oss/python/langgraph/persistence): Add memory and checkpointing to your graphs.
-- [Add Memory](https://docs.langchain.com/oss/python/langgraph/add-memory): Implement short-term and long-term memory.
-- [Workflows & Agents](https://docs.langchain.com/oss/python/langgraph/workflows-agents): Build agents and workflows with LangGraph.
-
-## How-To Guides
-
-- [Use Subgraphs](https://docs.langchain.com/oss/python/langgraph/use-subgraphs): Compose graphs using subgraphs.
-- [Observability](https://docs.langchain.com/oss/python/langgraph/observability): Add tracing and debugging to your graphs.
-- [Common Errors](https://docs.langchain.com/oss/python/langgraph/common-errors): Troubleshoot common LangGraph errors.
-
-## Tutorials
-
-- [Agentic RAG](https://docs.langchain.com/oss/python/langgraph/agentic-rag): Build an agentic RAG system with LangGraph.
-- [SQL Agent](https://docs.langchain.com/oss/python/langgraph/sql-agent): Create a SQL agent with LangGraph.
-
-## Reference
-
-- [API Reference](https://reference.langchain.com/python/langgraph/): Complete API documentation for LangGraph.
-
-## LangGraph Platform
-
-For deploying LangGraph applications in production, see the [LangSmith documentation](https://docs.langchain.com/langsmith/agent-server).
+# Docs by LangChain: LangGraph (Python)
+
+> Markdown index of the LangGraph (Python) documentation.
+
+## LangGraph (Python)
+
+- [Memory](https://docs.langchain.com/oss/python/langgraph/add-memory.md)
+- [Build a custom RAG agent with LangGraph](https://docs.langchain.com/oss/python/langgraph/agentic-rag.md)
+- [Application structure](https://docs.langchain.com/oss/python/langgraph/application-structure.md)
+- [Backward compatibility](https://docs.langchain.com/oss/python/langgraph/backward-compatibility.md)
+- [Case studies](https://docs.langchain.com/oss/python/langgraph/case-studies.md)
+- [Changelog](https://docs.langchain.com/oss/python/langgraph/changelog-js.md)
+- [Changelog](https://docs.langchain.com/oss/python/langgraph/changelog-py.md)
+- [Checkpointers](https://docs.langchain.com/oss/python/langgraph/checkpointers.md)
+- [Choosing between Graph and Functional APIs](https://docs.langchain.com/oss/python/langgraph/choosing-apis.md)
+- [Deployment](https://docs.langchain.com/oss/python/langgraph/deploy.md)
+- [GRAPH_RECURSION_LIMIT](https://docs.langchain.com/oss/python/langgraph/errors/GRAPH_RECURSION_LIMIT.md)
+- [INVALID_CHAT_HISTORY](https://docs.langchain.com/oss/python/langgraph/errors/INVALID_CHAT_HISTORY.md)
+- [INVALID_CONCURRENT_GRAPH_UPDATE](https://docs.langchain.com/oss/python/langgraph/errors/INVALID_CONCURRENT_GRAPH_UPDATE.md)
+- [INVALID_GRAPH_NODE_RETURN_VALUE](https://docs.langchain.com/oss/python/langgraph/errors/INVALID_GRAPH_NODE_RETURN_VALUE.md)
+- [MISSING_CHECKPOINTER](https://docs.langchain.com/oss/python/langgraph/errors/MISSING_CHECKPOINTER.md)
+- [MULTIPLE_SUBGRAPHS](https://docs.langchain.com/oss/python/langgraph/errors/MULTIPLE_SUBGRAPHS.md)
+- [Event streaming](https://docs.langchain.com/oss/python/langgraph/event-streaming.md)
+- [Fault tolerance](https://docs.langchain.com/oss/python/langgraph/fault-tolerance.md)
+- [Custom stream channels](https://docs.langchain.com/oss/python/langgraph/frontend/custom-stream-channels.md)
+- [Graph execution](https://docs.langchain.com/oss/python/langgraph/frontend/graph-execution.md)
+- [Overview](https://docs.langchain.com/oss/python/langgraph/frontend/overview.md)
+- [Functional API overview](https://docs.langchain.com/oss/python/langgraph/functional-api.md)
+- [Graph API overview](https://docs.langchain.com/oss/python/langgraph/graph-api.md)
+- [Install LangGraph](https://docs.langchain.com/oss/python/langgraph/install.md)
+- [Interrupts](https://docs.langchain.com/oss/python/langgraph/interrupts.md)
+- [Run a local server](https://docs.langchain.com/oss/python/langgraph/local-server.md)
+- [LangSmith Observability](https://docs.langchain.com/oss/python/langgraph/observability.md)
+- [LangGraph overview](https://docs.langchain.com/oss/python/langgraph/overview.md)
+- [Persistence](https://docs.langchain.com/oss/python/langgraph/persistence.md)
+- [LangGraph runtime](https://docs.langchain.com/oss/python/langgraph/pregel.md)
+- [Quickstart](https://docs.langchain.com/oss/python/langgraph/quickstart.md)
+- [Build a custom SQL agent](https://docs.langchain.com/oss/python/langgraph/sql-agent.md)
+- [Stores](https://docs.langchain.com/oss/python/langgraph/stores.md)
+- [Streaming](https://docs.langchain.com/oss/python/langgraph/streaming.md)
+- [LangSmith Studio](https://docs.langchain.com/oss/python/langgraph/studio.md)
+- [Test](https://docs.langchain.com/oss/python/langgraph/test.md)
+- [Thinking in LangGraph](https://docs.langchain.com/oss/python/langgraph/thinking-in-langgraph.md)
+- [Agent Chat UI](https://docs.langchain.com/oss/python/langgraph/ui.md)
+- [Use the functional API](https://docs.langchain.com/oss/python/langgraph/use-functional-api.md)
+- [Use the graph API](https://docs.langchain.com/oss/python/langgraph/use-graph-api.md)
+- [Subgraphs](https://docs.langchain.com/oss/python/langgraph/use-subgraphs.md)
+- [Use time-travel](https://docs.langchain.com/oss/python/langgraph/use-time-travel.md)
+- [Workflows and agents](https://docs.langchain.com/oss/python/langgraph/workflows-agents.md)