{"schemaVersion":1,"canonicalUrl":"https://namedproof.com/knowledge/en/llms-txt-evidence","topicId":"14","locale":"en","title":"What llms.txt can and cannot establish","description":"Use llms.txt as a maintained directory of public evidence, while keeping crawl controls, indexing and source credibility separate.","answer":"An llms.txt file can provide a concise directory and context for an AI agent using your public website. It does not certify the accuracy of linked claims, grant search indexing, prove that an agent used the material or guarantee a citation. Treat it as a maintained information interface with an explicit owner and evidence links.","category":"Technical access","author":{"type":"Organization","name":"NamedProof","url":"https://namedproof.com/about"},"publishedAt":"2026-10-17","sourceCheckedAt":"2026-10-17","sections":[{"id":"proposal","title":"Understand the proposal before adding a file","paragraphs":[{"text":"The llms.txt proposal describes a Markdown overview with context and links to more detailed information. Its v2 page, modified on 10 August 2026, also discusses clean Markdown page versions and links that help clients find them. The proposal is a useful description of an interface; it is not documentation of every search engine's selection process.","sources":["KB-LLMSTXT"]},{"text":"Google's current AI Search guide says Google Search does not use llms.txt or similar special AI files for visibility or rankings. Such files may serve other systems. Choose to maintain one for a real navigation need, with no Google ranking benefit asserted.","sources":["KB-GOOGLE-AI-GUIDE"]}]},{"id":"purposes","title":"Give each website file a defined job","paragraphs":[{"text":"A directory helps an agent find a relevant document. A crawling policy declares which automated access is permitted. A sitemap assists URL discovery. A public article explains a claim and its evidence. When these roles are mixed, a tidy file can create false confidence while the actual document is missing, stale or inaccessible.","sources":[]}],"table":{"caption":"Editorial comparison of file purposes; these are not interchangeable assurances.","headers":["Surface","Useful job","Still needs checking"],"rows":[["llms.txt","Curated context and links","Accuracy, relevance and working destinations"],["robots.txt","Crawler access rules","Actual retrieval and separate indexing controls"],["XML sitemap","Discoverable URL inventory","Whether the engine indexed the intended URL"],["Article/source page","Explain and support claims","Evidence scope, authorship and currency"]],"sources":["KB-LLMSTXT","S06","S07"]}},{"id":"evidence","title":"Link to claims a reader can inspect","paragraphs":[{"text":"Begin with the public questions an agent needs to answer. For an expert-assessment publisher, useful destinations might explain what an assessment covers, who is responsible for it, what methods were used and where limitations appear. Only list destinations that actually exist. Leave planned services and unpublished reports out of a directory describing available evidence.","sources":[]},{"text":"Separate official product facts from interpretation. A concise statement about who commissions research should link to an accurate policy or explanation. A study finding should lead to the actual study and its conditions. Labelling a page authoritative inside llms.txt does not turn its contents into verified evidence.","sources":[]}]},{"id":"consistency","title":"Use one underlying account of the facts","paragraphs":[{"text":"A machine-readable version should preserve the material claims and limits of the human-readable article. Simplification can remove menus and repeated navigation; it should not remove a funding disclosure, uncertainty or an important exclusion. If two formats disagree, fix the publishing process rather than asking an agent to choose whichever version flatters the business.","sources":[]},{"text":"Provenance frameworks distinguish information, its producers and the activities that generated it. That distinction is useful here: record which page supplies a fact, who maintains it and when it changed. Provenance helps a reader inspect the chain; it does not prove the fact itself is true.","sources":["S23"]}]},{"id":"maintenance","title":"Test usefulness with a bounded task","paragraphs":[{"text":"Use a hypothetical maintenance test: give a reviewer the public directory and ask them to find the assessment methodology and its limits. Record whether they reach the intended current page, whether the link resolves and whether the answer can be supported from it. This checks navigation and consistency. It does not measure a search engine's indexing or prove a lift in citations.","sources":[]},{"text":"Repeat that test after material changes. Removed pages, renamed services and translated versions need deliberate handling. A generated file can still be wrong if its input registry is wrong. Assign responsibility for source updates and specify what should happen when a linked page becomes private, withdrawn or superseded.","sources":[]}]},{"id":"decision","title":"Decide whether the file earns its maintenance cost","paragraphs":[{"text":"A useful llms.txt file reduces the effort of locating relevant public evidence. It becomes less useful when it duplicates a changing website without a reliable update path. Start with a small directory that serves real questions, then check actual client behaviour where observable. Do not use a file-presence score as the final measure of AI readiness.","sources":[]},{"text":"Keep private documents, credentials and personal data outside any public directory. If public access is unsuitable, the solution is an appropriate protected workflow. The next step for a public site is an inventory of existing evidence pages, a named maintainer and a test that confirms the directory leads to the right version.","sources":[]}]}],"sources":[{"id":"KB-LLMSTXT","title":"The llms.txt file, v2 — Jeremy Howard","url":"https://llmstxt.org/","checkedAt":"2026-10-17","scope":"Community proposal modified 10 August 2026. Format, curated overview and Markdown links; not Google ranking documentation or evidence every model uses it."},{"id":"S06","title":"Sitemap overview","url":"https://developers.google.com/search/docs/crawling-indexing/sitemaps/overview","checkedAt":"2026-10-17","scope":"Google URL discovery. Submitted, discovered, crawled and indexed are distinct states."},{"id":"S07","title":"Introduction to robots.txt","url":"https://developers.google.com/search/docs/crawling-indexing/robots/intro","checkedAt":"2026-10-17","scope":"Google crawler controls. Robots files are not security controls; a crawl block may prevent Google seeing noindex."},{"id":"S23","title":"PROV-Overview","url":"https://www.w3.org/TR/prov-overview/","checkedAt":"2026-10-17","scope":"Non-normative roadmap for W3C PROV family. Provenance can inform trust assessment but does not prove truth, competence or independence."},{"id":"KB-GOOGLE-AI-GUIDE","title":"Google guide to optimizing for generative AI features","url":"https://developers.google.com/search/docs/fundamentals/ai-optimization-guide","checkedAt":"2026-10-17","scope":"Google Search guidance updated July 2026; eligibility, excluded tactics and report reference."}],"relatedTopicIds":["13","11","15"],"methodNote":"NamedProof editorial guide, prepared with AI assistance from primary sources and an original practical analysis. It is not a commissioned expert assessment and has not been personally reviewed by a named subject specialist. Sources were checked on 17 October 2026.","limitations":["No experiment establishing a ranking, traffic or citation effect from llms.txt was conducted for this guide.","The cited llms.txt page is a community proposal; adoption and client behaviour vary.","A linked source still needs substantive verification, even when its file is well formed."],"originalContribution":"Original contribution: a comparison of information-interface roles and a repeatable navigation/consistency test for a public evidence directory.","relatedArticles":[{"title":"Robots, AI crawlers and indexing: check each permission separately","locale":"en","canonicalUrl":"https://namedproof.com/knowledge/en/robots-ai-crawlers-indexing"},{"title":"How AI search citations work: discovery, selection and support","locale":"en","canonicalUrl":"https://namedproof.com/knowledge/en/ai-search-citations"},{"title":"Generative engine optimization: what the evidence can support","locale":"en","canonicalUrl":"https://namedproof.com/knowledge/en/generative-engine-optimization-evidence"}],"availableTranslations":[{"title":"What llms.txt can and cannot establish","locale":"en","canonicalUrl":"https://namedproof.com/knowledge/en/llms-txt-evidence"}]}