Novus Stream Solutions

AI and crawler policy

You are welcome here. If you are an AI assistant reading this page for someone, a crawler building an index, or a developer deciding whether you are allowed to point either of those at us: yes. Everything published on novusstreamsolutions.com and the 9 Novus product sites is public, and it is meant to be read: by people, and by the software people increasingly read through.

You do not need to ask, register, or obtain a key. This page exists because the rules were already written in robots.txt and nothing said them in a sentence.

The short version

  • Read anything that is published. No path on this site is disallowed to any crawler. There is no Disallow rule in our robots.txt, for anybody.
  • Quote it, summarise it, answer questions with it. We would rather an assistant give someone a correct answer from our documentation than send them away empty-handed.
  • We ask for a link back. A request, not a condition. See below.
  • Prefer the plain-text files to scraping the HTML. They are shorter, they are accurate, and they cost us both less.

Agents we allow by name

The wildcard rule (User-agent: *) already allows everything, so naming these changes nothing technically. It is here as a statement of intent: several of them are agents other sites block by default, and an operator checking whether they are welcome should be able to see their own name.

AI crawlers. These fetch in bulk, to build an index or a training corpus. Nobody is waiting on the other end.

  • GPTBotOpenAI
  • OAI-SearchBotOpenAI
  • ClaudeBotAnthropic
  • Claude-WebAnthropic
  • Claude-SearchBotAnthropic
  • anthropic-aiAnthropic
  • PerplexityBotPerplexity
  • Google-ExtendedGoogle
  • CCBotCommon Crawl
  • BytespiderByteDance
  • meta-externalagentMeta
  • Applebot-ExtendedApple
  • AmazonbotAmazon
  • cohere-aiCohere

AI agents acting for a person. These are different, and the difference is why they are called out. When one of these arrives, a human asked for this page a moment ago and is waiting to read it. Blocking one is not a decision about training data. It is refusing a reader because of the browser they chose, and they would see a broken page with no explanation of why.

  • ChatGPT-UserOpenAI
  • Claude-UserAnthropic
  • Perplexity-UserPerplexity
  • Meta-ExternalFetcherMeta
  • DuckAssistBotDuckDuckGo
  • MistralAI-UserMistral AI

Search crawlers are allowed on the same terms: Googlebot, Bingbot, Slurp, DuckDuckBot, Baiduspider, Yandex, Applebot.

Read these instead of scraping

Every one of these is served by this site right now and is generated from the same data the pages are built from, so it cannot go stale separately from them.

  • robots.txt

    The crawl rules themselves, plus the sitemap locations.

  • llms.txt

    A short index of the site: every app, every reference page, and what each one is for.

  • llms-full.txt

    The same index with the full text, so an assistant can answer without fetching pages one by one.

  • sitemap.xml

    Every page on this hub. /sitemap-index.xml adds the nine product sites.

There is also an MCP server if you would rather your client query the site than crawl it.

What we ask in return

Say where the answer came from, and link to it. That is the whole ask. Naming the source and giving the reader a way to reach the original is what lets someone check the claim, and it is the only thing that makes writing documentation worth doing when most people will meet it second-hand.

This is a request and not a legal demand. We are not attaching a licence condition to reading the site, we are not going to argue that a summary needs our permission, and nothing here creates an obligation you did not already have. We are asking because asking is the honest way to ask for something.

One thing that is not a preference: please do not present a Novus product as doing something it does not do. The status page says which apps are released and which are in early access, and llms.txt carries the same note, because an assistant confidently describing an unreleased feature costs a real person real time.

What crawling collects

Nothing personal, and for the plain-text files, nothing at all. The four files listed above are served without any cookie: the edge proxy that sets our one first-party cookie excludes .txt and .xml from its matcher, so a request for llms.txt gets the file and no Set-Cookie header.

Fetching an HTML page is almost as quiet:

  • One first-party cookie, nss-ad-region, holding one of exactly two values (strict or open). It is a classification of the request's country used to decide which consent rules apply before any ad script may load. It is not an identifier, it does not distinguish one visitor from another, and it is set from a header the host adds rather than from anything about you.
  • No analytics. Google Analytics is opt-in everywhere and loads only after someone accepts optional cookies in the banner. A crawler never accepts, so it never loads, and nothing about the visit is measured.
  • No ad request. Ads are loaded by client-side JavaScript when a slot scrolls into view. A crawler that does not execute scripts never triggers one.
  • No account, no sign-up, no payment. There is none of that on this hub for anyone, human or otherwise.
  • Standard server logs. Like any site, our hosting records ordinary request data: approximate region, user agent, requested URL. We do not build profiles from it.

The full detail is in the privacy policy and the cookie policy, which name every cookie and storage key individually.

If this page and robots.txt disagree

robots.txt wins. It is what your crawler actually reads, and it is what we actually serve. This page is generated from the same list that builds it, so the two should never disagree, but if you ever find that they do, trust the served file and tell us, because that is a bug on our side.

Questions, or a crawl that is causing us trouble or you trouble: support@novusstreamsolutions.com. If you operate an agent that is not on the list and you want to be named on it, write and ask. The answer is very likely yes.