Assistant docsite rag

[hanna-paasivirta]

Short Description

Add v1 docsite RAG for the AI assistant.

Fixes #

Implementation Details

Add a docsite semantic search at the beginning of an AI assistant conversation.

Search trigger & process
This search uses the user's first message (and adaptor info as context) to decide whether to search our documentation. If so, it will generate one or more queries (with optional adaptor only/general docs only filters). These are used to perform a semantic search with a threshold & max docs limit. The returned documents are injected in the prompt.

Versioning, monitoring

• Automatically gets the latest docsite
• Returns doc retrieval info
• Uses yaml config to track RAG settings and prompts version

AI Usage

Please disclose how you've used AI in this work (it's cool, we just want to know!):

• Code generation (copilot but not intellisense)
• Learning or fact checking
• Strategy / design
• Optimisation / refactoring
• Translation / spellchecking / doc gen
• Other
• I have not used AI

You can read more details in our Responsible AI Policy

[hanna-paasivirta]

Todo:

• merge other PRs into this (claude version, docsite updates)
• Check when prompt generation cadence
• Check if search results formatting is optimal
• Test run & performance, possibly adjust threshold/max doc slightly

[josephjclark]

@hanna-paasivirta before releasing this, and if we haven't already, can we look at removing the describe_adaptor call from generate_system_message? The RAG stuff should be giving us a much more robust solution (that's something I need to test in detail actually)