Exceptions from Sphinx-Needs when clean builds are not used

[twodrops]

This is a user experience problem while using Sphinx-Needs that, when a Sphinx incremental build is used, Sphinx-Needs often raises exceptions. Most of the time this happens when there is a version upgrade of Sphinx-Needs or when the doctrees are in some inconsistent state. Most of the time, the solution is to do a clean Sphinx build. However, many users struggle until they realize that a clean build is the solution to the problem.

I see two possible solutions.

1. Every time Sphinx-Needs detects inconsistencies in cached doctrees, it automatically does a full build. If Sphinx-Needs can detect this early and request a full build from Sphinx, this would be ideal. I am unsure if this is technically feasible from an extension as they come in during "later" stages of a Sphinx build. Else we could create a PR for Sphinx to support this. To start with, a check could be as simple as, "is there a version difference of Sphinx-Needs that was used for the last build"
2. Sphinx-Needs catches all its exceptions and throws an error to the user with a hint to do a clean build.

[danwos]

1. wouldn't be so easy to detect a problem.
   We could store the used Sphinx-Needs version in the Sphinx env object, and compare with one from the currently use Sphinx-Needs package.
   But I'm afraid of a false-positive check and Sphinx-Needs requested a full build to often.

2. Could be doable. We could register a function for build-finished and check for exceptions.
   If this the case we can write some short text at the end of the log.

[danwos]

I want to close this issue because I see it more as something that should be solved in Sphinx itself.

What is described for Sphinx-Needs (inconsistent data, version upgrade, config change) is something all other Sphinx extensions have as well.
So for me Sphinx should be aware of if an extension raised its version between two builds. All extension must report their current version to Sphinx, so it should be doable.
Or if not done by Sphinx, then this check should be part of another extension.

We have made some tests for this with Sphinx-Needs (PR #936) and one of the problems was, that our implementation fetches problems of other extensions as well and Sphinx-Needs is unable to tell if these are a problem for the user or not.

[chrisjsewell]

a check could be as simple as, "is there a version difference of Sphinx-Needs that was used for the last build"

@twodrops @danwos This is already possible with env_version: https://www.sphinx-doc.org/en/master/extdev/index.html#extension-metadata

You can see here, that it simply excepts if this version has changed since the last incremental build: https://github.com/sphinx-doc/sphinx/blob/3ffde92c54c6e9d594b2a41a365a90dc0dcbac6c/sphinx/environment/__init__.py#L226

Obviously you would need to decide when to increment env_version, which should only be if something stored on the environment has changed