Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

chore: Add tooling to detect missing specification tests #23

Merged
merged 1 commit into from
Sep 21, 2023

Conversation

justinabrahms
Copy link
Member

This PR

This adds a tool so we know which spec tests are missing. It is nearly identical to https://github.com/open-feature/java-sdk/blob/main/spec_finder.py, save the regexps.

How to test

$ python spec_finder.py
In the spec, but not in our tests: 
  1.1.1: The API, and any state it maintains SHOULD exist as a global singleton, even in cases wherein multiple versions of the API are present at runtime.
  1.1.2.1: The API MUST define a provider mutator, a function to set the default provider, which accepts an API-conformant provider implementation.
  1.1.2.2: The provider mutator function MUST invoke the initialize function on the newly registered provider before using it to resolve flag values.
  1.1.2.3: The provider mutator function MUST invoke the shutdown function on the previously registered provider once it's no longer being used to resolve flag values.
  1.1.3: The API MUST provide a function to bind a given provider to one or more client names. If the client-name already has a bound provider, it is overwritten with the new mapping.
  1.1.4: The API MUST provide a function to add hooks which accepts one or more API-conformant hooks, and appends them to the collection of any previously added hooks. When new hooks are added, previously added hooks are not removed.
  1.1.5: The API MUST provide a function for retrieving the metadata field of the configured provider.
  1.1.6: The API MUST provide a function for creating a client which accepts the following options:  - name (optional): A logical string identifier for the client.
  1.1.7: The client creation function MUST NOT throw, or otherwise abnormally terminate.
  1.1.8: The API SHOULD provide functions to set a provider and
....a bunch more....

Verified

This commit was signed with the committer’s verified signature.
justinabrahms Justin Abrahms
This is nearly identical to https://github.com/open-feature/java-sdk/blob/main/spec_finder.py, save the regexps

Signed-off-by: Justin Abrahms <justin@abrah.ms>
Copy link
Member

@beeme1mr beeme1mr left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm a bit bummed you didn't write this in Rust.

@sheepduke
Copy link
Contributor

Just one question: How do you handle the Conditions in the spec? :-)

@justinabrahms
Copy link
Member Author

justinabrahms commented Sep 20, 2023

@sheepduke https://github.com/open-feature/java-sdk/blob/main/src/test/java/dev/openfeature/sdk/FlagEvaluationSpecTest.java#L318-L325

All specification items need to be accounted for. For some, I annotate a bare test w/ a name that explains why this doesn't apply to us.

@sheepduke
Copy link
Contributor

@sheepduke https://github.com/open-feature/java-sdk/blob/main/src/test/java/dev/openfeature/sdk/FlagEvaluationSpecTest.java#L318-L325

All specification items need to be accounted for. For some, I annotate a bare test w/ a name that explains why this doesn't apply to us.

That makes sense. Thank you!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

3 participants