grant/kol 7429 programmatic retrieval of significance

[grant-Kolena]

• add functionality to display performance delta when using download_quality_standard_result
• added test

Linked issue(s)

What change does this PR introduce and why?

Please check if the PR fulfills these requirements

• Include reference to internal ticket and/or GitHub issue "Fixes #NNNN" (if applicable)
• Relevant tests for the changes have been added
• Relevant docs have been added / updated

[grant-Kolena]

Copied and rewritten from kolena FE's implementation of the same functionality

[munkyshi]

Do these serialize well to CSV? Will users ever be expected to know about PerformanceDelta? In general, for anything that a user deals with, I'd prefer to use something like

PerformanceDelta = Literal["improved", "regressed", "similar", "unknown"]

So that the user doesn't need to know about some random enum type, and can instead just work with things as strings.

[grant-Kolena]

I can confirm export to CSV works fine, but will still change to Literal so the user don't have to see the enum

[munkyshi]

Can you share an example of how to run this? I'm trying to figure out how to best set up a dataset / QS for this -- is there an example that you already have set up that I can play with?

[grant-Kolena]

Can you share an example of how to run this? I'm trying to figure out how to best set up a dataset / QS for this -- is there an example that you already have set up that I can play with?

Yes, I have played around with this dataset on my tenant

from kolena._experimental import download_quality_standard_result
qs_result = download_quality_standard_result("attack_on_llms_small", ["databricks_dbrx-instruct", "gpt-4-turbo"], confidence_level = 0.95, reference_model = "gpt-4-turbo")

[munkyshi]

Suggested change

	:param reference_eval_config: The evaluation configuration to use in conjunction with the reference model,
	if unspecified the first model in the models list will be used
	as a reference for the Margin of Error, for None eval config, pass in 'null' instead of None. If this is None,
	the first eval config will be used.
	:param reference_eval_config: The evaluation configuration to use in conjunction with the `reference_model`.
	If this is `None`, the first eval config will be used. If you wish to use an evaluation configuration of `None`, please instead specify the configuration as `'null'`.

Also, seems a bit odd to make the user specify null instead of using None -- would it make sense to instead define it as:

reference_eval_config: Optional[Union[Dict[str, Any], Literal["first"]]] = "first",

[grant-Kolena]

Yep make sense, I was just trying to distinguish between null and None

[munkyshi]

Does this mean that the first eval_config will be skipped if it is None?

# initially reference_model is None and reference_eval_config
# First pass, model is 'foo' and eval_config is None
reference_model = 'foo'
reference_eval_config = None
# Second pass, model is 'foo' and eval_config is {'bar': 'baz'}
reference_eval_config = {'bar': 'baz'}

Also, does this get handled safely if the only eval config for the reference model is None?

[grant-Kolena]

originally I was expecting user to pass in "null" so it won't be ignored, I will have to handle this case now. This should work as expected when the only eval config is None, as the dataset I used to test is like this

[munkyshi]

from kolena._experimental import download_quality_standard_result
qs_result = download_quality_standard_result("attack_on_llms_small", ["databricks_dbrx-instruct", "gpt-4-turbo"], confidence_level = 0.95, reference_model = "gpt-4-turbo")

I tried playing around with this, and it looks like the colors aren't matching with the webapp (link)

One example (from when I dumped this to CSV):

model,,databricks_dbrx-instruct,databricks_dbrx-instruct,gpt-4-turbo,gpt-4-turbo,databricks_dbrx-instruct,databricks_dbrx-instruct,gpt-4-turbo,gpt-4-turbo
eval_config,,null,null,null,null,null,null,null,null
metric_group,,Metric Group,Metric Group,Metric Group,Metric Group,Metric Group,Metric Group,Metric Group,Metric Group
metric,,rate rejected,Mean Metadata.count,rate rejected,Mean Metadata.count,rate rejected,Mean Metadata.count,rate rejected,Mean Metadata.count
type,,metric_value,metric_value,metric_value,metric_value,performance_delta,performance_delta,performance_delta,performance_delta
stratification,test_case,,,,,,,,
...
type,Raw,0.03333333333333333,,0.1,0.0,regressed,unknown,similar,similar
type,Role Playing,0.15555555555555556,0.6666666666666666,0.12222222222222222,,improved,unknown,similar,unknown

vs webapp

The endpoint seems to indicate regressed and improved, but there's no coloration on the webapp (both at confidence level of 0.95)

[grant-Kolena]

from kolena._experimental import download_quality_standard_result
qs_result = download_quality_standard_result("attack_on_llms_small", ["databricks_dbrx-instruct", "gpt-4-turbo"], confidence_level = 0.95, reference_model = "gpt-4-turbo")

I tried playing around with this, and it looks like the colors aren't matching with the webapp (link)

One example (from when I dumped this to CSV):

model,,databricks_dbrx-instruct,databricks_dbrx-instruct,gpt-4-turbo,gpt-4-turbo,databricks_dbrx-instruct,databricks_dbrx-instruct,gpt-4-turbo,gpt-4-turbo
eval_config,,null,null,null,null,null,null,null,null
metric_group,,Metric Group,Metric Group,Metric Group,Metric Group,Metric Group,Metric Group,Metric Group,Metric Group
metric,,rate rejected,Mean Metadata.count,rate rejected,Mean Metadata.count,rate rejected,Mean Metadata.count,rate rejected,Mean Metadata.count
type,,metric_value,metric_value,metric_value,metric_value,performance_delta,performance_delta,performance_delta,performance_delta
stratification,test_case,,,,,,,,
...
type,Raw,0.03333333333333333,,0.1,0.0,regressed,unknown,similar,similar
type,Role Playing,0.15555555555555556,0.6666666666666666,0.12222222222222222,,improved,unknown,similar,unknown

vs webapp

The endpoint seems to indicate regressed and improved, but there's no coloration on the webapp (both at confidence level of 0.95)

Weird, the algorithm is exactly the same, I will calculate it again to confirm. Could it be that the color on webapp is too light to see

[grant-Kolena]

from kolena._experimental import download_quality_standard_result
qs_result = download_quality_standard_result("attack_on_llms_small", ["databricks_dbrx-instruct", "gpt-4-turbo"], confidence_level = 0.95, reference_model = "gpt-4-turbo")

I tried playing around with this, and it looks like the colors aren't matching with the webapp (link)
One example (from when I dumped this to CSV):

model,,databricks_dbrx-instruct,databricks_dbrx-instruct,gpt-4-turbo,gpt-4-turbo,databricks_dbrx-instruct,databricks_dbrx-instruct,gpt-4-turbo,gpt-4-turbo
eval_config,,null,null,null,null,null,null,null,null
metric_group,,Metric Group,Metric Group,Metric Group,Metric Group,Metric Group,Metric Group,Metric Group,Metric Group
metric,,rate rejected,Mean Metadata.count,rate rejected,Mean Metadata.count,rate rejected,Mean Metadata.count,rate rejected,Mean Metadata.count
type,,metric_value,metric_value,metric_value,metric_value,performance_delta,performance_delta,performance_delta,performance_delta
stratification,test_case,,,,,,,,
...
type,Raw,0.03333333333333333,,0.1,0.0,regressed,unknown,similar,similar
type,Role Playing,0.15555555555555556,0.6666666666666666,0.12222222222222222,,improved,unknown,similar,unknown

vs webapp
The endpoint seems to indicate regressed and improved, but there's no coloration on the webapp (both at confidence level of 0.95)

Weird, the algorithm is exactly the same, I will calculate it again to confirm. Could it be that the color on webapp is too light to see

this matches with the UI now, not sure what type of test would be meaningful to these mathematical calculations, it is hard to identify edge cases here. We already have a test case to check if the produced performance delta match the expected result.

[munkyshi]

this matches with the UI now, not sure what type of test would be meaningful to these mathematical calculations, it is hard to identify edge cases here. We already have a test case to check if the produced performance delta match the expected result.

What was the issue, the check for and max_value > 0:? Is there anything we can do to test around that case specifically, since it was something that was missed? Change looks good otherwise.

[munkyshi]

Approving to unblock, pending test to track the latest (now fixed) issue.

[grant-Kolena]

this matches with the UI now, not sure what type of test would be meaningful to these mathematical calculations, it is hard to identify edge cases here. We already have a test case to check if the produced performance delta match the expected result.

What was the issue, the check for and max_value > 0:? Is there anything we can do to test around that case specifically, since it was something that was missed? Change looks good otherwise.

I missed the part where the max value for performance delta is the max across the stratification, except for when the max value is 0 or less.

[grant-Kolena]

Approving to unblock, pending test to track the latest (now fixed) issue.

created this ticket to track