[SPARK-51847][PYTHON] Extend PySpark testing framework util functions with basic data tests

[stanlocht]

What changes were proposed in this pull request?

This PR extends the PySpark testing framework with four new utility functions for data quality and integrity testing:

1. assertColumnUnique: Verifies that specified column(s) contain only unique values
2. assertColumnNonNull: Checks that specified column(s) do not contain null values
3. assertColumnValuesInSet: Ensures all values in specified column(s) are within a given set of accepted values
4. assertReferentialIntegrity: Validates that all non-null values in a source column exist in a target column (similar to foreign key constraints)

Why are the changes needed?

These new utility functions address this gap by providing standardized, well-tested implementations of the most common data quality checks. They reduce boilerplate code, improve test readability, and enable testing patterns similar to those in popular data testing frameworks like dbt.

Does this PR introduce any user-facing change?

Yes, this PR introduces new public utility functions in the pyspark.testing module. These are additive changes that don't modify existing functionality.

Example usage:

from pyspark.testing import assertColumnUnique, assertReferentialIntegrity

# Check that 'id' column contains only unique values
assertColumnUnique(df, "id")

# Check that all customer_ids in orders exist in customers.id
assertReferentialIntegrity(orders, "customer_id", customers, "id")

How was this patch tested?

Comprehensive tests were added for all new functions in python/pyspark/sql/tests/test_utils.py. The tests cover:

• Basic functionality with valid inputs
• Error cases with invalid inputs
• Edge cases (e.g., null values, empty DataFrames)
• Different DataFrame types (Spark, pandas, pandas-on-Spark)
• Detailed validation of error messages

Each function has multiple test methods that verify both positive and negative test cases. For example, assertReferentialIntegrity has tests for valid relationships, invalid relationships with a single missing value, multiple missing values, and proper handling of null values.

All tests pass on the current master branch.

Was this patch authored or co-authored using generative AI tooling?

Generated-by: Claude 3.7 Sonnet

[HyukjinKwon]

cc @asl3 fyi

[zhengruifeng]

remove [TESTS] from title since it seems a new user-facing feature

[stanlocht]

Hi @HyukjinKwon, @zhengruifeng, @asl3 — just following up to see if you might have a chance to review the PR when time allows. Appreciate your time and input!

[asl3]

thank you for the contributions! these will be helpful test utils

[stanlocht]

@asl3 thanks for you review! i've implemented your suggestions. would be great if you could check it out when you have the time!

[stanlocht]

Hi @HyukjinKwon, @zhengruifeng, @asl3 — just checking in on this PR. I’ve made the changes based on the earlier feedback, so let me know if there’s anything else you’d like to see. Would be great to get this moving if/when you have a moment. Thanks again!

[asl3]

LGTM, thanks!!