Start moving ctlrender tests to CTest

[ThomasWilshaw]

This PR is an experiment in moving the ctlrender tests from a bash script to CTest to make them more cross platform compatible.

[ThomasWilshaw]

I'm not really sure why these tests are failing, It seems any time ctlrender is run via the CI with tiff files it fails and I can't work out why. I appreciate this might not be the way you want to go anyway but it seems worth pursuing.

[michaeldsmith]

the CI test without TIF works, but the others fail. I will look into why the other fail

[michaeldsmith]

running "Build" on the "RUN TESTS" project in Visual Studio works great, this is a big improvement

[michaeldsmith]

i get 100% tests pass in Ubuntu docker as well

[michaeldsmith]

@ThomasWilshaw does your PR include creating the output folder for the ctlrender conversions? In the existing test.sh, the output folder is created using
mkdir output

[ThomasWilshaw]

No it doesn't actually, good spot. Are you thinking different libraries handle none existing folders differently?

[michaeldsmith]

yeah, i think there may not be good handling of invalid input or output filenames. I added one message to exr_read() last month. It certainly helps for debugging to have a message on why something fails, rather than just silently failing.

[michaeldsmith]

it looks like when I run ubuntu docker on my local system the output folder is copied over to the docker, which didn't result in failures yesterday, but if I remove it, then I get the failures just like the CI fails.

[michaeldsmith]

maybe a better approach would be to store the output files somewhere the \CTL\build directory or whatever build directory is used by CMake, instead of embedded in the \CTL\unittest\ctlrender directory ?

[michaeldsmith]

i added exception throwing to dpx_write(), now the dpx outputs also fail when output directory doesn't exist

[ThomasWilshaw]

That seems ot have work pretty well. A new folder is created a build time in the unit test ctlrender subdirectory and all the outputs are stored in there. Most of the CI tests seem to be passing now.

[michaeldsmith]

it looks like docker fedora 35 failed for reasons unrelated to this, problems downloading. I'll re-run the failed job

[michaeldsmith]

@ThomasWilshaw now that this approach seems to be working, can you also add the exr -> exr and tiff32->exr and exr->tiff32 tests from existing test.sh reproduced below ?

Also, since we discovered the silent failure for invalid filenames, it might be good to also have tests that we expect to fail included like you did for the ctlrender-ctl-function-name-is-incorrect test ? For invalid input filenames with DPX, TIFF and EXR format and for invalid output filenames for DPX, TIFF and EXR outtput. We may need to add additional ctlrender code to catch invalid filenames, maybe we could do that after we have invalid filename tests.

# test EXR to EXR support
if [[ $IS_OPENEXR_FOUND == 1 ]] ; then
	printf 'bars_photoshop.exr -> output/bars_exr_exr.exr \n'
	$CTLRENDER -ctl unity.ctl -format exr -force bars_photoshop.exr output/bars_exr_exr.exr
	printf 'bars_photoshop.exr -> output/bars_exr_exr16.exr \n'
	$CTLRENDER -ctl unity.ctl -format exr16 -force bars_photoshop.exr output/bars_exr_exr16.exr
	printf 'bars_photoshop.exr -> output/bars_exr_exr32.exr \n'
	$CTLRENDER -ctl unity.ctl -format exr32 -force bars_photoshop.exr output/bars_exr_exr32.exr

	# RGB tests
	printf 'colorbars_nuke_rgb_exr16.exr -> output/bars_rgb_exr16_to_exr16.exr \n'
	$CTLRENDER -force -format exr16 -ctl unity.ctl colorbars_nuke_rgb_exr16.exr output/bars_rgb_exr16_to_exr16.exr
	printf 'colorbars_nuke_rgb_exr16.exr -> output/bars_rgb_exr16_to_exr32.exr \n'
	$CTLRENDER -force -format exr32 -ctl unity.ctl colorbars_nuke_rgb_exr16.exr output/bars_rgb_exr16_to_exr32.exr
	printf 'colorbars_nuke_rgb_exr32.exr -> output/bars_rgb_exr32_to_exr16.exr \n'
	$CTLRENDER -force -format exr16 -ctl unity.ctl colorbars_nuke_rgb_exr32.exr output/bars_rgb_exr32_to_exr16.exr
	printf 'colorbars_nuke_rgb_exr32.exr -> output/bars_rgb_exr32_to_exr32.exr \n'
	$CTLRENDER -force -format exr32 -ctl unity.ctl colorbars_nuke_rgb_exr32.exr output/bars_rgb_exr32_to_exr32.exr

	# RGBA tests
	printf 'colorbars_nuke_rgba_exr16.exr -> output/bars_rgba_exr16_to_exr16.exr \n'
	$CTLRENDER -force -format exr16 -ctl unity_with_alpha.ctl colorbars_nuke_rgba_exr16.exr output/bars_rgba_exr16_to_exr16.exr
	printf 'colorbars_nuke_rgba_exr16.exr -> output/bars_rgba_exr16_to_exr32.exr \n'
	$CTLRENDER -force -format exr32 -ctl unity_with_alpha.ctl colorbars_nuke_rgba_exr16.exr output/bars_rgba_exr16_to_exr32.exr
	printf 'colorbars_nuke_rgba_exr32.exr -> output/bars_rgba_exr32_to_exr16.exr \n'
	$CTLRENDER -force -format exr16 -ctl unity_with_alpha.ctl colorbars_nuke_rgba_exr32.exr output/bars_rgba_exr32_to_exr16.exr
	printf 'colorbars_nuke_rgba_exr32.exr -> output/bars_rgba_exr32_to_exr32.exr \n'
	$CTLRENDER -force -format exr32 -ctl unity_with_alpha.ctl colorbars_nuke_rgba_exr32.exr output/bars_rgba_exr32_to_exr32.exr

fi

# test TIFF32 to EXR and EXR to TIFF32 support
if [[ $IS_OPENEXR_FOUND == 1 ]] && [[ $IS_TIFF_FOUND == 1 ]] ; then
	
	for J in exr exr16 exr32 tiff32; do

		for I in bars_photoshop_32_be_interleaved.tif bars_photoshop_32_be_planar.tif bars_photoshop_32_le_interleaved.tif bars_photoshop_32_le_planar.tif bars_photoshop.exr; do
			name=`echo $I | sed -e 's/\..*//'`
			ext=`echo $J | sed -e 's/[0-9]//g'`
			echo ${I} '->' ${J}
			if [[ $IS_TIFF_FOUND == 0 ]] && [[ $J == *"tif"* ]] ; then
				printf 'skipping %s because tiff support was not detected\n' $J
			elif [[ $IS_TIFF_FOUND == 0 ]] && [[ $I == *"tif"* ]] ; then
				printf 'skipping %s because tiff support was not detected\n' $I
			else
				$CTLRENDER -ctl unity.ctl -format ${J} -force ${I} output/${name}_${J}.${ext}
			fi

		done
	done
fi

[ThomasWilshaw]

Yes I can add all that. I wanted to make sure things were working before I went too far.

[ThomasWilshaw]

I've added all the requested tests. With regards to updating ctlrender's missing file code should that be a seperate PR to stop this one going to0 far out of scope?

[michaeldsmith]

I agree, lets get this PR merged and then add a check of invalid filename in dpx_read() in another PR, I added issue #122 to help track it