Issues with the validation of CTA-WAVE content using the Dash-IF-Conformance tool

[nlsdvl]

The development branch / staging deployment of Dash-If-Conformance software is being used to validate CTA-WAVE test content.

Test failures are being reported, however after initial manual inspection there seems to be failures reported where the content is found to be valid.

This issue is also tracked on CTA-WAVE's Test-Content-Generation/issues/55.

I see that running the validation on cta-wave vectors is part of the GitHub workflows : https://github.com/Dash-Industry-Forum/DASH-IF-Conformance/actions/workflows/run-cta-wave-test-vectors.yml

But the validation process isn't configured for CTA vectors. I used --cmaf --ctawave --segments. Here's the aggregated result of running JCCP on CTA-WAVE's database:
db.validation-staging.json

There are several different failure points that may be issues with the validator's staging deployment.

For instance, on cfhd_sets/12.5_25_50/t12/2023-09-01/ we are getting a test failure with the message : For a CMAF Track to comply with one of the media profiles in Table A.1, it SHALL conform to the colour_primaries, transfer_characteristics and matrix_coefficients values from the options listed in the table

But inspecting the content, it appears the vui is present and valid as described in ISO/IEC 23000-19:

gpac -i https://dash.akamaized.net/WAVE/vectors/cfhd_sets/12.5_25_50/t12/2023-09-01/stream.mpd inspect:interleave=false:analyze=on

<PIDConfigure PID="1" name="AS_1" SourcePath="gmem://0x11d774b48" URL="https://dash.akamaized.net/WAVE/vectors/cfhd_sets/12.5_25_50/t12/2023-09-01/1/init.mp4" Extension="mp4" MIMEType="video/mp4" Accept-Ranges="bytes" ETag=""e17ede3158e04b6164a0d78d003e2fda:1693769884.775624"" Last-Modified="Sun, 03 Sep 2023 19:38:04 GMT" Server="AkamaiNetStorage" Content-Length="917" Cache-Control="max-age=29642591" Date="Fri, 20 Dec 2024 12:29:28 GMT" Connection="keep-alive" Akamai-Mon-Iucid-Del="941106" Alt-Svc="h3-Q050=":443"; ma=93600,quic=":443"; ma=93600; v="46,43"" Content-Type="video/mp4" Cross-Origin-Opener-Policy="same-origin" Cross-Origin-Embedder-Policy="require-corp" Timing-Allow-Origin="*" Access-Control-Max-Age="86400" Access-Control-Allow-Credentials="true" Access-Control-Expose-Headers="Server,range,Date,hdntl,hdnts,Akamai-Mon-Iucid-Ing,Akamai-Mon-Iucid-Del,Akamai-Request-BC,Content-Length,Content-Range,Geo-Info,Quic-Version" Access-Control-Allow-Headers="origin,range,hdntl,hdnts,accept-encoding,referer,CMCD-Request,CMCD-Object,CMCD-Status,CMCD-Session" Access-Control-Allow-Methods="GET,POST,OPTIONS" Access-Control-Allow-Origin="*" ID="1" ClockID="1" StreamType="Visual" Timescale="12800" TrackNumber="1" NumFrames="0" StreamSubtype="vide" MediaDataSize="0" PlaybackMode="rewind" IsoAltBrands="cfhd,cmf2,iso6,dash" IsoBrand="iso6" TrackTemplate="245 bytes (CRC32 0xFD967AFE)" TrexTemplate="32 bytes (CRC32 0x5E44F4D3)" MovieUserData="244 bytes (CRC32 0x53C9159E)" MovieTime="0" Width="1920" Height="1080" TransX="0" TransY="0" ZOrder="0" HandlerName="VideoHandler" TrackFlags="7" TrackMatrix="65536,0,0,0,65536,0,0,0,1073741824" isom_creation_date="3776404751" isom_modification_date="3776404751" CodecID="MPEG-4 AVC|H264 Video" DecoderConfigIndex="1" ColorPrimaries="BT709" ColorTransfer="BT709" ColorMatrix="BT709" FullRange="false" Bitrate="4200000" Maxrate="73904738" DBSize="0" ISOMSubtype="avc3" STSDTemplate="160 bytes (CRC32 0x571A6D9B)" HasSync="true" DecoderConfig="57 bytes (CRC32 0x12CB80B5)" ServiceWidth="1920" ServiceHeight="1080" SAR="1" Duration="30" BufferLength="2000" has_selected="0" has_auto="1" has_tilemode="0" Representation="1" Codec="avc3.640028">
<AVCParameterSets>
  <AVCSPSArray location="decoderConfig">
   <NALU size="37" code="7" type="SequenceParameterSet" sps_id="0" frame_mbs_only_flag="1" mb_adaptive_frame_field_flag="0" vui_parameters_present_flag="1" max_num_ref_frames="4" gaps_in_frame_num_value_allowed_flag="0" chroma_format_idc="1" bit_depth_luma_minus8="0" bit_depth_chroma_minus8="0" width="1920" height="1080" crop_top="0" crop_left="0" crop_bottom="8" crop_right="0" vui_video_full_range_flag="0" vui_video_signal_type_present_flag="1" vui_aspect_ratio_info_present_flag="1" vui_aspect_ratio_num="65535" vui_aspect_ratio_den="65535" vui_overscan_info_present_flag="0" vui_colour_description_present_flag="1" vui_colour_primaries="1" vui_transfer_characteristics="1" vui_matrix_coefficients="1" vui_low_delay_hrd_flag="0" log2_max_poc_lsb="6" log2_max_frame_num="4" delta_pic_order_always_zero_flag="0" offset_for_non_ref_pic="0" nal_ref_idc="3"/>
  </AVCSPSArray>
  <AVCPPSArray location="decoderConfig">
   <NALU size="5" code="8" type="PictureParameterSet" pps_id="0" sps_id="0" entropy_coding_mode_flag="1" deblocking_filter_control_present_flag="1" mb_slice_group_map_type="0" num_ref_idx_l0_default_active_minus1="2" num_ref_idx_l1_default_active_minus1="0" pic_order_present="0" pic_size_in_map_units_minus1="0" redundant_pic_cnt_present="0" slice_group_change_rate_minus1="0" slice_group_count="1" weighted_pred_flag="1" weighted_bipred_idc="0" nal_ref_idc="3"/>
  </AVCPPSArray>
</AVCParameterSets>
</PIDConfigure>

cc @rbouqueau

[rbouqueau]

Yes, the colr box is present. Probably it uses the old IsoSegmentValidator that doesn't support it? Anyway the error message can be improved.

[rbouqueau]

Maybe I'm wrong as the table A.1 mentions the VUI, not the 'colr' box.

[Phencys]

When running the suggested stream through the current staging version, it gives me the same error about the colr box.

However, further down the list of checks, at SEGMENT_VALIDATION it also indeed shows us that the ISOSegmentValidator doesn't support this particular type of colr box.

We should be able to fix this once the modules are converted to all support the Validator Wrapper (e.g. it can then use MP4Box instead).