Overview on KPIs for WIGOS metadata

[fstuerzl]

Branch
https://github.com/wmo-im/wmdr/tree/issue42

Summary and Purpose
Discussion of an initial approach to define KPIs for WIGOS metadata on the basis of a station record from OSCAR/Surface.
Goals: agree on a structure, review proposal for individual KPIs

Stakeholder(s)
OSCAR/Surface, other users of WIGOS Metadata, WMO

Proposal
Structure:
The table below shows rating categories, which each have a different set of rules, to measure the completeness and quality of a metadata record.

Category no.	Subcategory no.	Name
1	0	Schema compliance
2	0	Station characteristics
2	1	Station characteristics (OSCAR/Surface)
3	0	Observations/measurements - Basic information
3	1	Deployment
3	2	Deployment (OSCAR/Surface)
3	3	Data generation
4	0	Station contacts
4	1	Station contact - individual
5	0	Bibliographic references and Documents (OSCAR/Surface)
6	0	Value of a station for WIGOS

Every category consists of a set of KPIs, in which every element has an ID, a list of rules and information on the scoring system.

Naming convention for individual KPIs:
[category no.]-[subcategory no.]-[number (two digit)]

General approach for scoring:
If a rule is fulfilled it gets the rating "1" if not "0". When an element has more than one rule, the individual ratings will be added up (for n rules: KPI maximum score = n). To calculate a rating of a certain category or the whole station record, the importance of individual KPIs needs to be taken into account. To do so, all individual scores are normalised (by dividing by the maximum score) and multiplied by a weighting factor.

[fstuerzl]

To do:

KPI ID	Open questions
2-0-06	Requirements for minumum and maximum length?
3-1-01	Plausibility check possible?
3-1-17	Criteria for plausibility check?
3-3-03	Plausibility check possible?
4-1-00	Criteria for evaluation of format?
4-1-01	Consistency check possible?
4-1-03	Validation possible? E.g. country prefix corresponds to the address.
6-0-02	Consider alternative: Evaluate number of application area(s) for all observations (and deployments).
6-0-xx	Include evaluation of timeline? (Length of timeline (begin and end date of one observation), completeness/coverage (analyse begin and end dates of a deployment), timeliness (active/not active?))

• Review tables and add KPIs if necessary
• Expand adoc files (see WIS template: https://github.com/wmo-im/wcmp/blob/master/kpi/template.adoc)

[IgorZahumensky]

My first comments & questions on Category 2 Station characteristics but they can be applied for all categories:
WMD-KPI-2-0_Station-characteristics.docx

[IgorZahumensky]

Maybe not right at all, maybe partly: when I compare Category 6 "Value of a station for WIGOS" with the other categories, I have a feeling that it is a different perspective being used for this category than for the other KPIs for WMD.
It is rather problematic to estimate "value of a station for WIGOS". From what aspects?
It could be like a summary of all KPIs above? Or whether all mandatory and conditional WMD elements are available in the station record in OSCAR/Surface? Please provide more detailed info, thanks.

[fstuerzl]

My first comments & questions on Category 2 Station characteristics but they can be applied for all categories:
WMD-KPI-2-0_Station-characteristics.docx

Thank you for your comments, @IgorZahumensky! Below my responses to the issues you addressed. I will update table 2-0 Station characteristics according to your revisions.

• Distinction between mandatory, conditional and optional elements (influence on weights)
  Maybe we could include information on the cardinality of the element in the WMDR schema.
• Name of elements consistent with WIGOS Metadata Standard
  I propose to include the names used in the WIGOS Schema in the tables.
• Relevance of begin date (XML: wmdr:validPeriod):
  I agree that a begin date is less important than the specified element itself, this should be reflected in the weights. I do not think it is in general irrelevant (a time zone can have changed for example) and since it is part of the Schema for various data types, it should also be considered in the KPIs.
• Calculation of the score
  I will try to improve the tables in order to make the calculation of the score more comprehensible.
  • 2-0-11: Value for population in 10 km range (1 point), value for population in 50 km range (1 point), date (1 point) -> score: 3 points
  • 2-0-12: Date (1 point), event name (1 point), description (1 point), author (1 point), online reference (1 point) -> score: 5 points

[ferrighi]

A first round of comments from my side as well:

• linking elements to the actual schema would be good, so that the check is clearly related to each element of the WIGOS categories.
• A general, yet minor comment, we could use "begin position of valid period is specified" instead of "a begin date is specified".
• A general comment, should the checks be done with respect to the controlled vocabularies/code table? for example 3-1-00 (or in wigos 5-01) the rule says: "Source of observation is specified and not "unknown", but not that is the provided element should be within the code table 5-01, or is that implicit?
• 2-0-00 for the geoposition, should we check that geolocation uses lat/lon/vertical?
• 2-0-06 I am not sure we need a max length.
• 2-1-00 Maybe it would be enough with provided/not provided
• 2-1-02 It can be that nothing is changing in the records, thus a station can be up-to-date, yet not updated
  in a long time
• 3-0-00/01 link to the elements in the schema
  3-1-17 The coordinates are plausible (in vicinity of the facility) is a very subjective, I am not sure it can/should be used.
  4-0-00 I think it can be ok to value the presence of more than one contact, although I do not see why this should be
  outside the country's national focal point
  4-1 don't we have the email address? I would say that phone/fax should have very low weight if any.
  5-0-00/01 does this refer to data/publications?

[fstuerzl]

Thanks for your review, @ferrighi!

• linking elements to the actual schema would be good, so that the check is clearly related to each element of the WIGOS categories.

  That is an imprtant proposal, which @IgorZahumensky has also mentioned in his comments. I am going to include the specific element of the WIGOS schema, to which the KPI refers, in the tables (see e.g. https://github.com/wmo-im/wmdr/blob/issue42/kpi/2-0.adoc).

• A general, yet minor comment, we could use "begin position of valid period is specified" instead of "a begin date is specified".

  Agreed, that is better and more specific!

• A general comment, should the checks be done with respect to the controlled vocabularies/code table? for example 3-1-00 (or in wigos 5-01) the rule says: "Source of observation is specified and not "unknown", but not that is the provided element should be within the code table 5-01, or is that implicit?

  That was implied, but maybe it is better to include it in the rules, so that it becomes more obvious, which elements come from a code list and which not.

• 2-0-00 for the geoposition, should we check that geolocation uses lat/lon/vertical?

  I left that out, because it is a mandatory element in the application and the XML record, and would therefore go into a KPI for schema compliance.

• 2-0-06 I am not sure we need a max length.

  I'll remove that.

• 2-1-00 Maybe it would be enough with provided/not provided

  The numbers in the rules are a bit arbitrary maybe, but I think it does make sense to distinguish between different amouts of station photos and reward station records with multiple station photos. How important this category itself is will be considered in the weights.

• 2-1-02 It can be that nothing is changing in the records, thus a station can be up-to-date, yet not updated
  in a long time

  I agree, that in a set of KPIs that evaluate the completeness this rule does not make sense. However in my opinion the date of the last update does tell something about the way the metadata are maintained in OSCAR. For an active facility it is unlikely that there were no updates in years that could be integrated in the station record (especially log entries).

• 3-0-00/01 link to the elements in the schema

  In progress!

• 3-1-17 The coordinates are plausible (in vicinity of the facility) is a very subjective, I am not sure it can/should be used.

  I agree, that it is difficult to define a radius for that. My intention was to add plausibility checks for a value whenever possible. But we can leave that out.

• 4-0-00 I think it can be ok to value the presence of more than one contact, although I do not see why this should be
  outside the country's national focal point

  My mistake. I got confused with the list of station contacts in the OSCAR/Surface application (which includes NFPs) and the wmdr section responsibleParty in a station record, which only lists individuals designated as station contacts for this specific station.

• 4-1 don't we have the email address? I would say that phone/fax should have very low weight if any.

  The e-mail address is again mandatory and therefore there is no need to check if it is specified.

• 5-0-00/01 does this refer to data/publications?

  This refers to the section "Bibliographic references" and "Documents" in OSCAR/Surface. That is how they are described in the Guide to WIGOS:

  3.2.4 Bibliographic references
  Where the data series or deployment, or methods relating to the data series or deployment,
  have been previously published or referenced, for example nationally or on the Internet, the
  references are recorded in this section. OSCAR/Surface allows for the upload of documents.
  There is no direct correspondence between this section and any particular metadata element of
  the Standard.
  3.2.5 Documents
  This section provides access to documents concerning the station/platform or the observed
  variables. These may include correspondence, instrument calibration certificates, network
  descriptions and so on. This section may be related with element 4-05 Site information and
  can be regarded as a historic archive of complementary documentation on the changes in the
  station/platform, its instruments and conditions of observation.

[fstuerzl]

Branch is updated (https://github.com/wmo-im/wmdr/tree/issue42/kpi) with @ferrighi and @IgorZahumensky's ideas for the KPIs regarding the completeness of a station record.

However we still need some ideas how a station record could be evaluated beyond its completeness.

Criteria	KPI sets
Schema compliance	1-1: Schema compliance
Completeness of a station record	2-0: Station characteristics
"	3-0: Observations/measurements - Basic information
"	3-1: Deployment
"	3-3: Data generation
"	4-0: Station contacts
"	4-1: Station contact - individual
Completeness of a station record (additional OSCAR/Surface elements)	2-1: Station characteristics (OSCAR/Surface)
"	3-2: Deployment (OSCAR/Surface)
"	3-4: Data generation (OSCAR/Surface)
"	5-0: Bibliographic references and Documents (OSCAR/Surface)
Quality	7-0: Maintenance of a station record -edit-
Quality	7-1: Use of log books -edit-
Value	6-0: Value of a station for WIGOS

[RMaerz]

Additional to the good ideas for KPI which are already collected let me bring another fact in discussion: would it be valuable to have even a flag or a weighed Indicator on how up-to-date a meta-dataset is. OSCAR has this 'last edited' timestamps on several places. If I think of regular inspections and/or calibrations of sensors on platforms, this should be documented in OSCAR too and could be used for a separate Indicator. Maintenance for other technical systems has rules and if not updated manx things 'expire'. This could be reflected in a quality within a KPI ...?

[fstuerzl]

That is a good point, @RMaerz, which @ferrighi has also adressed in her comment on KPI 2-1-02. I think, that to evaluate how up-to-date a station record is, we need a new (quality) KPI set e.g. "Timeliness of data".

As @ferrighi mentioned it is important to consider where updates are likely to occur, so that we don't "punish" station records for not being updated in a long time, when nothing actually changed. Therefore I'd propose to only apply such KPIs on active facilities and observations.

Some aspects relevant for the timeliness of data might be

• last inspection or equipment maintenance of the facility
• newest log entry (device replacement) for each deployed equipment
• ...

In OSCAR/Surface I noticed that the DWD stations are very accurate in their use of the log books (see e.g. 0-20000-0-10929), so maybe you (@RMaerz) could help us with establishing more concrete rules.

[gaochen-larc]

2-0-00 Coordinates: elevation should be specified for clarity. Also, it may be helpful to have an entry for the size of the site foot print, especially for the large sites.

2-0-06 Site description: It would be more helpful if some attributes are specified, similar to 3-1-15, to ensure the description is meaningful.

2-0-9 Surface roughness: It may need more than one value for certain sites to properly describe the surface roughness.

3-1-01 Distance from reference surface and 3-1-17 Coordinates: it may be more useful to specify the exact location of the sensor or the sampling inlet as the instrument location may not be same as the sampling location.

3-3 Data Generation: Is this the proper place to indicate the data revision history? This may help the data users to know if the most up-to-date data are used in analysis.

[ferrighi]

@fstuerzl a question about KPI 3-0, it looks for geometry and deployment for the OM and the max score is 1 per geometry and 1 per deployment. Should this be 1 per OM or 1 for all the OM combined? Meaning if each OM has at least one deployment for example you get 1, if you have several OM and one or some are missing deployment you get 0?

[fstuerzl]

@ferrighi, 1 per observation!
Example: If a record has 2 observations, one with a geometry and some deployments, the other one with geometry unknown and no deployments, then the first observation scores 2+"average score of all deployments below", the second 0. In a summary the KPI for the observations is the average value of all observations.