Add condition in fct get_metrics

[cleherny]

I had to add conditions to the get_metrics function to avoid errors when calculating metrics with the different methods.
However, the complexity is now judged B by the sonar cube... I've quickly tried to do some simplification, but have been unable to resolve it. Any improvement is welcome!

[GwenaelleSa]

I feel like SonarCube sometimes misses something, such as the high complexity of the code, and suddenly is triggered for an unknown reason and notices new stuff.
Like with the high complexity of generate_tilesets.py, I feel it is not an urgent problem. If the code is functional, we can merge it.
I will quickly check if I can see some new simplification.

[GwenaelleSa]

I would propose the following code, which is not much simpler, but that's the best I can do:

    for id_cl in id_classes:

       # Check dataframe existence for the determination of the FP and FN counts 
        pure_fp_count = 0 if fp_gdf.empty else len(fp_gdf[fp_gdf.det_class==id_cl])
        pure_fn_count = 0 if fn_gdf.empty else len(fn_gdf[fn_gdf.label_class==id_cl+1])  # label class starting at 1 and id class at 0

        if mismatch_gdf.empty:
            mismatched_fp_count = 0
            mismatched_fn_count = 0
        else:
            mismatched_fp_count = len(mismatch_gdf[mismatch_gdf.det_class==id_cl])
            mismatched_fn_count = len(mismatch_gdf[mismatch_gdf.label_class==id_cl+1])

        fp_count = pure_fp_count + mismatched_fp_count
        fn_count = pure_fn_count + mismatched_fn_count

        # Check dataframe existence for the determination of the TP count and set resulting values to 0 if necessary
        if tp_gdf.empty:
            tp_count = 0
            p_k, r_k, count_k, pw_k, rw_k = no_tp_for_class(p_k, r_k, count_k, pw_k, rw_k, id_cl, method)
        else:
            tp_count = len(tp_gdf[tp_gdf.det_class==id_cl])
            if tp_count == 0:
                p_k, r_k, count_k, pw_k, rw_k = no_tp_for_class(p_k, r_k, count_k, pw_k, rw_k, id_cl, method)
            else:
                p_k[id_cl] = tp_count / (tp_count + fp_count)
                r_k[id_cl] = tp_count / (tp_count + fn_count)
                count_k[id_cl] = tp_count + fn_count
                if method == 'macro-weighted-average':
                    pw_k[id_cl] = (count_k[id_cl] / sum(count_k.values())) * p_k[id_cl]
                    rw_k[id_cl] = (count_k[id_cl] / sum(count_k.values())) * r_k[id_cl] 

        fp_k[id_cl] = fp_count
        fn_k[id_cl] = fn_count
        tp_k[id_cl] = tp_count

    if method == 'macro-average':   
        precision = sum(p_k.values()) / len(id_classes)
        recall = sum(r_k.values()) / len(id_classes)
    elif method == 'macro-weighted-average':  
        precision = sum(pw_k.values()) / len(id_classes)
        recall = sum(rw_k.values()) / len(id_classes)
    elif method == 'micro-average':  
        if sum(tp_k.values()) == 0 and sum(fp_k.values()) == 0:
            precision = 0
            recall = 0
        else:
            precision = sum(tp_k.values()) / (sum(tp_k.values()) + sum(fp_k.values()))
            recall = sum(tp_k.values()) / (sum(tp_k.values()) + sum(fn_k.values()))

with

def no_tp_for_class(p_k, r_k, count_k, pw_k, rw_k, id_cl, method):
    p_k[id_cl] = 0
    r_k[id_cl] = 0
    count_k[id_cl] = 0
    if method == 'macro-weighted-average':
        pw_k[id_cl] = 0
        rw_k[id_cl] = 0 
    
    return p_k, r_k, count_k, pw_k, rw_k

Later, I will create the columns det_class and label_class even for empty dataframes. That way, we won't have to check if they are empty. However, that's not urgent right now.

[GwenaelleSa]

Does it really happen or is it for the hypothetical case?

[cleherny]

Yes it has occured.

[GwenaelleSa]

Actually, I found a better version. It would be simpler if the dict values where set to 0 instead of None.

def get_metrics(tp_gdf, fp_gdf, fn_gdf, mismatch_gdf, id_classes=0, method='macro-average'):
    """Determine the metrics based on the TP, FP and FN

    Args:
        tp_gdf (geodataframe): true positive detections
        fp_gdf (geodataframe): false positive detections
        fn_gdf (geodataframe): false negative labels
        mismatch_gdf (geodataframe): labels and detections intersecting with a mismatched class id
        id_classes (list): list of the possible class ids. Defaults to 0.
        method (str): method used to compute multi-class metrics. Default to macro-average
    
    Returns:
        tuple: 
            - dict: precision for each class
            - dict: recall for each class
            - float: precision;
            - float: recall;
            - float: f1 score.
    """

    by_class_dict = {key: 0 for key in id_classes}
    tp_k = by_class_dict.copy()
    fp_k = by_class_dict.copy()
    fn_k = by_class_dict.copy()
    p_k = by_class_dict.copy()
    r_k = by_class_dict.copy()
    count_k = by_class_dict.copy()
    pw_k = by_class_dict.copy()
    rw_k = by_class_dict.copy()

    for id_cl in id_classes:

        pure_fp_count = 0 if fp_gdf.empty else len(fp_gdf[fp_gdf.det_class==id_cl])
        pure_fn_count = 0 if fn_gdf.empty else len(fn_gdf[fn_gdf.label_class==id_cl+1])  # label class starting at 1 and id class at 0

        if mismatch_gdf.empty:
            mismatched_fp_count = 0
            mismatched_fn_count = 0
        else:
            mismatched_fp_count = len(mismatch_gdf[mismatch_gdf.det_class==id_cl])
            mismatched_fn_count = len(mismatch_gdf[mismatch_gdf.label_class==id_cl+1])

        fp_count = pure_fp_count + mismatched_fp_count
        fn_count = pure_fn_count + mismatched_fn_count
        tp_count = 0 if tp_gdf.empty else len(tp_gdf[tp_gdf.det_class==id_cl])

        fp_k[id_cl] = fp_count
        fn_k[id_cl] = fn_count
        tp_k[id_cl] = tp_count

        if tp_count > 0:
            p_k[id_cl] = tp_count / (tp_count + fp_count)
            r_k[id_cl] = tp_count / (tp_count + fn_count)
            count_k[id_cl] = tp_count + fn_count
            if method == 'macro-weighted-average':
                pw_k[id_cl] = (count_k[id_cl] / sum(count_k.values())) * p_k[id_cl]
                rw_k[id_cl] = (count_k[id_cl] / sum(count_k.values())) * r_k[id_cl] 

    if method == 'macro-average':   
        precision = sum(p_k.values()) / len(id_classes)
        recall = sum(r_k.values()) / len(id_classes)
    elif method == 'macro-weighted-average':  
        precision = sum(pw_k.values()) / len(id_classes)
        recall = sum(rw_k.values()) / len(id_classes)
    elif method == 'micro-average':  
        if sum(tp_k.values()) == 0 and sum(fp_k.values()) == 0:
            precision = 0
            recall = 0
        else:
            precision = sum(tp_k.values()) / (sum(tp_k.values()) + sum(fp_k.values()))
            recall = sum(tp_k.values()) / (sum(tp_k.values()) + sum(fn_k.values()))

    if precision==0 and recall==0:
        return tp_k, fp_k, fn_k, p_k, r_k, 0, 0, 0
    
    f1 = 2 * precision * recall / (precision + recall)
    
    return tp_k, fp_k, fn_k, p_k, r_k, precision, recall, f1

Sorry for all the mails...

[stdl-s-sonarqube]

• D Maintainability Rating on New Code (is worse than A)

Analysis Details

9 Issues

• 0 Bugs
• 0 Vulnerabilities
• 9 Code Smells

Coverage and Duplications

• No coverage information (0.00% Estimated after merge)
• 0.00% Duplicated Code (0.00% Estimated after merge)

Project ID: swiss-territorial-data-lab_object-detector_AYZ4zWIzr7JdaaSXwexc

View in SonarQube

[cleherny]

Thank you. I just kept the previous version of the part of the code below because otherwise it doesn't work properly. sum(count_k.values()) must be used once the dict has go through all the id_class.

        if tp_count > 0:
            p_k[id_cl] = tp_count / (tp_count + fp_count)
            r_k[id_cl] = tp_count / (tp_count + fn_count)
            count_k[id_cl] = tp_count + fn_count
            if method == 'macro-weighted-average':
                pw_k[id_cl] = (count_k[id_cl] / sum(count_k.values())) * p_k[id_cl]
                rw_k[id_cl] = (count_k[id_cl] / sum(count_k.values())) * r_k[id_cl] 

    if method == 'macro-average':   
        precision = sum(p_k.values()) / len(id_classes)
        recall = sum(r_k.values()) / len(id_classes)
    elif method == 'macro-weighted-average':  
        precision = sum(pw_k.values()) / len(id_classes)
        recall = sum(rw_k.values()) / len(id_classes)
    elif method == 'micro-average':  
        if sum(tp_k.values()) == 0 and sum(fp_k.values()) == 0:
            precision = 0
            recall = 0
        else:
            precision = sum(tp_k.values()) / (sum(tp_k.values()) + sum(fp_k.values()))
            recall = sum(tp_k.values()) / (sum(tp_k.values()) + sum(fn_k.values()))

We can figure out later how to make sonar cube happy...

[cleherny]

Yes it has occured.