test code coverage

[lslusarczyk]

https://clang.llvm.org/docs/SourceBasedCodeCoverage.html or something like this
we need to executed both shp and mhp tests so using gcov is probably not an option

We may add this as a separate scan.

But some projects (e.g. botocore using codecov) automatically checks code coverage on commit and rejects if coverage is less than required. It would be great have something like this here.

In that case we need to make gcov/lcov working in shp, which means compiling shp with gcc, then use tutorial how to use codecov with C++ and configure PR check that coverage is good enough.

[rscohn2]

It is not a simple task. I do not know of a way to compile SYCL programs with GCC. I do not know how to get the LLVM-based coverage tools to work with icpx or the intel/llvm project. Someone needs to investigate solutions.

We would be better off for the short term prioritizing testing coverage of algorithms, views, data types, and combinations. Code coverage is not the right tool for that task.

[BenBrock]

I think the way to do this is to bug the intel/llvm folks into getting the open-source build to support LLVM tools. I made an issue complaining about runtime tools not working, and never got a response. (I'm not sure if the coverage tools are part of runtime tools?)

Lukasz, you could make an additional issue.

[lslusarczyk]

Code coverage is not the right tool for that task.

Yes. It will just show which code is not executed at all by tests. If code is executed it still does not mean that tests are indeed good and test everything. Information if code is not executed at all by test is also useful.

So far I've put minor flag here.

[lslusarczyk]

Lukasz, you could make an additional issue.

First I'd like to check, that indeed the llvm option to create coverage doesn't work. Then report the bug. But I'd like not to start this task too much now. And the first step which I would start (when I start) is to try compile by gcc to have gcov/lcov working.

I heard advertisement that sycl is so cool, that it can be visible by C++ compiler as a regular library code, just not being optimized/parallelized by ordinary (not sycl-aware) compilers. This should be enough to detect uncovered code. @BenBrock , if you know the way to compile sycl with gcc, please share.

[lslusarczyk]

I'm pasting here discussion we had on slack:

Many llvm tools are supported for icpx now. Code coverage works with sycl!

From Chih-Ping Chen:
We have enabled the address, leak, memory, thread, and undefined behavior sanitizers in 2023.1. The fuzzer will be available in 2023.2. They are for C/C++ only. We are looking at getting them to work with SYCL host code, tracked by CMPLRLLVM-44232.
4 replies

Here is a code coverage sycl example:

options Buffers Tools Compile Help
        -:    0:Source:sycl.cpp
        -:    0:Graph:sycl.gcno
        -:    0:Data:sycl.gcda
        -:    0:Runs:1
        -:    0:Programs:1
        -:    1:#include <sycl/sycl.hpp>
        -:    2:
        -:    3:int a;
        -:    4:
    #####:    5:void foo() {
    #####:    6:  a = 2;
    #####:    7:}
    #####:    8:void bar() {
    #####:    9:  a = 1;
    #####:   10:}
        1:   11:int main(int argc, char *argv[]) {
        1:   12:  sycl::queue q;
        -:   13:
        1:   14:  auto foo = []() {
        -:   15:    int a = 1;
        -:   16:  };
        -:   17:
        1:   18:  auto bar = []() {
    #####:   19:    int a = 1;
    #####:   20:  };
        -:   21:
        1:   22:  q.single_task(bar);
        1:   23:  return 0;
        1:   24:}

Here is a JIRA specific to libFuzzer: https://jira.devtools.intel.com/browse/CMPLRLLVM-44636.
Can someone investigate code coverage, with the goal being integration into CI? We should be thinking about adding sanitizers and libfuzzer when they support SYCL host code.