Skip to content

Conversation

@sharder996
Copy link
Collaborator

@sharder996 sharder996 commented Feb 24, 2025

Adds Tiobe's TICS github action to the Multipass project

  • Runs on a weekly basis (the Desktop team SSDLC guide suggests nightly)
  • Each analysis takes about 4+hrs to run (may vary depending on the number of changed files between runs)
  • The TICS custom action has some specific requirements which are not always clear
    • The coverage report must be at ${{ github.workspace }}/coverage/coverage.xml
    • The project must be built beforehand with standard build tools and build directories
      • Build files should be in ${{ github.workspace }}/build
      • make must be used for compilation (using ninja generator will fail the action)
  • TICS analysis now includes and analyzes the tests/ subdirectory (this is inline with similar projects such as Mir)
  • Current metrics from test runs can be found here

MULTI-1759


Closes #3962

@sharder996 sharder996 requested a review from ricab February 24, 2025 18:55
@codecov
Copy link

codecov bot commented Feb 24, 2025

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 89.44%. Comparing base (3d1d7f6) to head (ac69de0).
⚠️ Report is 379 commits behind head on main.

Additional details and impacted files
@@           Coverage Diff           @@
##             main    #3953   +/-   ##
=======================================
  Coverage   89.44%   89.44%           
=======================================
  Files         259      259           
  Lines       14758    14758           
=======================================
  Hits        13200    13200           
  Misses       1558     1558           

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

@sharder996 sharder996 marked this pull request as draft April 4, 2025 15:05
@sharder996 sharder996 force-pushed the tiobe-integration branch 2 times, most recently from 3decb72 to 75a7763 Compare April 14, 2025 01:24
@sharder996 sharder996 marked this pull request as ready for review April 14, 2025 01:26
@ricab ricab requested review from Sploder12 and georgeliao and removed request for ricab April 14, 2025 14:44
@ricab
Copy link
Collaborator

ricab commented Apr 17, 2025

@sharder996
Copy link
Collaborator Author

@ricab I think its out of my control. I can see the same "error" in the initial TICS run that was run by Tiobe staff. Comparing between the different time stamps, I've noticed some times slight changes between runs even though the source code was the same. I also Tiobe is actively developing the backend of TICS and analysis results may be subject to changes. The TICS github action used here is simply a client that calls some backend API endpoint.

https://canonical.tiobe.com/tiobeweb/TICS/TqiDashboard.html#axes=Project(multipass),Date(1715703041),Sub()&metric=tqiTestCoverage&sort=T(1,worst)

Copy link
Collaborator

@ricab ricab left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks Scott, I have a few questions.

sudo apt-get update
sudo apt-get install -y devscripts equivs lcov
mk-build-deps -s sudo -i
sudo --preserve-env apt-get install --yes devscripts equivs lcov ninja-build
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Could you please explain why --preserve-env was necessary?

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

And why not update? Are these runner's apt cache guaranteed to be up to date?

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is how Mir had it set up so I copied it, but I just tested and it isn't necessary. The variables that --preserve-env is protecting isn't related to building Multipass anyways.

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

OK, makes sense. How about the missing apt update?

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We have this b/c:
https://github.com/canonical/mir/blob/df6674bd3686286aa0e422fdc26664e1fbef5adf/.github/workflows/tics.yml#L24-L25

You might get hit with some dependency that requires configuration, and the install will seem stuck b/c it will wait on user input.

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

No reason to exclude it so I added it back.

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I see, interesting. Thanks @Saviq, is this only a concern in these runners? I suppose GH could be addressing that under the hood.

@sharder996, might be a good idea to do the same then (i.e. avoid apt-get asking what services should be restarted.)

Copy link
Collaborator

@ricab ricab left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks @sharder996, this looks good to me now!

As discussed, we have a little margin to speed up with ccache if we need to in the future. Otherwise just needs a rebase.

@sharder996 sharder996 force-pushed the tiobe-integration branch from d82e81d to ac69de0 Compare May 13, 2025 12:10
@sharder996 sharder996 added this pull request to the merge queue May 13, 2025
Merged via the queue into main with commit a1df4c9 May 14, 2025
19 checks passed
@sharder996 sharder996 deleted the tiobe-integration branch May 14, 2025 01:46
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants