fix(general): escape default ignored directories

[tpvasconcelos]

User description

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.

Description

See #6737 for more details and context:

While trying to run checkov's test suit on my local machine I noticed that some files were being (unexpectedly) ignored. After a little digging, I realised that this is because I cloned the repo under a path that looks something like: ~/my_git_forks/checkov.

The reason this is a problem is because the following patterns are treated as regular expressions by checkov:

checkov/checkov/secrets/utils.py

Line 10 in 42d3178

EXCLUDED_PATHS = [*ignored_directories, DEFAULT_EXTERNAL_MODULES_DIR, ".idea", ".git", "venv"]

and the .git pattern obviously excludes any paths that include the word 'git' (incl. at least one preceding character)

Checklist:

• I have performed a self-review of my own code
• I have commented my code, particularly in hard-to-understand areas
• I have made corresponding changes to the documentation
• I have added tests that prove my feature, policy, or fix is effective and works
• New and existing tests pass locally with my changes

---

Generated description

Below is a concise technical summary of the changes proposed in this PR:

Enhances the directory exclusion mechanism by introducing a new re_dir function that creates regex patterns for matching paths containing specified directories at any level. Updates the IGNORED_DIRECTORIES_ENV and EXCLUDED_PATHS to use these new regex patterns, ensuring more accurate exclusion of directories like .git and venv. Adds comprehensive unit tests to validate the new functionality.

Topic	Details
Unit Testing	Adds unit tests to verify the functionality of the new re_dir function and its regex patterns. Modified files (1) tests/common/runners/test_base_runner.py Latest Contributors(2) User Commit Date 86768411+YaaraVerner@u... fix-general-fix-exclud... December 26, 2022 mike@bridgecrew.io add-file-types-for-all... April 23, 2022
Regex Dir Exclusion	Implements a new re_dir function to create regex patterns for directory exclusion and updates existing exclusion lists. Modified files (2) checkov/secrets/utils.py checkov/common/runners/base_runner.py Latest Contributors(2) User Commit Date 16597193+omryMen@users... chore-secrets-update-d... October 15, 2024 49649760+lirshindalman... platform-general-remov... January 28, 2024

This pull request is reviewed by Baz. Join @tpvasconcelos and the rest of your team on (Baz).

[MaryArmaly]

Hey @tpvasconcelos, could you please merge the latest changes from the main branch into your branch again? Thanks!

[tpvasconcelos]

@MaryArmaly done!

[tjwald]

@tpvasconcelos Please rebase and wait for CI to run again. Sorry for the delay, I will try to follow up when you rebase

[tpvasconcelos]

@tpvasconcelos Please rebase and wait for CI to run again. Sorry for the delay, I will try to follow up when you rebase

Done ✅

[tjwald]

@tpvasconcelos There are errors in the CI - can you please fix them?

[tpvasconcelos]

@tjwald I haven't looked at this in 1 year so, before I invest more time on it, could you first provide some feedback on the original issue I flagged (#6737) and on this solution and implementation? Cheers!

[tjwald]

I saw the issue and the solution and it LGTM. just need to make sure the CI passes 😄

[tpvasconcelos]

@tjwald the error seems to come from a flaky test that should not be directly affected by these changes.

I'm not sure what the expected value for this test duration is (apart from it having to be <1s) but for this run it was 1.01s

Have you seen this before?

=================================== FAILURES ===================================
_ TestCombinatorPluginMultilineYml.test_non_multiline_pair_time_limit_creating_report _
[gw1] linux -- Python 3.8.18 /home/runner/.local/share/virtualenvs/checkov-_hkiHoFg/bin/python

self = <tests.secrets.test_plugin_multiline_yml.TestCombinatorPluginMultilineYml testMethod=test_non_multiline_pair_time_limit_creating_report>

    def test_non_multiline_pair_time_limit_creating_report(self):
        # given
        test_files = [str(Path(__file__).parent / "yml_multiline/pomerium_compose.yml")]
        runner = Runner()
        runner_filter = RunnerFilter(framework=['secrets'])
    
        # when
        start_time = time.time()
        report = runner.run(root_folder=None, files=test_files, runner_filter=runner_filter)
        end_time = time.time()
    
        # then
>       assert end_time-start_time < 1  # assert the time limit is not too long for parsing long lines.
E       assert (1759153141.5744386 - 1759153140.5599039) < 1

tests/secrets/test_plugin_multiline_yml.py:185: AssertionError
=========================== short test summary info ============================
FAILED tests/secrets/test_plugin_multiline_yml.py::TestCombinatorPluginMultilineYml::test_non_multiline_pair_time_limit_creating_report - assert (1759153141.5744386 - 1759153140.5599039) < 1
= 1 failed, 4830 passed, 11 skipped, 2 xfailed, 6 warnings in 376.21s (0:06:16) =

[OfekShimko]

All checks have passed.
Thank you @tpvasconcelos for your contribution!

[OfekShimko]

Reverted due to this issue.