Skip to content

Conversation

@jfzunigac
Copy link
Contributor

Adding support for wildcard use in all parts of directory configs, it currently is limited to only last part of directory string. In addition, refactoring certain part of the code structure to improve readability. List of main changes:

  • Change logConfigMap to use Pair<> for keys: This map is essentially only used in Kubernetes mode by RecursiveFSEventProcessor. During the pod directory traversal, we use this map to determine if there are any configs left for directories in a sublevel. Now that we have wildcards in the directory names, we can no longer use subMap() function reliably. Thus, we change the approach to directly use directory depth for this purpose. The Pair consists of directory depth (for ordering) and directory (for tracking, can contain wildcard).

  • Change approach for Wildcard matching: Before we use to translate the wildcards to regex pattern, now we use Apache Commons FilenameUtils.wildcardMatch + Files.walkFileTree for traversal, which should be a bit more efficient than using regex.

  • MissingDirChecker: Removed extra data structure to hold wildcard directories, now we combine them into the existing map and perform dynamic discovery for wildcard directories only.

Tested the following scenarios in devapp:

  1. No wildcard directories and single directory (standard use case checking for regression)
  2. Wildcard directory + no wildcard directory
  3. Wildcard directory only
  4. Wildcard directory dynamic discovery: Create directory and make sure Singer picks it up

@jfzunigac jfzunigac requested a review from a team as a code owner March 17, 2025 23:49
@jfzunigac jfzunigac changed the title allow multiple directories with wildcards per log config Allow multiple directories per log config Mar 17, 2025
@jfzunigac jfzunigac merged commit 090ed7f into pinterest:master Mar 18, 2025
1 check passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants