Implement Content Updater JobScheduler #387

QU3B1M · 2025-04-28T20:37:10Z

Description

Job Scheduler implementation for the Content Updater module

Issues Resolved

Closes: #366

Add toXContent logic to ContentUpdater Job parameters Remove unnecesary code from build.gradle

abbonno

I have some proposals, but overall everything is right.

plugins/content-manager/src/main/java/com/wazuh/contentmanager/index/ContentIndex.java

plugins/content-manager/src/main/java/com/wazuh/contentmanager/client/CTIClient.java

...ent-manager/src/main/java/com/wazuh/contentmanager/jobscheduler/ContentUpdaterJobRunner.java

Signed-off-by: Álex Ruiz Becerra <alejandro.ruiz.becerra@wazuh.com>

AlexRuiz7 · 2025-05-26T17:25:13Z

plugins/content-manager/build.gradle

 integTest {
    // The --debug-jvm command-line option makes the cluster debuggable; this makes the tests debuggable
    if (System.getProperty("test.debug") != null) {
        jvmArgs '-agentlib:jdwp=transport=dt_socket,server=y,suspend=y,address=*:5005'
    }
+    environment "IS_DEV", System.getenv("IS_DEV") ?: "false"


What is this for?

Since the required indices are managed by the setup plugin I needed a way to enable creation of the indices only on development environments.

AlexRuiz7 · 2025-05-26T17:27:22Z

plugins/content-manager/src/main/java/com/wazuh/contentmanager/ContentManagerPlugin.java

+        // @TODO do we want an IS_DEV variable?
+        if (System.getenv("IS_DEV").equals("true") && localNode.isClusterManagerNode()) {
+            this.contextIndex.createIndex();
+            this.contentIndex.createIndex();


The content index wazuh-cve is created by the initialization plugin. We only check for its existence here.

I used the IS_DEV environment variable to ease development and testing. If the variable is set, the indices are created. Otherwise errors are thrown when running the plugin without the setup plugin.

plugins/content-manager/src/main/java/com/wazuh/contentmanager/ContentManagerPlugin.java

plugins/content-manager/src/main/java/com/wazuh/contentmanager/client/CTIClient.java

AlexRuiz7 · 2025-05-26T17:33:20Z

plugins/content-manager/src/main/java/com/wazuh/contentmanager/index/ContentIndex.java

+    public void createIndex() {
+        if (!this.exists()) {
+            client.admin().indices().create(new CreateIndexRequest(INDEX_NAME)).actionGet();
+        }
+    }
+
+    /**
+     * Checks whether the {@link ContentIndex#INDEX_NAME} index exists.
+     *
+     * @see ClusterInfo#indexExists(Client, String)
+     * @return true if the index exists, false otherwise.
+     */
+    public boolean exists() {
+        return ClusterInfo.indexExists(this.client, ContentIndex.INDEX_NAME);
+    }
+


The content index is created by the initialization plugin.

This is used by onNodeStarted() when the IS_DEV variable is set up. Otherwise the snapshot/update process is not triggered.

plugins/content-manager/src/main/java/com/wazuh/contentmanager/index/ContextIndex.java

AlexRuiz7 · 2025-05-26T17:42:25Z

plugins/content-manager/src/main/java/com/wazuh/contentmanager/index/ContextIndex.java

@@ -137,6 +195,7 @@ public ConsumerInfo get(String context, String consumer) {
                    "Fetched consumer from the [{}] index: {}", ContextIndex.INDEX_NAME, this.consumerInfo);
        } catch (InterruptedException | ExecutionException | TimeoutException e) {
            log.error("Failed to fetch consumer [{}][{}]: {}", context, consumer, e.getMessage());
+            this.consumerInfo = new ConsumerInfo(consumer, context, 0L, 0L, "");


We need to somehow distinguish between a valid consumer with offset 0, or having no consumer information at all (not yet initialized) We return null in this case, as documented in the method's signature.

Previously, null was returned here, but both a 0 offset and a null ConsumerInfo object were handled in the same conditional expression (both were ORed). So for all practical purposes, a zero offset and a null object were the same.
I considered this case and concluded that 0 is not a valid remote offset.

Let me know your thoughts on this.

...-manager/src/main/java/com/wazuh/contentmanager/jobscheduler/ContentUpdaterJobParameter.java

...ent-manager/src/main/java/com/wazuh/contentmanager/jobscheduler/ContentUpdaterJobRunner.java

...tent-manager/src/main/java/com/wazuh/contentmanager/jobscheduler/ContentUpdaterRunnable.java

plugins/content-manager/src/main/java/com/wazuh/contentmanager/updater/ContentUpdater.java

plugins/content-manager/src/main/java/com/wazuh/contentmanager/utils/Privileged.java

plugins/content-manager/src/main/java/com/wazuh/contentmanager/utils/SnapshotManager.java

AlexRuiz7 · 2025-05-27T07:54:10Z

plugins/content-manager/src/main/java/com/wazuh/contentmanager/utils/SnapshotManager.java

+    /**
+     * Alternate constructor that allows injecting CommandManagerClient for test purposes. Dependency
+     * injection.
+     *
+     * @param environment Needed for snapshot file handling.
+     * @param contextIndex Handles context and consumer related metadata.
+     * @param contentIndex Handles indexed content.
+     * @param privileged Handles privileged actions.
+     * @param ctiClient Instance of CTIClient.
+     * @param commandManagerClient Instance of CommandManagerClient.
+     */
+    public SnapshotManager(
+            Environment environment,
+            ContextIndex contextIndex,
+            ContentIndex contentIndex,
+            Privileged privileged,
+            CTIClient ctiClient,
+            CommandManagerClient commandManagerClient) {
+        this.ctiClient = ctiClient;
+        this.commandManagerClient = commandManagerClient;
+        this.environment = environment;
+        this.contextIndex = contextIndex;
+        this.contentIndex = contentIndex;
+        this.privileged = privileged;
+        this.pluginSettings = PluginSettings.getInstance();
+    }


Probably duplicated. This and the previous constructor are similar and can fit the same use case.

This was added after the merge

plugins/content-manager/src/main/java/com/wazuh/contentmanager/utils/SnapshotManager.java

AlexRuiz7 · 2025-05-27T07:58:55Z

plugins/content-manager/src/main/java/com/wazuh/contentmanager/utils/SnapshotManager.java

        log.debug("Latest consumer info: {}", latest);

        // Consumer is not yet initialized. Initialize to latest.
-        if (current == null || current.getOffset() == 0) {


Consumer's initialization should only happen when it's not ye initialized (null or offset == 0). We are losing this condition now.

the current object is created with a zeroed offset. It cannot be null by the point it reaches this expression

plugins/content-manager/src/test/java/com/wazuh/contentmanager/utils/SnapshotManagerTests.java

Signed-off-by: Fede Galland <99492720+f-galland@users.noreply.github.com>

QU3B1M · 2025-05-29T10:26:04Z

plugins/content-manager/src/main/java/com/wazuh/contentmanager/ContentManagerPlugin.java

+    /** Scheduled jobs index name. */
+    public static final String JOB_INDEX = ".content_updater_jobs";
+
+    /** Scheduled job ID. */
+    public static final String JOB_ID = "content_updater_job";


Could we store these constants into PluginSettings?

Implement JobScheduler boilerplate

db04746

QU3B1M self-assigned this Apr 28, 2025

QU3B1M and others added 8 commits April 28, 2025 17:49

Add JobScheduler plugin to build.gradle

0905525

Update build.gradle to avoid duplicated plugins IDs

9b69a07

Implement unit tests for ContentUpdaterJobParameters

e0b66d9

Add toXContent logic to ContentUpdater Job parameters Remove unnecesary code from build.gradle

Implement minimal required functionality to ContentUpdaterJobRunner

742ac9f

Add content manager logic

e0291fb

Load job-scheduler in tests

b4f847f

Add plugin workflow logic to runnable

9f9f13e

Add streaming download method

b5e50b2

f-galland self-assigned this May 8, 2025

AlexRuiz7 unassigned QU3B1M May 9, 2025

f-galland added 4 commits May 12, 2025 15:40

Include debug config in gradle

8e2e56b

Refactor to give CTIClient privileges

ee67948

Move jvmArgs to bottom of method

9c4779c

Merge branch 'main' into enhancement/366-content-manager-job-scheduler

5c2a358

abbonno mentioned this pull request May 13, 2025

Refactor snapshot download #413

Merged

f-galland added 13 commits May 13, 2025 13:47

Cleanup code after main merge

3870f65

Remove unneeded parameters

cf6a10e

Fix privileged

8e5acdd

Centralize offset comparisons to Runnable class

f56f321

Fix singleton

412ebe6

Fix context index creation upon initialization

0fd09ab

Create indices for dev environments

d1924fe

Use dev env variable

60affcd

Fix tests

b56b85b

Fix SnapshotManager tests

af28834

Fix link checker

6aa8b38

Merge branch '6.0.0' into enhancement/366-content-manager-job-scheduler

8cbcd82

Fix merge issues

8615b85

f-galland added 4 commits May 20, 2025 12:48

Fix tests

e9b490a

Merge branch '6.0.0' into enhancement/366-content-manager-job-scheduler

3c9b656

Merge changes from 6.0.0

63794f9

Fix tests after merge

a60869c

f-galland marked this pull request as ready for review May 20, 2025 16:09

f-galland requested a review from a team as a code owner May 20, 2025 16:09

abbonno previously approved these changes May 21, 2025

View reviewed changes

Delete old download method

b3a96f5

f-galland dismissed abbonno’s stale review via b3a96f5 May 21, 2025 15:18

Merge branch '6.0.0' into enhancement/366-content-manager-job-scheduler

7068691

Signed-off-by: Álex Ruiz Becerra <alejandro.ruiz.becerra@wazuh.com>

AlexRuiz7 reviewed May 26, 2025

View reviewed changes

AlexRuiz7 requested changes May 27, 2025

View reviewed changes

f-galland and others added 17 commits May 27, 2025 17:43

Fix after merge

81930bb

Use download()

9b4f925

Remove create()

1447817

Switch to OpenSearchStatusException

095e57f

Fix tests

3fba343

Delete unused get() method

01e6f0c

Delete unused methods

5d03a2d

Make ContentUpdaterJobRunner a pseudo-builder

6259c76

Make ContentUpdaterRunnable a pseudo-builder

f4c723d

Remove unused wrapper

7cbfce9

Remove privileged wrappers

3b3dc0f

Remove redundant constructor

a5f2134

Fix links

ed59528

Merge branch '6.0.0' into enhancement/366-content-manager-job-scheduler

a283c64

Signed-off-by: Fede Galland <99492720+f-galland@users.noreply.github.com>

Merge branch '6.0.0' into enhancement/366-content-manager-job-scheduler

39adfe6

Merge 6.0.0

2fa23d8

Fix after merge

c73b9a3

QU3B1M commented May 29, 2025

View reviewed changes

Implement Content Updater JobScheduler #387

Are you sure you want to change the base?

Implement Content Updater JobScheduler #387

Uh oh!

Conversation

QU3B1M commented Apr 28, 2025

Description

Issues Resolved

Uh oh!

abbonno left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

abbonno left a comment •

edited

Loading