Preprocessing #11

Mmorgenn · 2024-12-21T01:00:05Z

Module for sample preprocessing for EM algorithm includes:

Methods for estimating the number of components of a mixture:

-> Elbow method
-> Sihouette method
-> Peaks method
-> XMeans method

Mixture component classifiers:

-> Constant Classifier
-> Assymetry Classifier
-> KSTest Classifier
-> ICClassifier

Information criterion (AIC, BIC)
Random initialisation of parameters of the mixture of distributions

toxakaz · 2025-01-10T04:22:15Z

You can change file pyproject.toml, correcting [tool.pylint] section to disable too-many-positional-arguments pylint warnings. It will be done in future anyway.

[tool.pylint]
disable = "too-few-public-methods, too-many-locals, line-too-long, too-many-positional-arguments, fixme"

Copilot

Pull Request Overview

Adds a preprocessing module to set up tasks for the EM algorithm, including component number estimators, classifiers, and random parameter initialization.

Introduces new dependencies and utility types
Implements Preprocessing orchestration and Parameterization
Adds AIC/BIC criteria, four component number methods, and four classifiers

Reviewed Changes

Copilot reviewed 31 out of 31 changed files in this pull request and generated 10 comments.

Show a summary per file

File	Description
pyproject.toml	Added `kneed` dependency required by Elbow method
mpest/preprocessing/utils.py	Introduced `Model` union type for supported distributions
mpest/preprocessing/preprocessing.py	Added `Preprocessing` class orchestrating EM task creation
mpest/preprocessing/parameterization.py	Implemented random initialization of mixture parameters
mpest/preprocessing/criterions/bayesian_criterion.py	Added BIC implementation
mpest/preprocessing/criterions/akaike_criterion.py	Added AIC implementation
mpest/preprocessing/criterions/abstract_criterion.py	Defined base logic for information criteria
mpest/preprocessing/criterions/init.py	Exported criterion classes
mpest/preprocessing/components_number/xmeans.py	Implemented X-Means estimator
mpest/preprocessing/components_number/silhouette.py	Implemented Silhouette method estimator
mpest/preprocessing/components_number/peaks.py	Implemented Peaks method estimator
mpest/preprocessing/components_number/elbow.py	Implemented Elbow method estimator
mpest/preprocessing/components_number/abstract_estimator.py	Defined abstract estimator interface
mpest/preprocessing/components_number/init.py	Exported component number estimators
mpest/preprocessing/components_classifier/kstest.py	Added KS-test based classifier
mpest/preprocessing/components_classifier/icclassifier.py	Added IC-based classifier
mpest/preprocessing/components_classifier/constant.py	Added constant classifier for edge cases
mpest/preprocessing/components_classifier/assymetry.py	Added skewness-based classifier
mpest/preprocessing/components_classifier/abstract_classifier.py	Defined abstract classifier interface
mpest/preprocessing/components_classifier/init.py	Exported classifier classes

Comments suppressed due to low confidence (3)

mpest/preprocessing/parameterization.py:14

The method name get_componet_parameters has a typo; consider renaming to get_component_parameters for clarity.

def get_componet_parameters(model: Model, samples: Samples) -> np.ndarray:

mpest/preprocessing/criterions/abstract_criterion.py:19

The method name _loglikehood is misspelled; consider renaming to _loglikelihood.

def _loglikehood(

mpest/preprocessing/components_classifier/assymetry.py:12

The class name Assymetry is misspelled; consider renaming to Asymmetry to match the standard term.

class Assymetry(AComponentsClassifier):

Copilot · 2025-05-25T04:45:30Z

mpest/preprocessing/utils.py

@@ -0,0 +1,5 @@
+"""Module wich contains supported distributions"""


Fix the typo in the module docstring: replace "wich" with "which".

Suggested change

"""Module wich contains supported distributions"""

"""Module which contains supported distributions"""

Copilot · 2025-05-25T04:45:30Z

mpest/preprocessing/preprocessing.py

@@ -0,0 +1,38 @@
+"""Module wich contains the method of task creation for EM algorithm"""


Fix the typo in the module docstring: replace "wich" with "which".

Suggested change

"""Module wich contains the method of task creation for EM algorithm"""

"""Module which contains the method of task creation for EM algorithm"""

Copilot · 2025-05-25T04:45:31Z

mpest/preprocessing/parameterization.py

@@ -0,0 +1,45 @@
+"""Module wich contains a method for random parameterization of distributions"""


Fix the typo in the module docstring: replace "wich" with "which".

Suggested change

"""Module wich contains a method for random parameterization of distributions"""

"""Module which contains a method for random parameterization of distributions"""

Copilot · 2025-05-25T04:45:31Z

mpest/preprocessing/criterions/bayesian_criterion.py

+
+    @property
+    def name(self):
+        return "Bayesian_Infromation_Criterion"


Fix the typo in the name property: replace "Infromation" with "Information".

Suggested change

return "Bayesian_Infromation_Criterion"

return "Bayesian_Information_Criterion"

Copilot · 2025-05-25T04:45:31Z

mpest/preprocessing/criterions/akaike_criterion.py

+
+    @property
+    def name(self):
+        return "Akaike_Infromation_Criterion"


Fix the typo in the name property: replace "Infromation" with "Information".

Suggested change

return "Akaike_Infromation_Criterion"

return "Akaike_Information_Criterion"

Copilot · 2025-05-25T04:45:31Z

mpest/preprocessing/components_number/elbow.py

+
+    def estimate(self, samples: Samples) -> int:
+        samples = samples.reshape(-1, 1)
+        k_range = range(1, self.kmax + 2)  # possible components: [2, kmax]


[nitpick] The comment incorrectly states the range; update it to reflect range(1, kmax+2) or adjust code to match the intended range.

Suggested change

k_range = range(1, self.kmax + 2) # possible components: [2, kmax]

k_range = range(1, self.kmax + 2) # possible components: [1, kmax + 1]

Copilot · 2025-05-25T04:45:32Z

mpest/preprocessing/components_classifier/constant.py

+    X-Means method
+    -----


The class docstring mentions "X-Means method" but this is the Constant Classifier; update the description accordingly.

Suggested change

X-Means method

-----

Constant Classifier

-----

This classifier always predicts the same model for all inputs.

Copilot · 2025-05-25T04:45:32Z

mpest/preprocessing/components_number/peaks.py

+
+
+class Peaks(AComponentsNumber):
+    """Peaks Method with Emperical Density"""


Fix the typo: replace "Emperical" with "Empirical".

Suggested change

"""Peaks Method with Emperical Density"""

"""Peaks Method with Empirical Density"""

Copilot · 2025-05-25T04:45:32Z

mpest/preprocessing/components_classifier/__init__.py

+"""TODO"""
+


[nitpick] Replace the placeholder docstring with a brief module description.

Suggested change

"""TODO"""

"""This module provides various components and utilities for classifying and preprocessing data.

It includes abstract base classes, specific classifiers, and statistical tests such as:

- AComponentsClassifier: Abstract base class for component classifiers.

- Assymetry: Handles asymmetry-related classification tasks.

- CModel: Represents a constant model for classification.

- ICClassifier: Implements an information criterion-based classifier.

- KSTest: Performs the Kolmogorov-Smirnov test for statistical analysis.

"""

Copilot · 2025-05-25T04:45:32Z

mpest/preprocessing/components_number/__init__.py

@@ -0,0 +1,8 @@
+"""Module which represents method estimating components number and abstract classe"""


Fix the typo: replace "classe" with "classes".

Suggested change

"""Module which represents method estimating components number and abstract classe"""

"""Module which represents method estimating components number and abstract classes"""

feature: Added three estimator: Elbow, Silhouette, Peaks

f1cb843

Mmorgenn changed the title ~~Components number estimators~~ сomponents number estimators Dec 21, 2024

Mmorgenn changed the title ~~сomponents number estimators~~ Сomponents number estimators Dec 21, 2024

Mmorgenn added 5 commits December 21, 2024 08:13

feature: abstract estimator

ff214ff

Added estimator XMeans, AIC, BIC

9d8f02f

ci/cd: Fixed pylint

603fe9d

ci/cd: Fixed black

c885ce2

ci/cd: Fixed requirements

dfd27e7

Mmorgenn added 5 commits April 4, 2025 18:36

feature: Added three estimator: Elbow, Silhouette, Peaks

1277864

feature: abstract estimator

6b8e3ec

Added estimator XMeans, AIC, BIC

3607155

ci/cd: Fixed pylint

fdc8f5b

ci/cd: Fixed black

e0cff35

Mmorgenn force-pushed the components_number branch from dfd27e7 to e0cff35 Compare April 4, 2025 15:50

Mmorgenn added 2 commits April 11, 2025 02:47

upd: Modifications for new linters

f9b4125

ci/cd: Fixed dependencies

40ad774

Mmorgenn changed the title ~~Сomponents number estimators~~ Preprocessing May 12, 2025

feature: Added three classifier: Assymetry, KSTest, ICClassifier

c04596d

iraedeus requested review from Copilot and iraedeus May 25, 2025 04:42

Copilot AI reviewed May 25, 2025

View reviewed changes

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Preprocessing #11

Preprocessing #11

Uh oh!

Mmorgenn commented Dec 21, 2024 •

edited

Loading

Uh oh!

toxakaz commented Jan 10, 2025

Uh oh!

Copilot AI left a comment

Uh oh!

Copilot AI May 25, 2025

Uh oh!

Copilot AI May 25, 2025

Uh oh!

Copilot AI May 25, 2025

Uh oh!

Copilot AI May 25, 2025

Uh oh!

Copilot AI May 25, 2025

Uh oh!

Copilot AI May 25, 2025

Uh oh!

Copilot AI May 25, 2025

Uh oh!

Copilot AI May 25, 2025

Uh oh!

Copilot AI May 25, 2025

Uh oh!

Copilot AI May 25, 2025

Uh oh!

Uh oh!

		@@ -0,0 +1,5 @@
		"""Module wich contains supported distributions"""

	"""Module wich contains supported distributions"""
	"""Module which contains supported distributions"""

		@@ -0,0 +1,38 @@
		"""Module wich contains the method of task creation for EM algorithm"""

	"""Module wich contains the method of task creation for EM algorithm"""
	"""Module which contains the method of task creation for EM algorithm"""

		@@ -0,0 +1,45 @@
		"""Module wich contains a method for random parameterization of distributions"""

	"""Module wich contains a method for random parameterization of distributions"""
	"""Module which contains a method for random parameterization of distributions"""

	return "Bayesian_Infromation_Criterion"
	return "Bayesian_Information_Criterion"

	return "Akaike_Infromation_Criterion"
	return "Akaike_Information_Criterion"

	k_range = range(1, self.kmax + 2) # possible components: [2, kmax]
	k_range = range(1, self.kmax + 2) # possible components: [1, kmax + 1]

-    X-Means method
-    -----
+    Constant Classifier
+    -----
+    This classifier always predicts the same model for all inputs.



		class Peaks(AComponentsNumber):
		"""Peaks Method with Emperical Density"""

	"""Peaks Method with Emperical Density"""
	"""Peaks Method with Empirical Density"""

-"""TODO"""
+"""This module provides various components and utilities for classifying and preprocessing data.
+It includes abstract base classes, specific classifiers, and statistical tests such as:
+- AComponentsClassifier: Abstract base class for component classifiers.
+- Assymetry: Handles asymmetry-related classification tasks.
+- CModel: Represents a constant model for classification.
+- ICClassifier: Implements an information criterion-based classifier.
+- KSTest: Performs the Kolmogorov-Smirnov test for statistical analysis.
+"""

		@@ -0,0 +1,8 @@
		"""Module which represents method estimating components number and abstract classe"""

Preprocessing #11

Are you sure you want to change the base?

Preprocessing #11

Uh oh!

Conversation

Mmorgenn commented Dec 21, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

toxakaz commented Jan 10, 2025

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull Request Overview

Reviewed Changes

Uh oh!

Copilot AI May 25, 2025

Choose a reason for hiding this comment

Uh oh!

Copilot AI May 25, 2025

Choose a reason for hiding this comment

Uh oh!

Copilot AI May 25, 2025

Choose a reason for hiding this comment

Uh oh!

Copilot AI May 25, 2025

Choose a reason for hiding this comment

Uh oh!

Copilot AI May 25, 2025

Choose a reason for hiding this comment

Uh oh!

Copilot AI May 25, 2025

Choose a reason for hiding this comment

Uh oh!

Copilot AI May 25, 2025

Choose a reason for hiding this comment

Uh oh!

Copilot AI May 25, 2025

Choose a reason for hiding this comment

Uh oh!

Copilot AI May 25, 2025

Choose a reason for hiding this comment

Uh oh!

Copilot AI May 25, 2025

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Mmorgenn commented Dec 21, 2024 •

edited

Loading