Use / Contribute to moocore: https://multi-objective.github.io/moocore/python/

[MLopez-Ibanez]

The goal of moocore is not to replace jMetalPy but simply to provide high-quality basic building blocks to make building such libraries easier.

Even with its current non-state-of-the-art algorithms (and implementing the state-of-the-art ones is a goal of the project), moocore is order of magnitudes faster than most multi-objective python packages, not only in hypervolume but in all indicators.

If you find any benchmarks were moocore is slower or produces the wrong output, please let me know.

Feedback on how to make moocore easier to use in jMetalPy would be welcome.

Benchmark source code

The following benchmark compares the computation of the hypervolume in moocore and in jMetalPy:

import numpy as np
import moocore

from jmetal.core.quality_indicator import HyperVolume as jmetal_HV

import timeit
timeit.template = """
def inner(_it, _timer{init}):
    {setup}
    _t0 = _timer()
    for _i in _it:
        retval = {stmt}
    _t1 = _timer()
    return _t1 - _t0, retval
"""

def read_data(name):
    files = {"DTLZLinearShape.3d": "~/work/perfassess/moocore/testsuite/data/DTLZLinearShape.3d.front.1000pts.10",
             "DTLZLinearShape.4d" : "~/work/perfassess/moocore/testsuite/data/DTLZLinearShape.4d.front.1000pts.10"}
    x = moocore.read_datasets(files[name])[:, :-1]
    x = moocore.filter_dominated(x)
    return x


name = "DTLZLinearShape.4d"
x = read_data(name)

ref = np.ones(x.shape[1])
jmetal_hv = jmetal_HV(ref)
moocore_hv = moocore.Hypervolume(ref = ref)

n = np.arange(500, 3000 + 1, 500)

jmetal_values = []
moocore_values =  []
jmetal_times = []
moocore_times =  []
for maxrow in n:
    z = x[:maxrow, :]
    duration, values = timeit.timeit('jmetal_hv.compute(z)', globals=globals(), number = 2)
    jmetal_times += [duration]
    jmetal_values += [values]
    duration, values = timeit.timeit('moocore_hv(z)', globals=globals(), number = 2)
    moocore_times += [duration]
    moocore_values += [values]
    
jmetal_values = np.array(jmetal_values)
moocore_values = np.array(moocore_values)
assert np.allclose(jmetal_values, moocore_values)

jmetal_times = np.array(jmetal_times)
moocore_times = np.array(moocore_times)

import pandas as pd
import matplotlib.pyplot as plt
cpu = "Intel i5-6200U 2.30GHz"
df = pd.DataFrame(dict(n = n, jMetalPy = jmetal_times, moocore = moocore_times)).set_index('n')
df.plot(grid=True, ylabel='CPU time (seconds)', title = f'HV computation for {name} ({cpu})')
plt.savefig(f'jmetal-{name}-time.pdf')
plt.show()

You can find the benchmark files here: https://github.com/multi-objective/testsuite or use your own.

[ajnebro]

Hi Manuel.
Thanks of opening this issue; I'll check your tool to add some examples of use in jMetalPy. As you point out, the hypervolume implementation in jMetal (which is based on a proposal of yours) is very slow.

If you think than I could colaborate somehow in moocore (e.g., testing it with jMetalPy algorithms), please let me know.

[ajnebro]

After testing that it works properly, I have replaced the current HV code by a call to the implementation included in moocore.

The performance improvement would now make it possible to implement the SMS-EMOA algorithm, which was unfeasible with the old HV code.

[MLopez-Ibanez]

Great! moocore welcomes all contributions. The goal is to provide as a well-tested well-documented repository for highly-performing building blocks. For the hypervolume contributions, I am not sure yet what the best interface is.

There is the hv_contributions() function in the R interface and I could easily add the same function in the python interface but I am not sure if this is the API that you need for SMS-EMOA. Suggestions about useful APIs are welcome! Please open an issue in https://github.com/multi-objective/moocore/issues

[MLopez-Ibanez]

By the way, the performance improvements not only apply to hypervolume. See for example:

import numpy as np
import moocore
import pandas as pd
import pathlib
import matplotlib.pyplot as plt

from jmetal.core.quality_indicator import EpsilonIndicator as jmetal_eps

import timeit
timeit.template = """
def inner(_it, _timer{init}):
    {setup}
    _dt = float('inf')
    for _i in _it:
        _t0 = _timer()
        retval = {stmt}
        _t1 = _timer()
        _dt = min(_dt, _t1 - _t0)
    return _dt, retval
"""

# FIXME: How to get this info automatically?
cpu = "Intel i5-6200U 2.30GHz"
# See https://github.com/multi-objective/testsuite/tree/main/data
path_to_data = "~/work/perfassess/moocore/testsuite/data/" 
files = {"rmnk_10D_random_search": (path_to_data + "rmnk_0.0_10_16_1_0_random_search_1.txt.xz",
                                    path_to_data + "rmnk_0.0_10_16_1_0_ref.txt.xz"),
}

def read_data(filename):
    x = moocore.read_datasets(filename)[:, :-1]
    x = moocore.filter_dominated(x)
    return x

eps = {}
names = files.keys()
for name in files.keys():
    x = read_data(files[name][0])
    ref = np.loadtxt(pathlib.Path(files[name][1]).expanduser())
    n = np.arange(200, min(len(x),1000) + 1, 200)

    eps["moocore"] = lambda z, ref=ref: moocore.epsilon_additive(z, ref=ref)
    eps["jMetalPy"] = lambda z, eps=jmetal_eps(ref): eps.compute(z)
    
    values = { k:[] for k in eps.keys() }
    times = { k:[] for k in eps.keys() }
    for maxrow in n:
        z = x[:maxrow, :]
        for what in eps.keys():
            eps_fun = eps[what]
            test = 'eps_fun(z)'
            duration, value = timeit.timeit(test, globals=globals(), number = 3)
            times[what] += [duration]
            values[what] += [value]
            
    for what in eps.keys():
        values[what] = np.array(values[what])
        times[what] = np.array(times[what])

    for what in eps.keys():
        if what == "moocore":
            continue
        assert np.allclose(values["moocore"], values[what]), f"In {name}, {what}={values[what]}  not equal to moocore={values['moocore']}"

    df = pd.DataFrame(dict(n = n, **times)).set_index('n')
    df.plot(grid=True, ylabel='CPU time (seconds)', style='o-',
            title = f'eps+ computation for {name} ({cpu})',
            logy = True)
    plt.savefig(f'eps_bench-{name}-time.png')

    reltimes = {}
    for what in eps.keys():
        if what == "moocore":
            continue
        reltimes["Rel_" + what] = times[what]/times["moocore"]

    df = pd.DataFrame(dict(n = n, **reltimes)).set_index('n')
    df.plot(grid=True, ylabel='Time relative to moocore', style='o-',
            title = f'eps+ computation for {name} ({cpu})')
    plt.savefig(f'eps_bench-{name}-reltime.png')

plt.show()

[MLopez-Ibanez]

@ajnebro There is now a hv_contributions() function: https://multi-objective.github.io/moocore/python/reference/generated/moocore.hv_contributions.html

and it will be much faster in 0.1.9 for 3D inputs: https://multi-objective.github.io/moocore/python/whatsnew/index.html#what-s-new

[ajnebro]

Hi @MLopez-Ibanez

I have just implemented SMS-EMOA in jMetalPy using the hv_contributions() function of moocore 0.1.8. The computing time to optimize the DTLZ3 problem with three objectives using a population of 100 individuals and a stopping condition of 40000 function evaluation is about 100 seconds. This time is comparable with the one obtained with the implementation of jMetal, which is 64 seconds.

Thanks a lot for providing your library.

[MLopez-Ibanez]

Please try 0.1.9: https://multi-objective.github.io/moocore/python/whatsnew/index.html#version-0-1-9-dev
It should be significantly faster for 3D.

[MLopez-Ibanez]

You can install the development version easily. We provide pre-compiled wheels for all OS: https://github.com/multi-objective/moocore/tree/main/python#install