Performance Indicators
Metaheuristics.jl includes performance indicators to assess evolutionary optimization algorithms performance.
Available indicators:
Note that in Metaheuristics.jl
, minimization is always assumed. Therefore these indicators have been developed for minimization problems.
Metaheuristics.PerformanceIndicators
— ModulePerformanceIndicators
This module includes performance indicators to assess evolutionary multi-objective optimization algorithms.
gd
Generational Distance.igd
Inverted Generational Distance.gd_plus
Generational Distance plus.igd_plus
Inverted Generational Distance plus.covering
Covering indicator (C-metric).hypervolume
Hypervolume indicator.
Example
julia> import Metaheuristics: PerformanceIndicators, TestProblems
julia> A = [ collect(1:3) collect(1:3) ]
3×2 Array{Int64,2}:
1 1
2 2
3 3
julia> B = A .- 1
3×2 Array{Int64,2}:
0 0
1 1
2 2
julia> PerformanceIndicators.gd(A, B)
0.47140452079103173
julia> f, bounds, front = TestProblems.get_problem(:ZDT1);
julia> front
F space
┌────────────────────────────────────────┐
1 │⠁⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀│
│⠄⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀│
│⠈⠄⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀│
│⠀⠈⢆⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀│
│⠀⠀⠀⠢⡀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀│
│⠀⠀⠀⠀⠈⠢⡀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀│
│⠀⠀⠀⠀⠀⠀⠉⠢⡄⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀│
f_2 │⠀⠀⠀⠀⠀⠀⠀⠀⠈⠑⢤⡀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀│
│⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠈⠲⢄⡀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀│
│⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠈⠒⢤⡀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀│
│⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠈⠙⠢⢄⡀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀│
│⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠈⠑⠢⢄⡀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀│
│⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠈⠉⠢⠤⣀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀│
│⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠉⠑⠢⢤⣀⠀⠀⠀⠀⠀│
0 │⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠉⠒⠢⢄⣀│
└────────────────────────────────────────┘
0 1
f_1
julia> PerformanceIndicators.igd_plus(front, front)
0.0
Generational Distance
Metaheuristics.PerformanceIndicators.gd
— Functiongd(front, true_pareto_front; p = 1)
Returns the Generational Distance.
Parameters
front
and true_pareto_front
can be:
N×m
matrix whereN
is the number of points andm
is the number of objectives.State
Array{xFgh_indiv}
(usuallyState.population
)
Generational Distance Plus
Metaheuristics.PerformanceIndicators.gd_plus
— Functiongd_plus(front, true_pareto_front; p = 1)
Returns the Generational Distance Plus.
Parameters
front
and true_pareto_front
can be:
N×m
matrix whereN
is the number of points andm
is the number of objectives.State
Array{xFgh_indiv}
(usuallyState.population
)
Inverted Generational Distance
Metaheuristics.PerformanceIndicators.igd
— Functionigd(front, true_pareto_front; p = 1)
Returns the Inverted Generational Distance.
Parameters
front
and true_pareto_front
can be:
N×m
matrix whereN
is the number of points andm
is the number of objectives.State
Array{xFgh_indiv}
(usuallyState.population
)
Inverted Generational Distance Plus
Metaheuristics.PerformanceIndicators.igd_plus
— Functionigd_plus(front, true_pareto_front; p = 1)
Returns the Inverted Generational Distance Plus.
Parameters
front
and true_pareto_front
can be:
N×m
matrix whereN
is the number of points andm
is the number of objectives.State
Array{xFgh_indiv}
(usuallyState.population
)
Spacing Indicator
Metaheuristics.PerformanceIndicators.spacing
— Functionspacing(A)
Computes the Schott spacing indicator. spacing(A) == 0
means that vectors in A
are uniformly distributed.
Covering Indicator ($C$-metric)
Metaheuristics.PerformanceIndicators.covering
— Functioncovering(A, B)
Computes the covering indicator (percentage of vectors in B that are dominated by vectors in A) from two sets with non-dominated solutions.
A and B with size (n, m) where n is number of samples and m is the vector dimension.
Note that covering(A, B) == 1
means that all solutions in B are dominated by those in A. Moreover, covering(A, B) != covering(B, A)
in general.
If A::State
and B::State
, then computes covering(A.population, B.population)
after ignoring dominated solutions in each set.
Hypervolume
Metaheuristics.PerformanceIndicators.hypervolume
— Functionhypervolume(front, reference_point)
Computes the hypervolume indicator, i.e., volume between points in front
and reference_point
.
Note that each point in front
must (weakly) dominates to reference_point
. Also, front
is a non-dominated set.
If front::State
and reference_point::Vector
, then computes hypervolume(front.population, reference_point)
after ignoring solutions in front
that do not dominate reference_point
.
Examples
Computing hypervolume indicator from vectors in a Matrix
julia> import Metaheuristics.PerformanceIndicators: hypervolume
julia> f1 = collect(0:10); # objective 1
julia> f2 = 10 .- collect(0:10); # objective 2
julia> front = [ f1 f2 ]
11×2 Matrix{Int64}: 0 10 1 9 2 8 3 7 4 6 5 5 6 4 7 3 8 2 9 1 10 0
julia> reference_point = [11, 11]
2-element Vector{Int64}: 11 11
julia> hv = hypervolume(front, reference_point)
66.0
Now, let's compute the hypervolume implementation in Julia from the result of NSGA3
when solving DTLZ2 test problem.
julia> using Metaheuristics
julia> import Metaheuristics.PerformanceIndicators: hypervolume
julia> import Metaheuristics: TestProblems, get_non_dominated_solutions
julia> f, bounds, true_front = TestProblems.DTLZ2();
julia> result = optimize(f, bounds, NSGA3());
julia> approx_front = get_non_dominated_solutions(result.population)
100-element Vector{Metaheuristics.xFgh_solution{Vector{Float64}}}: (f = [0.9917916802035865, 1.2136578691520829e-5, 0.14325542722379989], g = [0.0], h = [0.0], x = [9.132e-02, 7.790e-06, …, 5.036e-01]) (f = [0.695695345065702, 0.6089089612211246, 0.40136918191082904], g = [0.0], h = [0.0], x = [2.607e-01, 4.577e-01, …, 4.920e-01]) (f = [0.891040420378389, 0.453001603345165, 0.06364969017001297], g = [0.0], h = [0.0], x = [4.048e-02, 2.994e-01, …, 4.779e-01]) (f = [0.9971627532883175, 0.09704629974511388, 0.00010095954017194353], g = [0.0], h = [0.0], x = [6.415e-05, 6.176e-02, …, 4.972e-01]) (f = [0.6996785927676169, 0.5098387932739061, 0.507021068811417], g = [0.0], h = [0.0], x = [3.373e-01, 4.009e-01, …, 5.039e-01]) (f = [0.643504645599712, 0.1870329165587611, 0.7456656176360811], g = [0.0], h = [0.0], x = [5.339e-01, 1.801e-01, …, 5.077e-01]) (f = [0.4117451894534776, 0.7074523198106492, 0.5850970758284032], g = [0.0], h = [0.0], x = [3.951e-01, 6.644e-01, …, 4.726e-01]) (f = [0.9993888443268505, 0.0592885969862795, 6.214275857863392e-5], g = [0.0], h = [0.0], x = [3.952e-05, 3.772e-02, …, 4.850e-01]) (f = [0.7545611333141676, 0.6605580220894377, 1.8789540626257055e-5], g = [0.0], h = [0.0], x = [1.193e-05, 4.578e-01, …, 5.023e-01]) (f = [0.9805256388420425, 0.203620760992724, 0.000517530390277091], g = [0.0], h = [0.0], x = [3.290e-04, 1.304e-01, …, 5.042e-01]) ⋮ (f = [0.408854468660186, 0.8166482870194989, 0.43855595146410126], g = [0.0], h = [0.0], x = [2.850e-01, 7.045e-01, …, 5.038e-01]) (f = [0.9985962462910579, 0.07191773092516894, 0.0034183482074281514], g = [0.0], h = [0.0], x = [2.174e-03, 4.577e-02, …, 4.853e-01]) (f = [0.9412337675934119, 0.2766062579978686, 0.19896809559695275], g = [0.0], h = [0.0], x = [1.274e-01, 1.820e-01, …, 5.085e-01]) (f = [6.138122235152718e-17, 1.002431434014496, 0.026087110728839374], g = [0.0], h = [0.0], x = [1.656e-02, 1.000e+00, …, 4.919e-01]) (f = [0.6185759441996121, 0.021740183498168007, 0.7931286458665786], g = [0.0], h = [0.0], x = [5.781e-01, 2.237e-02, …, 4.884e-01]) (f = [0.9218108883347504, 0.3910502537880052, 2.4437600739757835e-5], g = [0.0], h = [0.0], x = [1.554e-05, 2.554e-01, …, 4.853e-01]) (f = [0.7871080645155365, 0.6160701785245889, 0.06513774199360202], g = [0.0], h = [0.0], x = [4.143e-02, 4.228e-01, …, 4.804e-01]) (f = [0.3968191283528723, 0.46549898388409, 0.7931735784539267], g = [0.0], h = [0.0], x = [5.818e-01, 5.506e-01, …, 5.026e-01]) (f = [0.9878757483471329, 0.08754235830161197, 0.1430395935950811], g = [0.0], h = [0.0], x = [9.119e-02, 5.627e-02, …, 5.031e-01])
julia> reference_point = nadir(result.population)
3-element Vector{Float64}: 3.452892150434363 1.0068573410631054 1.0005281492293228
julia> hv = hypervolume(approx_front, reference_point)
2.869718929233642
$\Delta_p$ (Delta $p$)
Metaheuristics.PerformanceIndicators.deltap
— Functiondeltap(front, true_pareto_front; p = 1)
Δₚ(front, true_pareto_front; p = 1)
Returns the averaged Hausdorff distance indicator aka Δₚ (Delta p).
"Δₚ
" can be typed as \Delta<tab>\_p<tab>
.
Parameters
front
and true_pareto_front
can be:
N×m
matrix whereN
is the number of points andm
is the number of objectives.Array{xFgh_indiv}
(usuallyState.population
)
$\varepsilon$-Indicator
Unary and binary $\varepsilon$-indicator (epsilon-indicator). Details in E. Zitzler, L. Thiele, M. Laumanns, C.M. Fonseca, V.G. da Fonseca (2003)
Metaheuristics.PerformanceIndicators.epsilon_indicator
— Functionepsilon_indicator(A, B)
Computes the ε-indicator for non-dominated sets A
and B
. It is assumed that all values in A
and B
are positive. If negative, the sets are translated to positive values.
Interpretation
epsilon_indicator(A, PF)
is unary ifPF
is the Pareto-optimal front.epsilon_indicator(A, B) == 1
none is better than the other.epsilon_indicator(A, B) < 1
means that A is better than B.epsilon_indicator(A, B) > 1
means that B is better than A.- Values closer to 1 are preferable.
Examples
julia> A1 = [4 7;5 6;7 5; 8 4.0; 9 2];
julia> A2 = [4 7;5 6;7 5; 8 4.0];
julia> A3 = [6 8; 7 7;8 6; 9 5;10 4.0 ];
julia> PerformanceIndicators.epsilon_indicator(A1, A2)
1.0
julia> PerformanceIndicators.epsilon_indicator(A1, A3)
0.9
julia> f, bounds, pf = Metaheuristics.TestProblems.ZDT3();
julia> res = optimize(f, bounds, NSGA2());
julia> PerformanceIndicators.epsilon_indicator(res, pf)
1.00497701620997