plot_sparsity (generic function with 1 method)Alternating methods
In this example we will compare FrankWolfe.alternating_linear_minimization and FrankWolfe.alternating_projections for a very simple feasibility problem.
We consider the probability simplex
\[P = \{ x \in \mathbb{R}^n \colon \sum_{i=1}^n x_i = 1, x_i \geq 0 ~~ i=1,\dots,n\} ~.\]
and a scaled, shifted $\ell^{\infty}$ norm ball
\[Q = [-1,0]^n ~.\]
The goal is to find a point that lies both in $P$ and $Q$. We do this by reformulating the problem first. Instead of a finding a point in the intersection $P \cap Q$, we search for a pair of points, $(x_P, x_Q)$ in the cartesian product $P \times Q$, which attains minimal distance between $P$ and $Q$,
\[\|x_P - x_Q\|_2 = \min_{(x,y) \in P \times Q} \|x - y \|_2 ~.\]
using FrankWolfe
include("../examples/plot_utils.jl")plot_sparsity (generic function with 1 method)Setting up objective, gradient and linear minimization oracles
Alternating Linear Minimization (ALM) allows for an additional objective such that one can optimize over an intersection of sets instead of finding only feasible points. Since this example only considers the feasibility, we set the objective function as well as the gradient to zero.
n = 20
f(x) = 0
function grad!(storage, x)
@. storage = zero(x)
end
lmo1 = FrankWolfe.ProbabilitySimplexOracle(1.0)
lmo2 = FrankWolfe.ScaledBoundLInfNormBall(-ones(n), zeros(n))
lmos = (lmo1, lmo2)
x0 = rand(n)
target_tolerance = 1e-6
trajectories = [];Running Alternating Linear Minimization
The method FrankWolfe.alternating_linear_minimization is not a FrankWolfe method itself. It is a wrapper translating a problem over the intersection of multiple sets to a problem over the product space. ALM can be called with any FW method. The default choice though is FrankWolfe.block_coordinate_frank_wolfe as it allows to update the blocks separately. There are three different update orders implemented, FullUpdate, CyclicUpdate and Stochasticupdate. Accordingly both blocks are updated either simulatenously, sequentially or in random order.
for order in [FrankWolfe.FullUpdate(), FrankWolfe.CyclicUpdate(), FrankWolfe.StochasticUpdate()]
_, _, _, _, _, alm_trajectory = FrankWolfe.alternating_linear_minimization(
FrankWolfe.block_coordinate_frank_wolfe,
f,
grad!,
lmos,
x0,
update_order=order,
verbose=true,
trajectory=true,
epsilon=target_tolerance,
)
push!(trajectories, alm_trajectory)
end
Alternating Linear Minimization (ALM).
FW METHOD: block_coordinate_frank_wolfe
MEMORY_MODE: FrankWolfe.InplaceEmphasis() STEPSIZE: FrankWolfe.Adaptive{Float64, Int64} EPSILON: 1.0e-6 MAXITERATION: 10000
TYPE: Float64 GRADIENTTYPE: Float64
LAMBDA: 1.0
[ Info: In memory_mode memory iterates are written back into x0!
----------------------------------------------------------------------------------------------------------------
Type Iteration Primal Dual Dual Gap Time It/sec Dist2
----------------------------------------------------------------------------------------------------------------
I 1 3.778464e+00 -4.022154e+01 4.400000e+01 0.000000e+00 Inf 3.778464e+00
FW 1000 5.000047e-02 4.988536e-02 1.151053e-04 5.469825e-01 1.828212e+03 5.000047e-02
FW 2000 5.000000e-02 4.999558e-02 4.423306e-06 5.612889e-01 3.563227e+03 5.000000e-02
Last 2445 5.000000e-02 4.999898e-02 1.022986e-06 6.342005e-01 3.855248e+03 5.000000e-02
----------------------------------------------------------------------------------------------------------------
Alternating Linear Minimization (ALM).
FW METHOD: block_coordinate_frank_wolfe
MEMORY_MODE: FrankWolfe.InplaceEmphasis() STEPSIZE: FrankWolfe.Adaptive{Float64, Int64} EPSILON: 1.0e-6 MAXITERATION: 10000
TYPE: Float64 GRADIENTTYPE: Float64
LAMBDA: 1.0
[ Info: In memory_mode memory iterates are written back into x0!
----------------------------------------------------------------------------------------------------------------
Type Iteration Primal Dual Dual Gap Time It/sec Dist2
----------------------------------------------------------------------------------------------------------------
I 1 3.778464e+00 -4.022154e+01 4.400000e+01 0.000000e+00 Inf 3.778464e+00
FW 1000 5.000047e-02 4.988833e-02 1.121389e-04 8.180347e-02 1.222442e+04 5.000047e-02
FW 2000 5.000000e-02 4.999569e-02 4.308009e-06 1.063639e-01 1.880337e+04 5.000000e-02
Last 2446 5.000000e-02 4.999898e-02 1.018029e-06 1.178701e-01 2.075166e+04 5.000000e-02
----------------------------------------------------------------------------------------------------------------
Alternating Linear Minimization (ALM).
FW METHOD: block_coordinate_frank_wolfe
MEMORY_MODE: FrankWolfe.InplaceEmphasis() STEPSIZE: FrankWolfe.Adaptive{Float64, Int64} EPSILON: 1.0e-6 MAXITERATION: 10000
TYPE: Float64 GRADIENTTYPE: Float64
LAMBDA: 1.0
[ Info: In memory_mode memory iterates are written back into x0!
----------------------------------------------------------------------------------------------------------------
Type Iteration Primal Dual Dual Gap Time It/sec Dist2
----------------------------------------------------------------------------------------------------------------
I 1 5.732358e-01 -4.342676e+01 4.400000e+01 0.000000e+00 Inf 5.732358e-01
FW 1000 5.000032e-02 4.990096e-02 9.936080e-05 5.246559e-02 1.906011e+04 5.000032e-02
FW 2000 5.000000e-02 4.999595e-02 4.051228e-06 1.263864e-01 1.582449e+04 5.000000e-02
Last 2447 5.000000e-02 4.999903e-02 9.657121e-07 1.356903e-01 1.803371e+04 5.000000e-02
----------------------------------------------------------------------------------------------------------------As an alternative to Block-Coordiante Frank-Wolfe (BCFW), one can also run alternating linear minimization with standard Frank-Wolfe algorithm. These methods perform then the full (simulatenous) update at each iteration. In this example we also use FrankWolfe.away_frank_wolfe.
_, _, _, _, _, afw_trajectory = FrankWolfe.alternating_linear_minimization(
FrankWolfe.away_frank_wolfe,
f,
grad!,
lmos,
x0,
verbose=true,
trajectory=true,
epsilon=target_tolerance,
)
push!(trajectories, afw_trajectory);
Alternating Linear Minimization (ALM).
FW METHOD: away_frank_wolfe
MEMORY_MODE: FrankWolfe.InplaceEmphasis() STEPSIZE: FrankWolfe.Adaptive{Float64, Int64} EPSILON: 1.0e-6 MAXITERATION: 10000
TYPE: Float64 GRADIENTTYPE: Float64
LAMBDA: 1.0
[ Info: In memory_mode memory iterates are written back into x0!
-------------------------------------------------------------------------------------------------------------------------------
Type Iteration Primal Dual Dual Gap Time It/sec Dist2 #ActiveSet
-------------------------------------------------------------------------------------------------------------------------------
I 1 2.300000e+01 -Inf Inf 0.000000e+00 Inf 2.010582e+00 2
Last 158 5.000000e-02 4.999922e-02 7.835690e-07 1.856424e-01 8.510986e+02 5.000000e-02 86
-------------------------------------------------------------------------------------------------------------------------------
PP 158 5.000000e-02 4.999922e-02 7.835690e-07 2.373365e-01 6.657213e+02 5.000000e-02 86
-------------------------------------------------------------------------------------------------------------------------------Running Alternating Projections
Unlike ALM, Alternating Projections (AP) is only suitable for feasibility problems. One omits the objective and gradient as parameters.
_, _, _, _, ap_trajectory = FrankWolfe.alternating_projections(
lmos,
x0,
trajectory=true,
verbose=true,
print_iter=100,
epsilon=target_tolerance,
)
push!(trajectories, ap_trajectory);
Alternating Projections.
MEMORY_MODE: FrankWolfe.InplaceEmphasis() EPSILON: 1.0e-6 MAXITERATION: 10000 TYPE: Float64
GRADIENTTYPE: Vector{Vector{Float64}}
[ Info: In memory_mode memory iterates are written back into x0!
----------------------------------------------------------------------------------
Type Iteration Dual Gap Infeas Time It/sec
----------------------------------------------------------------------------------
I 1 5.092003e-01 2.546002e-01 0.000000e+00 Inf
FW 100 1.045308e-04 5.000040e-02 8.997606e-01 1.111407e+02
FW 200 2.524997e-05 5.000002e-02 2.213856e+00 9.034011e+01
FW 300 1.122811e-05 5.000000e-02 3.667039e+00 8.180988e+01
FW 400 6.334101e-06 5.000000e-02 5.304646e+00 7.540559e+01
FW 500 4.028960e-06 5.000000e-02 7.079036e+00 7.063109e+01
FW 600 2.823896e-06 5.000000e-02 8.974730e+00 6.685438e+01
FW 700 2.088682e-06 5.000000e-02 1.099258e+01 6.367932e+01
FW 800 1.569986e-06 5.000000e-02 1.309581e+01 6.108822e+01
FW 900 1.258972e-06 5.000000e-02 1.530990e+01 5.878551e+01
FW 1000 9.944106e-07 5.000000e-02 1.749776e+01 5.715017e+01
Last 1000 9.944106e-07 5.000000e-02 1.752006e+01 5.707744e+01
----------------------------------------------------------------------------------Plotting the resulting trajectories
labels = ["BCFW - Full", "BCFW - Cyclic", "BCFW - Stochastic", "AFW", "AP"]
plot_trajectories(trajectories, labels, xscalelog=true)This page was generated using Literate.jl.