Example from JOSS paper lags forever with reasonable dimensions

[gdalle]

The following example from the paper runs in very small dimension, but even for moderately-sized settings it lags forever (> 1 min). I tested it on a very bulky Macbook Pro so I really don't think the problem comes from the hardware. Do you have any clue why it is so slow?

using ExpFamilyPCA, Distributions

indim, outdim, n1 = 10, 5, 100

sample_from_gamma(n, m) = rand(Gamma(), n, m)

G(θ) = -log(-θ)
g(θ) = -1 / θ
gamma_epca = EPCA(indim, outdim, G, g, Val((:G, :g)); options = NegativeDomain())

X = sample_from_gamma(n1, indim)  # matrix of gamma-distributed data

X_compressed = fit!(gamma_epca, X)

[FlyingWorkshop]

Let me see, I'll do some testing now.

[FlyingWorkshop]

@gdalle, it appears to be a performance issue caused by a difficult loss landscape. The initial loss falls quickly (as expected), but slows down both in iteration and loss improvement. A solution could be adding early stopping, patience, or trying to get the package to better harness hardware.

using ExpFamilyPCA, Distributions

indim, outdim, n1 = 10, 5, 100

sample_from_gamma(n, m) = rand(Gamma(), n, m)

G(θ) = -log(-θ)
g(θ) = -1 / θ
gamma_epca = EPCA(indim, outdim, G, g, Val((:G, :g)); options = NegativeDomain())
X_compressed = fit!(gamma_epca1, X; verbose=true, steps_per_print=1)

julia> include("arena/epca/gamma_joss.jl")
Iteration: 1/100 | Loss: 96.55726454550793
Iteration: 2/100 | Loss: 96.5354008090762
Iteration: 3/100 | Loss: 96.5351939262657
Iteration: 4/100 | Loss: 96.53519151957907
Iteration: 5/100 | Loss: 96.5351914867849
Iteration: 6/100 | Loss: 93.6300590564199
Iteration: 7/100 | Loss: 81.77055667230692
Iteration: 8/100 | Loss: 76.85946370416866
Iteration: 9/100 | Loss: 74.92081719717324
Iteration: 10/100 | Loss: 74.05334898285854
Iteration: 11/100 | Loss: 73.70765717101862
Iteration: 12/100 | Loss: 73.51276079911916
Iteration: 13/100 | Loss: 73.29055526155825
Iteration: 14/100 | Loss: 73.1023643277939
Iteration: 15/100 | Loss: 72.97495140644317
Iteration: 16/100 | Loss: 72.89544144073695
Iteration: 17/100 | Loss: 72.76572894587677
Iteration: 18/100 | Loss: 72.66082215233749

[gdalle]

My bad, I should have switched on verbose mode. I forgot that we were solving a possibly challenging optimization problem. Perhaps a warning would be appropriate, telling users that the default fit! may be long and that they can play with verbosity or iteration number?

[FlyingWorkshop]

No worries, very valid concern. I can add a disclaimer to the documentation now.

[gdalle]

Also, if you add a yield() at the end of every iteration (or every couple of iterations), users will be able to interrupt optimization without spamming ctrl+c and eventually crashing their REPL

[gdalle]

Try it, it's life-changing

[FlyingWorkshop]

Just added a disclaimer :)

[FlyingWorkshop]

Will look into yield now too

[FlyingWorkshop]

I added the yields. I'm not familiar w/ this mechanic of Julia, so I'm not sure if this accomplishes the desired result, but it does seem to let me ctrl+c in the middle of fit! w/o crashing the repo, which is the desired behavior.

[FlyingWorkshop]

What do you think @gdalle?

[gdalle]

Looks good! Let's close this one