disk.frame is AMAZING! + quick Q ref dfglm()

[sfd99]

Thanks for disk.frame.
EXCELLENT!.

Q:
In pg:
https://diskframe.com/articles/07-glm.html
the GLM iris example for Logistic regression
works great!.

But...
when I tried the # commented-out line
(shown in that same iris example),
I get an error (syntax?):

iris_model = dfglm(Species == "setosa" ~ , data = iris.df, family=binomial())
Error: unexpected ',' in " iris_model = dfglm(Species == "setosa" ~ ,"

Help!
What's missing in that example syntax?.

SFd99
San Francisco
R 4.1.1 + latest Rstudio
Ubuntu LINUX 20.04
disk.frame V 0.50
===================

[xiaodaigh]

dfglm uses {biglm} as backend and biglm doesn't have flexible syntax like base's glm. So you must list the target and feature manually like

target~f1+f2

or generate them programmatically.

library(biglm)

iris2 = iris

iris2$target = iris2$Species == "setosa"

iris_model = bigglm(target ~ Sepal.Length, data = iris2, family=binomial())

[sfd99]

Hi,

just a follow-up, (if I may) in:
https://diskframe.com/articles/07-glm.html
GLM iris example.

Your Logistic Regression Iris example
works great with Species = "versicolor",
but if I change it to "setosa"
we get a Warning:

# fit a logistic regression model to predict Species == "setosa" using all variables

iris.df = as.disk.frame(iris)

 all_terms_except_species = setdiff(names(iris.df), "Species") 
 formula_rhs = paste0(all_terms_except_species, collapse = "+")
 formula = as.formula(paste("Species == 'setosa' ~ ", formula_rhs))

 iris_model = dfglm(formula , data = iris.df, family=binomial())
Warning message:
In bigglm.function(formula, data = streaming_fn, ...) :
  ran out of iterations and failed to converge

summary(iris_model)
Large data regression model: biglm::bigglm(formula, data = streaming_fn, ...)
Sample size =  150 
   failed to converge after 8  iterations
                Coef     (95%     CI)      SE      p
(Intercept)  -5.8900 -96.8243 85.0443 45.4671 0.8969
Sepal.Length  1.9019 -25.7887 29.5925 13.8453 0.8907
Sepal.Width   4.2452 -17.3080 25.7983 10.7766 0.6936
Petal.Length -5.0289 -31.0895 21.0316 13.0303 0.6995
Petal.Width  -3.7625 -53.9215 46.3966 25.0795 0.8807

Your example code works great
for Species = "versicolor"...
Why the Warning for "setosa"?.
Are the output model coeffs values reliable
after we get the above Warning ?.

Help!...
SFd99
San Francisco

[xiaodaigh]

hmm interesting. wonder what the result is if you just try glm. The p-value certainly suggest it's not significant. you might need to do some feature engineering cos a direct fit may not be well fitting.

[sfd99]

ok!

• I tried a completely different approach.
  using a target TRUE/FALSE column for "setosa".
  pls, do tell me if the results make any sense.

CLASSIFICATION of <> Iris Species w/ disk.frame PKG:

 iris2 <- iris
 iris2$target = iris2$Species == "setosa" # add a TF col f/Logistic Reg.!
 iris2.df = as.disk.frame(iris2)
 head(iris2.df)
   Sepal.Length Sepal.Width Petal.Length Petal.Width Species target
1:          5.1         3.5          1.4         0.2  setosa   TRUE
2:          4.9         3.0          1.4         0.2  setosa   TRUE
3:          4.7         3.2          1.3         0.2  setosa   TRUE
4:          4.6         3.1          1.5         0.2  setosa   TRUE
5:          5.0         3.6          1.4         0.2  setosa   TRUE
6:          5.4         3.9          1.7         0.4  setosa   TRUE

# where col:  target is TRUE f/rows with Species="setosa", 
# F otherwise... T/F or 0/1 REQUIRED f/LOGISTIC Regression target var!

# using: maxit=20 parameter against Warning: 
# "...ran out of iterations and failed to converge..."!

iris_model = dfglm(target ~ Sepal.Length , data = iris2.df, family=binomial(), maxit=20)

summary(iris_model)
Large data regression model: biglm::bigglm(formula, data = streaming_fn, ...)
Sample size =  150 
                Coef    (95%     CI)     SE p
(Intercept)  27.8285 18.1734 37.4837 4.8276 0
Sepal.Length -5.1757 -6.9625 -3.3889 0.8934 0

Does this approach make any sense?...
Are the results valid?.

SFd99
San Francisco.

[xiaodaigh]

i reckon u will get more luck if u ask in a modelling forum.

[sfd99]

Ok - will do, thanks
for your help.

by the way:
I confirmed the above dfglm() result
with a regular glm() result,
(as you had suggested).

Great.
Both model results
match perfectly!.

And no warnings or errors
with this approach...

So,
the above dfglm() approach
seems to work 100%!.
(just wanted to let you know...).

Thanks for the superb job
you have done with disk.frame.
Really appreciated.

Best,
SFd99
San Francisco.