Piecewise baseline predict hazard issue

[griffiri]

hi @CamDavidsonPilon

Noticed the following issue predicting hazard rate, seems unexpected and probably a bug. Repro below

from lifelines import CoxPHFitter
from lifelines.datasets import load_rossi

rossi = load_rossi()
cph = CoxPHFitter(baseline_estimation_method="piecewise", breakpoints=[25])
cph = cph.fit(rossi[["week", "arrest"]], "week", "arrest")

covariates = rossi[["week", "arrest"]].head()


#This works
z1 = cph.predict_cumulative_hazard(covariates)
print(z1.shape)
#This doesn't
z2 = cph.predict_hazard(covariates)

Had a look at fixing, but i seemed to break calibration in the process. Happy to help fix it but may need some guidance.

Full stack trace for the error below

Traceback (most recent call last):
  File "/Users/richardgriffiths/anaconda3/envs/p2pml39/lib/python3.9/site-packages/IPython/core/interactiveshell.py", line 3437, in run_code
    exec(code_obj, self.user_global_ns, self.user_ns)
  File "<ipython-input-2-52ce976ffed3>", line 1, in <module>
    runfile('/Users/richardgriffiths/Library/Application Support/JetBrains/PyCharm2021.1/scratches/scratch_61.py', wdir='/Users/richardgriffiths/Library/Application Support/JetBrains/PyCharm2021.1/scratches')
  File "/Applications/PyCharm.app/Contents/plugins/python/helpers/pydev/_pydev_bundle/pydev_umd.py", line 197, in runfile
    pydev_imports.execfile(filename, global_vars, local_vars)  # execute the script
  File "/Applications/PyCharm.app/Contents/plugins/python/helpers/pydev/_pydev_imps/_pydev_execfile.py", line 18, in execfile
    exec(compile(contents+"\n", file, 'exec'), glob, loc)
  File "/Users/richardgriffiths/Library/Application Support/JetBrains/PyCharm2021.1/scratches/scratch_61.py", line 15, in <module>
    z2 = cph.predict_hazard(covariates)
  File "/Users/richardgriffiths/projects/lifelines/lifelines/lifelines/fitters/coxph_fitter.py", line 2917, in predict_hazard
    return super(ParametricCoxModelFitter, self).predict_hazard(df, times=times, conditional_after=conditional_after)
  File "/Users/richardgriffiths/projects/lifelines/lifelines/lifelines/fitters/__init__.py", line 2421, in predict_hazard
    return pd.DataFrame(self._hazard(params_dict, np.tile(times, (n, 1)).T, Xs), index=times, columns=df.index)
  File "/Users/richardgriffiths/projects/lifelines/lifelines/lifelines/fitters/__init__.py", line 1369, in _hazard
    return egrad(self._cumulative_hazard, argnum=1)(params, T, Xs)  # pylint: disable=unexpected-keyword-arg
  File "/Users/richardgriffiths/anaconda3/envs/p2pml39/lib/python3.9/site-packages/autograd/wrap_util.py", line 20, in nary_f
    return unary_operator(unary_f, x, *nary_op_args, **nary_op_kwargs)
  File "/Users/richardgriffiths/anaconda3/envs/p2pml39/lib/python3.9/site-packages/autograd/differential_operators.py", line 38, in elementwise_grad
    vjp, ans = _make_vjp(fun, x)
  File "/Users/richardgriffiths/anaconda3/envs/p2pml39/lib/python3.9/site-packages/autograd/core.py", line 10, in make_vjp
    end_value, end_node =  trace(start_node, fun, x)
  File "/Users/richardgriffiths/anaconda3/envs/p2pml39/lib/python3.9/site-packages/autograd/tracer.py", line 10, in trace
    end_box = fun(start_box)
  File "/Users/richardgriffiths/anaconda3/envs/p2pml39/lib/python3.9/site-packages/autograd/wrap_util.py", line 15, in unary_f
    return fun(*subargs, **kwargs)
  File "/Users/richardgriffiths/projects/lifelines/lifelines/lifelines/fitters/coxph_fitter.py", line 2753, in _cumulative_hazard
    return self._cumulative_hazard_sans_strata(params, T, Xs)
  File "/Users/richardgriffiths/projects/lifelines/lifelines/lifelines/fitters/coxph_fitter.py", line 3146, in _cumulative_hazard_sans_strata
    T = T.reshape((n, 1))
  File "/Users/richardgriffiths/anaconda3/envs/p2pml39/lib/python3.9/site-packages/autograd/numpy/numpy_vjps.py", line 552, in wrapped_reshape
    return anp.reshape(x, *args, **kwargs)
  File "/Users/richardgriffiths/anaconda3/envs/p2pml39/lib/python3.9/site-packages/autograd/tracer.py", line 44, in f_wrapped
    ans = f_wrapped(*argvals, **kwargs)
  File "/Users/richardgriffiths/anaconda3/envs/p2pml39/lib/python3.9/site-packages/autograd/tracer.py", line 48, in f_wrapped
    return f_raw(*args, **kwargs)
  File "<__array_function__ internals>", line 5, in reshape
  File "/Users/richardgriffiths/anaconda3/envs/p2pml39/lib/python3.9/site-packages/numpy/core/fromnumeric.py", line 299, in reshape
    return _wrapfunc(a, 'reshape', newshape, order=order)
  File "/Users/richardgriffiths/anaconda3/envs/p2pml39/lib/python3.9/site-packages/numpy/core/fromnumeric.py", line 58, in _wrapfunc
    return bound(*args, **kwds)
ValueError: cannot reshape array of size 245 into shape (49,1)

thanks
Richard

[CamDavidsonPilon]

Hi @griffiri - yup looks like a bug. I've created an issue for this. I wonder if PiecewiseExponentialRegressionFitter has this problem, too.

[griffiri]

Yes @CamDavidsonPilon it does