feat: `ibis.count('*')`

[MarcoGorelli]

Is there any way to do a count('*') expression in Ibis?

Looks like there's no ibis.count

I'm looking at doing something like

ibis.count('*').over(ibis.window(group_by=group))

[NickCrews]

You are looking for Table.count() :). This also works with Ibis.Deferred. Eg

t.group_by("x").agg(x=t.count())
t.mutate(t.count().over(ibis.window(group_by=group))
# ps, you don't actually need the ibis.window call, you can just pass directly to .over():
t.mutate(t.count().over(group_by=group)
# or use the Deferred:
t.mutate(ibis._.count().over(group_by=group)

[NickCrews]

@cpcloud as a feature request, what if we exposed ibis._.count() as ibis.count()?

[MarcoGorelli]

thanks, table.count() does work

it's much more ergonomic to be able to create an expression that's not tied to a particular table though, so ibis._.count() would be helpful 🙏

[NickCrews]

To be clear, ibis._.count() should currently work. Just wanted to clarify because you said "WOULD be helpful", which made it sound like you didn't think it currently works. If it's not working for you, please chime back in here :)

[MarcoGorelli]

thanks - i haven't yet narrowed this down to a small reproducer, but i am getting

E       ibis.common.exceptions.InputTypeError: Unable to infer datatype of value ((_.count().over(LegacyWindowBuilder(how='rows', start=None, end=None, groupings=('b', <ibis.expr.operations.relations.Field object at 0x7dd1c595cb40>), orderings=())) - <scalar[int8]>) / <scalar[float64]>) with type <class 'ibis.common.deferred.Deferred'>

with that, will try to narrow it down

[NickCrews]

Ah, yes, I see. That is because most expressions in ibis know their datatype, eg float64. But deferred's don't by definition, because it is unclear what concrete value you are going to bind it to, and that will depend on the context. eg ibis._ + 2 would be an int64 if bound to an int64, but a float64 if bound to a float64. EDIT: this example is more clear with ibis._.my_column + 2.

So then some APIs in ibis require knowing the datatype of their inputs. You seem to be unlucky and using one of these. In general, it would be great if ibis reduced the number of these APIs whenever possible, because this error sucks. If you do figure out which API that is maybe I could get around to fixing it.

The more short term solution for this particular use case would be if we did make some specialized API for this. Or perhaps we made it so that we could optionally attach datatypes to Deferreds, and then these downstream datatype inferrences might work.

[MarcoGorelli]

Not sure I follow, sorry - isn't the output type of ibis._.count() always known upfront? So the type of ibis._.count() + 2 is also always known

[NickCrews]

Yes, for that special case (and some others I'm sure) of Deferreds we would know the d-type. But in general for Deferreds, eg ibis._.my_col, we don't. So it would be theoretically possible for us to track an optional d-type in each deferred object, but we currently don't do that.

[hottwaj]

Hi @MarcoGorelli could you provide an example that fails? e.g. I find that this works, so presumably your expression was different/more complicated

ibis._.count().over(ibis.window(group_by='foo'))

And this seems to work too:

t = ibis.table(name="t", schema={f"a": "int"})

(ibis._.count().over(ibis.window(group_by=('foo', t['a']))) - 10) / 100.0

Thanks!