Feature request/discussion

[davidbp]

Hi,

I am working on a somewhat related topic which would intersect with this package and maybe association rules.

problem 1

I have a list of search terms and I would like to understand "how associated" is a consequent to an antecedent.
For example, if I write "halloween" what are "the most likely" ngrams , "the most uniquely related to" n-grams, etc...

I have data of the form: text, counts, categorical variable (for example COSTUME_OUTFIT if the text is mostly related to a costume) and I am building a table similar to this (here the antecedent is "halloween")

| # | query                              | consequent                | category              | support        | confidence | lift         | conviction |
|---|------------------------------------|---------------------------|-----------------------|----------------|-------------|--------------|-------------|
| 0 | halloween decorations              | decorations               | LIGHTED_OUTDOOR_DISPLAY | 6.122921e-06   | 0.073535    | 24.696175    | 1.076157    |
| 1 | halloween costumes for women adult | costumes for women adult  | COSTUME_OUTFIT        | 6.576034e-07   | 0.007898    | 11147.867556 | 1.007960    |

Do you think something similar could be done with this package?

Here it is relevant to note that I have (text, counts) pairs, not just an streams of text, the text is very short (search queries).

problem 2

I have and extended table with date (text, counts, categorical_var, date) and I would like to find out which associations mostly occur on different dates (holidays, seasons) etc...

[atantos]

👋 Thanks for opening this discussion!

This is an interesting use case! And it indeed intersects with what TextAssociations.jl does conceptually, though the underlying logic differs from association-rule mining.

The package doesn’t operate directly on (text, count) pairs or on categorical tables.
It expects a reference corpus (either as a collection of documents or as a single long text), from which it automatically constructs contingency tables for co-occurrence within a context window (e.g. ±5 words).

From these contingency tables it then computes over 50 association metrics by now. These metrics quantify how strongly two words co-occur relative to chance, based on the corpus’s overall frequency distribution.

In other words, before an association can be computed, the package needs a distributional baseline. It can’t estimate association strength from (term, count) pairs alone.

I have a list of short search queries and would like to know “how associated” a consequent is to an antecedent (e.g. halloween → decorations, costumes …).

If your data are short independent queries, each line can be treated as a "mini-document".
Then you can build a corpus directly from them and compute associations for a given node word. This will produce collocates ranked by association strength, somewhat analogous to "lift" or "confidence" in association-rule mining, but grounded in corpus probabilities rather than transaction counts.

If you only have aggregated (text, count) data (without token-level text), the package currently cannot infer co-occurrence structure directly. In that case we’d need to know how those counts were computed (per term? per pair?) to see if a small conversion function could reconstruct the 2×2 counts (a,b,c,d) expected by the API.

I have (text, count, category, date) and want to find which associations occur on different dates or seasons.

That maps nicely to the built-in temporal analysis features (is almost done but a couple of things need to be improved/fixed):

ta = analyze_temporal(corpus, ["halloween"], :date, PMI; time_bins=12)

This would output something like the following:

Period	Node	Collocate	PMI
2023Q4	halloween	decorations	2.0
2024Q1	halloween	decorations	3.5
2024Q2	halloween	decorations	4.2
2023Q4	halloween	pumpkins	0.8
2024Q1	halloween	pumpkins	0.7
2024Q2	halloween	pumpkins	0.6
2023Q4	halloween	spooky sale	6.0

and then with

trend_df = compute_association_trends(ta.results_by_period, ["halloween"], PMI)

you would get the following output (ignore the values..the values below are not calculated values based on the previous table):

Node	Collocate	Correlation	Slope	MeanScore	StdScore	NumPeriods
halloween	decorations	0.98	1.10	3.23	1.12	3
halloween	pumpkins	−1.00	−0.10	0.70	0.10	3

This will slice the corpus into time periods (bins) and compute associations per period, letting you track which collocates peak around holidays or other events.

A crucial point: association metrics are unreliable at very low counts. For instance, a single co-occurrence can produce an extreme PMI simply because the expected value is near zero.