Mocking test data with functor

[tkawachi]

I tried to do a test data mock by reading https://opensource.googleblog.com/2021/04/logica-organizing-your-data-queries.html. The following code is a sample that is supposed to query the Google Analytics session table

mytable.l

GASession(..r) :- `ga_sessions_*`(..r);

# extract events from GASession
GAEvent(hit_type:, event_category:, event_action:, event_label:, screen_name:, operating_system:, user_id:, _table_suffix:) :-
  GASession(_table_suffix:, device:, hits:, customdimensions:),
  hit in hits,
  hit_type == hit.type,
  event_category == hit.eventinfo.eventcategory,
  event_action == hit.eventinfo.eventaction,
  event_label == hit.eventinfo.eventlabel,
  screen_name == hit.appinfo.screenname,
  operating_system == device.operatingsystem,
  user_id Max= (uid :- uid == If(cd.index == 1, cd.value, null), cd in customdimensions);

# predicate to test
UidCount(cnt? Count= user_id) distinct :- GAEvent(user_id:);

test.l

import mytable.UidCount;

MockEvent(user_id: 1);
MockEvent(user_id: 2);

# I want to mock GAEvent with MockEvent.
TestUidCount := UidCount(GAEvent: MockEvent);

However, when I tried to generate the SQL, I got the following error.

$ logica test.l print TestUidCount
Making:
TestUidCount
Traceback (most recent call last):
...
  File "/home/kawachi/ghq/github.com/EvgSkv/logica/compiler/functors.py", line 228, in CallFunctor
    raise FunctorError(
compiler.functors.FunctorError: Functor mytable_UidCount is applied to arguments GAEvent, which it does not have.
...

I can mock a reference to a BigQuery table, but can't I mock an arbitrary predicate?

[EvgSkv]

You can in fact mock an arbitrary predicate!

To mock a predicate that is defined in the program we need to import it in the test file. It doesn't need to be done for BQ tables because they are global.

Thus to make your test work we need to add import mytable.GAEvent; at the top.
So the test.l would look like:

import mytable.UidCount;
import mytable.GAEvent;  # <= New line.

MockEvent(user_id: 1);
MockEvent(user_id: 2);

# I want to mock GAEvent with MockEvent.
TestUidCount := UidCount(GAEvent: MockEvent);

I've tested it on my machine and it appears to work as intended.

Please let me know if this solves your problem and if you have further questions.

[tkawachi]

Thanks!

[tkawachi]

Let me ask one more question.

In the above example, if UidCount is defined as

UidCount(cnt? Count= user_id) distinct :- GAEvent(user_id:, _table_suffix: "20210401");

GAEvent needs _table_suffix, so I defined it as follows.

MockGAEvent(user_id: 1, _table_suffix: "20200401");
MockGAEvent(user_id: 2, _table_suffix: "20200401");

When I did a print AllCountTest, the SQL was generated, but the following runtime error was occured.

Invalid field name "_table_suffix". Field names are not allowed to start with the (case-insensitive) prefixes _PARTITION, TABLE, FILE, _ROW_TIMESTAMP, ROOT and _COLIDENTIFIER

Is it possible to mock a predicate that contains _table_suffix?

[EvgSkv]

Oh, I didn't know BQ disallows such column names. It adds friction and tricks are unfortunately needed.

One way is to make the mock predicate injectible, so there is never an explicit column of such name. Injectible means defined with a single non-aggregating conjunctive rule. You can do it as follows:

import mytable.UidCount;                                                                                                                             
import mytable.GAEvent;                                                                                                                              
                                                                                                                                                     
MockEventRow(user_id: 1);                                                                                                                            
MockEventRow(user_id: 2);                                                                                                                            
                                                                                                                                                     
MockEvent(_table_suffix: "20210404", ..r) :- MockEventRow(..r);                                                                                      
                                                                                                                                                     
# I want to mock GAEvent with MockEvent.                                                                                                             
TestUidCount := UidCount(GAEvent: MockEvent);    

I checked and as expected Logica handles the _table_suffix argument with substitutions, so the resulting SQL never addresses such column.

If you need rows with different table suffix then you would need to implement it as a function of some information in the row.
For example like this:

import mytable.UidCount;                                                                                                                             
import mytable.GAEvent;                                                                                                                              
                                                                                                                                                     
MockEventRow(user_id: 1);                                                                                                                            
MockEventRow(user_id: 2);                                                                                                                            
                                                                                                                                                     
MockEvent(_table_suffix:, ..r) :-
  MockEventRow(..r), 
  _table_suffix == (
    if r.user_id == 1 then
      "20210404"
    else
      "20210405"
    )
  );                                                                 
                                                                                                                                                     
# I want to mock GAEvent with MockEvent.                                                                                                             
TestUidCount := UidCount(GAEvent: MockEvent);    

Let me know if this helps.

[tkawachi]

I have confirmed that the generated SQL can be executed. I am not sure if I understand it correctly, but I will play with it more. Thank you.