############################################################### A/B testing procedure for Growth Service campaigns, using Pig.# ###############################################################
--BACKGROUND
The Growth Service is an event-triggered real-time user engagement framework that delivers messages, notifications and/or emails to a given user. The criteria for notifying a user, given some event, is flexible and easily modified.
Each GS message header comes with a system_generated tag that allows us to trap for the stimulated population.
--ANALYIS AND PROCEDURE
We wish to perform an A/B test, or in other words, a controlled experiment. Given that Voxer is a messaging app, the most important quantity to measure for changs is messages_sent. Note however, that we can measure other quantities e.g., profile updates, etc.,
We work under the hypothesis that messaging a user will increase the messaging generated by that very user, at some point after the GS notification is sent. Therefore, we measure the ratio of messaging of the population who received a GS message (A), divided by the population who did not recieve a GS message.
In order to maintain control of the experiment, we select the population (B) to resemble (A) as much as possible.
--LOGIC
((I)) -- Population A: the stimulated
(1) LOAD messages from time period of stimulation
(2) trap for relevant system_generated tag via FILTER
(3) project out to_id and then COUNT number of distinct users who received given stimulation
(4) JOIN to_id of stimulator recipients with user_id of messages
(5) FILTER out voxerbot, peoplematcher and system_generated messages from messages from population A in order to isolate user-genererated messages
(6) center time co-ordinate of individual user's message sequence about time_sent of stimulator message -- this is the MESSAGE_CURVE
((II)) -- Population B: the control sample
(1) LOAD inactive user list
(2) project out user_id and last_sent (in days)
(3) center time co-ordinate of last_sent about current day
(4) FILTER inactive list by last_sent = selection criteria of GS -- now we have the control sample (B)
(5) COUNT all users in control sample
(6) form union of A_count and B_count via CROSS
(7) compute ratio of A_count/B_count
(8) randomly down-SAMPLE control population to equal size of stimulated population
(9) JOIN control sample with all user-generated messages -- this is the un-normalized MESSAGE_CURVE the control sample
((III)) -- A/B daily reporting
(1) COUNT all messages from A
(2) COUNT all messages from B
(3) form union of A_messages_count and B_messages_count == A_B via CROSS
(4) STORE/DUMP A_B
(5) COUNT all senders from A
(6) COUNT all senders from B
(7) form union of A_senders_count and B_senders_count == alpha_beta via CROSS
(8) STORE/DUMP alpha_beta