-
Notifications
You must be signed in to change notification settings - Fork 4
meta.shadowTesting
Shadow testing is the practice of copying real traffic from a deployed instance to a testing instance. This second instance can then be tested with real user traffic in real conditions, this instance can only read the traffic, updates and modifications are not applied to production data. The actual requests are still answered by the production system, the testing system just runs in parallel.
Shadow testing is natively implemented in the Talium Bot, in terms of a Twitch Bot this looks like reading events form one chat, and sending all responses (and modifying api requests) to a second channel that is used for testing. The test result can then be observed in the difference in the output of the production and testing bot. While testing is occurring, chat users will not notice any disturbance or repeated responses.
Pure data sources (READ), or data sinks (WRITE) don't really need to add anything. They just need to connect to the right account. This can be done by using the information of another service, like using the READ name of the Twitch Input as the name of the account to use on other Services. Or the Input can just have a option to specify the credentials which are to be used. These can then be manually changed when shadow testing is done.
Inputs (or Services) that read and write data to external services need to split this functionality into different accounts, if the changes could technically be made on a different account. In normal operation these two read and write accounts would be the same, but could be separate if needed for testing. The mode of operation should be switched via a config parameter, and is not required to be able to be done at runtime.
Currently, it is not specified how to fail if a WRITE action can impossibly be made to work on a different account (without significantly changing the behavior of the program). (Changing an access token in a request would be fine, creating a dummy object just to be able to modify it would generally be unwanted) One solution would be to fail silently to the caller (with logging), and just return dummy information.
How this behavior is exposed to other parts of the application is not specified and is up to the implementer. One approach would be to add an transparent api that hides that the READ and WRITE actions happen on different accounts.