Merge branch 'release/0.53.0' into main

Author: ljupcovangelski

README.md

@@ -1,7 +1,7 @@
 <p align="center">
   <img src="https://global-uploads.webflow.com/5e9d5014fb5d85233d05fa23/5ea6ab4327484b79bdb4cea4_airy_primary_rgb.svg" alt="Airy-logo" width="240">
   <div align="center">The open source, fully-featured, production ready</div>
-  <div align="center">Conversational Platform</div>
+  <div align="center">Data Platform</div>
 </p>
 
 # Airy Core
@@ -15,10 +15,9 @@
 
 ---
 
-![Airy_Explainer_Highlevel_Readme](https://user-images.githubusercontent.com/124274/113720584-18a8d500-96ef-11eb-97c3-362eebd6253d.jpeg)
+![Airy_Explainer_Highlevel_Readme](https://airy.co/docs/core/img/getting-started/introduction-light.png)
 
-Airy Core is an open source, fully-featured, production ready conversational
-platform. With Airy you can process conversational data from a variety of
+Airy Core is an is an open-source streaming app framework to train ML models and supply them with historical and real-time data. With Airy you can process data from a variety of
 sources:
 
 - **Facebook**
@@ -27,26 +26,25 @@ sources:
 - **SMS**
 - **Website Chat Plugins, like our own open source Live Chat**
 - **Twilio**
-- **Your own conversational channels**
+- **Any source you want with Custom Connectors**
 
 You can then use Airy to:
 
-- **Unify your messaging channels**
-- **Stream your conversational data wherever you want**
-- **Integrate with different NLP frameworks**
-- **Mediate open requests with Agents via our messaging UI**
-- **Analyze your conversations**
+- **Join historical and real-time data in the stream to create smarter ML and AI applications.**
+- **Build real-time data pipelines and make real-time data universally accessible with our open-source streaming app framework.**
+- **Standardize complex data ingestion and consume data directly from Kafka. Stream it directly to standard and customized applications, using pre-built, easily configured connectors.**
+- **Significantly simplify deployment and reduce development times and increase the robustness of your infrastructure and apps.**
 
 Since Airy's infrastructure is built around Apache Kafka, it can process a large
-amount of conversations and messages simultaneously and stream the relevant
-conversational data to wherever you need it.
+amount of events simultaneously and stream the relevant
+real-time and historical data to wherever you need it.
 
 ---
 
 ## About Airy
 
 - **What does Airy do? 🚀**
-  [Learn more on our Website](https://airy.co/developers)
+  [Learn more on our Website](https://airy.co/)
 
 - **I'm new to Airy 😄**
   [Get Started with Airy](https://airy.co/docs/core/)
@@ -69,38 +67,30 @@ conversational data to wherever you need it.
 
 ![Airy_Explainer_Components_Readme (1)](https://user-images.githubusercontent.com/12533283/112460661-6de3fe80-8d5f-11eb-8274-8446fbfcf5c8.png)
 
-Airy Core contains the following components:
+Airy Core comes with all the components you need to stream historical and real-time data.
 
-- 💬 Connectors for all [conversational sources](https://airy.co/docs/core/sources/introduction)
+- 💬 Pre-built and easily configurable [connectors](https://airy.co/docs/core/sources/introduction)
 
-Connect anything from our free open-source [live chat
-plugin](https://airy.co/docs/core/sources/chat-plugin) to Facebook
-Messenger & Google's Business Messages to your Airy Core. This is
-all possible through an ingestion platform that heavily relies on [Apache
-Kafka](https://kafka.apache.org) to process incoming webhook data from different
-sources. We make sense of the data and reshape it into source independent
-contacts, conversations, and messages.
+By ingesting all real-time events and continuously processing, aggregating and joining them in the stream, development time can be significantly reduced. Through integrations with pre-built and easily configured connectors, events are consumed from any source, including business systems such as ERP/CRM, conversational sources, third party APIs. Airy also comes with an SDK to build custom connectors to any source.
 
 - ⚡[APIs](https://airy.co/docs/core/api/introduction) to access your data
 
-An [API](https://airy.co/docs/core/api/introduction) to access conversational
+An [API](https://airy.co/docs/core/api/introduction) to access
 data with blazing fast HTTP endpoints.
 
 - 🔌[WebSockets](https://airy.co/docs/core/api/websocket) to power real-time applications
 
 A [WebSocket server](https://airy.co/docs/core/api/websocket) that allows
 clients to receive near real-time updates about data flowing through the system.
 
-- 🎣[Webhook](https://airy.co/docs/core/api/webhook) to listen to events and participate programmatically in conversations
+- 🎣[Webhook](https://airy.co/docs/core/api/webhook) to listen to events and create actionable workflows
 
-A webhook integration server that allows its users to programmatically
-participate in conversations by sending messages (the webhook integration
+A webhook integration server that allows its users to create actionable workflows (the webhook integration
 exposes events users can "listen" to and react programmatically.)
 
-- 💎[UI: From an inbox to dashboards](https://airy.co/docs/core/apps/ui/introduction)
+- 💎[UI: From a control center to dashboards](https://airy.co/docs/core/apps/ui/introduction)
 
-Not every message can be handled by code, this is why Airy comes with different
-UIs ready for you and your teams to use.
+No-code interfaces to manage and control Airy, your connectors and your streams.
 
 ## How to contribute
 

VERSION

@@ -1 +1 @@
-0.52.0
+0.53.0

backend/components/cognigy/BUILD

@@ -15,6 +15,7 @@ app_deps = [
     "//backend/model/metadata",
     "//:feign",
     "//lib/java/log",
+    "//lib/java/sources-parser",
     "//lib/java/spring/kafka/core:spring-kafka-core",
     "//lib/java/spring/core:spring-core",
     "//lib/java/spring/kafka/streams:spring-kafka-streams",

backend/components/cognigy/src/main/java/co/airy/core/cognigy_connector/MessageHandler.java

@@ -4,11 +4,10 @@
 import co.airy.avro.communication.Message;
 import co.airy.core.cognigy.models.MessageSendResponse;
 import com.fasterxml.jackson.core.JsonProcessingException;
-import com.fasterxml.jackson.databind.JsonNode;
 import com.fasterxml.jackson.databind.ObjectMapper;
-import com.fasterxml.jackson.databind.node.JsonNodeFactory;
-import com.fasterxml.jackson.databind.node.ObjectNode;
+import com.fasterxml.jackson.databind.JsonNode;
 import org.springframework.stereotype.Service;
+import co.airy.sources_parser.SourcesParser;
 
 import java.time.Instant;
 import java.util.Map;
@@ -47,47 +46,10 @@ public String getContent(String source, MessageSendResponse response) throws Jso
         final String text = messageNode.get("text").textValue();
         final JsonNode data = messageNode.findValue("data");
 
-        final ObjectNode node = getNode();
-        switch (source) {
-            case "google": {
-                final ObjectNode representative = getNode();
-                representative.put("representativeType", "BOT");
-                node.set("representative", representative);
-                node.put("text", text);
-                return mapper.writeValueAsString(node);
-            }
-            case "viber": {
-                node.put("text", text);
-                node.put("type", "text");
-                return mapper.writeValueAsString(node);
-            }
-            case "chatplugin":
-            case "instagram":
-            case "facebook": {
-                node.put("text", text);
-                if(!data.isEmpty()){
-                    node.put("message", data);
-                }
-                return mapper.writeValueAsString(node);
-            }
-            case "twilio.sms":
-            case "twilio.whatsapp": {
-                node.put("Body", text);
-                return mapper.writeValueAsString(node);
-            }
-            case "whatsapp": {
-                node.put("Body", text);
-                return mapper.writeValueAsString(node);
-            }
-
-            default: {
-                return null;
-            }
+        if(text == null && data == null){
+            return null;
         }
-    }
 
-    private ObjectNode getNode() {
-        final JsonNodeFactory jsonNodeFactory = JsonNodeFactory.instance;
-        return jsonNodeFactory.objectNode();
+        return SourcesParser.mapContent(source, text, data);
     }
 }

backend/components/ibm-watson-assistant/BUILD

@@ -15,6 +15,7 @@ app_deps = [
     "//backend/model/metadata",
     "//:feign",
     "//lib/java/log",
+    "//lib/java/sources-parser",
     "//lib/java/spring/kafka/core:spring-kafka-core",
     "//lib/java/spring/core:spring-core",
     "//lib/java/spring/kafka/streams:spring-kafka-streams",

backend/components/ibm-watson-assistant/src/main/java/co/airy/core/ibm-watson-assistant-connector/IbmWatsonAssistantClient.java renamed to backend/components/ibm-watson-assistant/src/main/java/co/airy/core/ibm_watson_assistant_connector/IbmWatsonAssistantClient.java

@@ -4,11 +4,10 @@
 import co.airy.avro.communication.Message;
 import co.airy.core.ibm_watson_assistant.models.MessageSendResponse;
 import co.airy.log.AiryLoggerFactory;
+import co.airy.sources_parser.SourcesParser;
 import com.fasterxml.jackson.core.JsonProcessingException;
 import com.fasterxml.jackson.databind.JsonNode;
 import com.fasterxml.jackson.databind.ObjectMapper;
-import com.fasterxml.jackson.databind.node.JsonNodeFactory;
-import com.fasterxml.jackson.databind.node.ObjectNode;
 import org.springframework.stereotype.Service;
 import org.slf4j.Logger;
 
@@ -54,51 +53,17 @@ public String getContent(String source, MessageSendResponse response) throws Jso
                 text = nestedNode.get("text").textValue();
             }
 
-            if (text != "") {
-                final ObjectNode node = getNode();
-                switch (source) {
-                    case "google": {
-                        final ObjectNode representative = getNode();
-                        representative.put("representativeType", "BOT");
-                        node.set("representative", representative);
-                        node.put("text", text);
-                        return mapper.writeValueAsString(node);
-                    }
-                    case "viber": {
-                        node.put("text", text);
-                        node.put("type", "text");
-                        return mapper.writeValueAsString(node);
-                    }
-                    case "chatplugin":
-                    case "instagram":
-                    case "facebook": {
-                        node.put("text", text);
-                        return mapper.writeValueAsString(node);
-                    }
-                    case "twilio.sms":
-                    case "twilio.whatsapp": {
-                        node.put("Body", text);
-                        return mapper.writeValueAsString(node);
-                    }
-                    case "whatsapp": {
-                        node.put("Body", text);
-                        return mapper.writeValueAsString(node);
-                    }
-
-                    default: {
-                        return null;
-                    }
-                }
+            if (text == null) {
+                return null;
             }
+
+            return SourcesParser.mapContent(source, text, null);
+
+            
         } catch (Exception e) {
             log.error(String.format("could not find the text node in the response %s %s", response.toString(), e));
         }
 
         return null;
     }
-
-    private ObjectNode getNode() {
-        final JsonNodeFactory jsonNodeFactory = JsonNodeFactory.instance;
-        return jsonNodeFactory.objectNode();
-    }
 }

backend/components/ibm-watson-assistant/src/main/java/co/airy/core/ibm-watson-assistant-connector/IbmWatsonAssistantClientConfig.java renamed to backend/components/ibm-watson-assistant/src/main/java/co/airy/core/ibm_watson_assistant_connector/IbmWatsonAssistantClientConfig.java

@@ -15,6 +15,7 @@ app_deps = [
     "//backend/model/metadata",
     "//:feign",
     "//lib/java/log",
+    "//lib/java/sources-parser",
     "//lib/java/spring/kafka/core:spring-kafka-core",
     "//lib/java/spring/core:spring-core",
     "//lib/java/spring/kafka/streams:spring-kafka-streams",

backend/components/ibm-watson-assistant/src/main/java/co/airy/core/ibm-watson-assistant-connector/IbmWatsonAssistantConnectorService.java renamed to backend/components/ibm-watson-assistant/src/main/java/co/airy/core/ibm_watson_assistant_connector/IbmWatsonAssistantConnectorService.java

@@ -3,19 +3,20 @@
 import co.airy.avro.communication.DeliveryState;
 import co.airy.avro.communication.Message;
 import co.airy.core.rasa_connector.models.MessageSendResponse;
-import com.fasterxml.jackson.core.JsonProcessingException;
 import com.fasterxml.jackson.databind.ObjectMapper;
-import com.fasterxml.jackson.databind.node.JsonNodeFactory;
 import com.fasterxml.jackson.databind.node.ObjectNode;
+import com.fasterxml.jackson.databind.JsonNode;
+import com.fasterxml.jackson.core.JsonProcessingException;
 import org.springframework.stereotype.Service;
+import co.airy.sources_parser.SourcesParser;
+
 
 import java.time.Instant;
 import java.util.Map;
 import java.util.UUID;
 
 @Service
 public class MessageHandler {
-    private final ObjectMapper mapper = new ObjectMapper();
 
     MessageHandler() {
     }
@@ -40,50 +41,34 @@ public Message getMessage(Message contactMessage, MessageSendResponse response)
                 .build();
     }
 
+   
+
     public String getContent(String source, MessageSendResponse response) throws JsonProcessingException {
         final String text = response.getText();
-        if (text == null) {
+        final String image = response.getImage();
+
+        ObjectMapper mapper = new ObjectMapper();
+        ObjectNode rootNode = mapper.createObjectNode();
+        ObjectNode childNode1 = mapper.createObjectNode();
+        ObjectNode childNode2 = mapper.createObjectNode();
+
+        if (text == null && image == null) {
             return null;
         }
 
-        final ObjectNode node = getNode();
-        switch (source) {
-            case "google": {
-                final ObjectNode representative = getNode();
-                representative.put("representativeType", "BOT");
-                node.set("representative", representative);
-                node.put("text", text);
-                return mapper.writeValueAsString(node);
-            }
-            case "viber": {
-                node.put("text", text);
-                node.put("type", text);
-                return mapper.writeValueAsString(node);
-            }
-            case "chatplugin":
-            case "instagram":
-            case "facebook": {
-                node.put("text", text);
-                return mapper.writeValueAsString(node);
-            }
-            case "twilio.sms":
-            case "twilio.whatsapp": {
-                node.put("Body", text);
-                return mapper.writeValueAsString(node);
-            }
-            case "whatsapp": {
-                node.put("Body", text);
-                return mapper.writeValueAsString(node);
-            }
+        if(image != null){
+            childNode1.put("type", "image");
+            childNode2.put("url", image);
+            childNode1.put("payload", childNode2);
+            rootNode.put("attachment", childNode1);
+            JsonNode imageJsonNode = mapper.convertValue(rootNode, JsonNode.class);
 
-            default: {
-                return null;
-            }
+            return SourcesParser.mapContent(source, text, imageJsonNode);
+        } else {
+            return SourcesParser.mapContent(source, text, null);
         }
-    }
 
-    private ObjectNode getNode() {
-        final JsonNodeFactory jsonNodeFactory = JsonNodeFactory.instance;
-        return jsonNodeFactory.objectNode();
+       
     }
+
 }

backend/components/ibm-watson-assistant/src/main/java/co/airy/core/ibm-watson-assistant-connector/MessageHandler.java renamed to backend/components/ibm-watson-assistant/src/main/java/co/airy/core/ibm_watson_assistant_connector/MessageHandler.java

@@ -13,6 +13,7 @@ app_deps = [
     "//backend/model/metadata",
     "//backend/model/tag",
     "//lib/java/date",
+    "//lib/java/kafka/schema:ops-application-components",
     "//lib/java/spring/auth:spring-auth",
     "//lib/java/spring/web:spring-web",
     "//lib/java/spring/kafka/core:spring-kafka-core",

backend/components/ibm-watson-assistant/src/main/java/co/airy/core/ibm-watson-assistant-connector/Stores.java renamed to backend/components/ibm-watson-assistant/src/main/java/co/airy/core/ibm_watson_assistant_connector/Stores.java

@@ -8,6 +8,7 @@
 import co.airy.kafka.schema.application.ApplicationCommunicationMessages;
 import co.airy.kafka.schema.application.ApplicationCommunicationMetadata;
 import co.airy.kafka.schema.application.ApplicationCommunicationTags;
+import co.airy.kafka.schema.ops.OpsApplicationComponents;
 import co.airy.kafka.streams.KafkaStreamsWrapper;
 import co.airy.model.metadata.dto.MetadataMap;
 import org.apache.kafka.streams.KeyValue;
@@ -53,6 +54,12 @@ public void onApplicationEvent(ApplicationStartedEvent event) {
                 .toStream()
                 .peek((identifier, metadataMap) -> webSocketController.onMetadata(metadataMap));
 
+        builder.<String, Metadata>table(new OpsApplicationComponents().name())
+                .groupBy((metadataId, metadata) -> KeyValue.pair(getSubject(metadata).getIdentifier(), metadata))
+                .aggregate(MetadataMap::new, MetadataMap::adder, MetadataMap::subtractor)
+                .toStream()
+                .peek((identifier, metadataMap) -> webSocketController.onComponentUpdate(metadataMap));
+
         streams.start(builder.build(), appId);
     }