Skip to content

Home

Jump to bottom
Anton Pryamostanov edited this page Feb 14, 2020 · 50 revisions

Infinite Technology ∞ Pigeon 🕊

Pigeons have been (and sometimes still are) serving in Post for delivering Mail and other paper-based messages (such as text, drawings, maps) - thus the project name: Pigeon.

Contents

Table of contents generated with markdown-toc

Purpose

Pigeon is an end-user server application (HTTP Message Broker) designed for distribution of text messages in HTTP format.

In short

Pigeon allows to:

  • Accept a text message in a free format (it can be anything - XML, JSON, TLV, Base64, etc) from external source using "enqueue" REST API or direct DB insert by external app
  • Convert it into one or more HTTP messages with a specified body/query string parameters using one of the Plugins (written in Groovy Script)
    • HTTP message attributes such as Headers, Body, URL are controlled by the Plugin and there are no constraints on the produced output (as long as it within allowed HTTP format boundaries)
  • Send the resulting HTTP messages ansynchronously to one or more recipients (URLs) using a variety of HTTP connection and authentication mechanisms (such as AWS v4 signature)
  • If needed retry sending the message several times

Features

Extensibility

Pigeon supports programmability extension in a form of user-defined Plugins written using Groovy script.

Please refer to Plugins page for details on Pigeon Plugins.

Bulk Upload

Pigeon supports bulk upload (enqueue) of messages for sending, bypassing the enqueue REST API.

This is achieved via direct insert into Pigeon DB - messages table.

Duplicates prevention

Pigeon prevents same message to be sent more than 1 time (to the same URL) - even if it is enqueued multiple times by mistake from external system/user.

Performance

Pigeon can be explicitly configured to use a pre-defined number of sender threads - for optimizing the performance and preventing overloading the external recipient server in case of large number of messages - in case of Bulk Upload.

The messages are delivered in parallel to each other, but not exceeding a defined number at a time.

Scalability

The above configuration can be scaled up by increasing the number of sender threads - to address increased traffic over the time.

Furthermore - additional sending URLs can be added without affecting or interfering with delivery of messages to existing URLs.

Pigeon is fully multi-threaded application with a granularity of one thread oriented to take care of one URL at a time.

Retries

If the external server (URL) is not responding - Pigeon will retry sending message several times over a period of time.

Monitoring REST API

Pigeon provides a complete picture of its data via Spring Data Rest HATEOAS repositories and custom endpoints.

Such data as detailed HTTP message interchange logs is available in a readable form (using YAML format) including HTTP Headers, URL and Body.

Self-test diagnostics

Pigeon provides extensible capabilities for self-test and diagnostics using SelfTest plugin - allow to easily check the health of the application.

Closed loop copies

Pigeon can easily be configured for sending a copy of outgoing HTTP message to itself - preserving all the message attributes.

This is useful for monitoring and troubleshooting purposes, when external server is unavailable or there is uncertainty in the validity of outgoing message.

In this scenario Pigeon acts as a Mock server for itself.

Security

Currently Pigeon supports the following mechanisms for HTTP authentication:

  • Basic authentication
  • AWS v4 signing

Additionally Pigeon easily supports HTTP as well as unsecure (self-signed certificates) HTTPS connections for test environments - something which normally causes a headache in Java applications.

DBMS-agnostic

Pigeon uses Spring Data repositories (Hibernate JPA) and allows externalized configuration and change of JDBC driver by placing the needed JAR in the lib directory.

Ultimate logging

Pigeon uses Bobbin Slf4j Logger with a pre-defined well-structured configuretion.

It provides unprecedented level of insight and ease of analyzing the log data written in a logically grouped files.

Furthermore, Pigeon enables usage of BlackBox in the user-defined Pigeon Plugins - with help of that you literally do not have to write any debug code at all anymore!

Installation

Download jar

  1. Open latest version of pigeon-app on Bintray:

Download

  1. On the same page scroll down to the Downloads section
  2. Click on pigeon-app-<version>.jar to start the download
  3. Rename the file to pigeon.jar

Service installation

❗ This step is optional and is recommended for Production and unattended test environments.
You can directly run Pigeon using Command-line.

Windows

Install Pigeon as Windows service following instructions (using instructions: https://dzone.com/articles/spring-boot-as-a-windows-service-in-5-minutes)

Sample pigeon.xml

❗ Recommended Java Heap size parameters: at least -Xms1000m -Xmx1500m

<?xml version="1.0" encoding="UTF-8"?>
<service>
    <id>Pigeon</id>
    <name>Pigeon</name>
    <description>Pigeon Windows Service</description>
    <executable>"C:\Program Files\Java\jdk1.8.0_181\bin\java"</executable>
    <arguments>-Xms1000m -Xmx1500m -cp "pigeon.jar" -Dloader.path=lib org.springframework.boot.loader.PropertiesLauncher</arguments>	
    <logmode>rotate</logmode>
</service>

Other operating systems

Follow operating system documentation to install Pigeon jar as a service.

❗ It is needed to set loader.path to lib. Refer to Windows example. Recommended Java Heap size parameters: at least -Xms1000m -Xmx1500m

Database connector

❇ Optional Step (recommended)

By default Pigeon comes with H2 in-memory database.
While suitable for trying and certain testing cases, it is highly recommended (with H2 DB data is lost when Pigeon is restarted) to use stand-alone database.
Pigeon is using JPA which allows using a variety of JDBC connectors.

  1. Download JAR JDBC connector supporting your DBMS (e.g. mysql-connector-java-8.0.13.jar)

  2. Deploy it into same {Pigeon Home}/lib directory (create it first).

Note: refer to documentation of your DBMS/JDBC driver for more details.

Configuration

application.properties

❇ Optional Step (recommended)

❗ Refer to Spring and Spring Boot documentation for details on supported configuration in application.properties.

In the Pigeon working directory, create file application.properties.

Define the below mandatory properties (refer to sample application.properties file):

Pigeon configuration paths

pigeonConfFile=./conf/Pigeon.json
pigeonOutPluginsDir=./conf/plugins/output/
pigeonInputPluginsRestDir=./conf/plugins/input/rest/
pigeonInputPluginsHttpDir=./conf/plugins/input/http/

Database connection credentials

❗ The below parameters are required if you have installed JDBC connector JAR.

❇ If you are using default in-memory H2 DB (not recommended for Production/Test) - these parameters are optional.

Refer to your JDBC driver documentation for details on how to configure the connections credentials.
Below example is demonstrating the case for MySQL connector only.

spring.datasource.url=jdbc:mysql://ip:port/db
spring.datasource.username=username
spring.datasource.password=password

Web Application settings

Define how Pigeon will expose its Web services:

spring.data.rest.base-path=/pigeon
server.contextPath=/pigeon
server.port = 8089

JPA Hibernate DDL mode

❗ Recommended value is update.
Data loss can happen when other values are used.

spring.jpa.hibernate.ddl-auto=update

Other settings

spring.banner.image.width=30
spring.banner.image.height=20
spring.jackson.serialization.indent_output=true

Pigeon.json

❗ Mandatory Step

Create a file Pigeon.json in the previously configured path.

Refer to Pigeon.json Wiki article for details on the file structure and configuration.

You can use the sample Pigeon.json as a start.

Bobbin.json

❇ Optional Step

Pigeon comes with a pre-defined well-structured logging configuration.
Therefore this step is required only if you need to tweak the logging configuration.
Refer to the Default Bobbin.json used by Pigeon.

Pigeon is using Bobbin - a revolutionary Java/Groovy Slf4j Logger.

For details on how to configure Bobbin.json refer to Bobbin Documentation

Bobbin.json should be placed in the Pigeon Working directory.

Carburetor.json

❇ Optional Step

Pigeon allows usage of @BlackBox annotation in the Pigeon plugins user code.

With the BlackBox the users don't have to worry about writing logging code in the Pigeon plugins - it is automatically generated.

All you need to do is to place @BlackBox annotation on the needed methods.

For more details refer to BlackBox Documentation.

The granularity of logging can be controlled using externalized configuration: Carburetor.json.

Place the Carburetor.json in the Pigeon working directory and define its contents:

{
    "defaultLevel": "EXPRESSION"
}

You can set defaultLevel to one of 5 values:

  1. NONE
  2. ERROR
  3. METHOD
  4. STATEMENT
  5. EXPRESSION

For more details refer to:

Plugins

You can develop your own plugins or use a variety of community plugins available in Pigeon Plugins Repository.

However it is recommended to deploy a bare minimum set of plugins used for self-diagnostic and health-check of Pigeon.

The plugins are placed as per Pigeon Configuration Paths.

The relative directory structure of Plugins normally is the following:

  • {Pigeon Home}
    • plugins
      • input
        • http
        • rest
      • output

For detailed information on Pigeon Plugins please refer to Plugins Page.

Recommended REST Input Plugins

Place SelfTest and EchoTest plugins into:

  • {Pigeon Home}
    • plugins
      • input
        • rest

Recommended Output Plugins

Place PASSTHROUGH_GET and PASSTHROUGH_POST plugins into:

  • {Pigeon Home}
    • plugins
      • output

Running

Running from command-line

Refer to documentation of command-line of your operating system for exact syntax.

❗ It is needed to set loader.path to lib. Refer to Windows example. Recommended Java Heap size parameters: at least -Xms1000m -Xmx1500m

Example for Windows:

"C:\Program Files\Java\jdk1.8.0_181\bin\java" -Xms1000m  -Xmx1500m -cp "pigeon.jar" -Dloader.path=lib org.springframework.boot.loader.PropertiesLauncher

Sample run output

If everything is configured properly you will see output similar to the below:

Bobbin          : main: Application working dir: C:\AP\CATEGORIZED\PIGEON
Bobbin          : main: Searching for Bobbin.json in: ./Bobbin.json (full path: C:\AP\CATEGORIZED\PIGEON\Bobbin.json)
Bobbin          : main: Not found.
Bobbin          : main: Searching for Bobbin.json in application resource files using Thread classloader.
Bobbin          : main: Found: jar:file:/C:/AP/CATEGORIZED/PIGEON/pigeon.jar!/BOOT-INF/lib/pigeon-lib-1.0.0-RC2.jar!/Bobbin.json
SLF4J: Bobbin: main            : application working dir: C:\AP\CATEGORIZED\PIGEON


                             @ @
                        @   @@ @
                       @@ @@@@
                      @@@@@@@@ @
                     @@@@@@@@@
                   @@@@@@@@@@@
   @              @@@@@@@@@@@@@
                @@@@@@@@@@@@@
      @@@      @@@@@@@@@@@@@@
    @@&  @@@@@ @@@@@@@@@@@@@
       @@@@@@@@@@@@@@@@@@@@8
           @@@@@@@@@@@@@@:
           @@@@@@@@@@@@
            @@@@@@@@@@
              @@@@@@@@@@
                    @@@@@@
                      @@@@@@@@
                       @@@@@@
                       @@@@@
                        @@@

////////////////////////////////////////////////////////////////////
////////////////////////////////////////////////////////////////////
Infinite Technology Pigeon v1.0.0
https://i-t.io/Pigeon
https://github.com/INFINITE-TECHNOLOGY/PIGEON
////////////////////////////////////////////////////////////////////
////////////////////////////////////////////////////////////////////
Mar 19, 2019 1:28:47 PM org.apache.catalina.core.StandardService startInternal
INFO: Starting service [Tomcat]
Mar 19, 2019 1:28:47 PM org.apache.catalina.core.StandardEngine startInternal
INFO: Starting Servlet Engine: Apache Tomcat/9.0.13
Mar 19, 2019 1:28:47 PM org.apache.catalina.core.AprLifecycleListener lifecycleEvent
INFO: The APR based Apache Tomcat Native library which allows optimal performance in production environments was not found on the java.library.path: [C:\Program Files\Java\jdk1.8.0_181\bin;C:\WINDOWS\Sun\Java\bin;C:\WINDOWS\system32;C:\WINDOWS;C:\Program Files (x86)\Common Files\Oracle\Java\javapath;C:\ProgramData\Oracle\Java\javapath;C:\WINDOWS\system32;C:\WINDOWS;C:\WINDOWS\System32\Wbem;C:\WINDOWS\System32\WindowsPowerShell\v1.0\;C:\PROGRA~2\Groovy\GROOVY~1.10\bin;C:\Program Files (x86)\PuTTY\;C:\Program Files (x86)\Skype\Phone\;C:\Program Files\Git\cmd;C:\Program Files\nodejs\;C:\Users\anton.pryamostanov\scoop\shims;C:\Users\anton.pryamostanov\AppData\Local\Microsoft\WindowsApps;C:\Users\anton.pryamostanov\AppData\Local\Programs\Git\cmd;C:\Users\anton.pryamostanov\AppData\Roaming\npm;.]
Mar 19, 2019 1:28:47 PM org.apache.catalina.core.ApplicationContext log
INFO: Initializing Spring embedded WebApplicationContext
2019-03-19 13:28:53:304|warn|main|org.springframework.boot.autoconfigure.orm.jpa.JpaBaseConfiguration$JpaWebConfiguration$JpaWebMvcConfiguration|spring.jpa.open-in-view is enabled by default. Therefore, database queries may be performed during view rendering. Explicitly configure spring.jpa.open-in-view to disable this warning|null;null
2019-03-19 13:28:53:950|warn|main|org.springframework.boot.autoconfigure.groovy.template.GroovyTemplateAutoConfiguration|Cannot find template location: classpath:/templates/ (please add some templates, check your Groovy configuration, or set spring.groovy.template.check-template-location=false)|null;null

Running as a service

After Pigeon is installed as a service, you can start Pigeon service as per Operating System documentation.

Post-install Self Test

Once the installation is completed and the Pigeon is started, you can run the Self Test to ensure there are no sanity issues and the application is healthy.

❇ This step requires installation of Recommended Plugins

To initiate Self Test just open in browser URL:

http://localhost:8089/pigeon/plugins/input/rest/SelfTest?format=yaml

❇ IP, Port and context path may differ based on your application.properties configuration.

If everything is ok, the browser will output the response of SelfTest Input Plugin:

response:
  outputMessages: http://localhost:8089/pigeon/outputMessages/search/searchByInputExternalIdAndSourceName?sourceName=SELF_TEST_PLUGIN&externalId=1553522562549
  httpLogs: http://localhost:8089/pigeon/readableHttpLogs?format=yaml&sourceName=SELF_TEST_PLUGIN&externalId=1553522562549

Now navigate the above displayed URLs to ensure that all messages are having status delivered and review the HTTP logs.

The searchByInputExternalIdAndSourceName results should look like:

{
  "_embedded" : {
    "outputMessages" : [ {
      "outputQueueName" : "SELF_TEST_ECHO_HTTP",
      "url" : "http://localhost:8089/pigeon/plugins/input/rest/EchoTest?format=yaml",
      "attemptsCount" : 1,
      "status" : "delivered",
      "outputThreadName" : "SELF_TEST_ECHO_HTTP_OUTPUT",
      "lastSenderThreadName" : "SELF_TEST_ECHO_HTTP_OUTPUT_SENDER_1",
      "insertTime" : "2019-03-25T14:02:42.892+0000",
      "lastSendTime" : "2019-03-25T14:02:43.239+0000",
      "exceptionString" : null,
      "_links" : {
        "self" : {
          "href" : "http://localhost:8089/pigeon/outputMessages/1"
        },
        "outputMessage" : {
          "href" : "http://localhost:8089/pigeon/outputMessages/1"
        },
        "httpLogs" : {
          "href" : "http://localhost:8089/pigeon/outputMessages/1/httpLogs"
        },
        "inputMessage" : {
          "href" : "http://localhost:8089/pigeon/outputMessages/1/inputMessage"
        }
      }
    } ]
  },
  "_links" : {
    "self" : {
      "href" : "http://localhost:8089/pigeon/outputMessages/search/searchByInputExternalIdAndSourceName?sourceName=SELF_TEST_PLUGIN&externalId=1553522562549"
    }
  }
}

And the readableHttpLogs results should look like:

 -
  id: 1
  requestDate: 1553522562926
  requestHeaders: '{Content-Type=application/json}'
  requestBody: |-
    {
    "message": "Test 2019-03-25 18-02-42-534"
    }
  method: POST
  url: http://localhost:8089/pigeon/plugins/input/rest/EchoTest?format=yaml
  requestStatus: delivered
  requestExceptionString: null
  responseDate: 1553522563059
  responseHeaders: '{Transfer-Encoding=chunked, null=HTTP/1.1 200, Date=Mon, 25 Mar 2019 14:02:42 GMT, Content-Type=text/yaml;charset=UTF-8}'
  responseBody: |
    response: |-
      {
      "message": "Test 2019-03-25 18-02-42-534"
      }
  responseStatus: 200
  insertDate: 1553522563239
  outputMessage: null
  senderThreadName: SELF_TEST_ECHO_HTTP_OUTPUT_SENDER_1

Note: the readableHttpLogs is recommended to be used with format=yaml for readability.

Now the installation of Pigeon is complete. You may proceed configuring pigeon.json, writing your own Pigeon Plugins - or using plugins from Pigeon Plugin Collection.

Clone this wiki locally