Processors

Xatkit embeds processors: additional pieces of logic that can be plugged to tune the intent recognition process. Pre-processors operate on the user input to optimize it before intent extraction (e.g. by adding a question mark at the end of a sentence that is obviously a question). Post-processors are designed to operate after the intent recognition process, usually to set additional context parameters (e.g. perform some sentiment analysis, set whether the input is a yes/no question, etc).

Enable Xatkit Processors

Unless explicitly stated in the documentation, Xatkit processors can be used with any intent recognition engine. The bot properties file accepts the following keys to enable pre/post processors:

xatkit.recognition.preprocessors  = PreProcessor1, PreProcessor2
xatkit.recognition.postprocessors = PostProcessor1, PostProcessor2

The value of each property is a comma-separated list of processor names (see the table below for the list of processors embedded in Xatkit and their names).

📚 You can also directly use the processor class to configure your bot programmatically:

import static com.xatkit.core.recognition.IntentRecognitionProviderFactoryConfiguration.*;
 
[...]
Configuration configuration = new BaseConfiguration();
// Bot-specific configuration (NLP engine, database, etc)
configuration.addProperty(RECOGNITION_PREPROCESSORS_KEY, "PostProcessor1, PostProcessor2");

Access Processor Data

Data extracted from processors is attached to the current intent and stored in the state context. context.getIntent().getNlpData() is a map containing all the information extracted by Xatkit processors for the current intent.

The code below shows how to access the property nlp.stanford.isYesNo (extracted by the IsEnglishYesNoQuestion post-processor) and use it to tune the bot behavior:

state("myState")
    .body(context -> {
        if((Boolean) context.getIntent().getNlpData().get("nlp.stanford.isYesNo")) {
            // Post a reply matching a yes/no question
        } else {
            // Post a generic reply
        }
     })
     .next()
     [...]

Pre-Processors

Name	Description	Requirements
SpacePunctuationPreProcessor	Adds spaces around punctuation when needed (e.g. from "?" to " ?"). This processor is enabled by default when using the NlpjsIntentRecognitionProvider.

Post-Processors

Name	Description	Requirements
RemoveEnglishStopWords	Removes English stop words from recognized intent's parameter values that have been extracted from any entities. This processor helps normalizing DialogFlow values when using any entities.
IsEnglishYesNoQuestion	Sets the parameter nlp.stanford.isYesNo to true if the user input is a yes/no question, and false otherwise	See Stanford CoreNLP configuration
EnglishSentiment	Sets the parameter nlp.stanford.sentiment to a value in ["Very Negative", "Negative", "Neutral", "Positive", "Very Positive"] corresponding to the sentiment extracted from the user input.	See Stanford CoreNLP configuration
ToxicityPostProcessor	Sets nlp.perspectiveapi, nlp.detoxify or both (see ToxicityPostProcessor usage) to an object that contains scores for different toxicity labels.	For PerspectiveAPI you need an API key. You can ask for it here. For Detoxify you need to deploy a python server to make requests to the model (see this example server).
TrimParameterValuesPostProcessor	Removes leading/trailing spaces in extracted parameter values (e.g. from "Barcelona " to "Barcelona"). This processor is enabled by default when using the NlpjsIntentRecognitionProvider.
TrimPunctuationPostProcessor	Removes punctuation in extracted parameter values (e.g. from "Barcelona!" to "Barcelona"). This processor is enabled by default when using the NlpjsIntentRecognitionProvider.

Stanford CoreNLP Configuration

Stanford CoreNLP is not embedded by default in Xatkit. Add the following dependencies in your bot's pom.xml if you want to use a Stanford CoreNLP processor:

<dependency>
    <groupId>edu.stanford.nlp</groupId>
    <artifactId>stanford-corenlp</artifactId>
    <version>3.9.2</version>
    <exclusions>
        <exclusion>
            <groupId>com.google.protobuf</groupId>
            <artifactId>protobuf-java</artifactId>
        </exclusion>
    </exclusions>
</dependency>

<dependency>
    <groupId>edu.stanford.nlp</groupId>
    <artifactId>stanford-corenlp</artifactId>
    <version>3.9.2</version>
    <classifier>models</classifier>
    <exclusions>
        <exclusion>
            <groupId>com.google.protobuf</groupId>
            <artifactId>protobuf-java</artifactId>
        </exclusion>
    </exclusions>
</dependency>

📚 You can use models for specific language by adapting the classifier. For example <classifier>models-chinese</classifier> imports the chinese models.

ToxicityPostProcessor usage

See the PerspectiveAPI and Detoxify for more information about the language models.

If you want to use Detoxify, add these properties to your bot with the proper parameters:

botConfiguration.setProperty(USE_DETOXIFY, true);
botConfiguration.setProperty(DetoxifyConfiguration.DETOXIFY_SERVER_URL, "YOUR SERVER URL");

📚 You'll need to wrap your Detoxify model in a REST API. See our prototype implementation for more information.

If you want to use PerspectiveAPI, add these properties to your bot with the proper parameters:

botConfiguration.setProperty(RECOGNITION_POSTPROCESSORS_KEY,"ToxicityPostProcessor");
botConfiguration.setProperty(USE_PERSPECTIVE_API, true);
botConfiguration.setProperty(PerspectiveApiConfiguration.API_KEY, "YOUR PERSPECTIVEAPI KEY");
botConfiguration.setProperty(PerspectiveApiConfiguration.LANGUAGE, "YOUR LANGUAGE (en/es)");

PerspectiveAPI has other optional parameters that can be added to the bot: doNotStore, clientToken and sessionId. See the official description of the attributes and methods for more information.

You can access to the toxicity scores this way:

DetoxifyScore score = (DetoxifyScore) context.getIntent().getNlpData().get("nlp.detoxify");
Double toxicity = score.getToxicity();

and the same way with PerspectiveAPI. See PerspectiveAPI and Detoxify code and the toxicity example bot for more information.