Merge pull request #20 from kit-data-manager/19-Improper_handling_of_mapping_results

19 improper handling of mapping results

Author: ThomasJejkal

.github/workflows/CI.yml

@@ -45,10 +45,10 @@ jobs:
 #        run: ./gradlew build -DapplicationProperties="src\test\resources\test-config\application-test-windows.properties"
       - if: matrix.os != 'windows-latest'
         name: Test with Gradle on ${{ matrix.os }}
-        run: ./gradlew build -DapplicationProperties="src/test/resources/test-config/application-test.properties"
+        run: ./gradlew build -DapplicationProperties="src/test/resources/test-config/application-test.properties" -DpythonLocation=`which python3`
       - name: Generate report
         run: ./gradlew jacocoTestReport
       - name: Codecov
         uses: codecov/codecov-action@v1
         with:
-          files: ./build/reports/jacoco/test/jacocoTestReport.xml
+          files: ./build/reports/jacoco/test/jacocoTestReport.xml

.gitignore

@@ -4,6 +4,7 @@
 #!gradle/wrapper/gradle-wrapper.jar
 #!**/src/main/**/build/
 #!**/src/test/**/build/
+lib/
 #
 #### Spring boot ###
 **/application.properties

README.md

@@ -21,44 +21,61 @@ Dependencies that are needed to build and are not being downloaded via gradle:
 
 - OpenJDK 17
 - Python 3
-- pip
+- pip (runtime only)
 
 `./gradlew -Pclean-release build`
 
 ### Python Location
 
-Currently, mapping-service requires Python to be installed in order to build and to run. By default, the Python location is set to `/usr/bin/python3`. In case
-your Python installation is located elsewhere or you build the mapping-service under Windows, you can provide the Python location externally, i.e.:
+Currently, mapping-service requires Python to be installed in order to build and to run. At runtime, the Python executable is configured in 
+`application.properties`(see below). For building the mapping-service Python executable is set to `/usr/bin/python3` by default. In case you want to build 
+the mapping-service on a machine on which the Python installation is located elsewhere, e.g., under Windows, you can provide the Python location 
+used at compile time externally, i.e.:
 
 ```
-`./gradlew -Pclean-release build -DpythonLocation=file:///C:/Python310/python.EXE`
+.\gradlew -Pclean-release "-DpythonExecutable=file:///C:/Python310/python.exe" build
 ```
 
 ## How to start
 
 Before you can start the mapping-service, you  first have to create an `application.properties` file in the source folder. As an example you may use `config/application.default.properties`
 and modify it according to your needs. Espacially the following properties (at the end of the file) are important:
-- `spring.datasource.url=jdbc:h2:file:e:/tmp/mapping-service/database`
+- `spring.datasource.url=jdbc:h2:file:/tmp/mapping-service/database`
 The path points to the location of the database in which your configured mappings are stored.
-- `mapping-service.pythonLocation=${pythonLocation:'file:///usr/bin/python3'}` \
-If no pythonLocation is provided externally (see above) the default `/usr/bin/python3` is used.
+- `mapping-service.pythonExecutable=${pythonExecutable:'file:///usr/bin/python3'}` \
+If no pythonExecutable is provided externally (see above) the default `/usr/bin/python3` is used.
+- `mapping-service.pluginLocation=file:///tmp/mapping-service/plugins` \
+The local folder where available plugins are located.
 - `mapping-service.mappingsLocation:file:///tmp/mapping-service/` \
 Enter the location where you want to store your mappings. This folder will be created if it does not exist yet.
 
-You might want to add a plugin to make the service working. Normally, there should be a gemma-plugin-x.x.x.jar in the plugins folder.
-If not, you can find its source code [here](https://github.com/maximilianiKIT/gemma-plugin).
+In order to provide the mapping-service with mapping functionality, there are already some pre-compiled plugins available under in the `plugins` folder of this repository.
+Copy them to your configured `mapping-service.pluginLocation` to make them available to the mapping-service. 
+The source code of the gemma-plugin can be found [here](https://github.com/maximilianiKIT/gemma-plugin). The plugin shows how to integrate Python mappings easily.
+
+There is also the possibility to add new plugins directly at the source tree and create a pluggable Jar out of them. Therefor, check 
+`src/main/java/edu/kit/datamanager/mappingservice/plugins/impl`. Just add your new plugin, e.g., based on the `TestPlugin` example. 
+In order to make the plugin usable by the mapping service, you then have to build a plugin Jar out of it. In order to do that, just call:
+
+```
+./gradlew buildPluginJar
+```
+
+This task creates a file `default-plugins-<VERSION>` at `build/libs` which has to be copied to `mapping-service.pluginLocation` to make it available. 
 
 After doing this, the mapping-service is ready for the first start. This can be achieved by executing:
 
-`./gradlew bootRun`
+`java -jar build/lib/mapping-service-<VERSION>.jar`
 
-### Python Location
+This assumes, that the command is called from the source folder and that your `application.properties` is located in the same folder. 
+Otherwise, you may use:
 
-Similar to configuring the Python location for the build process, you may also do the same for running the mapping-service via:
+`java -jar build/lib/mapping-service-<VERSION>.jar --spring.config.location=/tmp/application.properties`
 
-```
-.\gradlew -DpythonLocation=file:///C:/Python310/python.EXE bootRun
-```
+Ideally, for production use, you place everything (`mapping-service-<VERSION>.jar`, `application.properties`, `mapping-service.pluginLocation`, `mapping-service.mappingsLocation`,
+and `spring.datasource.url`) in a separate folder from where you then call the mapping-service via: 
+
+`java -jar mapping-service-<VERSION>.jar`
 
 ## License
 

build.gradle

@@ -36,7 +36,8 @@ repositories {
 ext {
     set('snippetsDir', file('build/generated-snippets'))
     applicationProperties = System.getProperty('applicationProperties', './src/test/resources/test-config/application-test.properties')
-    pythonLocation = System.getProperty('pythonLocation', 'file:///usr/bin/python3')
+    pythonExecutable = System.getProperty('pythonExecutable', 'file:///usr/bin/python3')
+    userDir = System.getProperty('user.dir')
 }
 
 dependencies {
@@ -89,11 +90,13 @@ dependencies {
 test {
     outputs.dir snippetsDir
     finalizedBy jacocoTestReport
-    print("Running tests with configuration: ${applicationProperties}")
+    println("Running tests with configuration: ${applicationProperties}")
+    println("Running tests with python: ${pythonExecutable}")
+    println("Running tests in directory: ${userDir}")
     environment 'spring.config.location', applicationProperties
-    environment 'pythonLocation', pythonLocation
-//    environment 'spring.config.location', 'classpath:/test-config/'
+    environment 'pythonExecutable', pythonExecutable
     useJUnitPlatform()
+    testLogging.showStandardStreams = true
 }
 
 jacoco {
@@ -126,7 +129,7 @@ springBoot {
 
 bootRun {
     systemProperty "spring.config.location", "file:$projectDir/"
-    systemProperty "pythonLocation", pythonLocation
+    systemProperty "pythonLocation", pythonExecutable
 }
 
 bootJar {
@@ -140,3 +143,10 @@ bootJar {
 release {
     tagTemplate = 'v${version}'
 }
+
+task buildPluginJar(type: Jar) {
+ description = 'Bundeling only plugin classes'
+ archiveFileName.set("default-plugins-${version}.jar")
+ from sourceSets.main.output
+ include '**/plugins/impl/*.class'
+}

config/application.default.properties

@@ -6,6 +6,9 @@ server.port=8095
 # The properties max-file-size and max-request-size define the maximum size of files 
 # transferred to and from the repository. Setting them to -1 removes all limits.
 server.compression.enabled=false
+
+# Max sizes of requests and uploaded files. This value may has to be increased for
+# bigger mapping inputs, e.g., while extracting information for a zipped dataset.
 spring.servlet.multipart.max-file-size=100MB
 spring.servlet.multipart.max-request-size=100MB
 # Logging settings
@@ -32,8 +35,9 @@ spring.jpa.hibernate.ddl-auto=update
 ##################################################
 # Mapping-Service specific settings
 ##################################################
-# Absolute path to the local python interpreter
+# Absolute path to the local python interpreter.
 mapping-service.pythonLocation=${pythonLocation:'file:///usr/bin/python3'}
-
-# Absolute path to the local gemma mappings folder
+# Absolute path to the folder where all plugins are located.
+mapping-service.pluginLocation=file:///${user.dir}/plugins
+# Absolute path to the local gemma mappings folder.
 mapping-service.mappingSchemasLocation=file:///tmp/mapping-service/mappingSchemas

src/main/java/edu/kit/datamanager/mappingservice/MappingServiceApplication.java

@@ -24,11 +24,16 @@ public ApplicationProperties applicationProperties() {
         return new ApplicationProperties();
     }
 
+    @Bean
+    public PluginManager pluginManager(){
+        return new PluginManager(applicationProperties());
+    }
+    
     public static void main(String[] args) {
         SpringApplication.run(MappingServiceApplication.class, args);
 
-        PluginManager.soleInstance().getListOfAvailableValidators().forEach((value) -> LOG.info("Found validator: " + value));
-        PythonRunnerUtil.printPythonVersion();
+        //pluginManager().getListOfAvailableValidators().forEach((value) -> LOG.info("Found validator: " + value));
+        //PythonRunnerUtil.printPythonVersion();
 
         System.out.println("Mapping service is running! Access it at http://localhost:8095");
     }

src/main/java/edu/kit/datamanager/mappingservice/configuration/ApplicationProperties.java

@@ -17,7 +17,6 @@
 
 import edu.kit.datamanager.annotations.ExecutableFileURL;
 import edu.kit.datamanager.annotations.LocalFolderURL;
-import edu.kit.datamanager.configuration.GenericPluginProperties;
 import lombok.Data;
 import lombok.EqualsAndHashCode;
 import org.springframework.beans.factory.annotation.Value;
@@ -28,8 +27,8 @@
 import java.net.URL;
 
 /**
- * This class is used to configure the application.
- * It reads the values from the application.properties file.
+ * This class is used to configure the application. It reads the values from the
+ * application.properties file.
  *
  * @author maximilianiKIT
  */
@@ -39,12 +38,20 @@
 @Validated
 @EqualsAndHashCode
 public class ApplicationProperties {
+
     /**
      * The absolute path to the python interpreter.
      */
     @ExecutableFileURL
-    @Value("${mapping-service.pythonLocation}")
-    private URL pythonLocation;
+    @Value("${mapping-service.pythonExecutable}")
+    private URL pythonExecutable;
+
+    /**
+     * The absolute path where the plugins are stored.
+     */
+    @LocalFolderURL
+    @Value("${mapping-service.pluginLocation}")
+    private URL pluginLocation;
 
     /**
      * The absolute path where the mappings are stored.

src/main/java/edu/kit/datamanager/mappingservice/exception/DuplicateMappingException.java

@@ -0,0 +1,61 @@
+/*
+ * Copyright 2022 Karlsruhe Institute of Technology.
+ *
+ * Licensed under the Apache License, Version 2.0 (the "License");
+ * you may not use this file except in compliance with the License.
+ * You may obtain a copy of the License at
+ *
+ *      http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package edu.kit.datamanager.mappingservice.exception;
+
+import org.springframework.http.HttpStatus;
+import org.springframework.web.bind.annotation.ResponseStatus;
+
+/**
+ * Invalid json format of data.
+ */
+@ResponseStatus(value = HttpStatus.CONFLICT)
+public class DuplicateMappingException extends RuntimeException {
+
+    /**
+     * Default constructor.
+     */
+    public DuplicateMappingException() {
+        super();
+    }
+
+    /**
+     * Constructor with given message and cause.
+     *
+     * @param message Message.
+     * @param cause   Cause.
+     */
+    public DuplicateMappingException(String message, Throwable cause) {
+        super(message, cause);
+    }
+
+    /**
+     * Constructor with given message.
+     *
+     * @param message Message.
+     */
+    public DuplicateMappingException(String message) {
+        super(message);
+    }
+
+    /**
+     * Constructor with given message and cause.
+     *
+     * @param cause Cause.
+     */
+    public DuplicateMappingException(Throwable cause) {
+        super(cause);
+    }
+}

src/main/java/edu/kit/datamanager/mappingservice/exception/MappingNotFoundException.java

@@ -0,0 +1,61 @@
+/*
+ * Copyright 2022 Karlsruhe Institute of Technology.
+ *
+ * Licensed under the Apache License, Version 2.0 (the "License");
+ * you may not use this file except in compliance with the License.
+ * You may obtain a copy of the License at
+ *
+ *      http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package edu.kit.datamanager.mappingservice.exception;
+
+import org.springframework.http.HttpStatus;
+import org.springframework.web.bind.annotation.ResponseStatus;
+
+/**
+ * Invalid json format of data.
+ */
+@ResponseStatus(value = HttpStatus.NOT_FOUND)
+public class MappingNotFoundException extends RuntimeException {
+
+    /**
+     * Default constructor.
+     */
+    public MappingNotFoundException() {
+        super();
+    }
+
+    /**
+     * Constructor with given message and cause.
+     *
+     * @param message Message.
+     * @param cause   Cause.
+     */
+    public MappingNotFoundException(String message, Throwable cause) {
+        super(message, cause);
+    }
+
+    /**
+     * Constructor with given message.
+     *
+     * @param message Message.
+     */
+    public MappingNotFoundException(String message) {
+        super(message);
+    }
+
+    /**
+     * Constructor with given message and cause.
+     *
+     * @param cause Cause.
+     */
+    public MappingNotFoundException(Throwable cause) {
+        super(cause);
+    }
+}