Fix #18, externalized plugin library folder, changed PluginManager to use spring autowire instead of custom singleton, made PluginLoader using current classloader such that packaging the interface with plugins is not longer required, fixed some UI issues causing HTTP 404 and not loaded list of mappings, updated README, added Gradle task buildPluginJar to allow to maintain some plugins in the mapping-service and to extract their classes as separate JAR without effort

Author: ThomasJejkal

README.md

@@ -21,44 +21,61 @@ Dependencies that are needed to build and are not being downloaded via gradle:
 
 - OpenJDK 17
 - Python 3
-- pip
+- pip (runtime only)
 
 `./gradlew -Pclean-release build`
 
 ### Python Location
 
-Currently, mapping-service requires Python to be installed in order to build and to run. By default, the Python location is set to `/usr/bin/python3`. In case
-your Python installation is located elsewhere or you build the mapping-service under Windows, you can provide the Python location externally, i.e.:
+Currently, mapping-service requires Python to be installed in order to build and to run. At runtime, the Python location is configured in 
+`application.properties`(see below). For building the mapping-service Python location is set to `/usr/bin/python3` by default. In case you want to build 
+the mapping-service on a machine on which the Python installation is located elsewhere, e.g., under Windows, you can provide the Python location 
+used at compile time externally, i.e.:
 
 ```
-`./gradlew -Pclean-release build -DpythonLocation=file:///C:/Python310/python.EXE`
+.\gradlew -Pclean-release "-DpythonLocation=file:///C:/Python310/python.exe" build
 ```
 
 ## How to start
 
 Before you can start the mapping-service, you  first have to create an `application.properties` file in the source folder. As an example you may use `config/application.default.properties`
 and modify it according to your needs. Espacially the following properties (at the end of the file) are important:
-- `spring.datasource.url=jdbc:h2:file:e:/tmp/mapping-service/database`
+- `spring.datasource.url=jdbc:h2:file:/tmp/mapping-service/database`
 The path points to the location of the database in which your configured mappings are stored.
 - `mapping-service.pythonLocation=${pythonLocation:'file:///usr/bin/python3'}` \
 If no pythonLocation is provided externally (see above) the default `/usr/bin/python3` is used.
+- `mapping-service.pluginLocation=file:///tmp/mapping-service/plugins` \
+The local folder where available plugins are located.
 - `mapping-service.mappingsLocation:file:///tmp/mapping-service/` \
 Enter the location where you want to store your mappings. This folder will be created if it does not exist yet.
 
-You might want to add a plugin to make the service working. Normally, there should be a gemma-plugin-x.x.x.jar in the plugins folder.
-If not, you can find its source code [here](https://github.com/maximilianiKIT/gemma-plugin).
+In order to provide the mapping-service with mapping functionality, there are already some pre-compiled plugins available under in the `plugins` folder of this repository.
+Copy them to your configured `mapping-service.pluginLocation` to make them available to the mapping-service. 
+The source code of the gemma-plugin can be found [here](https://github.com/maximilianiKIT/gemma-plugin). The plugin shows how to integrate Python mappings easily.
+
+There is also the possibility to add new plugins directly at the source tree and create a pluggable Jar out of them. Therefor, check 
+`src/main/java/edu/kit/datamanager/mappingservice/plugins/impl`. Just add your new plugin, e.g., based on the `TestPlugin` example. 
+In order to make the plugin usable by the mapping service, you then have to build a plugin Jar out of it. In order to do that, just call:
+
+```
+./gradlew buildPluginJar
+```
+
+This task creates a file `default-plugins-<VERSION>` at `build/libs` which has to be copied to `mapping-service.pluginLocation` to make it available. 
 
 After doing this, the mapping-service is ready for the first start. This can be achieved by executing:
 
-`./gradlew bootRun`
+`java -jar build/lib/mapping-service-<VERSION>.jar`
 
-### Python Location
+This assumes, that the command is called from the source folder and that your `application.properties` is located in the same folder. 
+Otherwise, you may use:
 
-Similar to configuring the Python location for the build process, you may also do the same for running the mapping-service via:
+`java -jar build/lib/mapping-service-<VERSION>.jar --spring.config.location=/tmp/application.properties`
 
-```
-.\gradlew -DpythonLocation=file:///C:/Python310/python.EXE bootRun
-```
+Ideally, for production use, you place everything (`mapping-service-<VERSION>.jar`, `application.properties`, `mapping-service.pluginLocation`, `mapping-service.mappingsLocation`,
+and `spring.datasource.url`) in a separate folder from where you then call the mapping-service via: 
+
+`java -jar mapping-service-<VERSION>.jar`
 
 ## License
 

build.gradle

@@ -140,3 +140,10 @@ bootJar {
 release {
     tagTemplate = 'v${version}'
 }
+
+task buildPluginJar(type: Jar) {
+ description = 'Bundeling only plugin classes'
+ archiveFileName.set("default-plugins-${version}.jar")
+ from sourceSets.main.output
+ include '**/plugins/impl/*.class'
+}

config/application.default.properties

@@ -6,6 +6,9 @@ server.port=8095
 # The properties max-file-size and max-request-size define the maximum size of files 
 # transferred to and from the repository. Setting them to -1 removes all limits.
 server.compression.enabled=false
+
+# Max sizes of requests and uploaded files. This value may has to be increased for
+# bigger mapping inputs, e.g., while extracting information for a zipped dataset.
 spring.servlet.multipart.max-file-size=100MB
 spring.servlet.multipart.max-request-size=100MB
 # Logging settings
@@ -32,8 +35,9 @@ spring.jpa.hibernate.ddl-auto=update
 ##################################################
 # Mapping-Service specific settings
 ##################################################
-# Absolute path to the local python interpreter
+# Absolute path to the local python interpreter.
 mapping-service.pythonLocation=${pythonLocation:'file:///usr/bin/python3'}
-
-# Absolute path to the local gemma mappings folder
+# Absolute path to the folder where all plugins are located.
+mapping-service.pluginLocation=file:///${user.dir}/plugins
+# Absolute path to the local gemma mappings folder.
 mapping-service.mappingSchemasLocation=file:///tmp/mapping-service/mappingSchemas

src/main/java/edu/kit/datamanager/mappingservice/MappingServiceApplication.java

@@ -24,11 +24,16 @@ public ApplicationProperties applicationProperties() {
         return new ApplicationProperties();
     }
 
+    @Bean
+    public PluginManager pluginManager(){
+        return new PluginManager(applicationProperties());
+    }
+    
     public static void main(String[] args) {
         SpringApplication.run(MappingServiceApplication.class, args);
 
-        PluginManager.soleInstance().getListOfAvailableValidators().forEach((value) -> LOG.info("Found validator: " + value));
-        PythonRunnerUtil.printPythonVersion();
+        //pluginManager().getListOfAvailableValidators().forEach((value) -> LOG.info("Found validator: " + value));
+        //PythonRunnerUtil.printPythonVersion();
 
         System.out.println("Mapping service is running! Access it at http://localhost:8095");
     }

src/main/java/edu/kit/datamanager/mappingservice/configuration/ApplicationProperties.java

@@ -17,7 +17,6 @@
 
 import edu.kit.datamanager.annotations.ExecutableFileURL;
 import edu.kit.datamanager.annotations.LocalFolderURL;
-import edu.kit.datamanager.configuration.GenericPluginProperties;
 import lombok.Data;
 import lombok.EqualsAndHashCode;
 import org.springframework.beans.factory.annotation.Value;
@@ -28,8 +27,8 @@
 import java.net.URL;
 
 /**
- * This class is used to configure the application.
- * It reads the values from the application.properties file.
+ * This class is used to configure the application. It reads the values from the
+ * application.properties file.
  *
  * @author maximilianiKIT
  */
@@ -39,13 +38,21 @@
 @Validated
 @EqualsAndHashCode
 public class ApplicationProperties {
+
     /**
      * The absolute path to the python interpreter.
      */
     @ExecutableFileURL
     @Value("${mapping-service.pythonLocation}")
     private URL pythonLocation;
 
+    /**
+     * The absolute path where the plugins are stored.
+     */
+    @LocalFolderURL
+    @Value("${mapping-service.pluginLocation}")
+    private URL pluginLocation;
+
     /**
      * The absolute path where the mappings are stored.
      */

src/main/java/edu/kit/datamanager/mappingservice/impl/MappingService.java

@@ -57,6 +57,12 @@ public class MappingService {
      */
     @Autowired
     private IMappingRecordDao mappingRepo;
+
+    /**
+     * The Plugin Manager.
+     */
+    @Autowired
+    private PluginManager pluginManager;
     /**
      * Path to directory holding all mapping files.
      */
@@ -176,7 +182,7 @@ public Optional<Path> executeMapping(URI contentUrl, String mappingId) throws Ma
             Path resultFile;
             resultFile = FileUtil.createTempFile(mappingId + "_" + srcFile.hashCode(), ".result");
             LOGGER.trace("Temporary output file available at {}. Performing mapping.", resultFile);
-            MappingPluginState result = PluginManager.soleInstance().mapFile(mappingRecord.getMappingType(), mappingFile, srcFile, resultFile);
+            MappingPluginState result = pluginManager.mapFile(mappingRecord.getMappingType(), mappingFile, srcFile, resultFile);
             LOGGER.trace("Mapping returned with result {}. Returning result file.", result);
             returnValue = Optional.of(resultFile);
             // remove downloaded file

src/main/java/edu/kit/datamanager/mappingservice/plugins/PluginLoader.java

@@ -60,29 +60,29 @@ public static void unload() {
      * the input.
      */
     public static Map<String, IMappingPlugin> loadPlugins(File plugDir) throws IOException, MappingPluginException {
-        if (plugDir == null || plugDir.getAbsolutePath().isBlank()) {
-            throw new MappingPluginException(MappingPluginState.INVALID_INPUT, "Empty input!");
-        }
-        File[] plugJars = plugDir.listFiles(new JARFileFilter());
-        if (plugJars == null || plugJars.length < 1) {
-            throw new MappingPluginException(MappingPluginState.NOT_FOUND, "No plugins found.");
-        }
-        cl = new URLClassLoader(PluginLoader.fileArrayToURLArray(plugJars));
-        
-        List<Class<IMappingPlugin>> plugClasses = PluginLoader.extractClassesFromJARs(plugJars, cl);
-
-        List<IMappingPlugin> IMappingPluginList = PluginLoader.createPluggableObjects(plugClasses);
         Map<String, IMappingPlugin> result = new HashMap<>();
-        for (IMappingPlugin i : IMappingPluginList) {
-            i.setup();
-            result.put(i.id(), i);
+        if (plugDir == null || plugDir.getAbsolutePath().isBlank()) {
+            LOG.warn("Plugin folder " + plugDir + " is null. Unable to load plugins.");
+        } else {
+            File[] plugJars = plugDir.listFiles(new JARFileFilter());
+            if (plugJars == null || plugJars.length < 1) {
+                LOG.warn("Plugin folder " + plugDir + " is empty. Unable to load plugins.");
+            } else {
+                cl = new URLClassLoader(PluginLoader.fileArrayToURLArray(plugJars), Thread.currentThread().getContextClassLoader());
+
+                List<Class<IMappingPlugin>> plugClasses = PluginLoader.extractClassesFromJARs(plugJars, cl);
+                List<IMappingPlugin> IMappingPluginList = PluginLoader.createPluggableObjects(plugClasses);
+
+                for (IMappingPlugin i : IMappingPluginList) {
+                    i.setup();
+                    result.put(i.id(), i);
+                }
+            }
         }
-
         return result;
     }
 
     private static URL[] fileArrayToURLArray(File[] files) throws MalformedURLException {
-
         URL[] urls = new URL[files.length];
         for (int i = 0; i < files.length; i++) {
             urls[i] = files[i].toURI().toURL();
@@ -91,27 +91,30 @@ private static URL[] fileArrayToURLArray(File[] files) throws MalformedURLExcept
     }
 
     private static List<Class<IMappingPlugin>> extractClassesFromJARs(File[] jars, ClassLoader cl) throws IOException, MappingPluginException {
-
+        LOG.trace("Extracting classes from plugin JARs.");
         List<Class<IMappingPlugin>> classes = new ArrayList<>();
         for (File jar : jars) {
+            LOG.trace("Processing file {}.", jar.getAbsolutePath());
             classes.addAll(PluginLoader.extractClassesFromJAR(jar, cl));
         }
         return classes;
     }
 
     private static List<Class<IMappingPlugin>> extractClassesFromJAR(File jar, ClassLoader cl) throws IOException, MappingPluginException {
-
+        LOG.trace("Extracting plugin classes from file {}.", jar.getAbsolutePath());
         List<Class<IMappingPlugin>> classes = new ArrayList<>();
         try (JarInputStream jaris = new JarInputStream(new FileInputStream(jar))) {
             JarEntry ent;
             while ((ent = jaris.getNextJarEntry()) != null) {
                 if (ent.getName().toLowerCase().endsWith(".class")) {
                     try {
                         Class<?> cls = cl.loadClass(ent.getName().substring(0, ent.getName().length() - 6).replace('/', '.'));
+                        LOG.trace("Checking {}.", cls);
                         if (PluginLoader.isPluggableClass(cls)) {
+                            LOG.trace("Plugin class found.");
                             classes.add((Class<IMappingPlugin>) cls);
                         }
-                    } catch (ClassNotFoundException e) {
+                    } catch (ClassNotFoundException | NoClassDefFoundError e) {
                         LOG.info("Can't load Class " + ent.getName());
                         throw new MappingPluginException(MappingPluginState.UNKNOWN_ERROR, "Can't load Class " + ent.getName(), e);
                     }
@@ -122,18 +125,22 @@ private static List<Class<IMappingPlugin>> extractClassesFromJAR(File jar, Class
     }
 
     private static boolean isPluggableClass(Class<?> cls) {
-
         for (Class<?> i : cls.getInterfaces()) {
+            LOG.trace("Checking {} against {}.", i, IMappingPlugin.class);
+LOG.trace("ASSIGN {}", IMappingPlugin.class.isAssignableFrom(cls));
             if (i.equals(IMappingPlugin.class)) {
+                LOG.trace("IMappingPlugin interface found.");
                 return true;
             }
         }
         return false;
     }
 
     private static List<IMappingPlugin> createPluggableObjects(List<Class<IMappingPlugin>> pluggable) throws MappingPluginException {
+        LOG.trace("Instantiating plugins from list: {}", pluggable);
         List<IMappingPlugin> plugs = new ArrayList<>(pluggable.size());
         for (Class<IMappingPlugin> plug : pluggable) {
+            LOG.trace("Instantiating plugin from class {}.", plug);
             try {
                 plugs.add(plug.getDeclaredConstructor().newInstance());
             } catch (InstantiationException | NoSuchMethodException | InvocationTargetException e) {