Merge pull request #606 from DSC-SPIDAL/connected-dataflow

Connected dataflow

Author: supunkamburugamuve

docs/docs/architecture/README.md

@@ -30,7 +30,7 @@ Job Master manages the job related activities during job execution such as fault
 
 The processes that perform the computations in a job.
 
-### Twister2 Dashboard:
+### Twister2 Dashboard
 
 Twister2 Dashboard is a web site that helps users to monitor their jobs. 
 Only one instance of Dashboard runs in the cluster and it will provide data for all jobs running in the cluster.

docs/docs/concepts/task-system/task-system.md

@@ -27,8 +27,8 @@ task graphs in the task layer which could be created either statically or dynami
 the task graph represents a task whereas an edge represents the communication link between the vertices.
 Each node in the task graph holds the information about the input and its output. A task could be 
 long-running (streaming graph) or short-running (dataflow graph without loops) depending on the type 
-of application. A task graph 'TG' generally consists of set of Task Vertices'TV' and Task Edges \(TE\) which is 
-mathematically denoted as Task Graph
+of application. A task graph 'TG' generally consists of set of Task Vertices'TV' and Task Edges \(TE\) 
+which is mathematically denoted as Task Graph
 
 ```text
 (TG) -> (TV, TE)
@@ -203,12 +203,12 @@ message TaskSchedulePlanList {
 ### YAML file
 
 The task scheduler has task.yaml in the config directory. The task scheduler mode represents either 
-'roundrobin' or 'firstfit' or 'datalocalityaware'. The default task instances represents the default 
-memory, disk, and cpu values assigned to the task instances. The default container padding values 
+'roundrobin' or 'firstfit' or 'datalocalityaware'. The default task instance represents the default 
+memory, disk, and cpu values assigned to the task instances. The default container padding value 
 represents the percentage of values to be added to each container. The default container instance 
-values represents the default size of memory, disk, and cpu of the container. The task parallelism 
+value represents the default size of memory, disk, and cpu of the container. The task parallelism 
 represents the default parallelism value assigned to each task instance. The task type represents 
-the streaming or batch task.The task scheduler dynamically loads the respective streaming and batch 
+the streaming or batch task. The task scheduler dynamically loads the respective streaming and batch 
 task schedulers based on the configuration values specified in the task.yaml.
 
 \`\`yaml
@@ -231,6 +231,12 @@ twister2.task.instance.ram: 512.0
 twister2.task.instance.disk: 500.0
 twister2.task.instance.cpu: 2.0
 
+#Default Container Padding Values
+twister2.ram.padding.container: 2.0
+twister2.disk.padding.container: 12.0
+twister2.cpu.padding.container: 1.0
+twister2.container.padding.percentage: 2
+
 #Default Container Instance Values
 twister2.container.instance.ram: 2048.0
 twister2.container.instance.disk: 2000.0
@@ -314,7 +320,8 @@ operations.
 * The task executor will receive the tasks as serialized objects and it will deserialize the objects 
   before processing them. 
 * A thread pool will be maintained by the task executors to manage the core in an optimal manner. 
-  * The size of the thread pool will be determined by the number of cores that are available to the executor. 
+  * The size of the thread pool will be determined by the number of cores that are available to the 
+  executor. 
 
 ### Types of Task Executors
 

docs/docs/deployment/docker/running-jobs.md

@@ -26,7 +26,7 @@ If you want to build docker image by yourself, you can download the [Dockerfile]
 docker build -t repositoryname:tag .
 
 # example: 
-# docker build -t twister2/standalone:0.2.1 .
+# docker build -t twister2/standalone:0.3.0 .
 ```
 
 ## Running The Container
@@ -39,10 +39,10 @@ docker run -it twister2/standalone bash
 
 ## Running Jobs
 
-Docker image contains the source code and prebuilt twister2. Prebuilt twister2 is in the `~/twister2-0.2.1`  So, make sure that you are in that directory
+Docker image contains the source code and prebuilt twister2. Prebuilt twister2 is in the `~/twister2-0.3.0`  So, make sure that you are in that directory
 
 ```bash
-cd ~/twister2-0.2.1
+cd ~/twister2-0.3.0
 ```
 
 In order to run a job you can use the following command on the interactive bash session
@@ -71,7 +71,7 @@ bazel build --config=ubuntu scripts/package:tarpkgs
 This will build twister2 distribution in the file:
 
 ```bash
-bazel-bin/scripts/package/twister2-0.2.1.tar.gz
+bazel-bin/scripts/package/twister2-0.3.0.tar.gz
 ```
 
 If you want to modify the twister2 code and test without installing all the dependencies to your machine, you can mount the directory to the docker container:

docs/docs/deployment/mesos/running-jobs.md

@@ -10,7 +10,7 @@ In order to run twister2 jobs you should have a running Mesos cluster.
 
 **Running HelloWorld Example**
 
-Please first familiarize yourself with [HelloWorld job](https://github.com/DSC-SPIDAL/twister2/blob/master/docs/quickstart.md).
+Please first familiarize yourself with [HelloWorld job](../../quickstart.md)
 You can submit jobs to Mesos cluster by using twister2 executable:
 
 ```bash
@@ -36,8 +36,7 @@ is on maser ip:5050) or persistent logs. The workers in HelloWorld job
 prints a log message and sleeps 1 minutes before exiting.
 
 You can view the log files in persistent storage if persistent storage is
-enabled. Details can be found here on [installation
-document](https://github.com/DSC-SPIDAL/twister2/blob/master/docs/deployment/mesos/twister2-mesos-install.md).
+enabled. Further details of the Mesos installation could be found below. 
 
 **Terminating a Running Job**
 

docs/docs/deployment/mesos/twister2-mesos-install.md

@@ -1,4 +1,4 @@
-**Twister2 Mesos Installation**
+# Twister2 Mesos Installation
 
 In order to use Mesos, you have to install it on the machines that you
 want to use. One of the nodes will be the Mesos master and the others

docs/docs/deployment/slurm/running-jobs.md

@@ -29,7 +29,7 @@ In this mode, the job is killed immediately when you terminate the client using
 
 ## Installing OpenMPI
 
-When you compile Twister2, it builds OpenMPI 3.1.2 version with it. This version is
+When you compile Twister2, it builds OpenMPI 4.0.1 version with it. This version is
 used by Twister2 for its standalone deployment by default.
 
 You can use your own OpenMPI installation when running the jobs. In order to do that, you

docs/docs/deployment/standalone/running-jobs.md

@@ -43,7 +43,7 @@ to this file in order to run the job on them.
 
 ## Installing OpenMPI
 
-When you compile Twister2, it builds OpenMPI 3.1.2 version with it. This version is
+When you compile Twister2, it builds OpenMPI 4.0.1 version with it. This version is
 used by Twister2 for its standalone deployment by default.
 
 You can use your own OpenMPI installation when running the jobs. In order to do that, you

docs/docs/examples/ml/kmeans/kmeans.md

@@ -393,24 +393,17 @@ K-Means clustering process.
 ### Sample Output 
 
 ```bash
-[2019-03-25 15:27:01 -0400] [INFO] [worker-0] [main] edu.iu.dsc.tws.examples.batch.kmeans.KMeansWorker: 
-Final Centroids After	100	iterations	[[0.2535406313735363, 0.25640515489554255], 
-[0.7236140928643464, 0.7530306848028933], [0.7481226889281528, 0.24480221871888594], 
-[0.2203460821168371, 0.754988220991043]]  
+9-08-22 16:03:16 -0400] [INFO] [-] [JM] edu.iu.dsc.tws.master.server.WorkerMonitor: All 2 workers joined the job.  
+[2019-08-22 16:03:16 -0400] [INFO] [-] [JM] edu.iu.dsc.tws.master.server.WorkerMonitor: Sending WorkersJoined messages ...  
 
-[2019-03-25 15:27:01 -0400] [INFO] [worker-1] [main] edu.iu.dsc.tws.examples.batch.kmeans.KMeansWorker: 
-Final Centroids After	100	iterations	[[0.2535406313735363, 0.25640515489554255], 
-[0.7236140928643464, 0.7530306848028933], [0.7481226889281528, 0.24480221871888594], 
-[0.2203460821168371, 0.754988220991043]]  
+[2019-08-22 16:03:21 -0400] [INFO] [worker-0] [main] edu.iu.dsc.tws.examples.batch.kmeans.KMeansWorker: Final Centroids After	100	iterations	
+[[0.2586896371541501, 0.26352894494071144], [0.7226999170143885, 0.7523701053237408], [0.7481226890456432, 0.2448022186721992], [0.21755114043859655, 0.7589559817543858]]  
 
-[2019-03-25 15:27:01 -0400] [INFO] [worker-1] [main] edu.iu.dsc.tws.rsched.schedulers.standalone.MPIWorker: 
-Worker finished executing - 1  
-
-[2019-03-25 15:27:01 -0400] [INFO] [worker-0] [main] edu.iu.dsc.tws.rsched.schedulers.standalone.MPIWorker: 
-Worker finished executing - 0  
-
-[2019-03-25 15:27:01 -0400] [INFO] [-] [JM] edu.iu.dsc.tws.master.server.JobMaster: All 2 workers have completed. 
-JobMaster is stopping.  
+[2019-08-22 16:03:21 -0400] [INFO] [worker-0] [main] edu.iu.dsc.tws.rsched.schedulers.standalone.MPIWorker: Worker finished executing - 0  
 
+[2019-08-22 16:03:21 -0400] [INFO] [worker-1] [main] edu.iu.dsc.tws.examples.batch.kmeans.KMeansWorker: Final Centroids After	100	iterations	
+[[0.2586896371541501, 0.26352894494071144], [0.7226999170143885, 0.7523701053237408], [0.7481226890456432, 0.2448022186721992], [0.21755114043859655, 0.7589559817543858]]  
 
+[2019-08-22 16:03:21 -0400] [INFO] [worker-1] [main] edu.iu.dsc.tws.rsched.schedulers.standalone.MPIWorker: Worker finished executing - 1  
+[2019-08-22 16:03:21 -0400] [INFO] [-] [JM] edu.iu.dsc.tws.master.server.JobMaster: All 2 workers have completed. JobMaster is stopping.  
 ```

docs/docs/examples/terasort/terasort.md

@@ -1,10 +1,10 @@
 ---
 id: terasort
-title: Tera Sort
-sidebar_label: Tera Sort
+title: TeraSort
+sidebar_label: TeraSort
 ---
 
-Tera Sort is a common benchmark to measure and compare high performance big data frameworks such 
+TeraSort is a common benchmark to measure and compare high performance big data frameworks such 
 as Twister2. The idea is to measure the time to sort one terabyte of randomly distributed data.
 
 This terasort is implemented according to the requirements listed in the 
@@ -39,7 +39,7 @@ operation is configured with a comparator to sort by key.
 ./bin/twister2 submit standalone jar examples/libexamples-java.jar edu.iu.dsc.tws.examples.batch.terasort.TeraSort -size .5 -valueSize 90 -keySize 10 -instances 8 -instanceCPUs 1 -instanceMemory 1024 -sources 8 -sinks 8 -memoryBytesLimit 4000000000 -fileSizeBytes 100000000 -inputFile /path/to/file-%d
 ```
 
-### Tera Sort parameters
+### TeraSort parameters
 
 #### Data Configuration - File Based mode
 

docs/docs/quickstart.md

@@ -122,7 +122,7 @@ examples/src/java/edu/iu/dsc/tws/examples/comms
 You can run them with a simple command such as
 
 ```text
-./bin/twister2 submit standalone jar examples/libexamples-java.jar edu.iu.dsc.tws.examples.comms.ExampleMain -op "reduce" -stages 8,1
+./bin/twister2 submit standalone jar examples/libexamples-java.jar edu.iu.dsc.tws.examples.comms.ExampleMain -op "reduce" -stages 8,1 -workers 4
 ```
 
 Now, lets focus on a simple communication example where we try to do a word count.
@@ -132,7 +132,7 @@ Now, lets focus on a simple communication example where we try to do a word coun
 Lets look at a word count example. This is a standard example in every other big data system. The code related to example can be found in
 
 ```bash
-examples/src/java/edu/iu/dsc/tws/examples/batch/wordcount
+examples/src/java/edu/iu/dsc/tws/examples/batch/wordcount/task
 ```
 
 We are using a Keyed Reduce communication operation to calculate the global counts of words, which are emitted from parallel workers.
@@ -142,19 +142,19 @@ The example has three main classes.
 WordCountWorker that implements the IWorker interface and runs the code.
 
 ```java
-edu.iu.dsc.tws.examples.batch.wordcount.WordCountWorker
+edu.iu.dsc.tws.examples.batch.wordcount.task.WordCountWorker
 ```
 
 BatchWordSource, where we use a thread to generate words and put them into the communication.
 
 ```java
-edu.iu.dsc.tws.examples.batch.wordcount.BatchWordSouce
+edu.iu.dsc.tws.examples.batch.wordcount.task.BatchWordSouce
 ```
 
 WordAggregator, where it receives the counts of the words.
 
 ```java
-edu.iu.dsc.tws.examples.batch.wordcount.WordAggregator
+edu.iu.dsc.tws.examples.batch.wordcount.task.WordAggregator
 ```
 
 The WordCountWorker sets up communications and task ids. Then it sets up the communication operation.
@@ -207,14 +207,19 @@ return true;
 Here is the command to run this example
 
 ```bash
-./bin/twister2 submit standalone jar examples/libexamples-java.jar edu.iu.dsc.tws.examples.batch.wordcount.WordCountJob
+./bin/twister2 submit standalone jar examples/libexamples-java.jar edu.iu.dsc.tws.examples.batch.wordcount.task.WordCountJob
 ```
 
 It will output the words and their counts on the console and below is an small sample.
 
 ```bash
-[INFO] edu.iu.dsc.tws.examples.batch.wordcount.WordAggregator: 12 Word MWf count 83
-[INFO] edu.iu.dsc.tws.examples.batch.wordcount.WordAggregator: 12 Word mFu count 105
-[INFO] edu.iu.dsc.tws.examples.batch.wordcount.WordAggregator: 12 Word JyDA count 105
+[2019-08-22 16:41:22 -0400] [INFO] [worker-3] [main] edu.iu.dsc.tws.examples.batch.wordcount.task.WordCountJob: 100003 Word aw count 1  
+[2019-08-22 16:41:22 -0400] [INFO] [worker-3] [main] edu.iu.dsc.tws.examples.batch.wordcount.task.WordCountJob: 100003 Word z count 5  
+[2019-08-22 16:41:22 -0400] [INFO] [worker-3] [main] edu.iu.dsc.tws.examples.batch.wordcount.task.WordCountJob: 100003 Word DWm count 3  
+[2019-08-22 16:41:22 -0400] [INFO] [worker-3] [main] edu.iu.dsc.tws.examples.batch.wordcount.task.WordCountJob: 100003 Word wU count 1  
+[2019-08-22 16:41:22 -0400] [INFO] [worker-3] [main] edu.iu.dsc.tws.examples.batch.wordcount.task.WordCountJob: 100003 Word Ge count 1  
+[2019-08-22 16:41:22 -0400] [INFO] [worker-3] [main] edu.iu.dsc.tws.examples.batch.wordcount.task.WordCountJob: 100003 Word yW count 2  
+[2019-08-22 16:41:22 -0400] [INFO] [worker-3] [main] edu.iu.dsc.tws.rsched.schedulers.standalone.MPIWorker: Worker finished executing - 3  
+[2019-08-22 16:41:22 -0400] [INFO] [-] [JM] edu.iu.dsc.tws.master.server.JobMaster: All 4 workers have completed. JobMaster is stopping.  
 ```