Issue 342 | TCP/UD Socket based Table Streamer (#345)

* - Implements TCP socket streamer. Opens on port 31416.
- Added a test case which also shows how to use it.
- The output is in JSON format.

* it is a good idea to zip the data with time for server.

* Temp commit. Needs to make sure that recv socket can read the data.

* Ready for a PR.

* Added moose server.

* This is nice but I don't know what is going on behind the scene. Must implement
from the scratch.

* Using raw sockets from python.

* I can now execute script on server.

* Updated a bit more.

* This works with ghevar [skip ci]

* A working version [skip ci]

* Extra code which saves all images to pdf file.

* A working version [skip ci]
Better path handling and result streaming. Needs to have a protocol for sending
messages.
python -m moose.server run the server.
removed pyflakes reported error [skip ci]
remove default -h (help) which was in conflict with -h (--host). [skip ci]

* Pipeline is complete. Works fine.

* Fixed pylint error.

* Socket works on OSX as well.

* Added logger. Need to test from different network.

* block socket till specified data is received.

Removed old dreprecated scripts. They are part of parent moose repo and
no longer required here.

Some more corner cases are handled.

Don't use fullpath.

Autosetup SocketStreamer when environment variable MOOSE_TCP_STREAMER_ADDRESS is
set to e.g. localhost:31416 .

Try till all data is recieved.

* Squashed commit of the following:

commit 206baccabb1d7c7b5b90accf9961376febcaab47
Author: Dilawar Singh <dilawars@ncbs.res.in>
Date:   Tue Feb 5 10:11:27 2019 +0530

    removed the background client. Accept only 1 client.

commit b71dbb3d3f312ca1eec4fb456e8ea758123ad505
Author: Dilawar Singh <dilawars@ncbs.res.in>
Date:   Mon Feb 4 15:55:44 2019 +0530

    changed the implementation. Now process function does nothing. We launch a
    thread in the background which monitor connections and stream the data. If no
    environment variable is set then no data is streamed and tables are safe.
    Tables are flushed when data is streamed. Need to change this behaviour.

* Socket streamer are alos available.

* added unix domain socket as well.

* Renamed some tests.

* On OSX, I can't send all the data [skip ci].

* Needs some more work [skip ci].

* Squashed commit of the following:

commit ff2b989659fdeb34ff5095177e17325286bd6746
Author: Dilawar Singh <dilawars@ncbs.res.in>
Date:   Wed Feb 6 21:26:03 2019 +0530

    Test passes on linux.

commit 8ca89dc
Author: Dilawar Singh <dilawars@ncbs.res.in>
Date:   Wed Feb 6 19:27:05 2019 +0530

    Latency in TCP socket is bad.

commit c3e15d8
Author: Dilawar Singh <dilawars@ncbs.res.in>
Date:   Wed Feb 6 18:56:36 2019 +0530

    Fixed both tests. Now test on OSX.

commit c731328
Author: Dilawar Singh <dilawars@ncbs.res.in>
Date:   Wed Feb 6 18:52:08 2019 +0530

    Time is handled by table.

commit 701ea55
Author: Dilawar Singh <dilawars@ncbs.res.in>
Date:   Wed Feb 6 16:26:35 2019 +0530

    Fixed frame size helped.

commit b168681
Author: Dilawar Singh <dilawars@ncbs.res.in>
Date:   Wed Feb 6 15:59:38 2019 +0530

    Test on osx [skip ci].

* Squashed commit of the following:

commit c3e15d8
Author: Dilawar Singh <dilawars@ncbs.res.in>
Date:   Wed Feb 6 18:56:36 2019 +0530

    Fixed both tests. Now test on OSX.

commit c731328
Author: Dilawar Singh <dilawars@ncbs.res.in>
Date:   Wed Feb 6 18:52:08 2019 +0530

    Time is handled by table.

commit 701ea55
Author: Dilawar Singh <dilawars@ncbs.res.in>
Date:   Wed Feb 6 16:26:35 2019 +0530

    Fixed frame size helped.

commit b168681
Author: Dilawar Singh <dilawars@ncbs.res.in>
Date:   Wed Feb 6 15:59:38 2019 +0530

    Test on osx [skip ci].

* TCP socket latency is high for small simulations.

* TCP socket works with latency but OK to use.

* Environment variable is working file. Time to merge.

Author: dilawar

CMakeLists.txt

@@ -97,7 +97,7 @@ else()
     message(STATUS "Building for Release/No unit tests.")
     set(CMAKE_BUILD_TYPE Release)
 		add_definitions(-UDO_UNIT_TESTS -O3 -DDISABLE_DEBUG)
-    
+
     # DO NOT Treat all warnings as errors. With some compilers and newer versions
     # this often causes headache.
     # add_definitions(-Werror)
@@ -353,7 +353,7 @@ if(SYSTEM_SHARED_LIBS)
     list(REMOVE_DUPLICATES SYSTEM_SHARED_LIBS)
 endif( )
 
-# cmake --help-policy CMP0042. Also in pymoose/CMakeLists.txt 
+# cmake --help-policy CMP0042. Also in pymoose/CMakeLists.txt
 # More details:
 # https://gitlab.kitware.com/cmake/community/wikis/doc/cmake/RPATH-handling
 # especially section 'Mac OS X and the RPATH'
@@ -528,8 +528,8 @@ foreach( _py_script ${PY_SCRIPTS} )
     file( READ ${_py_script} pytext)
     string(MD5 _md5 "${pytext}")
     set(TGT_NAME "${_py_name}-${_md5}" )
-    set(PYLINT_OPTIONS --disable=no-member --disable=no-name-in-module 
-            --disable=invalid-unary-operand-type 
+    set(PYLINT_OPTIONS --disable=no-member --disable=no-name-in-module
+            --disable=invalid-unary-operand-type
             --disable=import-error
             --disable=no-method-argument
         )

basecode/ObjId.cpp

@@ -17,111 +17,111 @@
 // Doesn't know how to deal with off-node bad fields.
 bool ObjId::bad() const
 {
-	Element* elm = id.element();
-	return ( elm == 0 ||
-		dataIndex == BADINDEX ||
-		fieldIndex == BADINDEX ||
-		dataIndex >= elm->numData()
-		/*
-		 * We have a bit of a problem here. The FieldElement can exist
-		 * with zero fields, and is invalid for field lookups but valid
-		 * for the element level lookups. I should rather not create the
-		 * FieldElements this way.
-		|| (
-		 	elm->getNode( dataIndex ) == Shell::myNode() &&
-		 	fieldIndex >= elm->numField( dataIndex - elm->localDataStart() )
-		)
-		 */
-	);
+    Element* elm = id.element();
+    return ( elm == 0 ||
+             dataIndex == BADINDEX ||
+             fieldIndex == BADINDEX ||
+             dataIndex >= elm->numData()
+             /*
+              * We have a bit of a problem here. The FieldElement can exist
+              * with zero fields, and is invalid for field lookups but valid
+              * for the element level lookups. I should rather not create the
+              * FieldElements this way.
+             || (
+              	elm->getNode( dataIndex ) == Shell::myNode() &&
+              	fieldIndex >= elm->numField( dataIndex - elm->localDataStart() )
+             )
+              */
+           );
 }
 
 ostream& operator <<( ostream& s, const ObjId& i )
 {
-	if ( i.dataIndex == 0 && i.fieldIndex == 0 )
-		s << i.id;
-	else if ( i.fieldIndex == 0 )
-		s << i.id << "[" << i.dataIndex << "]";
-	else
-		s << i.id << "[" << i.dataIndex << "][" << i.fieldIndex << "]";
-	return s;
+    if ( i.dataIndex == 0 && i.fieldIndex == 0 )
+        s << i.id;
+    else if ( i.fieldIndex == 0 )
+        s << i.id << "[" << i.dataIndex << "]";
+    else
+        s << i.id << "[" << i.dataIndex << "][" << i.fieldIndex << "]";
+    return s;
 }
 
 /**
  * need to complete implementation
  */
 istream& operator >>( istream& s, ObjId& i )
 {
-	s >> i.id;
-	return s;
+    s >> i.id;
+    return s;
 }
 
 //////////////////////////////////////////////////////////////
 ObjId::ObjId( const string& path )
 {
-	Shell* shell = reinterpret_cast< Shell* >( Id().eref().data() );
-	assert( shell );
-	*this = shell->doFind( path );
+    Shell* shell = reinterpret_cast< Shell* >( Id().eref().data() );
+    assert( shell );
+    *this = shell->doFind( path );
 }
 
 Eref ObjId::eref() const
 {
-	return Eref( id.element(), dataIndex, fieldIndex );
+    return Eref( id.element(), dataIndex, fieldIndex );
 }
 
 bool ObjId::operator==( const ObjId& other ) const
 {
-	return ( id == other.id && dataIndex == other.dataIndex &&
-					fieldIndex == other.fieldIndex );
+    return ( id == other.id && dataIndex == other.dataIndex &&
+             fieldIndex == other.fieldIndex );
 }
 
 bool ObjId::operator!=( const ObjId& other ) const
 {
-	return ( id != other.id || dataIndex != other.dataIndex ||
-					fieldIndex != other.fieldIndex );
+    return ( id != other.id || dataIndex != other.dataIndex ||
+             fieldIndex != other.fieldIndex );
 }
 
 bool ObjId::operator<( const ObjId& other ) const
 {
-	return ( id < other.id ||
-		(id == other.id && (
-			dataIndex < other.dataIndex  ||
-			( dataIndex == other.dataIndex &&
-					fieldIndex < other.fieldIndex )
-			)
-		)
-	);
+    return ( id < other.id ||
+             (id == other.id && (
+                  dataIndex < other.dataIndex  ||
+                  ( dataIndex == other.dataIndex &&
+                    fieldIndex < other.fieldIndex )
+              )
+             )
+           );
 }
 
 bool ObjId::isDataHere() const
 {
-	return ( id.element()->getNode( dataIndex ) == Shell::myNode() );
+    return ( id.element()->getNode( dataIndex ) == Shell::myNode() );
 }
 
 bool ObjId::isGlobal() const
 {
-	return ( id.element()->isGlobal() );
+    return ( id.element()->isGlobal() );
 }
 
 bool ObjId::isOffNode() const
 {
-	return ( Shell::numNodes() > 1 &&
-		( id.element()->isGlobal() ||
-		  id.element()->getNode( dataIndex ) != Shell::myNode() )
-	);
+    return ( Shell::numNodes() > 1 &&
+             ( id.element()->isGlobal() ||
+               id.element()->getNode( dataIndex ) != Shell::myNode() )
+           );
 }
 
 char* ObjId::data() const
 {
-	return id.element()->data( id.element()->rawIndex( dataIndex ),
-					fieldIndex );
+    return id.element()->data( id.element()->rawIndex( dataIndex ),
+                               fieldIndex );
 }
 
 string ObjId::path() const
 {
-	return Neutral::path( eref() );
+    return Neutral::path( eref() );
 }
 
 Element* ObjId::element() const
 {
-	return id.element();
+    return id.element();
 }

basecode/ObjId.h

@@ -19,95 +19,102 @@
  */
 class ObjId
 {
-	friend ostream& operator <<( ostream& s, const ObjId& i );
-	friend istream& operator >>( istream& s, ObjId& i );
-	public:
-		//////////////////////////////////////////////////////////////
-		//	ObjId creation
-		//////////////////////////////////////////////////////////////
-		/**
-		 * Returns the root Id
-		 */
-		ObjId()
-			: id(), dataIndex( 0 ), fieldIndex( 0 )
-		{;}
-
-		/**
-		 * Creates a ObjId using specified Id and DataIndex
-		 */
-		ObjId( Id i, unsigned int d, unsigned int f = 0 )
-			: id( i ), dataIndex( d ), fieldIndex( f )
-		{;}
-
-		ObjId( Id i )
-			: id( i ), dataIndex( 0 ), fieldIndex( 0 )
-		{;}
-
-		/**
-		 * Creates an ObjId by reading the path string
-		 * Returns bad on failure.
-		 */
-		ObjId( const string& path );
-
-		/**
-		 * Returns the absolute path including all array indices.
-		 */
-		string path() const;
-
-		/**
-		 * Returns the Eref matching itself.
-		 */
-		Eref eref() const;
-
-		/**
-		 * For equality check
-		 */
-		bool operator==( const ObjId& other ) const;
-		bool operator!=( const ObjId& other ) const;
-
-		/**
-		 * For sorting
-		 */
-		bool operator<( const ObjId& other ) const;
-
-		/**
-		 * True if the data is present on the current node. Always true for
-		 * globals, which confuses the matter somewhat.
-		 */
-		bool isDataHere() const;
-
-		/// Returns true if the Element is global.
-		bool isGlobal() const;
-
-		/// Returns true if we need to go off-node for calling operations
-		bool isOffNode() const;
-
-		/**
-		 * Returns data entry for this object
-		 */
-		char* data() const;
-
-		/**
-		 * Returns Element part
-		 */
-		Element* element() const;
-
-		/**
-		 * Here are the data values.
-		 */
-		Id id;
-		unsigned int dataIndex;
-		unsigned int fieldIndex;
-
-		/**
-		 * True if the return value is bad: either returning a failure,
-		 * or the DataIndex or FieldIndex is out of range. However, this
-		 * is a node-local funtion so it can't report the FieldIndex status
-		 * in all cases.
-		 */
-		bool bad() const;
-
-	private:
+    friend ostream& operator <<( ostream& s, const ObjId& i );
+    friend istream& operator >>( istream& s, ObjId& i );
+
+public:
+    //////////////////////////////////////////////////////////////
+    //	ObjId creation
+    //////////////////////////////////////////////////////////////
+    /**
+     * Returns the root Id
+     */
+    ObjId()
+        : id(), dataIndex( 0 ), fieldIndex( 0 )
+    {
+        ;
+    }
+
+    /**
+     * Creates a ObjId using specified Id and DataIndex
+     */
+    ObjId( Id i, unsigned int d, unsigned int f = 0 )
+        : id( i ), dataIndex( d ), fieldIndex( f )
+    {
+        ;
+    }
+
+    ObjId( Id i )
+        : id( i ), dataIndex( 0 ), fieldIndex( 0 )
+    {
+        ;
+    }
+
+    /**
+     * Creates an ObjId by reading the path string
+     * Returns bad on failure.
+     */
+    ObjId( const string& path );
+
+    /**
+     * Returns the absolute path including all array indices.
+     */
+    string path() const;
+
+    /**
+     * Returns the Eref matching itself.
+     */
+    Eref eref() const;
+
+    /**
+     * For equality check
+     */
+    bool operator==( const ObjId& other ) const;
+    bool operator!=( const ObjId& other ) const;
+
+    /**
+     * For sorting
+     */
+    bool operator<( const ObjId& other ) const;
+
+    /**
+     * True if the data is present on the current node. Always true for
+     * globals, which confuses the matter somewhat.
+     */
+    bool isDataHere() const;
+
+    /// Returns true if the Element is global.
+    bool isGlobal() const;
+
+    /// Returns true if we need to go off-node for calling operations
+    bool isOffNode() const;
+
+    /**
+     * Returns data entry for this object
+     */
+    char* data() const;
+
+    /**
+     * Returns Element part
+     */
+    Element* element() const;
+
+    /**
+     * Here are the data values.
+     */
+    Id id;
+    unsigned int dataIndex;
+    unsigned int fieldIndex;
+
+    /**
+     * True if the return value is bad: either returning a failure,
+     * or the DataIndex or FieldIndex is out of range. However, this
+     * is a node-local funtion so it can't report the FieldIndex status
+     * in all cases.
+     */
+    bool bad() const;
+
+private:
 };
 
 #endif // _OBJ_ID_H