Network Streamers in MOOSE

Table Streamer MOOSE already has file based streamer; it writes to a CSV file to local disk and cleans up the internal std::vector in moose.Table/moose.Table2. One does not run out of RAM on long simulation. See ExampleExample2Implementation.

In addition to above CSV streamer, a network streamer has been added to MOOSE PR1[PR2]. This streamer streams data over either UNIX DOMAIN SOCKET or TCP SOCKET depending on the address given by the user. The streamer can be set by user manually or by setting the environment variable MOOSE_SOCKET_ADDRESS. If the variable has the form http://address:port then a TCP socket is created at this port (note address is irrelevant, it is always 127.0.0.1 or localhost), and a UNIX domain socket is created if the address has the form file:///tmp/MOOSE_SOCKET. The UNIX domain socket are preferred since they are faster than TCP sockets and easy to secure.

The main purpose of this streamer is to be used over network (see below). To make sure that streaming is efficient, table data is serialized using the following protocol before streaming.

• Data is arranged in table path followed by data (vector<double>). Not that data of table also contains time value as well. For example, if a table moose.Table('/tab/a') has value v0 and t0 and v1 at time t1 then the data is arranged as H #H / t a b / a V #V t0 v0 t1 v1 where H means start of header, followed by size of header #H (in this case 6), and V means start of data, followed by number of entries to follow #V (in this case 4). See the implementation here. The decoder in python is available here.

In moose, there are two servers based on socket. Table Streamer Server (TSS): Server to stream data from tables. This server supports both TCP and Unix domain socket (henceforth UDS). UDS are file-based socket and useful only on intranet/local machine. This is the default socket for streaming data to another local machine. It is faster (~7x) then TCP socket and security management is easy. MOOE Server (MS): Server to handle simulation request from another client. This is by nature a TCP socket since the client machine may be on different network. This server run the simulation on server, collect and sends back artifact created; it also stream back data to client by query socket described above.

If MOOSE_STREAMER_ADDRESS is set to a valid address e.g., file:///tmp/MOOSE or http://127.0.0.1:76543 then a Streamer is automatically created during moose.reinit and all exsiting tables are added to the streamer.

Following is the rough diagram on his works.

Most client would only see the MOOSE server and not TableStreamer server. Streamer server is usually local to the host machine. MOOSE server fetches data from TableStreamer