load topology at startup and save each modification to memory and file

This feature allows to load a topology from file at daemon startup,
either from the default folder or from a user selected file, like

polycubed --cubes-dump-enable --cubes-dump-file ~/Desktop/cubes.yaml

keeping the whole configuration in memory.
To enable this feature we need to use the flag --cubes-dump-enable.
After each modification in any of the topology of the cube,
the configuration in memory is updated and another thread saves it
to file updating it too.
It is possible to start the daemon with an empty topology
by using the flag --cubes-dump-clean-init: if the file contains a topology,
at the first update it is overwritten.
--cubes-dump-enable is required when using one of the other two related flags.

Signed-off-by: Riccardo Marchi <riccardo.marchi4@gmail.com>

Author: richiMarchi

Documentation/polycubed/polycubed.rst

@@ -78,7 +78,36 @@ If the same parameter is specified in both, the configuration file and the comma
     daemon: true
     #p: 6000 <-- this is NOT supported, only long options are
 
+
+
+Persistency
+^^^^^^^^^^^
+
+Polycubed has persistent capabilities which means that (1) it can automatically load the configuration that was present when the daemon was shut down, (2) each time a configuration command is issued, it is automatically dumped on disk.
+This allows polycubed also to recover from failures, such as rebooting the machine.
+To enable this feature we need to start polycubed with the ``--cubes-dump-enable`` flag.
+The daemon keeps in memory an instance of all the topology, including the configuration of each individual service.
+Topology and configuration are automatically updated at each new command; the configuration is also dumped to disk, on file ``/etc/polycube/cubes.yaml``.
+The standard behavior of the daemon at startup is to load, if present, the latest topology that was active at the end of the previous execution.
+Users can load a different topology file by using the ``--cubes-dump-file`` flag followed by the path to the file.
+In case we want to start polycubed with an empty topology, avoiding any possible load at startup, we can launch polycubed with the ``--cubes-dump-clean-init`` flag. Beware that in this case any existing configuration in the default file will be overwritten.
+``--cubes-dump-enable`` is required if we want to use any of the other two related flags.
+There are some limitations: (1) YANG actions, such as "append" for firewall and nat rules, are not supported, (2) some services fail to load the full configuration at once and (3) transparent services attached to netdevs are not saved in the cubes dump file.
+
+::
+
+    # start daemon with topology saving functionalities
+    polycubed --cubes-dump-enable
+
+    # start polycubed with custom cubes configuration
+    polycubed --cubes-dump-enable --cubes-dump-file ~/Desktop/myCubes.yaml
+
+    # start polycubed with an empty topology
+    polycubed --cubes-dump-enable --cubes-dump-clean-init
+
+
+
 Rest API
 ^^^^^^^^
 
-TODO...
+TODO...

src/polycubed/src/CMakeLists.txt

@@ -62,6 +62,7 @@ set(polycubed_sources
   utils/veth.cpp
   utils/netlink.cpp
   utils/utils.cpp
+  cubes_dump.cpp
   ${server_sources}
   ${CMAKE_CURRENT_BINARY_DIR}/load_services.cpp
   ${CMAKE_CURRENT_BINARY_DIR}/version.cpp)

src/polycubed/src/config.cpp

@@ -32,13 +32,19 @@ namespace configuration {
 
 #define LOGLEVEL spdlog::level::level_enum::info
 #define DAEMON false
+#define CUBESDUMPCLEANINIT false
+//#define CUBESNODUMP false
+#define CUBESDUMPENABLED false
+#define CUBESDUMPFILEFLAG false
 #define PIDFILE "/var/run/polycube.pid"
 #define SERVER_PORT 9000
 #define SERVER_IP "localhost"
 #define LOGFILE "/var/log/polycube/polycubed.log"
 #define CONFIGFILEDIR "/etc/polycube"
 #define CONFIGFILENAME "polycubed.conf"
+#define CUBESDUMPFILENAME "cubes.yaml"
 #define CONFIGFILE (CONFIGFILEDIR "/" CONFIGFILENAME)
+#define CUBESDUMPFILEPATH (CONFIGFILEDIR "/" CUBESDUMPFILENAME)
 
 static void show_usage(const std::string &name) {
   std::cout << std::boolalpha;
@@ -64,6 +70,14 @@ static void show_usage(const std::string &name) {
             << ")" << std::endl;
   std::cout << "--configfile: configuration file (default: " << CONFIGFILE
             << ")" << std::endl;
+  std::cout << "--cubes-dump-file: file that keeps the last topology, "
+               "including the configuration of all cubes (default: "
+            << CUBESDUMPFILEPATH << ")" << std::endl;
+  std::cout << "--cubes-dump-clean-init: starts the daemon with an empty "
+               "topology" << std::endl;
+  //std::cout << "--cubes-nodump: starts the daemon without dumping updates to file" << std::endl;
+  std::cout << "--cubes-dump-enable: enables dumping updates to file"
+            << std::endl;
   std::cout << "--cert-blacklist: path to black listed certificates"
             << std::endl;
   std::cout << "--cert-whitelist: path to white listed certificates"
@@ -95,7 +109,12 @@ Config::Config()
       server_port(SERVER_PORT),
       server_ip(SERVER_IP),
       logfile(LOGFILE),
-      configfile(CONFIGFILE) {}
+      configfile(CONFIGFILE),
+      cubes_dump_file(CUBESDUMPFILEPATH),
+      cubes_dump_clean_init(CUBESDUMPCLEANINIT),
+      //cubes_nodump(CUBESNODUMP)
+      cubes_dump_enabled(CUBESDUMPENABLED),
+      cubes_dump_file_flag(CUBESDUMPFILEFLAG) {}
 
 Config::~Config() {}
 
@@ -180,6 +199,40 @@ void Config::setLogFile(const std::string &value) {
   logfile = value;
 }
 
+std::string Config::getCubesDumpFile() const {
+  return cubes_dump_file;
+}
+
+void Config::setCubesDumpFile(const std::string &value) {
+  cubes_dump_file_flag = true;
+  CHECK_OVERWRITE("cubes-dump-file", value, cubes_dump_file, CUBESDUMPFILEPATH);
+  cubes_dump_file = value;
+}
+
+bool Config::getCubesDumpCleanInit() const {
+  return cubes_dump_clean_init;
+}
+
+void Config::setCubesDumpCleanInit() {
+  cubes_dump_clean_init = true;
+}
+
+//bool Config::getCubesNoDump() const {
+//  return cubes_nodump;
+//}
+
+//void Config::setCubesNoDump() {
+//  cubes_nodump = true;
+//}
+
+bool Config::getCubesDumpEnabled() const {
+  return cubes_dump_enabled;
+}
+
+void Config::setCubesDumpEnabled() {
+  cubes_dump_enabled = true;
+}
+
 std::string Config::getCertPath() const {
   return cert_path;
 }
@@ -246,6 +299,8 @@ void Config::create_configuration_file(const std::string &path) {
   file << "addr: " << server_ip << std::endl;
   file << "# file to save polycube logs" << std::endl;
   file << "logfile: " << logfile << std::endl;
+  file << "# file to save last topology" << std::endl;
+  file << "cubes-dump-file: " << cubes_dump_file << std::endl;
   file << "# Security related:" << std::endl;
   file << "# server certificate " << std::endl;
   file << "#cert: path_to_certificate_file" << std::endl;
@@ -270,6 +325,10 @@ void Config::dump() {
   logger->info(" port: {}", server_port);
   logger->info(" addr: {}", server_ip);
   logger->info(" logfile: {}", logfile);
+  logger->info(" cubes-dump-file: {}", cubes_dump_file);
+  logger->info(" cubes-dump-clean-init: {}", cubes_dump_clean_init);
+  //logger->info(" cubes-nodump: {}", cubes_nodump);
+  logger->info(" cubes-dump-enable: {}", cubes_dump_enabled);
   if (!cert_path.empty()) {
     logger->info(" cert: {}", cert_path);
   }
@@ -363,6 +422,10 @@ static struct option options[] = {
     {"cacert", required_argument, NULL, 5},
     {"cert-whitelist", required_argument, NULL, 6},
     {"cert-blacklist", required_argument, NULL, 7},
+    {"cubes-dump-file", required_argument, NULL, 8},
+    {"cubes-dump-clean-init", no_argument, NULL, 9},
+    //{"cubes-nodump", no_argument, NULL, 10},
+    {"cubes-dump-enable", no_argument, NULL, 10},
     {NULL, 0, NULL, 0},
 };
 
@@ -408,6 +471,16 @@ void Config::load_from_cli(int argc, char *argv[]) {
     case 7:
       setCertBlacklistPath(optarg);
       break;
+    case 8:
+      setCubesDumpFile(optarg);
+      break;
+    case 9:
+      setCubesDumpCleanInit();
+      break;
+    case 10:
+      //setCubesNoDump();
+      setCubesDumpEnabled();
+      break;
     }
   }
 }
@@ -437,6 +510,15 @@ bool Config::load(int argc, char *argv[]) {
   load_from_file(configfile);
   load_from_cli(argc, argv);
   check();
+
+  if (cubes_dump_clean_init) {
+    std::ofstream output(cubes_dump_file);
+    if (output.is_open()) {
+      output << "{}";
+      output.close();
+    }
+  }
+
   return true;
 }
 
@@ -472,5 +554,10 @@ void Config::check() {
   if (cert_blacklist_path != "" && cacert_path == "") {
     throw std::runtime_error("--cert-blacklist requires --cacert");
   }
+
+  if ((cubes_dump_clean_init || cubes_dump_file_flag) && !cubes_dump_enabled) {
+    throw std::runtime_error("--cubes-dump-file and --cubes-dump-clean-init "
+                             "require --cubes-dump-enable");
+  }
 }
 }

src/polycubed/src/config.h

@@ -53,6 +53,22 @@ class Config {
   std::string getLogFile() const;
   void setLogFile(const std::string &value);
 
+  // file where last topology is saved
+  std::string getCubesDumpFile() const;
+  void setCubesDumpFile(const std::string &value);
+
+  // set to get an empty topology
+  bool getCubesDumpCleanInit() const;
+  void setCubesDumpCleanInit();
+
+  // set to avoid daemon to dump updates to file
+  //bool getCubesNoDump() const;
+  //void setCubesNoDump();
+
+  // set to let daemon to dump updates to file
+  bool getCubesDumpEnabled() const;
+  void setCubesDumpEnabled();
+
   // path of certificate & key to be used in server
   std::string getCertPath() const;
   void setCertPath(const std::string &value);
@@ -80,11 +96,17 @@ class Config {
 
   spdlog::level::level_enum loglevel;
   bool daemon;
+  bool cubes_dump_clean_init;
+  //bool cubes_nodump;
+  bool cubes_dump_enabled;
+  // true if --cubes-dump-file flag is used
+  bool cubes_dump_file_flag;
   std::string pidfile;
   uint16_t server_port;
   std::string server_ip;
   std::string logfile;
   std::string configfile;
+  std::string cubes_dump_file;
   std::string cert_path, key_path;
   std::string cacert_path;
   std::string cert_whitelist_path;