Merge pull request #223 from pynbody/mysql-optimize

MySQL optimization

Author: apontzen

docs/advanced.md

@@ -4,6 +4,7 @@ Advanced topics
 _Tangos_ is a highly flexible, customisable system. Tutorials are available covering the following
 topics:
 
+ - Working with [different database systems](dbms.md) (e.g. MySQL and PostgreSQL)
  - Writing code to [calculate your own properties](custom_properties.md)
  - [Tracking](tracking.md) groups of particles across timesteps
  - [Parallelisation strategies](mpi.md)

docs/index.md

@@ -77,45 +77,6 @@ MySQL / MariaDB below.
 Remember, you will need to set these environment variables *every* time you start a new session on your computer prior
 to booting up the database, either with the webserver or the python interface (see below).
 
-Using PostgreSQL, MySQL or MariaDB
-----------------------------------
-
-As stated above, tangos is agnostic to the underlying SQL flavour. It is easiest to get start with
-SQLite which doesn't need any special server. But version 1.5+ should also work well with [MySQL](https://www.mysql.com),
-[MariaDB](https://mariadb.org) and version 1.7+ also with [PostgreSQL](https://www.postgresql.org).
-
-To try this out, if you have [docker](https://docker.com), you can run a test
-MySQL server very easily:
-
-```bash
-docker pull mysql
-docker run -d --name=mysql-server -p3306:3306 -e MYSQL_ROOT_PASSWORD=my_secret_password mysql
-echo "create database database_name;" | docker exec -i mysql-server mysql -pmy_secret_password
-```
-
-Or, just as easily, you can get going with PostgreSQL:
-```bash
-docker pull postgres
-docker run --name tangos-postgres -e POSTGRES_USER=tangos -e POSTGRES_PASSWORD=my_secret_password -e POSTGRES_DB=database_name -p 5432:5432 -d postgres
-```
-
-To be sure that python can connect to MySQL or PostgreSQL, install the appropriate modules:
-```bash
-pip install PyMySQL # for MySQL
-pip install psycopg2-binary # for PostgreSQL
-```
-
-Tangos can now connect to your test MySQL server using the connection:
-```bash
-export TANGOS_DB_CONNECTION=mysql+pymysql://root:my_secret_password@localhost:3306/database_name
-```
-or for PostgreSQL:
-```bash
-export TANGOS_DB_CONNECTION=postgresql+psycopg2://tangos:my_secret_password@localhost/database_name
-```
-
-You can now use all the tangos tools as normal, and they will populate the MySQL/PostgreSQL database
-instead of a SQLite file.
 
 
 Where next?

docs/rdbms.md

@@ -0,0 +1,60 @@
+Working with different database systems
+=======================================
+
+Tangos is built on sqlalchemy, which means that it is in principle possible to use any database system supported by sqlalchemy. However, different database systems have different features and limitations of which it is worth being aware.
+
+The tangos tests are run with SQLite, mySQL and postgresql. Other databases, while supported by sqlalchemy, have not been directly tested. The following contain some notes on using these different systems.
+
+SQLite
+------
+
+SQLite is the default database. It is simple in the sense that it keeps your entire database within a single file which can easily be transferred to different systems. Additionally, the SQLite driver is included with Python and so it's quick to get started.
+
+There are two major, related drawbacks to SQLite. The first is that the
+
+PostgreSQL and MySQL
+--------------------
+
+PostgreSQL and MySQL are both server-based systems, and as such take a little more effort to set up and maintain. If one exposes PostgreSQL to the outside world, there are potential security implications. One can of course run it on a firewalled computer and manage access appropriately, but this takes some expertise of its own (that will not be covered here). The major advantage is that you can host your data in a single location and allow multiple users to connect.
+
+
+
+MySQL
+-----
+
+MySQL is a server-based system, and as such takes a little more effort to set up. The advantage is that you can host your data in a single location and allow multiple users to connect. Additionally, it is able to cope much better with complex parallel writes than SQLite.
+
+For most users, MySQL and PostgreSQL are
+
+To try this out, if you have [docker](https://docker.com), you can run a test
+MySQL server very easily:
+
+```bash
+docker pull mysql
+docker run -d --name=mysql-server -p3306:3306 -e MYSQL_ROOT_PASSWORD=my_secret_password mysql
+echo "create database database_name;" | docker exec -i mysql-server mysql -pmy_secret_password
+```
+
+Or, just as easily, you can get going with PostgreSQL:
+```bash
+docker pull postgres
+docker run --name tangos-postgres -e POSTGRES_USER=tangos -e POSTGRES_PASSWORD=my_secret_password -e POSTGRES_DB=database_name -p 5432:5432 -d postgres
+```
+
+To be sure that python can connect to MySQL or PostgreSQL, install the appropriate modules:
+```bash
+pip install PyMySQL # for MySQL
+pip install psycopg2-binary # for PostgreSQL
+```
+
+Tangos can now connect to your test MySQL server using the connection:
+```bash
+export TANGOS_DB_CONNECTION=mysql+pymysql://root:my_secret_password@localhost:3306/database_name
+```
+or for PostgreSQL:
+```bash
+export TANGOS_DB_CONNECTION=postgresql+psycopg2://tangos:my_secret_password@localhost/database_name
+```
+
+You can now use all the tangos tools as normal, and they will populate the MySQL/PostgreSQL database
+instead of a SQLite file.

setup.py

@@ -21,7 +21,8 @@
     'hupper',
     'scipy >= 0.14.0',
     'more_itertools >= 8.0.0',
-    'matplotlib >= 3.0.0' # for web interface
+    'matplotlib >= 3.0.0', # for web interface
+    'tqdm >= 4.59.0'
     ]
 
 tests_require = [

tangos/config.py

@@ -87,6 +87,11 @@
 # Default rtol for assert_amost_equal when using the diff tool
 diff_default_rtol = 1e-3
 
+
+# Database import: how many rows to copy at a time, and when to issue a commit
+DB_IMPORT_CHUNK_SIZE = 10
+DB_IMPORT_COMMIT_AFTER_CHUNKS = 500
+
 try:
     from .config_local import *
 except:

tangos/core/__init__.py

@@ -68,6 +68,7 @@ def get_default_engine() -> sqlalchemy.engine.Engine:
 Index("halo_finder_index", SimulationObjectBase.__table__.c.finder_id)
 Index("haloproperties_creator_index", HaloProperty.__table__.c.creator_id)
 Index("halolink_index", HaloLink.__table__.c.halo_from_id)
+Index("halolink_bidirectional_index", HaloLink.__table__.c.halo_to_id, HaloLink.__table__.c.halo_from_id)
 Index("named_halolink_index", HaloLink.__table__.c.relation_id, HaloLink.__table__.c.halo_from_id)
 
 

tangos/core/halo.py

@@ -1,5 +1,5 @@
 import numpy as np
-from sqlalchemy import Column, ForeignKey, Integer, orm, types
+from sqlalchemy import BigInteger, Column, ForeignKey, Integer, orm, types
 from sqlalchemy.orm import Session, backref, relationship
 
 from . import Base, creator, extraction_patterns
@@ -30,9 +30,9 @@ class SimulationObjectBase(Base):
     __tablename__= "halos"
 
     id = Column(Integer, primary_key=True) #the unique ID value of the database object created for this halo
-    halo_number = Column(Integer) #by default this will be the halo's rank in terms of particle count
-    finder_id = Column(UnsignedInteger) #raw halo ID from the halo catalog
-    finder_offset = Column(Integer) #index of halo within halo catalog, primary identifier used when reading catalog/simulation data
+    halo_number = Column(BigInteger) #by default this will be the halo's rank in terms of particle count
+    finder_id = Column(BigInteger) #raw halo ID from the halo catalog
+    finder_offset = Column(BigInteger) #index of halo within halo catalog, primary identifier used when reading catalog/simulation data
     timestep_id = Column(Integer, ForeignKey('timesteps.id'))
     timestep = relationship(TimeStep, backref=backref(
         'objects', order_by=halo_number, cascade_backrefs=False, lazy='dynamic'), cascade='')

tangos/parallel_tasks/testing.py

@@ -0,0 +1,19 @@
+from .message import Message
+
+FILENAME = "parallel_tasks_test_log.txt"
+
+def initialise_log():
+    with open(FILENAME, "w") as f:
+        f.write("")
+
+def get_log():
+    with open(FILENAME) as f:
+        return f.readlines()
+
+def log(message):
+    ServerLogMessage(message).send(0)
+
+class ServerLogMessage(Message):
+    def process(self):
+        with open(FILENAME, "a") as f:
+            f.write(f"[{self.source:d}] {self.contents:s}\r\n")

tangos/relation_finding/multi_hop.py

@@ -13,6 +13,8 @@
 
 from .. import config, core, temporary_halolist
 from ..config import DOUBLE_PRECISION
+from ..log import logger
+from ..util.timing_monitor import TimingMonitor
 from .one_hop import HopStrategy
 
 
@@ -89,6 +91,7 @@ def __init__(self, halo_from, nhops_max=None, directed=None, target=None,
         self._combine_routes = combine_routes
         self._debug_output = False # set to True to see information about discovered links as hops progress
 
+        self.timing_monitor = TimingMonitor()
     def temp_table(self):
         """Execute the strategy and return results as a temp_table (see temporary_halolist module)"""
         if self._all is None:
@@ -119,7 +122,10 @@ def _execute_query(self):
         with self._manage_temp_table():
             self._generate_multihop_results()
             try:
-                results = self._order_query(self._generate_query(halo_ids_only=False)).all()
+                q = self._order_query(self._generate_query(halo_ids_only=False))
+                results = q.all()
+                # NB the time here seems to be mainly making the ORM objects
+                # rather than the query itself
             except sqlalchemy.exc.ResourceClosedError:
                 results = []
 
@@ -230,6 +236,7 @@ def _create_temp_table(self):
         )
 
         self._table_index = Index('temp.source_id_index_' + rstr, multi_hop_link_table.c.source_id, multi_hop_link_table.c.nhops)
+        self._table_nhop_index = Index('temp.nhop_index_' + rstr, multi_hop_link_table.c.nhops)
 
         self._table = multi_hop_link_table
         self._prelim_table = multi_hop_link_prelim_table
@@ -252,12 +259,15 @@ def _seed_temp_table(self):
 
     def _make_hops(self):
         for i in range(0, self.nhops_max):
-            self._nhops_taken = i
-            generated_count = self._generate_next_level_prelim_links(i)
-            if generated_count != 0:
-                filtered_count = self._filter_prelim_links_into_final()
-            else:
-                filtered_count = 0
+            with self.timing_monitor(self):
+                self._nhops_taken = i
+                generated_count = self._generate_next_level_prelim_links(i)
+                if generated_count != 0:
+                    filtered_count = self._filter_prelim_links_into_final()
+                else:
+                    filtered_count = 0
+
+            # for performance info: self.timing_monitor.summarise_timing(logger)
 
             if self._hopping_finished(filtered_count):
                 break
@@ -267,7 +277,7 @@ def _hopping_finished(self, filtered_count):
         return filtered_count==0
 
     def _generate_next_level_prelim_links(self, from_nhops=0):
-
+        self.timing_monitor.mark('prelim-insert')
         new_weight = self._table.c.weight * core.halo_data.HaloLink.weight
 
         recursion_query = \
@@ -289,6 +299,7 @@ def _generate_next_level_prelim_links(self, from_nhops=0):
 
         num_inserted = self._connection.execute(insert).rowcount
 
+        self.timing_monitor.mark('prelim-thin')
         if self._combine_routes:
             # Ideally, before self._prelim_table.insert(), one would adapt recursion_query to return the argmax of
             # new_weight, grouped by halo_to_id and source_id. That could have been achieved using:
@@ -327,6 +338,7 @@ def _debug_print_links(self, tab):
             print(f"[s{row[2]} i{row[4]}] {row[0].path} -> {row[1].path} w={row[3]:.2f}")
 
     def _filter_prelim_links_into_final(self):
+        self.timing_monitor.mark('final-insert')
         if self._debug_output:
             print()
             print(f"[{self._nhops_taken}] Preliminary links:")
@@ -346,9 +358,10 @@ def _filter_prelim_links_into_final(self):
                                              q)).rowcount
 
         if self._debug_output:
-            print(f"[{self._nhops_taken}] Accepted links:")
+            logger.info(f"[{self._nhops_taken}] Accepted links:")
             self._debug_print_links(self._table)
 
+        self.timing_monitor.mark('final-thin')
         self._connection.execute(self._prelim_table.delete())
         return added_rows