Thread-safe and more performant parser based on ExprTK (#387)

* Update to install instructions...
Point release 3.2.1. The version bump is due to using new manylinux2010 images.

* - Added ExprTK based parser and removed muParser. EXPRTK is faster than
  muParser (see labnotes).
- All tests have been passing.
- Tweaks are made to exprtk operators/functions to make sure old
  expressions used scripts remains valid.

* Added function ln(double v) to EXPRTK. Compatibility with muParser. All tests are passing.

* It is working fine now with Vinu model. Needs to fix the sbmlParserReader which uses numVars_.

* Some more cleanup. Testing on travis. Removed moose.Func(); user should
use `moose.Function`. moose.Func was disabled long back.

* GSolve also honors MOOSE_NUM_THREADS ...

* Local tests are passing.  Fixed memory leak issue.  A little refactoring and memory segv at the end of the test is now clean.
Removed unneeded/outdated tests and other scripts.
Added new tests based on Vinu model. It is very comprehensive which
tests multithreaded Ksolve/Gsolve, SBML, and moose.Function
Update to install instructions... (#386)
Point release 3.2.1. The version bump is due to using new manylinux2010 images.

* Thanks to clang++; caught another memory related bug. If this passes, do
a memory profiling.

* Minor cleanup. Removed memory allocation and copying at couple of places.
Still memory leaks in exprtk. I think I can't get rid of them.
Parser is a member object rather than member pointer; still some leak but better version than before.

* Renamed test folder for regular tests. Added custom test which uses 4 threads to run Ksolve.
Modified atol and rtol in assertion in one of the tests. Using parallel version of Ksolve/Gsolve everywhere by setting MOOSE_NUM_THREADS=$(nproc)
Increased `rtol` value to make sure tests passes with BOOST solver as well. When the boost's ode solvers are used, the number generated are slightly different.

* - Added a docker file to run travis pipeline locally.
- Fixed python2 seg-fault in new tests. The segv was because / means int division in python2.

* Converting some rdesigneur scripts to tests; since we need to be able to
plot in nonblocking mode, I've added `block` option in `rdes.display`
function.
Multiple tests using pydoc. They are based on rdesigneur tutorials.
install requires matplotlib for rdesigneur.

* - The seg-fault in rdesigneur tests are due to #388.

* In the ksolve, we partition the interval beforehand now. The
single-threaded and multi-threaded numbers are not the same.  Fixed
tests to accomodate slightly different values resulted n multi-threaded
mode.

* Categorised tests. Also turned all doctest to normal assertion tests.
Moved alpha tests to fixme tests.

* All tests passing locally with both GSL and BOOST.

* Compilation fixes for the OSX build.
One test marked as python3 only. The pyrun expression uses python3
specific division and print function. Causes segfault with python2
(wontfix)

Author: dilawar

.ci/travis_build_linux.sh

@@ -13,10 +13,10 @@ PYTHON2="/usr/bin/python2"
 PYTHON3="/usr/bin/python3"
 
 $PYTHON2 -m pip install pip --upgrade --user
-$PYTHON2 -m pip install libNeuroML pyNeuroML --upgrade --user
+$PYTHON2 -m pip install libNeuroML pyNeuroML python-libsbml --upgrade --user
 
 $PYTHON3 -m pip install pip --upgrade --user
-$PYTHON3 -m pip install libNeuroML pyNeuroML --upgrade --user
+$PYTHON3 -m pip install libNeuroML pyNeuroML python-libsbml --upgrade --user
 
 NPROC=$(nproc)
 MAKE="make -j$NPROC"
@@ -30,15 +30,13 @@ unset PYTHONPATH
 $PYTHON2 -m compileall -q .
 $PYTHON3 -m compileall -q . 
 
-# Python3 with GSL. 
-echo "Python3: Removed python2-networkx and install python3"
-
-# GSL.
+# Python3 with GSL in debug more.
 (
     mkdir -p _GSL_BUILD_PY3 && cd _GSL_BUILD_PY3 && \
         cmake -DPYTHON_EXECUTABLE=$PYTHON3 \
         -DCMAKE_INSTALL_PREFIX=/usr -DDEBUG=ON ..
-    $MAKE && ctest -j$NPROC --output-on-failure -E ".*socket_streamer.*"
+    # Don't run test_long prefix. They take very long time in DEBUG mode.
+    $MAKE && MOOSE_NUM_THREADS=$NPROC ctest -j$NPROC --output-on-failure -E ".*test_long*"
     make install || sudo make install 
     cd /tmp
     $PYTHON3 -c 'import moose;print(moose.__file__);print(moose.version())'
@@ -49,7 +47,7 @@ echo "Python3: Removed python2-networkx and install python3"
     mkdir -p _BOOST_BUILD_PY3 && cd _BOOST_BUILD_PY3 && \
         cmake -DWITH_BOOST_ODE=ON -DPYTHON_EXECUTABLE="$PYTHON3" \
         -DCMAKE_INSTALL_PREFIX=/usr ..
-    $MAKE && ctest -j$NPROC --output-on-failure -E ".*socket_streamer.*"
+    $MAKE && MOOSE_NUM_THREADS=$NPROC ctest -j$NPROC --output-on-failure 
 )
 
 # GSL and python2, failure is allowed
@@ -58,7 +56,7 @@ set +e
     BUILDDIR=_GSL_PY2
     mkdir -p $BUILDDIR && cd $BUILDDIR && \
         cmake -DPYTHON_EXECUTABLE=$PYTHON2 -DCMAKE_INSTALL_PREFIX=/usr ..
-    $MAKE && ctest -j$NPROC --output-on-failure -E ".*socket_streamer.*"
+    $MAKE && MOOSE_NUM_THREADS=$NPROC ctest -j$NPROC --output-on-failure
 )
 set -e
 

.ci/travis_build_osx.sh

@@ -20,7 +20,6 @@
 set -o nounset                              # Treat unset variables as an error
 set -e
 
-# NOTE: On travis, don't enable -j`nproc` option. It may not compile properly.
 NPROC=$(nproc)
 (
     # Make sure not to pick up python from /opt.
@@ -34,17 +33,16 @@ NPROC=$(nproc)
     $PYTHON3 -m pip install pyneuroml --user
 
     mkdir -p _GSL_BUILD && cd _GSL_BUILD \
-        && cmake -DDEBUG=ON \
-        -DPYTHON_EXECUTABLE=$PYTHON3 \
+        && cmake -DPYTHON_EXECUTABLE=$PYTHON3 \
         ..
     make pylint -j$NPROC
-    make -j$NPROC && ctest --output-on-failure -$NPROC
+    make -j$NPROC && MOOSE_NUM_THREAD=$NPROC ctest --output-on-failure -j$NPROC
 
     cd .. # Now with boost.
     mkdir -p _BOOST_BUILD && cd _BOOST_BUILD \
-        && cmake -DWITH_BOOST_ODE=ON -DDEBUG=ON \
+        && cmake -DWITH_BOOST_ODE=ON \
         -DPYTHON_EXECUTABLE=`which python3` ..
 
-    make -j$NPROC && ctest -j$NPROC --output-on-failure 
+    make -j$NPROC && MOOSE_NUM_THREAD=$NPROC ctest -j$NPROC --output-on-failure 
 )
 set +e

.docker/travis/Dockerfile

@@ -5,6 +5,8 @@ matrix:
   include:
   - os: linux
     dist: bionic
+    services:
+        - xvfb
   - os: osx
     osx_image: xcode11.3
 
@@ -21,6 +23,7 @@ addons:
   apt:
     update: true
 
+
 before_script:
   - if [[ "$TRAVIS_OS_NAME" == "osx" ]]; then nvm get head || true; fi
   - if [[ "$TRAVIS_OS_NAME" == "osx" ]]; then ./.ci/travis_prepare_osx.sh; fi

.docker/travis/Makefile

@@ -4,10 +4,9 @@ cmake_minimum_required(VERSION 2.8 FATAL_ERROR)
 project(PyMOOSE)
 
 # cmake related macros.
-set(CMAKE_MODULE_PATH "${CMAKE_SOURCE_DIR}/cmake")
+set(CMAKE_MODULE_PATH "${CMAKE_CURRENT_SOURCE_DIR}/cmake")
 include(CheckCXXCompiler.cmake)
 include(CheckIncludeFileCXX)
-include(FindPythonModule)
 
 # We find python executable here. Though mainly used inside pymoose. 
 # FIXME: When cmake 3.12 is widely available use the following:
@@ -19,12 +18,17 @@ if(NOT PYTHONINTERP_FOUND)
   find_package(PythonInterp 2.7)
 endif()
 
+# Disable rpath on OSX.
+if(APPLE)
+  set(CMAKE_MACOSX_RPATH OFF)
+endif()
+
 
 # NOTE: version should be changed in setup.py file.
 # If moose version is not given, use setup.py file to get the default version.
 if(NOT VERSION_MOOSE)
     execute_process(COMMAND ${PYTHON_EXECUTABLE} setup.py --version
-        WORKING_DIRECTORY ${CMAKE_SOURCE_DIR}
+        WORKING_DIRECTORY ${CMAKE_CURRENT_SOURCE_DIR}
         OUTPUT_VARIABLE VERSION_MOOSE
         OUTPUT_STRIP_TRAILING_WHITESPACE)
 endif()
@@ -69,7 +73,6 @@ option(WITH_ASAN            "Use AddressSanitizer in DEBUG mode." OFF)
 
 # Default definitions.
 add_definitions(-DUSE_GENESIS_PARSER)
-
 if(DEBUG OR "${CMAKE_BUILD_TYPE}" STREQUAL "Debug")
     message(STATUS "Building for Debug/Unit testing")
     add_definitions(-DDO_UNIT_TESTS -O)
@@ -109,6 +112,7 @@ link_directories(${CMAKE_BINARY_DIR})
 add_library(libmoose SHARED basecode/main.cpp)
 add_executable(moose.bin basecode/main.cpp)
 
+
 ################################### SETUP BUILD ################################
 
 # default include paths.
@@ -225,10 +229,6 @@ if(WITH_MPI)
     endif()
 endif(WITH_MPI)
 
-# Always use private version of muparser. We have some custom edits.
-add_subdirectory(external/muparser)
-list(APPEND MOOSE_LIBRARIES muparser)
-
 # Add subdirectroeis
 add_subdirectory(basecode)
 add_subdirectory(msg)
@@ -368,9 +368,8 @@ endif()
 
 ############################ CTEST ######################################
 include( CTest )
-# If CTEST_OUTPUT_ON_FAILURE environment variable is set, the output is printed
-# onto the console if a test fails.
-set(ENV{CTEST_OUTPUT_ON_FAILURE} ON)
+set(CTEST_NIGHTLY_START_TIME "05:30:00 UTC")
+set(CTEST_SUBMIT_URL "http://my.cdash.org/submit.php?project=moose")
 
 if(DEBUG OR "${CMAKE_BUILD_TYPE}" STREQUAL "Debug")
     # Run this test in debug mode. In Release mode, this does not do anything.
@@ -380,33 +379,46 @@ if(DEBUG OR "${CMAKE_BUILD_TYPE}" STREQUAL "Debug")
     add_test(NAME moose.bin-raw-run COMMAND moose.bin -u -q)
 endif()
 
-## PyMOOSE tests.
-set(PYMOOSE_TEST_DIRECTORY ${CMAKE_SOURCE_DIR}/tests/python)
-
-# Collect all python script in tests folder and run them.
+# PyMOOSE tests. These tests complete in reasonable time and should run by
+# default.
+set(PYMOOSE_TEST_DIRECTORY ${CMAKE_CURRENT_SOURCE_DIR}/tests/py_moose)
 file(GLOB PY_TEST_SCRIPTS "${PYMOOSE_TEST_DIRECTORY}/test_*.py" )
-
-
-# Run python tests.
-foreach( _test_script ${PY_TEST_SCRIPTS} )
-    get_filename_component( _test_name ${_test_script} NAME_WE)
-    add_test( NAME ${_test_name}
+foreach(_test_script ${PY_TEST_SCRIPTS} )
+    get_filename_component(_name_we ${_test_script} NAME_WE)
+    set(_test_name "pymoose_${_name_we}")
+    add_test(NAME ${_test_name}
         COMMAND ${PYTHON_EXECUTABLE} ${_test_script}
         WORKING_DIRECTORY ${PYMOOSE_TEST_DIRECTORY}
         )
-     set_tests_properties( ${_test_name}
+     set_tests_properties(${_test_name}
          PROPERTIES ENVIRONMENT "PYTHONPATH=${CMAKE_CURRENT_BINARY_DIR}/python"
         )
 endforeach( )
 
-# Alpha tests. These should not run by default.
-set(MOOSE_ALPHA_TEST_DIRECTORY ${CMAKE_CURRENT_SOURCE_DIR}/tests/alpha)
-file(GLOB PY_ALPHA_TEST_SCRIPTS "${PYMOOSE_ALPHA_TEST_DIRECTORY}/*.py" )
-foreach( _test_script ${PY_ALPHA_TEST_SCRIPTS} )
-    get_filename_component( _test_name ${_test_script} NAME_WE)
+# rdesigneur tests. These tests require matplotlib.
+set(RDES_TEST_DIRECTORY ${CMAKE_CURRENT_SOURCE_DIR}/tests/py_rdesigneur) 
+file(GLOB RDES_TEST_SCRIPTS "${RDES_TEST_DIRECTORY}/test_*.py" )
+foreach(_test_script ${RDES_TEST_SCRIPTS})
+    get_filename_component(_name_we ${_test_script} NAME_WE)
+    set(_test_name "rdes_${_name_we}")
+    add_test(NAME ${_test_name}
+        COMMAND ${PYTHON_EXECUTABLE} ${_test_script}
+        WORKING_DIRECTORY ${RDES_TEST_DIRECTORY}
+        )
+    set_tests_properties(${_test_name} PROPERTIES 
+        ENVIRONMENT "PYTHONPATH=${CMAKE_CURRENT_BINARY_DIR}/python;MOOSE_NUM_THREADS=4"
+        )
+endforeach()
+
+# FIXME TESTS. These should not run by default. We need to fix them.
+set(PYMOOSE_FIXME_TEST_DIRECTORY ${CMAKE_CURRENT_SOURCE_DIR}/tests/fixme)
+file(GLOB PY_FIXME_TEST_SCRIPTS "${PYMOOSE_FIXME_TEST_DIRECTORY}/*.py" )
+foreach( _test_script ${PY_FIXME_TEST_SCRIPTS} )
+    get_filename_component( _name_we ${_test_script} NAME_WE)
+    set(_test_name "alpha_${_name_we}")
     add_test( NAME ${_test_name}
         COMMAND ${PYTHON_EXECUTABLE} ${_test_script}
-	CONFIGURATIONS Devel
+        CONFIGURATIONS alpha
 	WORKING_DIRECTORY ${PYMOOSE_ALPHA_TEST_DIRECTORY}
         )
      set_tests_properties( ${_test_name}
@@ -415,41 +427,19 @@ foreach( _test_script ${PY_ALPHA_TEST_SCRIPTS} )
 endforeach( )
 
 # Regression and github issues. These should not run by default.
-set(PYMOOSE_ISSUES_DIRECTORY ${CMAKE_CURRENT_SOURCE_DIR}/tests/issues)
+set(PYMOOSE_ISSUES_DIRECTORY ${CMAKE_CURRENT_SOURCE_DIR}/devel/issues)
 file(GLOB PY_ISSUES_SCRIPTS "${PYMOOSE_ISSUES_DIRECTORY}/*.py" )
-foreach( _test_script ${PY_ISSUES_SCRIPTS} )
+foreach(_test_script ${PY_ISSUES_SCRIPTS})
     get_filename_component( _test_name ${_test_script} NAME_WE)
-    add_test( NAME ${_test_name}
+    add_test(NAME ${_test_name}
         COMMAND ${PYTHON_EXECUTABLE} ${_test_script}
 	CONFIGURATIONS Devel
 	WORKING_DIRECTORY ${PYMOOSE_ISSUES_DIRECTORY}
         )
-     set_tests_properties( ${_test_name}
+     set_tests_properties(${_test_name}
          PROPERTIES ENVIRONMENT "PYTHONPATH=${CMAKE_CURRENT_BINARY_DIR}/python"
         )
-endforeach( )
-
-# add pylint target.
-set(PYSCRIPTS_DIR ${CMAKE_CURRENT_SOURCE_DIR}/python )
-file(GLOB_RECURSE PY_SCRIPTS  "python/*.py")
-add_custom_target( pylint )
-foreach( _py_script ${PY_SCRIPTS} )
-    get_filename_component( _py_name ${_py_script} NAME_WE)
-    file( READ ${_py_script} pytext)
-    string(MD5 _md5 "${pytext}")
-    set(TGT_NAME "${_py_name}-${_md5}" )
-    set(PYLINT_OPTIONS --disable=no-member --disable=no-name-in-module
-            --disable=invalid-unary-operand-type
-            --disable=import-error
-            --disable=no-method-argument
-        )
-    add_custom_target( ${TGT_NAME}
-        COMMAND ${PYTHON_EXECUTABLE} -m pylint -E ${PYLINT_OPTIONS} ${_py_script}
-        COMMENT "Running pylint on ${_py_script}"
-        WORKING_DIRECTORY ${CMAKE_CURRENT_SOURCE_DIR}
-        )
-    add_dependencies(pylint ${TGT_NAME} )
-endforeach( )
+endforeach()
 
 ############################ CPACK ######################################
 # This is not maintained anymore since packaging is done using OpenBuildService 
@@ -461,9 +451,10 @@ set(PYMOOSE_SDIST_FILE ${CMAKE_BINARY_DIR}/pymoose-${VERSION_MOOSE}.tar.gz)
 add_custom_target(sdist
     COMMAND ${PYTHON_EXECUTABLE} setup.py sdist --formats=gztar 
         -d ${CMAKE_BINARY_DIR}
-    WORKING_DIRECTORY ${CMAKE_SOURCE_DIR}
+    WORKING_DIRECTORY ${CMAKE_CURRENT_SOURCE_DIR}
     COMMENT "Creating sdist ${PYMOOSE_SDIST_FILE}" 
     VERBATIM)
+
 add_custom_target(upload_test DEPENDS sdist
     COMMAND ${PYTHON_EXECUTABLE} -m twine upload 
         --user $ENV{PYMOOSE_PYPI_USER} --password $ENV{PYMOOSE_PYPI_PASSWORD}
@@ -478,3 +469,35 @@ add_custom_target(upload DEPENDS sdist
         ${PYMOOSE_SDIST_FILE}
     COMMENT "Uploading source distribution to PyPI"
     VERBATIM)
+
+####################### DEVELOPMENT ########################################
+
+# PYLINT target.
+set(PYSCRIPTS_DIR ${CMAKE_CURRENT_SOURCE_DIR}/python)
+file(GLOB_RECURSE PY_SCRIPTS  "python/*.py")
+add_custom_target(pylint)
+foreach(_py_script ${PY_SCRIPTS})
+    get_filename_component( _py_name ${_py_script} NAME_WE)
+    file( READ ${_py_script} pytext)
+    string(MD5 _md5 "${pytext}")
+    set(TGT_NAME "${_py_name}-${_md5}" )
+    set(PYLINT_OPTIONS --disable=no-member --disable=no-name-in-module
+            --disable=invalid-unary-operand-type
+            --disable=import-error
+            --disable=no-method-argument
+        )
+    add_custom_target( ${TGT_NAME}
+        COMMAND ${PYTHON_EXECUTABLE} -m pylint -E ${PYLINT_OPTIONS} ${_py_script}
+        COMMENT "Running pylint on ${_py_script}"
+        WORKING_DIRECTORY ${CMAKE_CURRENT_SOURCE_DIR}
+        )
+    add_dependencies(pylint ${TGT_NAME} )
+endforeach( )
+
+# Replicate Travis-CI building using docker
+add_custom_target(docker
+    COMMAND docker build -t bhallalab/travis:latest
+        -f ${CMAKE_CURRENT_SOURCE_DIR}/devel/docker/travis/Dockerfile .
+    COMMENT "Replicating Travis-CI building using Docker."
+    WORKING_DIRECTORY ${CMAKE_CURRENT_SOURCE_DIR}
+    VERBATIM)