Critter is a comprehensive data management solution to essentially replace the previous system of spreadsheets and repositories in ubiquitous use. The idea is to achieve sufficient long-term robustness, openness, accessibility and coherence to permit comprehensive exploitation by both modern Web technologies and ongoing "classical" research.
If you want to interrogate all your available data or subset thereof with questions like:
- What's the average benthic biomass where species A lives?
- What's the variance in species richness in communities with and without the presence of genus X
- What's the difference in median abundance of family Y between grab and trawl sampling methods in summer in area Z?
- How many samples are taken on a winter expedition with vessel B per day on average?
You can do it with only one or few lines of code.
You only have to wait a few seconds for a comprehensive answer.
You will get the same answer every time you ask the same question on the same data. 😉
You will, however, get an immediate updated answer once new data come in.
You should consider moving away from spreadsheet data management and embrace new solutions like this
- Provide a consistent data model
- Enable coherent interrogation of data sets
- Achieve reproducibility and reusability of research results
- Data accessibility for humans and machines (realizing programmatic machine actionability of cited data)
- Provide tools for data ingest and manipulation
- Check data plausibility
- Enforce data standards
- Provide a web-based data exploration and retrieval tool
- User driven design
- Modular architecture
The critter framework consists of the three modules (i) database, (ii) ingest and curation tool, and (iii) visualization and retrieval tool. Take a look at this poster for rationale and background information.
Based on an elaborated, but still simple enough data model , the operational data base is the framework's centerpiece which allows concise and coherent data interrogation and building sustainable services for data, visualization, or story telling of top of it.
To help researchers with data ingestion and curation, an ingest tool provides a graphical user interface from where standard operations are performed. The visualization and retrieval tool is a web-based prototype built to showcase visualization techniques and to enable user-driver demand analysis is currently under evaluation in the ESKP-project RoaStBiW with several stakeholders. You can check it out on janhoo.shinyapps.io or use the function runGitHub() to run it locally
if (!require('shiny')) install.packages("shiny")
if (!require('plyr')) install.packages("plyr")
if (!require('dplyr')) install.packages("dplyr")
if (!require('rlang')) install.packages("rlang")
if (!require('leaflet')) install.packages("leaflet")
if (!require('googleVis')) install.packages("googleVis")
if (!require('xtable')) install.packages("xtable")
if (!require('RcolorBrewer')) install.packages("RcolorBrewer")
shiny::runGitHub("critter", "janhoo", subdir = "shiny")
- PostgreSQL 9.x
- python3 including pyqt5, xlsxwriter, psycopg2, folium
additionally for development
- R version >= 3.4.x
- QT5 Designer
- install postgreSQL 9.x
- then set up the database
cat critterbase/db.ddl.sql | psql -d <YOURDATABASENAME> -U <YOURDBUSER> -h <YOURHOSTe.g.localhost> -p <PORTe.g.5432> -q
- install python3 and the dependencies.
./ingesttool/multiplatform/install_dependencies_<YOUROS>.sh
Windows user pls take a look at ./ingesttool/multiplatform/install_dependencies_win.txt
- Feel embarrassed because this section is a massive void
- Provide example data
- show how to use the ingest tool to connect to the database
- show how to use the ingest tool to produce template .csv files
- show how to use the ingest tool to ingest the example data
- show how to use the ingest tool to perform basic database operations
- show how to use the ingest tool to add root data to look-up-tables
- explain some basic assumptions and principles of the data model
- show how to connect the visualization to the database
- show how to properly cite data retrieved with the visualization tool
- point to further reading and the full fledged documentation
- submit suggestions and bug-reports at: https://github.com/janhoo/benthos/issues
- send a pull request on: https://github.com/janhoo/benthos/
- compose a friendly e-mail to:janmholstein[at]gmail(dot)com
- Jan Holstein - Data Model, Concept, Architecture & Programming
- Paul Kloss & Jan Holstein - Ingest Tool
This work is owned by Jan Holstein and partly by Paul Kloss (ingest tool)
- private use is permitted
- permission required otherwise