Configurable database backend in CGAT pipelines

[MikeDMorgan]

Currently CSV2DB allows the database backend to be set to either sqlite, mysql or postgres. There is currently no option in either conf.py or pipeline.ini to allow the user to set the database backend. This is particularly important for large single cell sequencing experiments where the number of samples exceeds the SQLite variable limit that is set at compilation.

I propose to add a new parameter to pipeline.ini that allows the backend to be set, by uses sqlite by default (current behaviour).

I will start with pipeline_mapping.py as this is where the problems are largely arising.

[Acribbs]

@MikeDMorgan was any progress made on this?

[MikeDMorgan]

It was partly resolved for pipeline_mapping.py at least, but then it got lost in the migration to Py3. I'm not sure where it stands currently, particularly with the move to SQLAlchemy

[Acribbs]

Ok, I will leave it open for the moment and when the new branches for python 3.6 are merged I can have a look at it.