Update documentation with new features.

We have a lot of new features to document. This is a first patch about that,
some more work is to be done. That said, it's better than nothing already.

Author: dimitri

docs/index.rst

@@ -22,6 +22,10 @@ Welcome to pgloader's documentation!
    ref/mysql
    ref/sqlite
    ref/mssql
+   ref/pgsql
+   ref/pgsql-citus-target
+   ref/pgsql-redshift-source
+   ref/pgsql-redshift-target
    ref/transforms
    bugreport
 

docs/intro.rst

@@ -17,6 +17,14 @@ pgloader knows how to read data from different kind of sources:
     * SQLite
     * MySQL
     * MS SQL Server
+    * PostgreSQL
+    * Redshift
+
+pgloader knows how to target different products using the PostgresQL Protocol:
+
+  * PostgreSQL
+  * `Citus <https://www.citusdata.com>`_
+  * Redshift
 
 The level of automation provided by pgloader depends on the data source
 type. In the case of CSV and Fixed Format files, a full description of the

docs/pgloader.rst

@@ -154,6 +154,18 @@ Those options are meant to tweak `pgloader` behavior when loading data.
     machine code) another version of itself, usually a newer one like a very
     recent git checkout.
 
+  * `--no-ssl-cert-verification`
+
+    Uses the OpenSSL option to accept a locally issued server-side
+    certificate, avoiding the following error message::
+
+      SSL verify error: 20 X509_V_ERR_UNABLE_TO_GET_ISSUER_CERT_LOCALLY
+
+    The right way to fix the SSL issue is to use a trusted certificate, of
+    course. Sometimes though it's useful to make progress with the pgloader
+    setup while the certificate chain of trust is being fixed, maybe by
+    another team. That's when this option is useful.
+
 Command Line Only Operations
 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
 
@@ -552,6 +564,22 @@ queries from a SQL file. Implements support for PostgreSQL dollar-quoting
 and the `\i` and `\ir` include facilities as in `psql` batch mode (where
 they are the same thing).
 
+AFTER CREATE SCHEMA DO
+^^^^^^^^^^^^^^^^^^^^^^
+
+Same format as *BEFORE LOAD DO*, the dollar-quoted queries found in that
+section are executed once the schema has been craeted by pgloader, and
+before the data is loaded. It's the right time to ALTER TABLE or do some
+custom implementation on-top of what pgloader does, like maybe partitioning.
+
+AFTER CREATE SCHEMA EXECUTE
+^^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+Same behaviour as in the *AFTER CREATE SCHEMA DO* clause. Allows you to read
+the SQL queries from a SQL file. Implements support for PostgreSQL
+dollar-quoting and the `\i` and `\ir` include facilities as in `psql` batch
+mode (where they are the same thing).
+
 Connection String
 ^^^^^^^^^^^^^^^^^
 

docs/ref/mysql.rst

@@ -1,10 +1,9 @@
 Migrating a MySQL Database to PostgreSQL
 ========================================
 
-This command instructs pgloader to load data from a database connection. The
-only supported database source is currently *MySQL*, and pgloader supports
-dynamically converting the schema of the source database and the indexes
-building.
+This command instructs pgloader to load data from a database connection.
+pgloader supports dynamically converting the schema of the source database
+and the indexes building.
 
 A default set of casting rules are provided and might be overloaded and
 appended to by the command.
@@ -609,6 +608,14 @@ Date::
     to timestamptz drop default
 	using zero-dates-to-null
 
+  type datetime with extra on update current timestamp when not null
+    to timestamptz drop not null drop default
+       using zero-dates-to-null
+
+  type datetime with extra on update current timestamp
+    to timestamptz drop default
+       using zero-dates-to-null
+
   type timestamp when default "0000-00-00 00:00:00" and not null
     to timestamptz drop not null drop default
 	using zero-dates-to-null

docs/ref/pgsql-citus-target.rst

@@ -0,0 +1,77 @@
+Migrating a PostgreSQL Database to Citus
+========================================
+
+This command instructs pgloader to load data from a database connection.
+Automatic discovery of the schema is supported, including build of the
+indexes, primary and foreign keys constraints. A default set of casting
+rules are provided and might be overloaded and appended to by the command.
+
+Automatic distribution column backfilling is supported, either from commands
+that specify what is the distribution column in every table, or only in the
+main table, then relying on foreign key constraints to discover the other
+distribution keys.
+
+Here's a short example of migrating a database from a PostgreSQL server to
+another:
+
+::
+
+   load database
+   from pgsql:///hackathon
+   into pgsql://localhost:9700/dim
+
+   with include drop, reset no sequences
+
+   cast column impressions.seen_at to "timestamp with time zone"
+
+   distribute companies using id
+   -- distribute campaigns using company_id
+   -- distribute ads using company_id from campaigns
+   -- distribute clicks using company_id from ads, campaigns
+   -- distribute impressions using company_id from ads, campaigns
+   ;
+
+Everything works exactly the same way as when doing a PostgreSQL to
+PostgreSQL migration, with the added fonctionality of this new `distribute`
+command.
+
+Distribute Command
+^^^^^^^^^^^^^^^^^^
+
+The distribute command syntax is as following::
+
+  distribute <table name> using <column name>
+  distribute <table name> using <column name> from <table> [, <table>, ...]
+  distribute <table name> as reference table
+
+When using the distribute command, the following steps are added to pgloader
+operations when migrating the schema:
+
+  - if the distribution column does not exist in the table, it is added as
+    the first column of the table
+
+  - if the distribution column does not exists in the primary key of the
+    table, it is added as the first column of the primary of the table
+
+  - all the foreign keys that point to the table are added the distribution
+    key automatically too, including the source tables of the foreign key
+    constraints
+  
+  - once the schema has been created on the target database, pgloader then
+    issues Citus specific command `create_reference_table()
+    <http://docs.citusdata.com/en/v8.0/develop/api_udf.html?highlight=create_reference_table#create-reference-table>`_
+    and `create_distributed_table()
+    <http://docs.citusdata.com/en/v8.0/develop/api_udf.html?highlight=create_reference_table#create-distributed-table>`_
+    to make the tables distributed
+
+Those operations are done in the schema section of pgloader, before the data
+is loaded. When the data is loaded, the newly added columns need to be
+backfilled from referenced data. pgloader knows how to do that by generating
+a query like the following and importing the result set of such a query
+rather than the raw data from the source table.
+
+Citus Migration: Limitations
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+The way pgloader implements *reset sequence* does not work with Citus at
+this point, so sequences need to be taken care of separately at this point.

docs/ref/pgsql-redshift-source.rst

@@ -0,0 +1,12 @@
+Migrating a Redhift Database to PostgreSQL
+==========================================
+
+This command instructs pgloader to load data from a database connection.
+Automatic discovery of the schema is supported, including build of the
+indexes, primary and foreign keys constraints. A default set of casting
+rules are provided and might be overloaded and appended to by the command.
+
+The command and behavior are the same as when migration from a PostgreSQL
+database source. pgloader automatically discovers that it's talking to a
+Redshift database by parsing the output of the `SELECT version()` SQL query.
+

docs/ref/pgsql-redshift-target.rst

@@ -0,0 +1,10 @@
+Migrating a PostgreSQL Database to Redshift
+===========================================
+
+This command instructs pgloader to load data from a database connection.
+Automatic discovery of the schema is supported, including build of the
+indexes, primary and foreign keys constraints. A default set of casting
+rules are provided and might be overloaded and appended to by the command.
+
+
+TODO: add details about S3 credentials and bucket configuration.