File tree Expand file tree Collapse file tree 2 files changed +27
-0
lines changed Expand file tree Collapse file tree 2 files changed +27
-0
lines changed Original file line number Diff line number Diff line change @@ -128,6 +128,17 @@ spark.readStream \
128
128
.load()
129
129
```
130
130
131
+ ## Error handling
132
+
133
+ When reading data with the connector, any error that occurs - whether it is before any data is read or during a request
134
+ to MarkLogic to read data - is immediately thrown to the user and the read operation ends without any result being
135
+ returned. This is consistent with the design of Spark connectors, where a partition reader is allowed to throw
136
+ exceptions and is not given any mechanism for reporting an error and continuing to read data. The connector will strive
137
+ to provide meaningful context when an error occurs to assist with debugging the cause of the error.
138
+
139
+ In practice, it is expected that most errors will be a result of a misconfiguration. For example, the connection and
140
+ authentication options may be incorrect, or the Optic DSL query may have a syntax error. Any errors that cannot be
141
+ fixed via changes to the options passed to the connector should be reported as new issues to this GitHub repository.
131
142
132
143
## Tuning performance
133
144
Original file line number Diff line number Diff line change @@ -121,6 +121,22 @@ connector and into MarkLogic. This will result 100 new JSON documents in the `st
121
121
The ability to stream data into MarkLogic can make Spark an effective tool for obtaining data from a variety of data
122
122
sources and writing it directly to MarkLogic.
123
123
124
+ ## Error handling
125
+
126
+ The connector may throw an error during one of two phases of operation - before it begins to write data to MarkLogic,
127
+ and during the writing of data to MarkLogic.
128
+
129
+ For the first kind of error, the error will be immediately returned to the user and no data will have been written.
130
+ Such errors are often due to misconfiguration of the connector options and should be fixable.
131
+
132
+ For the second kind of error, the error will eventually be returned to the user, usually within seconds of it
133
+ occurring. The slight delay is due to the asynchronous nature of data being written by the connector. The error will
134
+ be logged by the connector and the write operation will be aborted. Any batches of documents that were written
135
+ successfully prior to the error occurring will still exist in the database.
136
+
137
+ Similar to errors with reading data, the connector will strive to provide meaningful context when an error occurs to
138
+ assist with debugging the cause of the error. Any errors that cannot be fixed via changes to the options passed to the
139
+ connector should be reported as new issues to this GitHub repository.
124
140
125
141
## Tuning performance
126
142
You can’t perform that action at this time.
0 commit comments