Make serde optional

[iancoleman]

Following on from #42 (comment)

we should make serde optional. (optional=true). However, that can be done in a separate PR.

and from #42 (review)

and also get rid of SerdeSecret and probably serde_impl module. But that should be a separate PR.