@@ -19,19 +19,23 @@ SELECT protobuf_query_multi('MyProto:some_repeated_field[*].some_map[*].some_fie
19
19
20
20
## Why put protobufs in a database?
21
21
22
- Protobufs in a database is in many ways similar to JSON in a database.
23
- The main advantage of protobufs is a compact and efficient representation.
22
+ Protobufs in a database is in many ways similar to JSON in a database:
23
+ - less code needed to convert data structures into database rows and back
24
+ - less alter tables required as data fields are added and removed
25
+
26
+ The main advantage of protobufs over JSON is a compact and efficient representation.
24
27
The main advantage of JSON is human-readability without extra steps.
25
28
26
29
### What about storing protobufs as JSON?
27
30
28
31
Protobufs have a well-defined JSON representation supported by most implementations.
29
- Using that for storage is a valid strategy if efficiency is not a major concern.
32
+ Using that for storage (as [ ` json ` or ` jsonb ` ] ( https://www.postgresql.org/docs/current/datatype-json.html ) )
33
+ is a valid strategy if efficiency is not a major concern.
30
34
31
35
Note that JSON protobufs store fields by name instead of by number,
32
36
which means that field renames will break backwards- and forwards-compatibility.
33
37
34
- ## Installation
38
+ ## Installation from source
35
39
36
40
Requires Postgres 11 or newer.
37
41
@@ -53,6 +57,19 @@ Then in Postgres:
53
57
CREATE EXTENSION postgres_protobuf;
54
58
```
55
59
60
+ ## Installation from binary release
61
+
62
+ Prebuilt binaries are [ here] ( https://github.com/mpartel/postgres-protobuf/releases ) .
63
+
64
+ Currently they are built with and tested on Ubuntu 18.04 only.
65
+ You can build your own binary package from source with ` make dist ` .
66
+
67
+ On Ubuntu, install the contents of the binary package like this:
68
+ - copy the contents of ` lib/ ` to ` /usr/lib/postgresql/11/lib/ `
69
+ - copy the contents of ` extension/ ` to ` /usr/share/postgresql/11/extension/ `
70
+
71
+ Other distros may have those Postgres directories elsewhere.
72
+
56
73
## Usage
57
74
58
75
First, you need to tell the extension about your protobuf schema.
@@ -66,6 +83,7 @@ VALUES ('default', contents_of_file)
66
83
```
67
84
68
85
where ` contents_of_file ` is a Postgres [ byte array] ( https://www.postgresql.org/docs/current/datatype-binary.html ) .
86
+ Commit the transaction if you're in one.
69
87
70
88
Now you can query protobufs described by your schema like this:
71
89
@@ -105,8 +123,8 @@ before relying on it too heavily in a design.
105
123
106
124
This extension is fairly new and written in C++, so some caution is warranted.
107
125
It's may be unwise to give it untrusted queries or protobuf data that you haven't
108
- parsed and reserialized first. Conversion to/from JSON should be safe, but see
109
- the note about memory management below.
126
+ parsed and reserialized first. Conversion to/from JSON should be safer since it
127
+ thinly wraps the well-tested protobuf library, but see the note about memory management below.
110
128
111
129
### Performance
112
130
@@ -130,12 +148,14 @@ This memory might not be properly accounted for by Postgres's
130
148
memory management and monitoring systems.
131
149
132
150
While care has been taken to avoid memory leaks and to tolerate memory exhaustion at any point,
133
- this has not been rigorously tested.
151
+ this has not been rigorously tested, and I am unsure whether the protobuf library always cleans up
152
+ correctly after a ` bad_alloc ` exception.
134
153
135
154
In the current version, protobuf [ map] ( https://developers.google.com/protocol-buffers/docs/proto3#maps )
136
- values are buffered before being scanned. This means that huge protobufs whose bulk is under a
155
+ values are buffered before being scanned by queries . This means that huge protobufs whose bulk is under a
137
156
highly nested map may take a lot of memory. Note that an attacker can easily construct such a
138
- protobuf if the schema has recursion involving a map.
157
+ protobuf if the schema has recursion involving a map. This caveat applies only to queries,
158
+ not to JSON conversion.
139
159
140
160
Other than the above case, memory use is linear, or roughly
141
161
` O(|descriptor sets| + |largest protobuf queried| + |result set|) ` .
@@ -144,24 +164,24 @@ Other than the above case, memory use is linear, or roughly
144
164
145
165
Protobuf versions 2 and 3 should both work,
146
166
but [ groups] ( https://developers.google.com/protocol-buffers/docs/proto#groups )
147
- (a deprecated protobuf feature )
148
- are currently not supported and will cause the query to fail if encountered.
167
+ (which have been deprecated for a long time )
168
+ are currently not supported and will cause queries to fail if encountered.
149
169
150
170
Note that proto3 does not store default values for fields.
151
171
The current implementation returns no result for missing values.
152
172
This means that e.g. a numeric field whose value is 0 will not show up in results.
153
- A future version may change this default behaviour .
173
+ A future version may change this default.
154
174
155
175
This extension has so far only been tested on AMD64.
156
176
157
177
### Advanced operations
158
178
159
- The current version has no support for modifying the contents of protobufs,
179
+ The current version does not provide functions for modifying the contents of protobufs,
160
180
and there are no concrete plans to add such functonality.
161
181
There are also no concrete plans to significantly extend the query language.
162
182
163
- If you do need these features occasionally, consider converting to JSON,
164
- for which Postgres has [ a wide range of operations] ( https://www.postgresql.org/docs/current/functions-json.html ) .
183
+ If you do need these features occasionally, consider converting to JSON and back ,
184
+ since Postgres has [ a wide range of JSON operations] ( https://www.postgresql.org/docs/current/functions-json.html ) .
165
185
166
186
## Comparison with pg_protobuf
167
187
0 commit comments