Skip to content

Commit 224383b

Browse files
rustyrussellrustyrussell
committed
doc: document the sql command.
In particular, we generate the schema part from the plugin itself. Signed-off-by: Rusty Russell <rusty@rustcorp.com.au> Changelog-Added: Plugins: `sql` plugin command to perform server-side complex queries.
1 parent 472d1b2 commit 224383b

File tree

7 files changed

+358
-1
lines changed

7 files changed

+358
-1
lines changed

‎doc/.gitignore

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -5,3 +5,4 @@
55
*.log
66
*.out
77
*.tex
8+
.sqlgen

‎doc/Makefile

Lines changed: 9 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -85,6 +85,7 @@ MANPAGES := doc/lightning-cli.1 \
8585
doc/lightning-setchannel.7 \
8686
doc/lightning-sendcustommsg.7 \
8787
doc/lightning-signmessage.7 \
88+
doc/lightning-sql.7 \
8889
doc/lightning-staticbackup.7 \
8990
doc/lightning-txprepare.7 \
9091
doc/lightning-txdiscard.7 \
@@ -149,6 +150,14 @@ $(MANPAGES): doc/%: doc/%.md tools/md2man.sh version_gen.h
149150
$(MANPAGES): $(FORCE)
150151
$(MARKDOWN_WITH_SCHEMA): $(FORCE)
151152

153+
# Use awk for preamble, then again for post, with the new docs in the middle.
154+
# We can't put plugins/sql in deps directly, since they all get sha256!
155+
doc/.sqlgen: plugins/sql
156+
@plugins/sql --print-docs > $@
157+
158+
doc/lightning-sql.7.md: doc/.sqlgen $(FORCE)
159+
@if $(call SHA256STAMP_CHANGED); then $(call VERBOSE, "sql-print-docs $@", awk "/GENERATE-DOC-START/ { print \$$0; exit } { print \$$0 }" < $@ > $@.tmp && cat doc/.sqlgen >> $@.tmp && (awk "/GENERATE-DOC-END/ { PRINT=1 } { if (PRINT) { print \$$0 } }" | grep -v SHA256STAMP) < $@ >> $@.tmp && mv $@.tmp $@ && $(call SHA256STAMP,[comment]: # $(LBRACKET),$(RBRACKET))); else touch $@; fi
160+
152161
doc/protocol-%.svg: test/test_protocol
153162
test/test_protocol --svg < test/commits/$*.script > $@
154163

‎doc/index.rst

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -117,6 +117,7 @@ Core Lightning Documentation
117117
lightning-setchannel <lightning-setchannel.7.md>
118118
lightning-signmessage <lightning-signmessage.7.md>
119119
lightning-signpsbt <lightning-signpsbt.7.md>
120+
lightning-sql <lightning-sql.7.md>
120121
lightning-staticbackup <lightning-staticbackup.7.md>
121122
lightning-stop <lightning-stop.7.md>
122123
lightning-txdiscard <lightning-txdiscard.7.md>

‎doc/lightning-sql.7.md

Lines changed: 315 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,315 @@
1+
lightning-sql -- Command to do complex queries on list commands
2+
===============================================================
3+
4+
SYNOPSIS
5+
--------
6+
7+
**sql** *query*
8+
9+
DESCRIPTION
10+
-----------
11+
12+
The **sql** RPC command runs the given query across a sqlite3 database
13+
created from various list commands.
14+
15+
When tables are accessed, it calls the above commands, so it's no
16+
faster than any other local access (though it goes to great length to
17+
cache `listnodes` and `listchannels`) which then processes the results.
18+
19+
It is, however faster for remote access if the result of the query is
20+
much smaller than the list commands would be.
21+
22+
TREATMENT OF TYPES
23+
------------------
24+
25+
The following types are supported in schemas, and this shows how they
26+
are presented in the database. This matters: a JSON boolean is
27+
represented as an integer in the database, so a query will return 0 or
28+
1, not true or false.
29+
30+
* *hex*. A hex string.
31+
* JSON: a string
32+
* sqlite3: BLOB
33+
34+
* *hash*/*secret*/*pubkey*/*txid*: just like *hex*.
35+
36+
* *msat*/*integer*/*u64*/*u32*/*u16*/*u8*. Normal numbers.
37+
* JSON: an unsigned integer
38+
* sqlite3: INTEGER
39+
40+
* *boolean*. True or false.
41+
* JSON: literal **true** or **false**
42+
* sqlite3: INTEGER
43+
44+
* *number*. A floating point number (used for times in some places).
45+
* JSON: number
46+
* sqlite3: REAL
47+
48+
* *string*. Text.
49+
* JSON: string
50+
* sqlite3: TEXT
51+
52+
* *short\_channel\_id*. A short-channel-id of form 1x2x3.
53+
* JSON: string
54+
* sqlite3: TEXT
55+
56+
TABLES
57+
------
58+
[comment]: # (GENERATE-DOC-START)
59+
The following tables are currently supported:
60+
- `channels` indexed by `short_channel_id` (see lightning-listchannels(7))
61+
- `source` (type `pubkey`, sqltype `BLOB`)
62+
- `destination` (type `pubkey`, sqltype `BLOB`)
63+
- `short_channel_id` (type `short_channel_id`, sqltype `TEXT`)
64+
- `direction` (type `u32`, sqltype `INTEGER`)
65+
- `public` (type `boolean`, sqltype `INTEGER`)
66+
- `amount_msat` (type `msat`, sqltype `INTEGER`)
67+
- `message_flags` (type `u8`, sqltype `INTEGER`)
68+
- `channel_flags` (type `u8`, sqltype `INTEGER`)
69+
- `active` (type `boolean`, sqltype `INTEGER`)
70+
- `last_update` (type `u32`, sqltype `INTEGER`)
71+
- `base_fee_millisatoshi` (type `u32`, sqltype `INTEGER`)
72+
- `fee_per_millionth` (type `u32`, sqltype `INTEGER`)
73+
- `delay` (type `u32`, sqltype `INTEGER`)
74+
- `htlc_minimum_msat` (type `msat`, sqltype `INTEGER`)
75+
- `htlc_maximum_msat` (type `msat`, sqltype `INTEGER`)
76+
- `features` (type `hex`, sqltype `BLOB`)
77+
78+
- `forwards` indexed by `in_channel and in_htlc_id` (see lightning-listforwards(7))
79+
- `in_channel` (type `short_channel_id`, sqltype `TEXT`)
80+
- `in_htlc_id` (type `u64`, sqltype `INTEGER`)
81+
- `in_msat` (type `msat`, sqltype `INTEGER`)
82+
- `status` (type `string`, sqltype `TEXT`)
83+
- `received_time` (type `number`, sqltype `REAL`)
84+
- `out_channel` (type `short_channel_id`, sqltype `TEXT`)
85+
- `out_htlc_id` (type `u64`, sqltype `INTEGER`)
86+
- `style` (type `string`, sqltype `TEXT`)
87+
- `fee_msat` (type `msat`, sqltype `INTEGER`)
88+
- `out_msat` (type `msat`, sqltype `INTEGER`)
89+
- `resolved_time` (type `number`, sqltype `REAL`)
90+
- `failcode` (type `u32`, sqltype `INTEGER`)
91+
- `failreason` (type `string`, sqltype `TEXT`)
92+
93+
- `htlcs` indexed by `short_channel_id and id` (see lightning-listhtlcs(7))
94+
- `short_channel_id` (type `short_channel_id`, sqltype `TEXT`)
95+
- `id` (type `u64`, sqltype `INTEGER`)
96+
- `expiry` (type `u32`, sqltype `INTEGER`)
97+
- `amount_msat` (type `msat`, sqltype `INTEGER`)
98+
- `direction` (type `string`, sqltype `TEXT`)
99+
- `payment_hash` (type `hash`, sqltype `BLOB`)
100+
- `state` (type `string`, sqltype `TEXT`)
101+
102+
- `invoices` indexed by `payment_hash` (see lightning-listinvoices(7))
103+
- `label` (type `string`, sqltype `TEXT`)
104+
- `description` (type `string`, sqltype `TEXT`)
105+
- `payment_hash` (type `hash`, sqltype `BLOB`)
106+
- `status` (type `string`, sqltype `TEXT`)
107+
- `expires_at` (type `u64`, sqltype `INTEGER`)
108+
- `amount_msat` (type `msat`, sqltype `INTEGER`)
109+
- `bolt11` (type `string`, sqltype `TEXT`)
110+
- `bolt12` (type `string`, sqltype `TEXT`)
111+
- `local_offer_id` (type `hash`, sqltype `BLOB`)
112+
- `invreq_payer_note` (type `string`, sqltype `TEXT`)
113+
- `pay_index` (type `u64`, sqltype `INTEGER`)
114+
- `amount_received_msat` (type `msat`, sqltype `INTEGER`)
115+
- `paid_at` (type `u64`, sqltype `INTEGER`)
116+
- `payment_preimage` (type `secret`, sqltype `BLOB`)
117+
118+
- `nodes` indexed by `nodeid` (see lightning-listnodes(7))
119+
- `nodeid` (type `pubkey`, sqltype `BLOB`)
120+
- `last_timestamp` (type `u32`, sqltype `INTEGER`)
121+
- `alias` (type `string`, sqltype `TEXT`)
122+
- `color` (type `hex`, sqltype `BLOB`)
123+
- `features` (type `hex`, sqltype `BLOB`)
124+
- related table `nodes_addresses`
125+
- `row` (reference to `nodes.rowid`, sqltype `INTEGER`)
126+
- `arrindex` (index within array, sqltype `INTEGER`)
127+
- `type` (type `string`, sqltype `TEXT`)
128+
- `port` (type `u16`, sqltype `INTEGER`)
129+
- `address` (type `string`, sqltype `TEXT`)
130+
- `option_will_fund_lease_fee_base_msat` (type `msat`, sqltype `INTEGER`, from JSON object `option_will_fund`)
131+
- `option_will_fund_lease_fee_basis` (type `u32`, sqltype `INTEGER`, from JSON object `option_will_fund`)
132+
- `option_will_fund_funding_weight` (type `u32`, sqltype `INTEGER`, from JSON object `option_will_fund`)
133+
- `option_will_fund_channel_fee_max_base_msat` (type `msat`, sqltype `INTEGER`, from JSON object `option_will_fund`)
134+
- `option_will_fund_channel_fee_max_proportional_thousandths` (type `u32`, sqltype `INTEGER`, from JSON object `option_will_fund`)
135+
- `option_will_fund_compact_lease` (type `hex`, sqltype `BLOB`, from JSON object `option_will_fund`)
136+
137+
- `offers` indexed by `offer_id` (see lightning-listoffers(7))
138+
- `offer_id` (type `hash`, sqltype `BLOB`)
139+
- `active` (type `boolean`, sqltype `INTEGER`)
140+
- `single_use` (type `boolean`, sqltype `INTEGER`)
141+
- `bolt12` (type `string`, sqltype `TEXT`)
142+
- `used` (type `boolean`, sqltype `INTEGER`)
143+
- `label` (type `string`, sqltype `TEXT`)
144+
145+
- `peerchannels` indexed by `peer_id` (see lightning-listpeerchannels(7))
146+
- `peer_id` (type `pubkey`, sqltype `BLOB`)
147+
- `peer_connected` (type `boolean`, sqltype `INTEGER`)
148+
- `state` (type `string`, sqltype `TEXT`)
149+
- `scratch_txid` (type `txid`, sqltype `BLOB`)
150+
- `feerate_perkw` (type `u32`, sqltype `INTEGER`, from JSON object `feerate`)
151+
- `feerate_perkb` (type `u32`, sqltype `INTEGER`, from JSON object `feerate`)
152+
- `owner` (type `string`, sqltype `TEXT`)
153+
- `short_channel_id` (type `short_channel_id`, sqltype `TEXT`)
154+
- `channel_id` (type `hash`, sqltype `BLOB`)
155+
- `funding_txid` (type `txid`, sqltype `BLOB`)
156+
- `funding_outnum` (type `u32`, sqltype `INTEGER`)
157+
- `initial_feerate` (type `string`, sqltype `TEXT`)
158+
- `last_feerate` (type `string`, sqltype `TEXT`)
159+
- `next_feerate` (type `string`, sqltype `TEXT`)
160+
- `next_fee_step` (type `u32`, sqltype `INTEGER`)
161+
- related table `peerchannels_inflight`
162+
- `row` (reference to `peerchannels.rowid`, sqltype `INTEGER`)
163+
- `arrindex` (index within array, sqltype `INTEGER`)
164+
- `funding_txid` (type `txid`, sqltype `BLOB`)
165+
- `funding_outnum` (type `u32`, sqltype `INTEGER`)
166+
- `feerate` (type `string`, sqltype `TEXT`)
167+
- `total_funding_msat` (type `msat`, sqltype `INTEGER`)
168+
- `our_funding_msat` (type `msat`, sqltype `INTEGER`)
169+
- `scratch_txid` (type `txid`, sqltype `BLOB`)
170+
- `close_to` (type `hex`, sqltype `BLOB`)
171+
- `private` (type `boolean`, sqltype `INTEGER`)
172+
- `opener` (type `string`, sqltype `TEXT`)
173+
- `closer` (type `string`, sqltype `TEXT`)
174+
- related table `peerchannels_features`
175+
- `row` (reference to `peerchannels.rowid`, sqltype `INTEGER`)
176+
- `arrindex` (index within array, sqltype `INTEGER`)
177+
- `features` (type `string`, sqltype `TEXT`)
178+
- `funding_pushed_msat` (type `msat`, sqltype `INTEGER`, from JSON object `funding`)
179+
- `funding_local_funds_msat` (type `msat`, sqltype `INTEGER`, from JSON object `funding`)
180+
- `funding_remote_funds_msat` (type `msat`, sqltype `INTEGER`, from JSON object `funding`)
181+
- `funding_fee_paid_msat` (type `msat`, sqltype `INTEGER`, from JSON object `funding`)
182+
- `funding_fee_rcvd_msat` (type `msat`, sqltype `INTEGER`, from JSON object `funding`)
183+
- `to_us_msat` (type `msat`, sqltype `INTEGER`)
184+
- `min_to_us_msat` (type `msat`, sqltype `INTEGER`)
185+
- `max_to_us_msat` (type `msat`, sqltype `INTEGER`)
186+
- `total_msat` (type `msat`, sqltype `INTEGER`)
187+
- `fee_base_msat` (type `msat`, sqltype `INTEGER`)
188+
- `fee_proportional_millionths` (type `u32`, sqltype `INTEGER`)
189+
- `dust_limit_msat` (type `msat`, sqltype `INTEGER`)
190+
- `max_total_htlc_in_msat` (type `msat`, sqltype `INTEGER`)
191+
- `their_reserve_msat` (type `msat`, sqltype `INTEGER`)
192+
- `our_reserve_msat` (type `msat`, sqltype `INTEGER`)
193+
- `spendable_msat` (type `msat`, sqltype `INTEGER`)
194+
- `receivable_msat` (type `msat`, sqltype `INTEGER`)
195+
- `minimum_htlc_in_msat` (type `msat`, sqltype `INTEGER`)
196+
- `minimum_htlc_out_msat` (type `msat`, sqltype `INTEGER`)
197+
- `maximum_htlc_out_msat` (type `msat`, sqltype `INTEGER`)
198+
- `their_to_self_delay` (type `u32`, sqltype `INTEGER`)
199+
- `our_to_self_delay` (type `u32`, sqltype `INTEGER`)
200+
- `max_accepted_htlcs` (type `u32`, sqltype `INTEGER`)
201+
- `alias_local` (type `short_channel_id`, sqltype `TEXT`, from JSON object `alias`)
202+
- `alias_remote` (type `short_channel_id`, sqltype `TEXT`, from JSON object `alias`)
203+
- related table `peerchannels_state_changes`
204+
- `row` (reference to `peerchannels.rowid`, sqltype `INTEGER`)
205+
- `arrindex` (index within array, sqltype `INTEGER`)
206+
- `timestamp` (type `string`, sqltype `TEXT`)
207+
- `old_state` (type `string`, sqltype `TEXT`)
208+
- `new_state` (type `string`, sqltype `TEXT`)
209+
- `cause` (type `string`, sqltype `TEXT`)
210+
- `message` (type `string`, sqltype `TEXT`)
211+
- related table `peerchannels_status`
212+
- `row` (reference to `peerchannels.rowid`, sqltype `INTEGER`)
213+
- `arrindex` (index within array, sqltype `INTEGER`)
214+
- `status` (type `string`, sqltype `TEXT`)
215+
- `in_payments_offered` (type `u64`, sqltype `INTEGER`)
216+
- `in_offered_msat` (type `msat`, sqltype `INTEGER`)
217+
- `in_payments_fulfilled` (type `u64`, sqltype `INTEGER`)
218+
- `in_fulfilled_msat` (type `msat`, sqltype `INTEGER`)
219+
- `out_payments_offered` (type `u64`, sqltype `INTEGER`)
220+
- `out_offered_msat` (type `msat`, sqltype `INTEGER`)
221+
- `out_payments_fulfilled` (type `u64`, sqltype `INTEGER`)
222+
- `out_fulfilled_msat` (type `msat`, sqltype `INTEGER`)
223+
- related table `peerchannels_htlcs`
224+
- `row` (reference to `peerchannels.rowid`, sqltype `INTEGER`)
225+
- `arrindex` (index within array, sqltype `INTEGER`)
226+
- `direction` (type `string`, sqltype `TEXT`)
227+
- `id` (type `u64`, sqltype `INTEGER`)
228+
- `amount_msat` (type `msat`, sqltype `INTEGER`)
229+
- `expiry` (type `u32`, sqltype `INTEGER`)
230+
- `payment_hash` (type `hash`, sqltype `BLOB`)
231+
- `local_trimmed` (type `boolean`, sqltype `INTEGER`)
232+
- `status` (type `string`, sqltype `TEXT`)
233+
- `state` (type `string`, sqltype `TEXT`)
234+
- `close_to_addr` (type `string`, sqltype `TEXT`)
235+
- `last_tx_fee_msat` (type `msat`, sqltype `INTEGER`)
236+
- `direction` (type `u32`, sqltype `INTEGER`)
237+
238+
- `peers` indexed by `id` (see lightning-listpeers(7))
239+
- `id` (type `pubkey`, sqltype `BLOB`)
240+
- `connected` (type `boolean`, sqltype `INTEGER`)
241+
- related table `peers_netaddr`
242+
- `row` (reference to `peers.rowid`, sqltype `INTEGER`)
243+
- `arrindex` (index within array, sqltype `INTEGER`)
244+
- `netaddr` (type `string`, sqltype `TEXT`)
245+
- `remote_addr` (type `string`, sqltype `TEXT`)
246+
- `features` (type `hex`, sqltype `BLOB`)
247+
248+
- `sendpays` indexed by `payment_hash` (see lightning-listsendpays(7))
249+
- `id` (type `u64`, sqltype `INTEGER`)
250+
- `groupid` (type `u64`, sqltype `INTEGER`)
251+
- `partid` (type `u64`, sqltype `INTEGER`)
252+
- `payment_hash` (type `hash`, sqltype `BLOB`)
253+
- `status` (type `string`, sqltype `TEXT`)
254+
- `amount_msat` (type `msat`, sqltype `INTEGER`)
255+
- `destination` (type `pubkey`, sqltype `BLOB`)
256+
- `created_at` (type `u64`, sqltype `INTEGER`)
257+
- `amount_sent_msat` (type `msat`, sqltype `INTEGER`)
258+
- `label` (type `string`, sqltype `TEXT`)
259+
- `bolt11` (type `string`, sqltype `TEXT`)
260+
- `description` (type `string`, sqltype `TEXT`)
261+
- `bolt12` (type `string`, sqltype `TEXT`)
262+
- `payment_preimage` (type `secret`, sqltype `BLOB`)
263+
- `erroronion` (type `hex`, sqltype `BLOB`)
264+
265+
- `transactions` indexed by `hash` (see lightning-listtransactions(7))
266+
- `hash` (type `txid`, sqltype `BLOB`)
267+
- `rawtx` (type `hex`, sqltype `BLOB`)
268+
- `blockheight` (type `u32`, sqltype `INTEGER`)
269+
- `txindex` (type `u32`, sqltype `INTEGER`)
270+
- `locktime` (type `u32`, sqltype `INTEGER`)
271+
- `version` (type `u32`, sqltype `INTEGER`)
272+
- related table `transactions_inputs`
273+
- `row` (reference to `transactions.rowid`, sqltype `INTEGER`)
274+
- `arrindex` (index within array, sqltype `INTEGER`)
275+
- `txid` (type `txid`, sqltype `BLOB`)
276+
- `idx` (type `u32`, sqltype `INTEGER`, from JSON field `index`)
277+
- `sequence` (type `u32`, sqltype `INTEGER`)
278+
- `type` (type `string`, sqltype `TEXT`)
279+
- `channel` (type `short_channel_id`, sqltype `TEXT`)
280+
- related table `transactions_outputs`
281+
- `row` (reference to `transactions.rowid`, sqltype `INTEGER`)
282+
- `arrindex` (index within array, sqltype `INTEGER`)
283+
- `idx` (type `u32`, sqltype `INTEGER`, from JSON field `index`)
284+
- `amount_msat` (type `msat`, sqltype `INTEGER`)
285+
- `scriptPubKey` (type `hex`, sqltype `BLOB`)
286+
- `type` (type `string`, sqltype `TEXT`)
287+
- `channel` (type `short_channel_id`, sqltype `TEXT`)
288+
289+
[comment]: # (GENERATE-DOC-END)
290+
291+
RETURN VALUE
292+
------------
293+
294+
[comment]: # (FIXME: we don't handle this schema in fromschema.py)
295+
On success, an object containing **rows** is returned. It is an array. Each array entry contains an array of values, each an integer, real number, string or *null*, depending on the sqlite3 type.
296+
297+
The object may contain **warning\_db\_failure** if the database fails partway through its operation.
298+
299+
On failure, an error is returned.
300+
301+
AUTHOR
302+
------
303+
304+
Rusty Russell <<rusty@rustcorp.com.au>> is mainly responsible.
305+
306+
SEE ALSO
307+
--------
308+
309+
lightning-listtransactions(7), lightning-listchannels(7), lightning-listpeers(7), lightning-listnodes(7), lightning-listforwards(7).
310+
311+
RESOURCES
312+
---------
313+
314+
Main web site: <https://github.com/ElementsProject/lightning>
315+
[comment]: # ( SHA256STAMP:93309f8c45ea3aa153bfd8822f2748c1254812d41a408de39bacefa292e11374)

‎doc/schemas/sql.request.json

Lines changed: 12 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,12 @@
1+
{
2+
"$schema": "http://json-schema.org/draft-07/schema#",
3+
"type": "object",
4+
"required": [
5+
"query"
6+
],
7+
"properties": {
8+
"query": {
9+
"type": "string"
10+
}
11+
}
12+
}

‎doc/schemas/sql.schema.json

Lines changed: 20 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,20 @@
1+
{
2+
"$schema": "http://json-schema.org/draft-07/schema#",
3+
"type": "object",
4+
"additionalProperties": false,
5+
"required": [
6+
"rows"
7+
],
8+
"properties": {
9+
"rows": {
10+
"type": "array",
11+
"items": {
12+
"type": "array"
13+
}
14+
},
15+
"warning_db_failure": {
16+
"type": "string",
17+
"description": "A message if the database encounters an error partway through"
18+
}
19+
}
20+
}

‎plugins/sql.c

Lines changed: 0 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -27,7 +27,6 @@ static const char schemas[] =
2727

2828
/* TODO:
2929
* 2. Refresh time in API.
30-
* 5. documentation.
3130
* 6. test on mainnet.
3231
* 7. Some cool query for documentation.
3332
* 8. time_msec fields.

0 commit comments

Comments
 (0)