Skip to content

Commit 1d369bc

Browse files
waderwader
authored
Merge pull request #319 from wader/release-0.0.8
Release 0.0.8
2 parents 2cf8575 + db70fcc commit 1d369bc

File tree

2 files changed

+276
-3
lines changed

2 files changed

+276
-3
lines changed

‎CHANGES.md

Lines changed: 275 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -1,3 +1,278 @@
1+
# 0.0.8
2+
3+
## Changes
4+
5+
- Add support for some common structured serialization formats: #284 #335
6+
- XML, `toxml`, `fromxml` options for indent, jq mapping variants (object or array) and order preservation
7+
- HTML, `fromhtml` options for indent, jq mapping variants (object or array) and order preservation
8+
- TOML, `totoml`, `fromtoml`
9+
- YAML, `toyaml`, `fromyaml`
10+
- jq-flavored JSON (optional key quotes and trailing comma) `tojq`, `fromjq` options for indent #284
11+
```sh
12+
# query a YAML file
13+
$ fq '...' file.yml
14+
15+
# convert YAML to JSON
16+
# note -r for raw string output, without a JSON string with JSON would outputted
17+
$ fq -r 'tojson({indent:2})' file.yml
18+
19+
$ fq -nr '{hello: {world: "test"}} | toyaml, totoml, toxml, tojq({indent: 2})'
20+
hello:
21+
world: test
22+
23+
[hello]
24+
world = "test"
25+
26+
<hello>
27+
<world>test</world>
28+
</hello>
29+
{
30+
hello: {
31+
world: "test"
32+
}
33+
}
34+
$ echo '<doc><element a="b"></doc>' | fq -r '.doc.element."-a"'
35+
b
36+
$ echo '<doc><element a="b"></doc>' | fq -r '.doc.element."-a" = "<test>" | toxml({indent: 2})'
37+
<doc>
38+
<element a="&lt;test&gt;"></element>
39+
</doc>
40+
```
41+
- CSV, `tocsv`, `fromcsv` options for separator and comment character
42+
```sh
43+
$ echo -e '1,2\n3,4' | fq -rRs 'fromcsv | . + [["a","b"]] | tocsv'
44+
1,2
45+
3,4
46+
a,b
47+
```
48+
- Add support for binary encodings
49+
- Base64. `tobase64`, `frombase64` options for encoding variants.
50+
```sh
51+
$ echo -n hello | base64 | fq -rRs 'frombase64 | tostring'
52+
hello
53+
```
54+
- Hex string. `tohex`, `fromhex`
55+
- Add support for text formats
56+
- XML entities `toxmlentities`, `fromxmlentities`
57+
- URL `tourl`, `fromurl`
58+
```sh
59+
$ echo -n 'https://host/path/?key=value#fragment' | fq -Rs 'fromurl | ., (.host = "changed" | tourl)'
60+
{
61+
"fragment": "fragment",
62+
"host": "host",
63+
"path": "/path/",
64+
"query": {
65+
"key": "value"
66+
},
67+
"rawquery": "key=value",
68+
"scheme": "https"
69+
}
70+
"https://changed/path/?key=value#fragment"
71+
```
72+
- URL path encoding `tourlpath`, `fromurlpath`
73+
- URL encoding `tourlencode`, `fromurlencode`
74+
- URL query `tourlquery`, `fromurlquery`
75+
- Add support for common hash functions:
76+
- MD4 `tomd4`
77+
- MD5 `tomd5`
78+
```sh
79+
$ echo -n hello | fq -rRs 'tomd5 | tohex'
80+
5d41402abc4b2a76b9719d911017c592
81+
```
82+
- SHA1 `tosha1`
83+
- SHA256 `tosha256`
84+
- SHA512 `tosha512`
85+
- SHA3 224 `tosha3_224`
86+
- SHA3 256 `tosha3_256`
87+
- SHA3 384 `tosha3_384`
88+
- SHA3 512 `tosha3_512`
89+
- Add support for common text encodings:
90+
- ISO8859-1 `toiso8859_1`, `fromiso8859_1`
91+
- UTF8 `tutf8`, `fromutf8`
92+
- UTF16 `toutf16`, `fromutf16`
93+
- UTF16LE `toutf16le`, `fromutf16le`
94+
- UTF16BE `toutf16be`, `fromutf16be`
95+
```sh
96+
$ echo -n 00680065006c006c006f | fq -rRs 'fromhex | fromutf16be'
97+
hello
98+
```
99+
- Add `group` function, same as `group_by(.)` #299
100+
- Update/rebase readline dependency (based on @tpodowd https://github.com/chzyer/readline/pull/207) #305 #308
101+
- Less blinking/redraw in REPL
102+
- Lots of small bug fixes
103+
- Update/rebase gojq dependency #247
104+
- Fixes JQValue destructing issue (ex: `<some object JQValue> as {$key}`)
105+
- Major rewrite/refactor how native function are implemented. Less verbose and less error-prone as now shared code takes care of type casting and some argument errors. #316
106+
- Add `tojson($opts)` that support indent option. `tojson` still works as before (no indent).
107+
```sh
108+
$ echo '{a: 1}' | fq -r 'tojson({indent: 2})'
109+
{
110+
"a": 1
111+
}
112+
```
113+
- Rename `--decode-file` (will still work) to `--argdecode` be be more consistent with existing `--arg*` arguments. #309
114+
- On some decode error cases fq can now keep more of partial tree making it easier to know where it stopped #245
115+
- Build with go 1.18 #272
116+
117+
## Decoder changes
118+
119+
- `bitcoin` Add Bitcoin blkdat, block, transcation and script decoders #239
120+
- `elf` Use correct offset to dynamic linking string table #304
121+
- `tcp` Restructure into separate client/server objects and add `skipped_bytes` (number of bytes with known missing ACK), `has_start` (has first byte in stream) and `has_end` (has last byte in stream) per direction #251
122+
- Old:
123+
```
124+
│00 01 02 03 04 05 06 07│01234567│.tcp_connections[0]{}: tcp_connection
125+
│ │ │ source_ip: "192.168.69.2"
126+
│ │ │ source_port: 34059
127+
│ │ │ destination_ip: "192.168.69.1"
128+
│ │ │ destination_port: "http" (80) (World Wide Web HTTP)
129+
│ │ │ has_start: true
130+
│ │ │ has_end: true
131+
0x000│47 45 54 20 2f 74 65 73│GET /tes│ client_stream: raw bits
132+
0x008│74 2f 65 74 68 65 72 65│t/ethere│
133+
* │until 0x1bc.7 (end) (44│ │
134+
0x000│48 54 54 50 2f 31 2e 31│HTTP/1.1│ server_stream: raw bits
135+
0x008│20 32 30 30 20 4f 4b 0d│ 200 OK.│
136+
* │until 0x191.7 (end) (40│ │
137+
```
138+
- New:
139+
```
140+
│00 01 02 03 04 05 06 07│01234567│.tcp_connections[0]{}: tcp_connection
141+
│ │ │ client{}:
142+
│ │ │ ip: "192.168.69.2"
143+
│ │ │ port: 34059
144+
│ │ │ has_start: true
145+
│ │ │ has_end: true
146+
│ │ │ skipped_bytes: 0
147+
0x000│47 45 54 20 2f 74 65 73│GET /tes│ stream: raw bits
148+
0x008│74 2f 65 74 68 65 72 65│t/ethere│
149+
* │until 0x1bc.7 (end) (44│ │
150+
│ │ │ server{}:
151+
│ │ │ ip: "192.168.69.1"
152+
│ │ │ port: "http" (80) (World Wide Web HTTP)
153+
│ │ │ has_start: true
154+
│ │ │ has_end: true
155+
│ │ │ skipped_bytes: 0
156+
0x000│48 54 54 50 2f 31 2e 31│HTTP/1.1│ stream: raw bits
157+
0x008│20 32 30 30 20 4f 4b 0d│ 200 OK.│
158+
* │until 0x191.7 (end) (40│ │
159+
```
160+
- `zip` Add 64-bit support and add `uncompress` option #278
161+
- `matroska` Update and regenerate based on latest spec and also handle unknown ids better #291
162+
- `mp4` Changes:
163+
- Fix PSSH decode issue #283
164+
- Add track for track_id references without tfhd box
165+
- Makes it possible to see samples in fragments without having an init segment.
166+
Note it is possible to decode samples in a fragment file by concatenating the init and fragment file ex: `cat init frag | fq ...`.
167+
- Add `senc` box support #290
168+
- Don't decode encrypted samples #311
169+
- Add `track_id` to tracks #254
170+
- Add fairplay PSSH system ID #310
171+
- Properly handle `trun` data offset #294
172+
- Skip decoding of individual PCM samples for now #268
173+
- Add `mvhd`, `tkhd`, `mdhd` and `mehd` version 1 support #258
174+
- Make sure to preserve sample table order #330
175+
- `fairplay_spc` Add basic FairPlay Server Playback Context decoder #310
176+
- `avc_pps` Correctly check for more rbsp data
177+
178+
## Changelog
179+
180+
* 210940a4 Update docker-golang from 1.18.1 to 1.18.2
181+
* fbeabdc3 Update docker-golang from 1.18.2 to 1.18.3
182+
* 51a414db Update docker-golang from 1.18.3 to 1.18.4
183+
* 3017e8b4 Update github-go-version from 1.18.1, 1.18.1, 1.18.1 to 1.18.2
184+
* c597f7f7 Update github-go-version from 1.18.2, 1.18.2, 1.18.2 to 1.18.3
185+
* dd283923 Update github-go-version from 1.18.3, 1.18.3, 1.18.3 to 1.18.4
186+
* d10a3616 Update github-golangci-lint from 1.45.2 to 1.46.0
187+
* 75b5946c Update github-golangci-lint from 1.46.0 to 1.46.1
188+
* 3ffa9efb Update github-golangci-lint from 1.46.1 to 1.46.2
189+
* 4be8cb91 Update github-golangci-lint from 1.46.2 to 1.47.0
190+
* 1b8f4be8 Update github-golangci-lint from 1.47.0 to 1.47.1
191+
* fc596a7a Update github-golangci-lint from 1.47.1 to 1.47.2
192+
* 62be9223 Update gomod-BurntSushi/toml from 1.1.0 to 1.2.0
193+
* 5db7397a Update make-golangci-lint from 1.45.2 to 1.46.0
194+
* 456742ea Update make-golangci-lint from 1.46.0 to 1.46.1
195+
* 06757119 Update make-golangci-lint from 1.46.1 to 1.46.2
196+
* 3d69e9d0 Update make-golangci-lint from 1.46.2 to 1.47.0
197+
* 2170925d Update make-golangci-lint from 1.47.0 to 1.47.1
198+
* c4199c0f Update make-golangci-lint from 1.47.1 to 1.47.2
199+
* 02f00be9 Update usage.md
200+
* 75169a65 asn1: Add regression test for range decode fix ##330
201+
* b0096bc1 avc_pps: Correct check if there is more rbsp data
202+
* 5d67df47 avro_ocf: Fix panic on missing meta schema
203+
* 417255b7 bitcoin: Add blkdat, block, transcation and script decoder
204+
* a6a97136 decode: Cleanup Try<f>/<f> pairs
205+
* 3ce660a2 decode: Keep decode tree on RangeFn error
206+
* c4dd518e decode: Make compound range sort optional
207+
* 8bb4a6d2 decode: Range decode with new decoder to preserve bit reader
208+
* 342612eb dev: Cleanup linters and fix some unused args
209+
* 78aa96b0 dev: Cleanup some code to fix a bunch of new linter warnings
210+
* 3570f1f0 doc: Add more related tools
211+
* 7aff654a doc: Clarify decode, slurp and spew args
212+
* 0863374f doc: Correct bencode spec URL
213+
* 10cc5518 doc: Improve and cleanup text formats
214+
* b1006119 doc: Typos and add note about Try* functions
215+
* c27646a6 doc: Update and shorten README.md a bit
216+
* b0388722 doc: Use singular jq value to refer to jq value
217+
* a980656c doc: go 1.18 and improve intro text a bit
218+
* a64c28d9 dump: Skip JQValueEx if there are not options
219+
* 40481f66 elf,fuzz: Error on too large string table
220+
* f66a359c elf: Use correct offset to dynamic linking string table
221+
* 64f3e5c7 fairplay: Add basic SPC decoder and PSSH system id
222+
* cae288e6 format,intepr: Refactor json, yaml, etc into formats also move out related functions
223+
* e9d9f8ae fq: Use go 1.18
224+
* 377af133 fqtest: Cleanup path usage
225+
* 2464ebc2 fuzz: Replace built tag with FUZZTEST env and use new interp api
226+
* 0f78687b gojq: Fix JQValue index and destructuring issue and rebase fq fork
227+
* 59c7d0df gojq: Rebase fq fork
228+
* c57dc17d gojq: Rebase fq fork
229+
* 9a7ce148 gojq: Update rebased fq fork
230+
* c1a0cda5 gojq: Update rebased fq fork
231+
* 32361dee gojqextra: Cleanup gojq type cast code
232+
* 9b2e474e gojqextra: Simplify function type helpers
233+
* fd302093 hevc_vps,fuzz: Error on too many vps layers
234+
* efa5e23a icc_profile: Correctly clamp align padding on EOF
235+
* 1ddea1ad interp,format: Refactor registry usage and use function helpers
236+
* a3c33fc1 interp: Add group/0
237+
* 95e61965 interp: Add internal _is_<type> helpers
238+
* 3b717c3b interp: Add to/from<encoding> for some common serialzations, encodings and hashes
239+
* 6b088000 interp: Cast jq value to go value properly for encoding functions
240+
* f5be5180 interp: Cleanup and clarify some format naming
241+
* c7701851 interp: Extract to/from map/struct to own package
242+
* 8dde3ef5 interp: Fix crash when including relatve path when no search paths are set
243+
* 735c443b interp: Improve type normalization and use it for toyaml and totoml
244+
* 81a014ce interp: Make empty _finally fin error on error
245+
* 2dc509ab interp: Refactor dump and revert #259 for now
246+
* ab8c728a interp: Rename --decode-file to --argdecode to be more consistent
247+
* dff3cc11 interp: dump: Fix column truncate issue with unicode bars
248+
* 5109df4a interp: dump: Show address bar for nested roots
249+
* 80214921 interp: help: Fix incorrect options example
250+
* 76714349 mapstruct: Handle nested values when converting to camel case
251+
* c92f4f13 matroska: Update ebml_matroska.xml and allow unknown ids
252+
* c2a359bd mod: Update golang.org/x/{crypto,net}
253+
* 3780375d mp3: Use d.FieldValueU and some cleanup
254+
* 7b27e506 mp4,bitio: Fix broken pssh decoding and add proper reader cloning to bitio
255+
* 6b00297e mp4,senc: Refactor current track/moof tracking and add senc box support
256+
* 8228ecae mp4: Add track id field and add track for tfhd with unseen track_id
257+
* ea2cc3c2 mp4: Don't decode encrypted samples
258+
* c6d0d89c mp4: Don't range sort samples, keep sample table order
259+
* 7d25fbfd mp4: Properly use trun data offset
260+
* ba844eb0 mp4: Skip fields for pcm samples for now
261+
* 0e02bb66 mp4: iinf: Only assume sub boxes for version 0
262+
* 2e328180 mp4: mvhd,tkhd,mdhd,mehd: Add version 1 support
263+
* 44bab274 readline: Rebase on top of tpodowd's redraw/completion fixes PR
264+
* a5122690 readline: Rebase on top of tpodowd's update PR
265+
* 54dcdce9 readline: Update fq fork
266+
* 6e7267d2 readme: add MacPorts install details
267+
* 76161a1b scalar,mp4,gzip,tar: Add timestamp to description
268+
* 9133f0e5 scalar: Add *Fn type to map value and clearer naming
269+
* 34cf5442 tcp: Split into client/server structs and add skipped_bytes and has_start/end per direction
270+
* 1aaaefb0 wav,bencode,mpeg_ps_packet,id3v1: Random fixes
271+
* 47350e46 zip: Add uncompress=false test and some docs
272+
* e6412744 zip: Add zip64 support and uncompress option
273+
* aa694e3f zip: s/Decompress/Uncompress/
274+
275+
1276
# 0.0.7
2277
3278
## Changes
@@ -458,8 +733,6 @@ Also thanks to @Doctor-love @mathieu-aubin for keeping things tidy.
458733
459734
(Some commits have been removed from list for clarity)
460735
461-
462-
463736
# 0.0.2
464737
465738
## Changelog

‎fq.go

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -6,7 +6,7 @@ import (
66
"github.com/wader/fq/pkg/interp"
77
)
88

9-
const version = "0.0.7"
9+
const version = "0.0.8"
1010

1111
func main() {
1212
cli.Main(interp.DefaultRegistry, version)

0 commit comments

Comments
 (0)