1 | QSN: A Familiar String Interchange Format
|
2 | =========================================
|
3 |
|
4 | <style>
|
5 | .q {
|
6 | color: darkred;
|
7 | }
|
8 | .comment {
|
9 | color: green;
|
10 | font-style: italic;
|
11 | }
|
12 | .terminal {
|
13 | color: darkred;
|
14 | font-family: monospace;
|
15 | }
|
16 | .an {
|
17 | color: darkgreen;
|
18 | }
|
19 |
|
20 | .attention {
|
21 | font-size: x-large;
|
22 | background-color: #DEE;
|
23 | margin-left: 1em;
|
24 | margin-right: 1em;
|
25 | padding-left: 1em;
|
26 | padding-right: 1em;
|
27 | }
|
28 | </style>
|
29 |
|
30 |
|
31 |
|
32 |
|
33 |
|
34 | <div class=attention>
|
35 |
|
36 |
|
37 |
|
38 | As of January 2024, QSN has been replaced by [J8 Notation](j8-notation.html).
|
39 | They're very similar, but J8 Notation is more "harmonized" with JSON.
|
40 |
|
41 |
|
42 |
|
43 | </div>
|
44 |
|
45 |
|
46 |
|
47 |
|
48 |
|
49 |
|
50 |
|
51 |
|
52 |
|
53 |
|
54 |
|
55 | QSN ("quoted string notation") is a data format for **byte strings**.
|
56 | Examples:
|
57 |
|
58 | <pre>
|
59 | '' <span class=comment># empty string</span>
|
60 | 'my favorite song.mp3'
|
61 | 'bob<span class=q>\t</span>1.0<span class=q>\n</span>carol<span class=q>\t</span>2.0<span class=q>\n</span>' <span class=comment># tabs and newlines</span>
|
62 | 'BEL = <span class=q>\x07</span>' <span class=comment># byte escape</span>
|
63 | 'mu = <span class=q>\u{03bc}</span>' <span class=comment># Unicode char escape</span>
|
64 | 'mu = μ' <span class=comment># represented literally, not escaped</span>
|
65 | </pre>
|
66 |
|
67 | It's an adaptation of Rust's string literal syntax with a few use cases:
|
68 |
|
69 | - To print filenames to a terminal. Printing arbitrary bytes to a
|
70 | terminal is bad, so programs like [coreutils]($xref) already have [informal
|
71 | QSN-like formats][coreutils-quotes].
|
72 | - To exchange data between different programs, like [JSON][] or UTF-8. Note
|
73 | that JSON can't express arbitrary byte strings.
|
74 | - To solve the "[framing problem](framing.html)" over pipes. QSN represents
|
75 | newlines like `\n`, so literal newlines can be used to delimit records.
|
76 |
|
77 | Oil uses QSN because it's well-defined and parsable. It's both human- and
|
78 | machine-readable.
|
79 |
|
80 | Any programming language or tool that understands JSON should also understand
|
81 | QSN.
|
82 |
|
83 | [JSON]: https://json.org
|
84 |
|
85 | <div id="toc">
|
86 | </div>
|
87 |
|
88 | <!--
|
89 | ### The Terminal Use Case
|
90 |
|
91 | Filenames may contain arbitrary bytes, including ones that will <span
|
92 | class=terminal>change your terminal color</span>, and more. Most command line
|
93 | programs need something like QSN, or they'll have subtle bugs.
|
94 |
|
95 | For example, as of 2016, [coreutils quotes funny filenames][coreutils] to avoid
|
96 | the same problem. However, they didn't specify the format so it can be parsed.
|
97 | In contrast, QSN can be parsed and printed like JSON.
|
98 |
|
99 | -->
|
100 |
|
101 | <!--
|
102 | The quoting only happens when `isatty()`, so it's not really meant
|
103 | to be parsed.
|
104 | -->
|
105 |
|
106 | ## Important Properties
|
107 |
|
108 | - QSN can represent **any byte sequence**.
|
109 | - Given a QSN-encoded string, any 2 decoders must produce the same byte string.
|
110 | (On the other hand, encoders have flexibility with regard to escaping.)
|
111 | - An encoded string always fits on a **single line**. Newlines must be encoded as
|
112 | `\n`, not literal.
|
113 | - A encoded string always fits in a **TSV cell**. Tabs must be encoded as `\t`,
|
114 | not literal.
|
115 | - An encoded string can itself be **valid UTF-8**.
|
116 | - Example: `'μ \xff'` is valid UTF-8, even though the decoded string is not.
|
117 | - An encoded string can itself be **valid ASCII**.
|
118 | - Example: `'\xce\xbc'` is valid ASCII, even though the decoded string is
|
119 | not.
|
120 |
|
121 | ## More QSN Use Cases
|
122 |
|
123 | - To pack arbitrary bytes on a **single line**, e.g. for line-based tools like
|
124 | [grep]($xref), [awk]($xref), and [xargs]($xref). QSN strings never contain
|
125 | literal newlines or tabs.
|
126 | - For `set -x` in shell. Like filenames, Unix `argv` arrays may contain
|
127 | arbitrary bytes. There's an example in the appendix.
|
128 | - `ps` has to display untrusted `argv` arrays.
|
129 | - `ls` has to display untrusted filenames.
|
130 | - `env` has to display untrusted byte strings. (Most versions of `env` don't
|
131 | handle newlines well.)
|
132 | - As a building block for larger specifications, like [QTT][].
|
133 | - To transmit arbitrary bytes over channels that can only represent ASCII or
|
134 | UTF-8 (e.g. e-mail, Twitter).
|
135 |
|
136 | [QTT]: qtt.html
|
137 | [surrogate pairs]: https://en.wikipedia.org/wiki/UTF-16#Code_points_from_U+010000_to_U+10FFFF
|
138 | [coreutils]: https://www.gnu.org/software/coreutils/quotes.html
|
139 |
|
140 | ## Specification
|
141 |
|
142 | ### A Short Description
|
143 |
|
144 | 1. Start with [Rust String Literal Syntax](https://doc.rust-lang.org/reference/tokens.html#string-literals)
|
145 | 2. Use **single quotes** instead of double quotes to surround the string. This
|
146 | is mainly to to avoid confusion with JSON.
|
147 |
|
148 | ### An Analogy
|
149 |
|
150 | <pre>
|
151 |
|
152 | <span class=an>JavaScript Object Literals</span> are to <span class=an>JSON</span>
|
153 | as <span class=an>Rust String Literals</span> are to <span class=an>QSN</span>
|
154 |
|
155 | </pre>
|
156 |
|
157 | But QSN is **not** tied to either Rust or shell, just like JSON isn't tied to
|
158 | JavaScript.
|
159 |
|
160 | It's a **language-independent format** like UTF-8 or HTML. We're only
|
161 | borrowing a design, so that it's well-specified and familiar.
|
162 |
|
163 | ### Full Spec
|
164 |
|
165 | TODO: The short description above should be sufficient, but we might want to
|
166 | write it out.
|
167 |
|
168 | - Special escapes:
|
169 | - `\t` `\r` `\n`
|
170 | - `\'` `\"`
|
171 | - `\\`
|
172 | - `\0`
|
173 | - Byte escapes: `\x7F`
|
174 | - Character escapes: `\u{03bc}` or `\u{0003bc}`. These are encoded as UTF-8.
|
175 |
|
176 | ## Advantages Over JSON Strings
|
177 |
|
178 | - QSN can represent any byte string, like `'\x00\xff\x00'`. JSON can't
|
179 | represent **binary data** directly.
|
180 | - QSN can represent any code point, like `'\u{01f600}'` for 😀. JSON
|
181 | needs awkward [surrogate pairs][] to represent this code point.
|
182 |
|
183 | ## Implementation Issues
|
184 |
|
185 | ### How Does a QSN Encoder Deal with Unicode?
|
186 |
|
187 | The input to a QSN encoder is a raw **byte string**. However, the string may
|
188 | have additional structure, like being UTF-8 encoded.
|
189 |
|
190 | The encoder has three options to deal with this structure:
|
191 |
|
192 | 1. **Don't decode** UTF-8. Walk through bytes one-by-one, showing unprintable
|
193 | ones with escapes like `\xce\xbc`. Never emit escapes like `\u{3bc}` or
|
194 | literals like <code>μ</code>. This option is OK for machines, but
|
195 | isn't friendly to humans who can read Unicode characters.
|
196 |
|
197 | Or **speculatively decode** UTF-8. After decoding a valid UTF-8 sequence,
|
198 | there are two options:
|
199 |
|
200 | 2. Show **escaped code points**, like `\u{3bc}`. The encoded string is limited
|
201 | to the ASCII subset, which is useful in some contexts.
|
202 |
|
203 | 3. Show them **literally**, like <code>μ</code>.
|
204 |
|
205 | QSN encoding should never fail; it should only fall back to byte escapes like
|
206 | `\xff`. TODO: Show the state machine for detecting and decoding UTF-8.
|
207 |
|
208 | Note: Strategies 2 and 3 indicate whether the string is valid UTF-8.
|
209 |
|
210 | ### Which Bytes Should Be Hex-Escaped?
|
211 |
|
212 | The reference implementation has two functions:
|
213 |
|
214 | - `IsUnprintableLow`: any byte below an ASCII space `' '` is escaped
|
215 | - `IsUnprintableHigh`: the byte `\x7f` and all bytes above are escaped, unless
|
216 | they're part of a valid UTF-8 sequence.
|
217 |
|
218 | In theory, only escapes like `\'` `\n` `\\` are strictly necessary, and no
|
219 | bytes need to be hex-escaped. But that strategy would defeat the purpose of
|
220 | QSN for many applications, like printing filenames in a terminal.
|
221 |
|
222 | ### List of Syntax Errors
|
223 |
|
224 | QSN decoders must enforce (at least) these syntax errors:
|
225 |
|
226 | - Literal newline or tab in a string. Should be `\t` or `\n`. (The lack of
|
227 | literal tabs and newlines is essential for [QTT][].)
|
228 | - Invalid character escape, e.g. `\z`
|
229 | - Invalid hex escape, e.g. `\xgg`
|
230 | - Invalid unicode escape, e.g. `\u{123` (incomplete)
|
231 |
|
232 | Separate messages aren't required for each error; the only requirement is that
|
233 | they not accept these sequences.
|
234 |
|
235 | ## Reference Implementation in Oil
|
236 |
|
237 | - Oil's **encoder** is in [qsn_/qsn.py]($oils-src), including the state machine
|
238 | for the UTF-8 strategies.
|
239 | - The **decoder** has a lexer in [frontend/lexer_def.py]($oils-src), and a
|
240 | "parser" / validator in [qsn_/qsn_native.py]($oils-src). (Note that QSN is a
|
241 | [regular language]($xref:regular-language)).
|
242 |
|
243 | The encoder has options to emit shell-compatible strings, which you probably
|
244 | **don't need**. That is, C-escaped strings in bash look `$'like this\n'`.
|
245 |
|
246 | A **subset** of QSN is compatible with this syntax. Example:
|
247 |
|
248 | $'\x01\n' # A valid bash string. Removing $ makes it valid QSN.
|
249 |
|
250 | Something like `$'\0065'` is never emitted, because QSN doesn't contain octal
|
251 | escapes. It can be encoded with hex or character escapes.
|
252 |
|
253 | ## Appendices
|
254 |
|
255 | ### Design Notes
|
256 |
|
257 | The general idea: Rust string literals are like C and JavaScript string
|
258 | literals, without cruft like octal (`\755` or `\0755` — which is it?) and
|
259 | vertical tabs (`\v`).
|
260 |
|
261 | Comparison with shell strings:
|
262 |
|
263 | - `'Single quoted strings'` in shell can't represent arbitrary byte strings.
|
264 | - `$'C-style shell strings\n'` strings are similar to QSN, but have cruft like
|
265 | octal and `\v`.
|
266 | - `"Double quoted strings"` have unneeded features like `$var` and `$(command
|
267 | sub)`.
|
268 |
|
269 | Comparison with Python's `repr()`:
|
270 |
|
271 | - A single quote in Python is `"'"`, whereas it's `'\''` in QSN
|
272 | - Python has both `\uxxxx` and `\Uxxxxxxxx`, whereas QSN has the more natural
|
273 | `\u{xxxxxx}`.
|
274 |
|
275 | ### Related Links
|
276 |
|
277 | - [GNU Coreutils - Quoting File names][coreutils-quotes]. *Starting with GNU
|
278 | coreutils version 8.25 (released Jan. 2016), ls's default output quotes
|
279 | filenames with special characters*
|
280 | - [In-band signaling][in-band] is the fundamental problem with filenames and
|
281 | terminals. Code (control codes) and data are intermingled.
|
282 | - [QTT][] is a cleanup of CSV/TSV, built on top of QSN.
|
283 |
|
284 | [coreutils-quotes]: https://www.gnu.org/software/coreutils/quotes.html
|
285 |
|
286 | [in-band]: https://en.wikipedia.org/wiki/In-band_signaling
|
287 |
|
288 |
|
289 | ### `set -x` example
|
290 |
|
291 | When arguments don't have any spaces, there's no ambiguity:
|
292 |
|
293 |
|
294 | $ set -x
|
295 | $ echo two args
|
296 | + echo two args
|
297 |
|
298 | Here we need quotes to show that the `argv` array has 3 elements:
|
299 |
|
300 | $ set -x
|
301 | $ x='a b'
|
302 | $ echo "$x" c
|
303 | + echo 'a b' c
|
304 |
|
305 | And we want the trace to fit on a single line, so we print a QSN string with
|
306 | `\n`:
|
307 |
|
308 | $ set -x
|
309 | $ x=$'a\nb'
|
310 | $ echo "$x" c
|
311 | + echo $'a\nb' c
|
312 |
|
313 | Here's an example with unprintable characters:
|
314 |
|
315 | $ set -x
|
316 | $ x=$'\e\001'
|
317 | $ echo "$x"
|
318 | + echo $'\x1b\x01'
|