OILS / doc / ref / chap-expr-lang.md View on Github | oilshell.org

762 lines, 461 significant
1---
2title: YSH Expression Language (Oils Reference)
3all_docs_url: ..
4body_css_class: width40
5default_highlighter: oils-sh
6preserve_anchor_case: yes
7---
8
9<div class="doc-ref-header">
10
11[Oils Reference](index.html) &mdash;
12Chapter **YSH Expression Language**
13
14</div>
15
16This chapter describes the YSH expression language, which includes [Egg
17Expressions]($xref:eggex).
18
19<div id="dense-toc">
20</div>
21
22## Assignment
23
24### assign
25
26The `=` operator is used with assignment keywords:
27
28 var x = 42
29 setvar x = 43
30
31 const y = 'k'
32
33 setglobal z = 'g'
34
35### aug-assign
36
37The augmented assignment operators are:
38
39 += -= *= /= **= //= %=
40 &= |= ^= <<= >>=
41
42They are used with `setvar` and `setglobal`. For example:
43
44 setvar x += 2
45
46is the same as:
47
48 setvar x = x + 2
49
50Likewise, these are the same:
51
52 setglobal a[i] -= 1
53
54 setglobal a[i] = a[i] - 1
55
56## Literals
57
58### atom-literal
59
60YSH uses JavaScript-like spellings for these three "atoms":
61
62 null # type Null
63 true false # type Bool
64
65Note: to signify "no value", you may sometimes use an empty string `''`,
66instead of `null`.
67
68### int-literal
69
70Examples of integer literals:
71
72 var decimal = 42
73 var big = 42_000
74
75 var hex = 0x0010_ffff
76
77 var octal = 0o755
78
79 var binary = 0b0001_0000
80
81### float-lit
82
83Examples of float literals:
84
85 var myfloat = 3.14
86
87 var f2 = -1.5e-100
88
89### char-literal
90
91Three kinds of unquoted backslash escapes are allowed in expression mode. They
92match what's available in quoted J8-style strings:
93
94 var backslash = \\
95 var quotes = \' ++ \" # same as u'\'' ++ '"'
96
97 var mu = \u{3bc} # same as u'\u{3bc}'
98
99 var nul = \y00 # same as b'\y00'
100
101### ysh-string
102
103YSH has single and double-quoted strings borrowed from Bourne shell, and
104C-style strings borrowed from J8 Notation.
105
106Double quoted strings respect `$` interpolation:
107
108 var dq = "hello $world and $(hostname)"
109
110You can add a `$` before the left quote to be explicit: `$"x is $x"` rather
111than `"x is $x"`.
112
113Single quoted strings may be raw:
114
115 var s = r'line\n' # raw string means \n is literal, NOT a newline
116
117Or *J8 strings* with backslash escapes:
118
119 var s = u'line\n \u{3bc}' # unicode string means \n is a newline
120 var s = b'line\n \u{3bc} \yff' # same thing, but also allows bytes
121
122Both `u''` and `b''` strings evaluate to the single `Str` type. The difference
123is that `b''` strings allow the `\yff` byte escape.
124
125#### Notes
126
127There's no way to express a single quote in raw strings. Use one of the other
128forms instead:
129
130 var sq = "single quote: ' "
131 var sq = u'single quote: \' '
132
133Sometimes you can omit the `r`, e.g. where there are no backslashes and thus no
134ambiguity:
135
136 echo 'foo'
137 echo r'foo' # same thing
138
139The `u''` and `b''` strings are called *J8 strings* because the syntax in YSH
140**code** matches JSON-like **data**.
141
142 var strU = u'mu = \u{3bc}' # J8 string with escapes
143 var strB = b'bytes \yff' # J8 string that can express byte strings
144
145More examples:
146
147 var myRaw = r'[a-z]\n' # raw strings can be used for regexes (not
148 # eggexes)
149
150### triple-quoted
151
152Triple-quoted string literals have leading whitespace stripped on each line.
153They come in the same variants:
154
155 var dq = """
156 hello $world and $(hostname)
157 no leading whitespace
158 """
159
160 var myRaw = r'''
161 raw string
162 no leading whitespace
163 '''
164
165 var strU = u'''
166 string that happens to be unicode \u{3bc}
167 no leading whitespace
168 '''
169
170 var strB = b'''
171 string that happens to be bytes \u{3bc} \yff
172 no leading whitespace
173 '''
174
175Again, you can omit the `r` prefix if there's no backslash, because it's not
176ambiguous:
177
178 var myRaw = '''
179 raw string
180 no leading whitespace
181 '''
182
183### str-template
184
185String templates use the same syntax as double-quoted strings:
186
187 var mytemplate = ^"name = $name, age = $age"
188
189Related topics:
190
191- [Str => replace](chap-type-method.html#replace)
192- [ysh-string](chap-expr-lang.html#ysh-string)
193
194### list-literal
195
196Lists have a Python-like syntax:
197
198 var mylist = ['one', 'two', [42, 43]]
199
200And a shell-like syntax:
201
202 var list2 = :| one two |
203
204The shell-like syntax accepts the same syntax as a simple command:
205
206 ls $mystr @ARGV *.py {foo,bar}@example.com
207
208 # Rather than executing ls, evaluate words into a List
209 var cmd = :| ls $mystr @ARGV *.py {foo,bar}@example.com |
210
211### dict-literal
212
213Dicts look like JavaScript.
214
215 var d = {
216 key1: 'value', # key can be unquoted if it looks like a var name
217 'key2': 42, # or quote it
218
219 ['key2' ++ suffix]: 43, # bracketed expression
220 }
221
222Omitting a value means that the corresponding key takes the value of a var of
223the same name:
224
225 ysh$ var x = 42
226 ysh$ var y = 43
227
228 ysh$ var d = {x, y} # values omitted
229 ysh$ = d
230 (Dict) {x: 42, y: 43}
231
232### range
233
234A range is a sequence of numbers that can be iterated over:
235
236 for i in (0 .. 3) {
237 echo $i
238 }
239 => 0
240 => 1
241 => 2
242
243As with slices, the last number isn't included. To iterate from 1 to n, you
244can use this idiom:
245
246 for i in (1 .. n+1) {
247 echo $i
248 }
249
250### block-expr
251
252In YSH expressions, we use `^()` to create a [Command][] object:
253
254 var myblock = ^(echo $PWD; ls *.txt)
255
256It's more common for [Command][] objects to be created with block arguments,
257which are not expressions:
258
259 cd /tmp {
260 echo $PWD
261 ls *.txt
262 }
263
264[Command]: chap-type-method.html#Command
265
266### expr-literal
267
268An expression literal is an object that holds an unevaluated expression:
269
270 var myexpr = ^[1 + 2*3]
271
272[Expr]: chap-type-method.html#Expr
273
274## Operators
275
276### op-precedence
277
278YSH operator precedence is identical to Python's operator precedence.
279
280New operators:
281
282- `++` has the same precedence as `+`
283- `->` and `=>` have the same precedence as `.`
284
285<!-- TODO: show grammar -->
286
287
288<h3 id="concat">concat <code>++</code></h3>
289
290The concatenation operator works on `Str` objects:
291
292 ysh$ var s = 'hello'
293 ysh$ var t = s ++ ' world'
294
295 ysh$ = t
296 (Str) "hello world"
297
298and `List` objects:
299
300 ysh$ var L = ['one', 'two']
301 ysh$ var M = L ++ ['three', '4']
302
303 ysh$ = M
304 (List) ["one", "two", "three", "4"]
305
306String interpolation can be nicer than `++`:
307
308 var t2 = "${s} world" # same as t
309
310Likewise, splicing lists can be nicer:
311
312 var M2 = :| @L three 4 | # same as M
313
314### ysh-equals
315
316YSH has strict equality:
317
318 a === b # Python-like, without type conversion
319 a !== b # negated
320
321And type converting equality:
322
323 '3' ~== 3 # True, type conversion
324
325The `~==` operator expects a string as the left operand.
326
327---
328
329Note that:
330
331- `3 === 3.0` is false because integers and floats are different types, and
332 there is no type conversion.
333- `3 ~== 3.0` is an error, because the left operand isn't a string.
334
335You may want to use explicit `int()` and `float()` to convert numbers, and then
336compare them.
337
338---
339
340Compare objects for identity with `is`:
341
342 ysh$ var d = {}
343 ysh$ var e = d
344
345 ysh$ = d is d
346 (Bool) true
347
348 ysh$ = d is {other: 'dict'}
349 (Bool) false
350
351To negate `is`, use `is not` (like Python:
352
353 ysh$ d is not {other: 'dict'}
354 (Bool) true
355
356### ysh-in
357
358The `in` operator tests if a key is in a dictionary:
359
360 var d = {k: 42}
361 if ('k' in d) {
362 echo yes
363 } # => yes
364
365Unlike Python, `in` doesn't work on `Str` and `List` instances. This because
366those operations take linear time rather than constant time (O(n) rather than
367O(1)).
368
369TODO: Use `includes() / contains()` methods instead.
370
371### ysh-compare
372
373The comparison operators apply to integers or floats:
374
375 4 < 4 # => false
376 4 <= 4 # => true
377
378 5.0 > 5.0 # => false
379 5.0 >= 5.0 # => true
380
381Example in context:
382
383 if (x < 0) {
384 echo 'x is negative'
385 }
386
387### ysh-logical
388
389The logical operators take boolean operands, and are spelled like Python:
390
391 not
392 and or
393
394Note that they are distinct from `! && ||`, which are part of the [command
395language](chap-cmd-lang.html).
396
397### ysh-arith
398
399YSH supports most of the arithmetic operators from Python. Notably, `/` and `%`
400differ from Python as [they round toward zero, not negative
401infinity](https://www.oilshell.org/blog/2024/03/release-0.21.0.html#integers-dont-do-whatever-python-or-c-does).
402
403Use `+ - *` for `Int` or `Float` addition, subtraction and multiplication. If
404any of the operands are `Float`s, then the output will also be a `Float`.
405
406Use `/` and `//` for `Float` division and `Int` division, respectively. `/`
407will _always_ result in a `Float`, meanwhile `//` will _always_ result in an
408`Int`.
409
410 = 1 / 2 # => (Float) 0.5
411 = 1 // 2 # => (Int) 0
412
413Use `%` to compute the _remainder_ of integer division. The left operand must
414be an `Int` and the right a _positive_ `Int`.
415
416 = 1 % 2 # -> (Int) 1
417 = -4 % 2 # -> (Int) 0
418
419Use `**` for exponentiation. The left operand must be an `Int` and the right a
420_positive_ `Int`.
421
422All arithmetic operators may coerce either of their operands from strings to a
423number, provided those strings are formatted as numbers.
424
425 = 10 + '1' # => (Int) 11
426
427Operators like `+ - * /` will coerce strings to _either_ an `Int` or `Float`.
428However, operators like `// ** %` and bit shifts will coerce strings _only_ to
429an `Int`.
430
431 = '1.14' + '2' # => (Float) 3.14
432 = '1.14' % '2' # Type Error: Left operand is a Str
433
434### ysh-bitwise
435
436Bitwise operators are like Python and C:
437
438 ~ # unary complement
439
440 & | ^ # binary and, or, xor
441
442 >> << # bit shift
443
444### ysh-ternary
445
446The ternary operator is borrowed from Python:
447
448 display = 'yes' if len(s) else 'empty'
449
450### ysh-index
451
452`Str` objects can be indexed by byte:
453
454 ysh$ var s = 'cat'
455 ysh$ = mystr[1]
456 (Str) 'a'
457
458 ysh$ = mystr[-1] # index from the end
459 (Str) 't'
460
461`List` objects:
462
463 ysh$ var mylist = [1, 2, 3]
464 ysh$ = mylist[2]
465 (Int) 3
466
467`Dict` objects are indexed by string key:
468
469 ysh$ var mydict = {'key': 42}
470 ysh$ = mydict['key']
471 (Int) 42
472
473### ysh-attr
474
475The expression `mydict.key` is short for `mydict['key']`.
476
477(Like JavaScript, but unlike Python.)
478
479### ysh-slice
480
481Slicing gives you a subsequence of a `Str` or `List`, like Python.
482
483Negative indices are relative to the end.
484
485### func-call
486
487A function call expression looks like Python:
488
489 ysh$ = f('s', 't', named=42)
490
491A semicolon `;` can be used after positional args and before named args, but
492isn't always required:
493
494 ysh$ = f('s', 't'; named=42)
495
496In these cases, the `;` is necessary:
497
498 ysh$ = f(...args; ...kwargs)
499
500 ysh$ = f(42, 43; ...kwargs)
501
502### thin-arrow
503
504The thin arrow is for mutating methods:
505
506 var mylist = ['bar']
507 call mylist->pop()
508
509<!--
510TODO
511 var mydict = {name: 'foo'}
512 call mydict->erase('name')
513-->
514
515### fat-arrow
516
517The fat arrow is for transforming methods:
518
519 if (s => startsWith('prefix')) {
520 echo 'yes'
521 }
522
523If the method lookup on `s` fails, it looks for free functions. This means it
524can be used for "chaining" transformations:
525
526 var x = myFunc() => list() => join()
527
528### match-ops
529
530YSH has four pattern matching operators: `~ !~ ~~ !~~`.
531
532Does string match an **eggex**?
533
534 var filename = 'x42.py'
535 if (filename ~ / d+ /) {
536 echo 'number'
537 }
538
539Does a string match a POSIX regular expression (ERE syntax)?
540
541 if (filename ~ '[[:digit:]]+') {
542 echo 'number'
543 }
544
545Negate the result with the `!~` operator:
546
547 if (filename !~ /space/ ) {
548 echo 'no space'
549 }
550
551 if (filename !~ '[[:space:]]' ) {
552 echo 'no space'
553 }
554
555Does a string match a **glob**?
556
557 if (filename ~~ '*.py') {
558 echo 'Python'
559 }
560
561 if (filename !~~ '*.py') {
562 echo 'not Python'
563 }
564
565Take care not to confuse glob patterns and regular expressions.
566
567- Related doc: [YSH Regex API](../ysh-regex-api.html)
568
569## Eggex
570
571### re-literal
572
573An eggex literal looks like this:
574
575 / expression ; flags ; translation preference /
576
577The flags and translation preference are both optional.
578
579Examples:
580
581 var pat = / d+ / # => [[:digit:]]+
582
583You can specify flags passed to libc `regcomp()`:
584
585 var pat = / d+ ; reg_icase reg_newline /
586
587You can specify a translation preference after a second semi-colon:
588
589 var pat = / d+ ; ; ERE /
590
591Right now the translation preference does nothing. It could be used to
592translate eggex to PCRE or Python syntax.
593
594- Related doc: [Egg Expressions](../eggex.html)
595
596### re-primitive
597
598There are two kinds of eggex primitives.
599
600"Zero-width assertions" match a position rather than a character:
601
602 %start # translates to ^
603 %end # translates to $
604
605Literal characters appear within **single** quotes:
606
607 'oh *really*' # translates to regex-escaped string
608
609Double-quoted strings are **not** eggex primitives. Instead, you can use
610splicing of strings:
611
612 var dq = "hi $name"
613 var eggex = / @dq /
614
615### class-literal
616
617An eggex character class literal specifies a set. It can have individual
618characters and ranges:
619
620 [ 'x' 'y' 'z' a-f A-F 0-9 ] # 3 chars, 3 ranges
621
622Omit quotes on ASCII characters:
623
624 [ x y z ] # avoid typing 'x' 'y' 'z'
625
626Sets of characters can be written as strings
627
628 [ 'xyz' ] # any of 3 chars, not a sequence of 3 chars
629
630Backslash escapes are respected:
631
632 [ \\ \' \" \0 ]
633 [ \xFF \u0100 ]
634
635Splicing:
636
637 [ @str_var ]
638
639Negation always uses `!`
640
641 ![ a-f A-F 'xyz' @str_var ]
642
643### named-class
644
645Perl-like shortcuts for sets of characters:
646
647 [ dot ] # => .
648 [ digit ] # => [[:digit:]]
649 [ space ] # => [[:space:]]
650 [ word ] # => [[:alpha:]][[:digit:]]_
651
652Abbreviations:
653
654 [ d s w ] # Same as [ digit space word ]
655
656Valid POSIX classes:
657
658 alnum cntrl lower space
659 alpha digit print upper
660 blank graph punct xdigit
661
662Negated:
663
664 !digit !space !word
665 !d !s !w
666 !alnum # etc.
667
668### re-repeat
669
670Eggex repetition looks like POSIX syntax:
671
672 / 'a'? / # zero or one
673 / 'a'* / # zero or more
674 / 'a'+ / # one or more
675
676Counted repetitions:
677
678 / 'a'{3} / # exactly 3 repetitions
679 / 'a'{2,4} / # between 2 to 4 repetitions
680
681### re-compound
682
683Sequence expressions with a space:
684
685 / word digit digit / # Matches 3 characters in sequence
686 # Examples: a42, b51
687
688(Compare `/ [ word digit ] /`, which is a set matching 1 character.)
689
690Alternation with `|`:
691
692 / word | digit / # Matches 'a' OR '9', for example
693
694Grouping with parentheses:
695
696 / (word digit) | \\ / # Matches a9 or \
697
698### re-capture
699
700To retrieve a substring of a string that matches an Eggex, use a "capture
701group" like `<capture ...>`.
702
703Here's an eggex with a **positional** capture:
704
705 var pat = / 'hi ' <capture d+> / # access with _group(1)
706 # or Match => _group(1)
707
708Captures can be **named**:
709
710 <capture d+ as month> # access with _group('month')
711 # or Match => group('month')
712
713Captures can also have a type **conversion func**:
714
715 <capture d+ : int> # _group(1) returns Int
716
717 <capture d+ as month: int> # _group('month') returns Int
718
719Related docs and help topics:
720
721- [YSH Regex API](../ysh-regex-api.html)
722- [`_group()`](chap-builtin-func.html#_group)
723- [`Match => group()`](chap-type-method.html#group)
724
725### re-splice
726
727To build an eggex out of smaller expressions, you can **splice** eggexes
728together:
729
730 var D = / [0-9][0-9] /
731 var time = / @D ':' @D / # [0-9][0-9]:[0-9][0-9]
732
733If the variable begins with a capital letter, you can omit `@`:
734
735 var ip = / D ':' D /
736
737You can also splice a string:
738
739 var greeting = 'hi'
740 var pat = / @greeting ' world' / # hi world
741
742Splicing is **not** string concatenation; it works on eggex subtrees.
743
744### re-flags
745
746Valid ERE flags, which are passed to libc's `regcomp()`:
747
748- `reg_icase` aka `i` - ignore case
749- `reg_newline` - 4 matching changes related to newlines
750
751See `man regcomp`.
752
753### re-multiline
754
755Multi-line eggexes aren't yet implemented. Splicing makes it less necessary:
756
757 var Name = / <capture [a-z]+ as name> /
758 var Num = / <capture d+ as num> /
759 var Space = / <capture s+ as space> /
760
761 # For variables named like CapWords, splicing @Name doesn't require @
762 var lexer = / Name | Num | Space /