| 1 | ---
|
| 2 | default_highlighter: oils-sh
|
| 3 | ---
|
| 4 |
|
| 5 | A Tour of YSH
|
| 6 | =============
|
| 7 |
|
| 8 | <!-- author's note about example names
|
| 9 |
|
| 10 | - people: alice, bob
|
| 11 | - nouns: ale, bean
|
| 12 | - peanut, coconut
|
| 13 | - 42 for integers
|
| 14 | -->
|
| 15 |
|
| 16 | This doc describes the [YSH]($xref) language from **clean slate**
|
| 17 | perspective. We don't assume you know Unix shell, or the compatible
|
| 18 | [OSH]($xref). But shell users will see the similarity, with simplifications
|
| 19 | and upgrades.
|
| 20 |
|
| 21 | Remember, YSH is for Python and JavaScript users who avoid shell! See the
|
| 22 | [project FAQ][FAQ] for more color on that.
|
| 23 |
|
| 24 | [FAQ]: https://www.oilshell.org/blog/2021/01/why-a-new-shell.html
|
| 25 | [path dependence]: https://en.wikipedia.org/wiki/Path_dependence
|
| 26 |
|
| 27 | This document is **long** because it demonstrates nearly every feature of the
|
| 28 | language. You may want to read it in multiple sittings, or read [The Simplest
|
| 29 | Explanation of
|
| 30 | Oil](https://www.oilshell.org/blog/2020/01/simplest-explanation.html) first.
|
| 31 | (Until 2023, YSH was called the "Oil language".)
|
| 32 |
|
| 33 |
|
| 34 | Here's a summary of what follows:
|
| 35 |
|
| 36 | 1. YSH has interleaved *word*, *command*, and *expression* languages.
|
| 37 | - The command language has Ruby-like *blocks*, and the expression language
|
| 38 | has Python-like *data types*.
|
| 39 | 2. YSH has both builtin *commands* like `cd /tmp`, and builtin *functions* like
|
| 40 | `join()`.
|
| 41 | 3. Languages for *data*, like [JSON][], are complementary to YSH code.
|
| 42 | 4. OSH and YSH share both an *interpreter data model* and a *process model*
|
| 43 | (provided by the Unix kernel). Understanding these common models will make
|
| 44 | you both a better shell user and YSH user.
|
| 45 |
|
| 46 | Keep these points in mind as you read the details below.
|
| 47 |
|
| 48 | [JSON]: https://json.org
|
| 49 |
|
| 50 | <div id="toc">
|
| 51 | </div>
|
| 52 |
|
| 53 | ## Preliminaries
|
| 54 |
|
| 55 | Start YSH just like you start bash or Python:
|
| 56 |
|
| 57 | <!-- oils-sh below skips code block extraction, since it doesn't run -->
|
| 58 |
|
| 59 | ```sh-prompt
|
| 60 | bash$ ysh # assuming it's installed
|
| 61 |
|
| 62 | ysh$ echo 'hello world' # command typed into YSH
|
| 63 | hello world
|
| 64 | ```
|
| 65 |
|
| 66 | In the sections below, we'll save space by showing output **in comments**, with
|
| 67 | `=>`:
|
| 68 |
|
| 69 | echo 'hello world' # => hello world
|
| 70 |
|
| 71 | Multi-line output is shown like this:
|
| 72 |
|
| 73 | echo one
|
| 74 | echo two
|
| 75 | # =>
|
| 76 | # one
|
| 77 | # two
|
| 78 |
|
| 79 | ## Examples
|
| 80 |
|
| 81 | ### Hello World Script
|
| 82 |
|
| 83 | You can also type commands into a file like `hello.ysh`. This is a complete
|
| 84 | YSH program, which is identical to a shell program:
|
| 85 |
|
| 86 | echo 'hello world' # => hello world
|
| 87 |
|
| 88 | ### A Taste of YSH
|
| 89 |
|
| 90 | Unlike shell, YSH has `var` and `const` keywords:
|
| 91 |
|
| 92 | const name = 'world' # const is rarer, used the top-level
|
| 93 | echo "hello $name" # => hello world
|
| 94 |
|
| 95 | They take rich Python-like expressions on the right:
|
| 96 |
|
| 97 | var x = 42 # an integer, not a string
|
| 98 | setvar x = x * 2 + 1 # mutate with the 'setvar' keyword
|
| 99 |
|
| 100 | setvar x += 5 # Increment by 5
|
| 101 | echo $x # => 6
|
| 102 |
|
| 103 | var mylist = [x, 7] # two integers [6, 7]
|
| 104 |
|
| 105 | Expressions are often surrounded by `()`:
|
| 106 |
|
| 107 | if (x > 0) {
|
| 108 | echo 'positive'
|
| 109 | } # => positive
|
| 110 |
|
| 111 | for i, item in (mylist) { # 'mylist' is a variable, not a string
|
| 112 | echo "[$i] item $item"
|
| 113 | }
|
| 114 | # =>
|
| 115 | # [0] item 6
|
| 116 | # [1] item 7
|
| 117 |
|
| 118 | YSH has Ruby-like blocks:
|
| 119 |
|
| 120 | cd /tmp {
|
| 121 | echo hi > greeting.txt # file created inside /tmp
|
| 122 | echo $PWD # => /tmp
|
| 123 | }
|
| 124 | echo $PWD # prints the original directory
|
| 125 |
|
| 126 | And utilities to read and write JSON:
|
| 127 |
|
| 128 | var person = {name: 'bob', age: 42}
|
| 129 | json write (person)
|
| 130 | # =>
|
| 131 | # {
|
| 132 | # "name": "bob",
|
| 133 | # "age": 42,
|
| 134 | # }
|
| 135 |
|
| 136 | echo '["str", 42]' | json read # sets '_reply' variable by default
|
| 137 |
|
| 138 | The `=` keyword evaluates and prints an expression:
|
| 139 |
|
| 140 | = _reply
|
| 141 | # => (List) ["str", 42]
|
| 142 |
|
| 143 | (Think of it like `var x = _reply`, without the `var`.)
|
| 144 |
|
| 145 | ## Word Language: Expressions for Strings (and Arrays)
|
| 146 |
|
| 147 | Let's describe the word language first, and then talk about commands and
|
| 148 | expressions. Words are a rich language because **strings** are a central
|
| 149 | concept in shell.
|
| 150 |
|
| 151 | ### Three Kinds of String Literals
|
| 152 |
|
| 153 | You can choose the quoting style that's most convenient to write a given
|
| 154 | string.
|
| 155 |
|
| 156 | #### Double-Quoted, Single-Quoted, and J8 strings (like JSON)
|
| 157 |
|
| 158 | Double-quoted strings allow **interpolation with `$`**:
|
| 159 |
|
| 160 | var person = 'alice'
|
| 161 | echo "hi $person, $(echo bye)" # => hi alice, bye
|
| 162 |
|
| 163 | Write operators by escaping them with `\`:
|
| 164 |
|
| 165 | echo "\$ \" \\ " # => $ " \
|
| 166 |
|
| 167 | In single-quoted strings, all characters are **literal** (except `'`, which
|
| 168 | can't be expressed):
|
| 169 |
|
| 170 | echo 'c:\Program Files\' # => c:\Program Files\
|
| 171 |
|
| 172 | If you want C-style backslash **character escapes**, use a J8 string, which is
|
| 173 | like JSON, but with single quotes::
|
| 174 |
|
| 175 | echo u' A is \u{41} \n line two, with backslash \\'
|
| 176 | # =>
|
| 177 | # A is A
|
| 178 | # line two, with backslash \
|
| 179 |
|
| 180 | The `u''` strings are guaranteed to be valid Unicode (unlike JSON), but you can
|
| 181 | also use `b''` strings:
|
| 182 |
|
| 183 | echo b'byte \yff' # byte that's not valid unicode, like \xff in other languages
|
| 184 | # do not confuse with \u{ff}
|
| 185 |
|
| 186 | #### Multi-line Strings
|
| 187 |
|
| 188 | Multi-line strings are surrounded with triple quotes. They come in the same
|
| 189 | three varieties, and leading whitespace is stripped in a convenient way.
|
| 190 |
|
| 191 | sort <<< """
|
| 192 | var sub: $x
|
| 193 | command sub: $(echo hi)
|
| 194 | expression sub: $[x + 3]
|
| 195 | """
|
| 196 | # =>
|
| 197 | # command sub: hi
|
| 198 | # expression sub: 9
|
| 199 | # var sub: 6
|
| 200 |
|
| 201 | sort <<< '''
|
| 202 | $2.00 # literal $, no interpolation
|
| 203 | $1.99
|
| 204 | '''
|
| 205 | # =>
|
| 206 | # $1.99
|
| 207 | # $2.00
|
| 208 |
|
| 209 | sort <<< u'''
|
| 210 | C\tD
|
| 211 | A\tB
|
| 212 | ''' # b''' strings also supported
|
| 213 | # =>
|
| 214 | # A B
|
| 215 | # C D
|
| 216 |
|
| 217 | (Use multiline strings instead of shell's [here docs]($xref:here-doc).)
|
| 218 |
|
| 219 | ### Three Kinds of Substitution
|
| 220 |
|
| 221 | YSH has syntax for 3 types of substitution, all of which start with `$`. These
|
| 222 | things can all be converted to a **string**:
|
| 223 |
|
| 224 | 1. Variables
|
| 225 | 2. The output of commands
|
| 226 | 3. The value of expressions
|
| 227 |
|
| 228 | #### Variable Sub
|
| 229 |
|
| 230 | The syntax `$a` or `${a}` converts a variable to a string:
|
| 231 |
|
| 232 | var a = 'ale'
|
| 233 | echo $a # => ale
|
| 234 | echo _${a}_ # => _ale_
|
| 235 | echo "_ $a _" # => _ ale _
|
| 236 |
|
| 237 | The shell operator `:-` is occasionally useful in YSH:
|
| 238 |
|
| 239 | echo ${not_defined:-'default'} # => default
|
| 240 |
|
| 241 | #### Command Sub
|
| 242 |
|
| 243 | The `$(echo hi)` syntax runs a command and captures its `stdout`:
|
| 244 |
|
| 245 | echo $(hostname) # => example.com
|
| 246 | echo "_ $(hostname) _" # => _ example.com _
|
| 247 |
|
| 248 | #### Expression Sub
|
| 249 |
|
| 250 | The `$[myexpr]` syntax evaluates an expression and converts it to a string:
|
| 251 |
|
| 252 | echo $[a] # => ale
|
| 253 | echo $[1 + 2 * 3] # => 7
|
| 254 | echo "_ $[1 + 2 * 3] _" # => _ 7 _
|
| 255 |
|
| 256 | <!-- TODO: safe substitution with "$[a]"html -->
|
| 257 |
|
| 258 | ### Arrays of Strings: Globs, Brace Expansion, Splicing, and Splitting
|
| 259 |
|
| 260 | There are four constructs that evaluate to an **list of strings**, rather than
|
| 261 | a single string.
|
| 262 |
|
| 263 | #### Globs
|
| 264 |
|
| 265 | Globs like `*.py` evaluate to a list of files.
|
| 266 |
|
| 267 | touch foo.py bar.py # create the files
|
| 268 | write *.py
|
| 269 | # =>
|
| 270 | # foo.py
|
| 271 | # bar.py
|
| 272 |
|
| 273 | If no files match, it evaluates to an empty list (`[]`).
|
| 274 |
|
| 275 | #### Brace Expansion
|
| 276 |
|
| 277 | The brace expansion mini-language lets you write strings without duplication:
|
| 278 |
|
| 279 | write {alice,bob}@example.com
|
| 280 | # =>
|
| 281 | # alice@example.com
|
| 282 | # bob@example.com
|
| 283 |
|
| 284 | #### Splicing
|
| 285 |
|
| 286 | The `@` operator splices an array into a command:
|
| 287 |
|
| 288 | var myarray = :| ale bean |
|
| 289 | write S @myarray E
|
| 290 | # =>
|
| 291 | # S
|
| 292 | # ale
|
| 293 | # bean
|
| 294 | # E
|
| 295 |
|
| 296 | You also have `@[]` to splice an expression that evaluates to a list:
|
| 297 |
|
| 298 | write -- @[split('ale bean')]
|
| 299 | # =>
|
| 300 | # ale
|
| 301 | # bean
|
| 302 |
|
| 303 | Each item will be converted to a string.
|
| 304 |
|
| 305 | #### Split Command Sub / Split Builtin Sub
|
| 306 |
|
| 307 | There's also a variant of *command sub* that splits first:
|
| 308 |
|
| 309 | write @(seq 3) # write gets 3 arguments
|
| 310 | # =>
|
| 311 | # 1
|
| 312 | # 2
|
| 313 | # 3
|
| 314 |
|
| 315 | <!-- TODO: This should decode J8 notation, which includes "" j"" and b"" -->
|
| 316 |
|
| 317 | ## Command Language: I/O, Control Flow, Abstraction
|
| 318 |
|
| 319 | ### Simple Commands and Redirects
|
| 320 |
|
| 321 | A simple command is a space-separated list of words, which are often unquoted.
|
| 322 | YSH looks up the first word to determine if it's a `proc` or shell builtin.
|
| 323 |
|
| 324 | echo 'hello world' # The shell builtin 'echo'
|
| 325 |
|
| 326 | proc greet (name) { # A proc is like a procedure or process
|
| 327 | echo "hello $name"
|
| 328 | }
|
| 329 |
|
| 330 | # Now the first word will resolve to the proc
|
| 331 | greet alice # => hello alice
|
| 332 |
|
| 333 | If it's neither, then it's assumed to be an external command:
|
| 334 |
|
| 335 | ls -l /tmp # The external 'ls' command
|
| 336 |
|
| 337 | Commands accept traditional string arguments, as well as typed arguments in
|
| 338 | parentheses:
|
| 339 |
|
| 340 | # 'write' is a string arg; 'x' is a typed expression arg
|
| 341 | json write (x)
|
| 342 |
|
| 343 | You can **redirect** `stdin` and `stdout` of simple commands:
|
| 344 |
|
| 345 | echo hi > tmp.txt # write to a file
|
| 346 | sort < tmp.txt
|
| 347 |
|
| 348 | Idioms for using stderr (identical to shell):
|
| 349 |
|
| 350 | ls /tmp 2>errors.txt
|
| 351 | echo 'fatal error' 1>&2
|
| 352 |
|
| 353 | "Simple" commands in YSH can also have typed `()` and block `{}` args, which
|
| 354 | we'll see in the section on "procs".
|
| 355 |
|
| 356 | ### Pipelines
|
| 357 |
|
| 358 | Pipelines are a powerful method manipulating data streams:
|
| 359 |
|
| 360 | ls | wc -l # count files in this directory
|
| 361 | find /bin -type f | xargs wc -l # count files in a subtree
|
| 362 |
|
| 363 | The stream may contain (lines of) text, binary data, JSON, TSV, and more.
|
| 364 | Details below.
|
| 365 |
|
| 366 | ### Multi-line Commands
|
| 367 |
|
| 368 | The YSH `...` prefix lets you write long commands, pipelines, and `&&` chains
|
| 369 | without `\` line continuations.
|
| 370 |
|
| 371 | ... find /bin # traverse this directory and
|
| 372 | -type f -a -executable # print executable files
|
| 373 | | sort -r # reverse sort
|
| 374 | | head -n 30 # limit to 30 files
|
| 375 | ;
|
| 376 |
|
| 377 | When this mode is active:
|
| 378 |
|
| 379 | - A single newline behaves like a space
|
| 380 | - A blank line (two newlines in a row) is illegal, but a line that has only a
|
| 381 | comment is allowed. This prevents confusion if you forget the `;`
|
| 382 | terminator.
|
| 383 |
|
| 384 | ### `var`, `setvar`, `const` to Declare and Mutate
|
| 385 |
|
| 386 | Constants can't be modified:
|
| 387 |
|
| 388 | const myconst = 'mystr'
|
| 389 | # setvar myconst = 'foo' would be an error
|
| 390 |
|
| 391 | Modify variables with the `setvar` keyword:
|
| 392 |
|
| 393 | var num_beans = 12
|
| 394 | setvar num_beans = 13
|
| 395 |
|
| 396 | A more complex example:
|
| 397 |
|
| 398 | var d = {name: 'bob', age: 42} # dict literal
|
| 399 | setvar d.name = 'alice' # d.name is a synonym for d['name']
|
| 400 | echo $[d.name] # => alice
|
| 401 |
|
| 402 | That's most of what you need to know about assignments. Advanced users may
|
| 403 | want to use `setglobal` or `call myplace->setValue(42)` in certain situations.
|
| 404 |
|
| 405 | <!--
|
| 406 | var g = 1
|
| 407 | var h = 2
|
| 408 | proc demo(:out) {
|
| 409 | setglobal g = 42
|
| 410 | setref out = 43
|
| 411 | }
|
| 412 | demo :h # pass a reference to h
|
| 413 | echo "$g $h" # => 42 43
|
| 414 | -->
|
| 415 |
|
| 416 | More details: [Variable Declaration and Mutation](variables.html).
|
| 417 |
|
| 418 | ### `for` Loop
|
| 419 |
|
| 420 | Shell-style for loops iterate over **words**:
|
| 421 |
|
| 422 | for word in 'oils' $num_beans {pea,coco}nut {
|
| 423 | echo $word
|
| 424 | }
|
| 425 | # =>
|
| 426 | # oils
|
| 427 | # 13
|
| 428 | # peanut
|
| 429 | # coconut
|
| 430 |
|
| 431 | You can also request the loop index:
|
| 432 |
|
| 433 | for i, word in README.md *.py {
|
| 434 | echo "$i - $word"
|
| 435 | }
|
| 436 | # =>
|
| 437 | # 0 - README.md
|
| 438 | # 1 - __init__.py
|
| 439 |
|
| 440 | To iterate over lines of `stdin`, use:
|
| 441 |
|
| 442 | for line in (stdin) {
|
| 443 | echo $line
|
| 444 | }
|
| 445 | # lines are buffered, so it's much faster than `while read --rawline`
|
| 446 |
|
| 447 | Ask for the loop index:
|
| 448 |
|
| 449 | for i, line in (stdin) {
|
| 450 | echo "$i $line"
|
| 451 | }
|
| 452 |
|
| 453 | To iterate over a typed data, use parentheses around an **expression**. The
|
| 454 | expression should evaluate to an integer `Range`, `List`, or `Dict`:
|
| 455 |
|
| 456 | for i in (3 .. 5) { # range operator ..
|
| 457 | echo "i = $i"
|
| 458 | }
|
| 459 | # =>
|
| 460 | # i = 3
|
| 461 | # i = 4
|
| 462 |
|
| 463 | List:
|
| 464 |
|
| 465 | var foods = ['ale', 'bean']
|
| 466 | for item in (foods) {
|
| 467 | echo $item
|
| 468 | }
|
| 469 | # =>
|
| 470 | # ale
|
| 471 | # bean
|
| 472 |
|
| 473 | Again, you can request the index with `for i, item in ...`.
|
| 474 |
|
| 475 | Here's the most general form of the loop over `Dict`:
|
| 476 |
|
| 477 | var mydict = {pea: 42, nut: 10}
|
| 478 | for i, k, v in (mydict) {
|
| 479 | echo "$i - $k - $v"
|
| 480 | }
|
| 481 | # =>
|
| 482 | # 0 - pea - 42
|
| 483 | # 1 - nut - 10
|
| 484 |
|
| 485 | There are two simpler forms:
|
| 486 |
|
| 487 | - One variable gives you the key: `for k in (mydict)`
|
| 488 | - Two variables gives you the key and value: `for k, v in (mydict)`
|
| 489 |
|
| 490 | (One way to think of it: `for` loops in YSH have the functionality Python's
|
| 491 | `enumerate()`, `items()`, `keys()`, and `values()`.)
|
| 492 |
|
| 493 | <!--
|
| 494 | TODO: Str loop should give you the (UTF-8 offset, rune)
|
| 495 | Or maybe just UTF-8 offset? Decoding errors could be exceptions, or Unicode
|
| 496 | replacement.
|
| 497 | -->
|
| 498 |
|
| 499 | ### `while` Loop
|
| 500 |
|
| 501 | While loops can use a **command** as the termination condition:
|
| 502 |
|
| 503 | while test --file lock {
|
| 504 | sleep 1
|
| 505 | }
|
| 506 |
|
| 507 | Or an **expression**, which is surrounded in `()`:
|
| 508 |
|
| 509 | var i = 3
|
| 510 | while (i < 6) {
|
| 511 | echo "i = $i"
|
| 512 | setvar i += 1
|
| 513 | }
|
| 514 | # =>
|
| 515 | # i = 3
|
| 516 | # i = 4
|
| 517 | # i = 5
|
| 518 |
|
| 519 | ### `if elif` Conditional
|
| 520 |
|
| 521 | If statements test the exit code of a command, and have optional `elif` and
|
| 522 | `else` clauses:
|
| 523 |
|
| 524 | if test --file foo {
|
| 525 | echo 'foo is a file'
|
| 526 | rm --verbose foo # delete it
|
| 527 | } elif test --dir foo {
|
| 528 | echo 'foo is a directory'
|
| 529 | } else {
|
| 530 | echo 'neither'
|
| 531 | }
|
| 532 |
|
| 533 | Invert the exit code with `!`:
|
| 534 |
|
| 535 | if ! grep alice /etc/passwd {
|
| 536 | echo 'alice is not a user'
|
| 537 | }
|
| 538 |
|
| 539 | As with `while` loops, the condition can also be an **expression** wrapped in
|
| 540 | `()`:
|
| 541 |
|
| 542 | if (num_beans > 0) {
|
| 543 | echo 'so many beans'
|
| 544 | }
|
| 545 |
|
| 546 | var done = false
|
| 547 | if (not done) { # negate with 'not' operator (contrast with !)
|
| 548 | echo "we aren't done"
|
| 549 | }
|
| 550 |
|
| 551 | ### `case` Conditional
|
| 552 |
|
| 553 | The case statement is a series of conditionals and executable blocks. The
|
| 554 | condition can be either an unquoted glob pattern like `*.py`, an eggex pattern
|
| 555 | like `/d+/`, or a typed expression like `(42)`:
|
| 556 |
|
| 557 | var s = 'README.md'
|
| 558 | case (s) {
|
| 559 | *.py { echo 'Python' }
|
| 560 | *.cc | *.h { echo 'C++' }
|
| 561 | * { echo 'Other' }
|
| 562 | }
|
| 563 | # => Other
|
| 564 |
|
| 565 | case (s) {
|
| 566 | / dot* '.md' / { echo 'Markdown' }
|
| 567 | (30 + 12) { echo 'the integer 42' }
|
| 568 | (else) { echo 'neither' }
|
| 569 | }
|
| 570 | # => Markdown
|
| 571 |
|
| 572 | <!-- TODO: document case on typed data -->
|
| 573 |
|
| 574 | (Shell style like `if foo; then ... fi` and `case $x in ... esac` is also legal,
|
| 575 | but discouraged in YSH code.)
|
| 576 |
|
| 577 | ### Error Handling
|
| 578 |
|
| 579 | If statements are also used for **error handling**. Builtins and external
|
| 580 | commands use this style:
|
| 581 |
|
| 582 | if ! test -d /bin {
|
| 583 | echo 'not a directory'
|
| 584 | }
|
| 585 |
|
| 586 | if ! cp foo /tmp {
|
| 587 | echo 'error copying' # any non-zero status
|
| 588 | }
|
| 589 |
|
| 590 | Procs use this style (because of shell's *disabled `errexit` quirk*):
|
| 591 |
|
| 592 | try {
|
| 593 | myproc
|
| 594 | }
|
| 595 | if failed {
|
| 596 | echo 'failed'
|
| 597 | }
|
| 598 |
|
| 599 | For a complete list of examples, see [YSH Error
|
| 600 | Handling](ysh-error.html). For design goals and a reference, see [YSH
|
| 601 | Fixes Shell's Error Handling](error-handling.html).
|
| 602 |
|
| 603 | #### `break`, `continue`, `return`, `exit`
|
| 604 |
|
| 605 | The `exit` **keyword** exits a process (it's not a shell builtin.) The other 3
|
| 606 | control flow keywords behave like they do in Python and JavaScript.
|
| 607 |
|
| 608 | ### Ruby-like Blocks
|
| 609 |
|
| 610 | Here's a builtin command that takes a literal block argument:
|
| 611 |
|
| 612 | shopt --unset errexit { # ignore errors
|
| 613 | cp ale /tmp
|
| 614 | cp bean /bin
|
| 615 | }
|
| 616 |
|
| 617 | Blocks are a special kind of typed argument passed to commands like `shopt`.
|
| 618 | Their type is `value.Command`.
|
| 619 |
|
| 620 | ### Shell-like `proc`
|
| 621 |
|
| 622 | You can define units of code with the `proc` keyword.
|
| 623 |
|
| 624 | proc mycopy (src, dest) {
|
| 625 | ### Copy verbosely
|
| 626 |
|
| 627 | mkdir -p $dest
|
| 628 | cp --verbose $src $dest
|
| 629 | }
|
| 630 |
|
| 631 | The `###` line is a "doc comment", and can be retrieved with `pp proc`. Simple
|
| 632 | procs like this are invoked like a shell command:
|
| 633 |
|
| 634 | touch log.txt
|
| 635 | mycopy log.txt /tmp # first word 'mycopy' is a proc
|
| 636 |
|
| 637 | Procs have more features, including **four** kinds of arguments:
|
| 638 |
|
| 639 | 1. Word args (which are always strings)
|
| 640 | 1. Typed, positional args (aka positional args)
|
| 641 | 1. Typed, named args (aka named args)
|
| 642 | 1. A final block argument, which may be written with `{ }`.
|
| 643 |
|
| 644 | At the call site, they can look like any of these forms:
|
| 645 |
|
| 646 | cd /tmp # word arg
|
| 647 |
|
| 648 | json write (d) # word arg, then positional arg
|
| 649 |
|
| 650 | # error 'failed' (status=9) # word arg, then named arg
|
| 651 |
|
| 652 | cd /tmp { echo $PWD } # word arg, then block arg
|
| 653 |
|
| 654 | var mycmd = ^(echo hi) # expression for a value.Command
|
| 655 | eval (mycmd) # positional arg
|
| 656 |
|
| 657 | <!-- TODO: lazy arg list: ls8 | where [age > 10] -->
|
| 658 |
|
| 659 | At the definition site, the kinds of parameters are separated with `;`, similar
|
| 660 | to the Julia language:
|
| 661 |
|
| 662 | proc p2 (word1, word2; pos1, pos2, ...rest_pos) {
|
| 663 | echo "$word1 $word2 $[pos1 + pos2]"
|
| 664 | json write (rest_pos)
|
| 665 | }
|
| 666 |
|
| 667 | proc p3 (w ; ; named1, named2, ...rest_named; block) {
|
| 668 | echo "$w $[named1 + named2]"
|
| 669 | eval (block)
|
| 670 | json write (rest_named)
|
| 671 | }
|
| 672 |
|
| 673 | proc p4 (; ; ; block) {
|
| 674 | eval (block)
|
| 675 | }
|
| 676 |
|
| 677 | YSH also has Python-like functions defined with `func`. These are part of the
|
| 678 | expression language, which we'll see later.
|
| 679 |
|
| 680 | For more info, see the [Informal Guide to Procs and Funcs](proc-func.html)
|
| 681 | (under construction).
|
| 682 |
|
| 683 | #### Builtin Commands
|
| 684 |
|
| 685 | **Shell builtins** like `cd` and `read` are the "standard library" of the
|
| 686 | command language. Each one takes various flags:
|
| 687 |
|
| 688 | cd -L . # follow symlinks
|
| 689 |
|
| 690 | echo foo | read --all # read all of stdin
|
| 691 |
|
| 692 | Here are some categories of builtin:
|
| 693 |
|
| 694 | - I/O: `echo write read`
|
| 695 | - File system: `cd test`
|
| 696 | - Processes: `fork wait forkwait exec`
|
| 697 | - Interpreter settings: `shopt shvar`
|
| 698 | - Meta: `command builtin runproc type eval`
|
| 699 |
|
| 700 | <!-- TODO: Link to a comprehensive list of builtins -->
|
| 701 |
|
| 702 | ## Expression Language: Python-like Types
|
| 703 |
|
| 704 | YSH expressions look and behave more like Python or JavaScript than shell. For
|
| 705 | example, we write `if (x < y)` instead of `if [ $x -lt $y ]`. Expressions are
|
| 706 | usually surrounded by `( )`.
|
| 707 |
|
| 708 | At runtime, variables like `x` and `y` are bounded to **typed data**, like
|
| 709 | integers, floats, strings, lists, and dicts.
|
| 710 |
|
| 711 | <!--
|
| 712 | [Command vs. Expression Mode](command-vs-expression-mode.html) may help you
|
| 713 | understand how YSH is parsed.
|
| 714 | -->
|
| 715 |
|
| 716 | ### Python-like `func`
|
| 717 |
|
| 718 | At the end of the *Command Language*, we saw that procs are shell-like units of
|
| 719 | code. Now let's talk about Python-like **functions** in YSH, which are
|
| 720 | different than `procs`:
|
| 721 |
|
| 722 | - They're defined with the `func` keyword.
|
| 723 | - They're called in expressions, not in commands.
|
| 724 | - They're **pure**, and live in the **interior** of a process.
|
| 725 | - In contrast, procs usually perform I/O, and have **exterior** boundaries.
|
| 726 |
|
| 727 | Here's a function that mutates its argument:
|
| 728 |
|
| 729 | func popTwice(mylist) {
|
| 730 | call mylist->pop()
|
| 731 | call mylist->pop()
|
| 732 | }
|
| 733 |
|
| 734 | var mylist = [3, 4]
|
| 735 |
|
| 736 | # The call keyword is an "adapter" between commands and expressions,
|
| 737 | # like the = keyword.
|
| 738 | call popTwice(mylist)
|
| 739 |
|
| 740 | Here's a pure function:
|
| 741 |
|
| 742 | func myRepeat(s, n; special=false) { # positional; named params
|
| 743 | var parts = []
|
| 744 | for i in (0 .. n) {
|
| 745 | append $s (parts)
|
| 746 | }
|
| 747 | var result = join(parts)
|
| 748 |
|
| 749 | if (special) {
|
| 750 | return ("$result !!") # parens required for typed return
|
| 751 | } else {
|
| 752 | return (result)
|
| 753 | }
|
| 754 | }
|
| 755 |
|
| 756 | echo $[myRepeat('z', 3)] # => zzz
|
| 757 |
|
| 758 | echo $[myRepeat('z', 3, special=true)] # => zzz !!
|
| 759 |
|
| 760 | Funcs are named using `camelCase`, while procs use `kebab-case`. See the
|
| 761 | [Style Guide](style-guide.html) for more conventions.
|
| 762 |
|
| 763 | #### Builtin Functions
|
| 764 |
|
| 765 | In addition, to builtin commands, YSH has Python-like builtin **functions**.
|
| 766 | These are like the "standard library" for the expression language. Examples:
|
| 767 |
|
| 768 | - Functions that take multiple types: `len() type()`
|
| 769 | - Conversions: `bool() int() float() str() list() ...`
|
| 770 | - Explicit word evaluation: `split() join() glob() maybe()`
|
| 771 |
|
| 772 | <!-- TODO: Make a comprehensive list of func builtins. -->
|
| 773 |
|
| 774 |
|
| 775 | ### Data Types: `Int`, `Str`, `List`, `Dict`, ...
|
| 776 |
|
| 777 | YSH has data types, each with an expression syntax and associated methods.
|
| 778 |
|
| 779 | ### Methods
|
| 780 |
|
| 781 | Mutating methods are looked up with a thin arrow `->`:
|
| 782 |
|
| 783 | var foods = ['ale', 'bean']
|
| 784 | var last = foods->pop() # bean
|
| 785 | write @foods # => ale
|
| 786 |
|
| 787 | You can ignore the return value with the `call` keyword:
|
| 788 |
|
| 789 | call foods->pop()
|
| 790 |
|
| 791 | Transforming methods use a fat arrow `=>`:
|
| 792 |
|
| 793 | var line = ' ale bean '
|
| 794 | var trimmed = line => trim() => upper() # 'ALE BEAN'
|
| 795 |
|
| 796 | If the `=>` operator doesn't find a method with the given name in the object's
|
| 797 | type, it looks for free functions:
|
| 798 |
|
| 799 | # list() is a free function taking one arg
|
| 800 | # join() is a free function taking two args
|
| 801 | var x = {k1: 42, k2: 43} => list() => join('/') # 'K1/K2'
|
| 802 |
|
| 803 | This allows a left-to-right "method chaining" style.
|
| 804 |
|
| 805 | ---
|
| 806 |
|
| 807 | Now let's go through the data types in YSH. We'll show the syntax for
|
| 808 | literals, and what **methods** they have.
|
| 809 |
|
| 810 | #### Null and Bool
|
| 811 |
|
| 812 | YSH uses JavaScript-like spellings these three "atoms":
|
| 813 |
|
| 814 | var x = null
|
| 815 |
|
| 816 | var b1, b2 = true, false
|
| 817 |
|
| 818 | if (b1) {
|
| 819 | echo 'yes'
|
| 820 | } # => yes
|
| 821 |
|
| 822 |
|
| 823 | #### Int
|
| 824 |
|
| 825 | There are many ways to write integers:
|
| 826 |
|
| 827 | var small, big = 42, 65_536
|
| 828 | echo "$small $big" # => 42 65536
|
| 829 |
|
| 830 | var hex, octal, binary = 0x0001_0000, 0o755, 0b0001_0101
|
| 831 | echo "$hex $octal $binary" # => 65536 493 21
|
| 832 |
|
| 833 | <!--
|
| 834 | "Runes" are integers that represent Unicode code points. They're not common in
|
| 835 | YSH code, but can make certain string algorithms more readable.
|
| 836 |
|
| 837 | # Pound rune literals are similar to ord('A')
|
| 838 | const a = #'A'
|
| 839 |
|
| 840 | # Backslash rune literals can appear outside of quotes
|
| 841 | const newline = \n # Remember this is an integer
|
| 842 | const backslash = \\ # ditto
|
| 843 |
|
| 844 | # Unicode rune literal is syntactic sugar for 0x3bc
|
| 845 | const mu = \u{3bc}
|
| 846 |
|
| 847 | echo "chars $a $newline $backslash $mu" # => chars 65 10 92 956
|
| 848 | -->
|
| 849 |
|
| 850 | #### Float
|
| 851 |
|
| 852 | Floats are written like you'd expect:
|
| 853 |
|
| 854 | var small = 1.5e-10
|
| 855 | var big = 3.14
|
| 856 |
|
| 857 | #### Str
|
| 858 |
|
| 859 | See the section above called *Three Kinds of String Literals*. It described
|
| 860 | `'single quoted'`, `"double ${quoted}"`, and `u'J8-style\n'` strings; as well
|
| 861 | as their multiline variants.
|
| 862 |
|
| 863 | Strings are UTF-8 encoded in memory, like strings in the [Go
|
| 864 | language](https://golang.org). There isn't a separate string and unicode type,
|
| 865 | as in Python.
|
| 866 |
|
| 867 | Strings are **immutable**, as in Python and JavaScript. This means they only
|
| 868 | have **transforming** methods:
|
| 869 |
|
| 870 | var x = s => trim()
|
| 871 |
|
| 872 | Other methods:
|
| 873 |
|
| 874 | - `trimLeft() trimRight()`
|
| 875 | - `trimPrefix() trimSuffix()`
|
| 876 | - `upper() lower()` (not implemented)
|
| 877 |
|
| 878 | <!--
|
| 879 | The syntax `:symbol` could be an interned string.
|
| 880 | -->
|
| 881 |
|
| 882 | #### List (and Arrays)
|
| 883 |
|
| 884 | All lists can be expressed with Python-like literals:
|
| 885 |
|
| 886 | var foods = ['ale', 'bean', 'corn']
|
| 887 | var recursive = [1, [2, 3]]
|
| 888 |
|
| 889 | As a special case, list of strings are called **arrays**. It's often more
|
| 890 | convenient to write them with shell-like literals:
|
| 891 |
|
| 892 | # No quotes or commas
|
| 893 | var foods = :| ale bean corn |
|
| 894 |
|
| 895 | # You can use the word language here
|
| 896 | var other = :| foo $s *.py {alice,bob}@example.com |
|
| 897 |
|
| 898 | Lists are **mutable**, as in Python and JavaScript. So they mainly have
|
| 899 | mutating methods:
|
| 900 |
|
| 901 | call foods->reverse()
|
| 902 | write -- @foods
|
| 903 | # =>
|
| 904 | # corn
|
| 905 | # bean
|
| 906 | # ale
|
| 907 |
|
| 908 | #### Dict
|
| 909 |
|
| 910 | Dicts use syntax that's more like JavaScript than Python. Here's a dict
|
| 911 | literal:
|
| 912 |
|
| 913 | var d = {
|
| 914 | name: 'bob', # unquoted keys are allowed
|
| 915 | age: 42,
|
| 916 | 'key with spaces': 'val'
|
| 917 | }
|
| 918 |
|
| 919 | There are two syntaxes for key lookup. If the key doesn't exist, it's a fatal
|
| 920 | error.
|
| 921 |
|
| 922 | var v1 = d['name']
|
| 923 | var v2 = d.name # shorthand for the above
|
| 924 | var v3 = d['key with spaces'] # no shorthand for this
|
| 925 |
|
| 926 | Keys names can be computed with expressions in `[]`:
|
| 927 |
|
| 928 | var key = 'alice'
|
| 929 | var d2 = {[key ++ '_z']: 'ZZZ'} # Computed key name
|
| 930 | echo $[d2.alice_z] # => ZZZ # Reminder: expression sub
|
| 931 |
|
| 932 | Omitting the value causes it to be taken from a variable of the same name:
|
| 933 |
|
| 934 | var d3 = {key} # value is taken from the environment
|
| 935 | echo "name is $[d3.key]" # => name is alice
|
| 936 |
|
| 937 | More:
|
| 938 |
|
| 939 | var empty = {}
|
| 940 | echo $[len(empty)] # => 0
|
| 941 |
|
| 942 | Dicts are **mutable**, as in Python and JavaScript. But the `keys()` and `values()`
|
| 943 | methods return new `List` objects:
|
| 944 |
|
| 945 | var keys = d2 => keys() # => alice_z
|
| 946 | # var vals = d3 => values() # => alice
|
| 947 |
|
| 948 | ### `Place` type / "out params"
|
| 949 |
|
| 950 | The `read` builtin can either set an implicit variable `_reply`:
|
| 951 |
|
| 952 | whoami | read --all # sets _reply
|
| 953 |
|
| 954 | Or you can pass a `value.Place`, created with `&`
|
| 955 |
|
| 956 | var x # implicitly initialized to null
|
| 957 | whoami | read --all (&x) # mutate this "place"
|
| 958 | echo who=$x # => who=andy
|
| 959 |
|
| 960 | #### Quotation Types: value.Command (Block) and value.Expr
|
| 961 |
|
| 962 | These types are for reflection on YSH code. Most YSH programs won't use them
|
| 963 | directly.
|
| 964 |
|
| 965 | - `Command`: an unevaluated code block.
|
| 966 | - rarely-used literal: `^(ls | wc -l)`
|
| 967 | - `Expr`: an unevaluated expression.
|
| 968 | - rarely-used literal: `^[42 + a[i]]`
|
| 969 |
|
| 970 | <!-- TODO: implement Block, Expr, ArgList types (variants of value) -->
|
| 971 |
|
| 972 | ### Operators
|
| 973 |
|
| 974 | Operators are generally the same as in Python:
|
| 975 |
|
| 976 | if (10 <= num_beans and num_beans < 20) {
|
| 977 | echo 'enough'
|
| 978 | } # => enough
|
| 979 |
|
| 980 | YSH has a few operators that aren't in Python. Equality can be approximate or
|
| 981 | exact:
|
| 982 |
|
| 983 | var n = ' 42 '
|
| 984 | if (n ~== 42) {
|
| 985 | echo 'equal after stripping whitespace and type conversion'
|
| 986 | } # => equal after stripping whitespace type conversion
|
| 987 |
|
| 988 | if (n === 42) {
|
| 989 | echo "not reached because strings and ints aren't equal"
|
| 990 | }
|
| 991 |
|
| 992 | <!-- TODO: is n === 42 a type error? -->
|
| 993 |
|
| 994 | Pattern matching can be done with globs (`~~` and `!~~`)
|
| 995 |
|
| 996 | const filename = 'foo.py'
|
| 997 | if (filename ~~ '*.py') {
|
| 998 | echo 'Python'
|
| 999 | } # => Python
|
| 1000 |
|
| 1001 | if (filename !~~ '*.sh') {
|
| 1002 | echo 'not shell'
|
| 1003 | } # => not shell
|
| 1004 |
|
| 1005 | or regular expressions (`~` and `!~`). See the Eggex section below for an
|
| 1006 | example of the latter.
|
| 1007 |
|
| 1008 | Concatenation is `++` rather than `+` because it avoids confusion in the
|
| 1009 | presence of type conversion:
|
| 1010 |
|
| 1011 | var n = 42 + 1 # string plus int does implicit conversion
|
| 1012 | echo $n # => 43
|
| 1013 |
|
| 1014 | var y = 'ale ' ++ "bean $n" # concatenation
|
| 1015 | echo $y # => ale bean 43
|
| 1016 |
|
| 1017 | <!--
|
| 1018 | TODO: change example above
|
| 1019 | var n = '42' + 1 # string plus int does implicit conversion
|
| 1020 | -->
|
| 1021 |
|
| 1022 | <!--
|
| 1023 |
|
| 1024 | #### Summary of Operators
|
| 1025 |
|
| 1026 | - Arithmetic: `+ - * / // %` and `**` for exponentatiation
|
| 1027 | - `/` always yields a float, and `//` is integer division
|
| 1028 | - Bitwise: `& | ^ ~`
|
| 1029 | - Logical: `and or not`
|
| 1030 | - Comparison: `== < > <= >= in 'not in'`
|
| 1031 | - Approximate equality: `~==`
|
| 1032 | - Eggex and glob match: `~ !~ ~~ !~~`
|
| 1033 | - Ternary: `1 if x else 0`
|
| 1034 | - Index and slice: `mylist[3]` and `mylist[1:3]`
|
| 1035 | - `mydict->key` is a shortcut for `mydict['key']`
|
| 1036 | - Function calls
|
| 1037 | - free: `f(x, y)`
|
| 1038 | - transformations and chaining: `s => startWith('prefix')`
|
| 1039 | - mutating methods: `mylist->pop()`
|
| 1040 | - String and List: `++` for concatenation
|
| 1041 | - This is a separate operator because the addition operator `+` does
|
| 1042 | string-to-int conversion
|
| 1043 |
|
| 1044 | TODO: What about list comprehensions?
|
| 1045 | -->
|
| 1046 |
|
| 1047 | ### Egg Expressions (YSH Regexes)
|
| 1048 |
|
| 1049 | An *Eggex* is a type of YSH expression that denote regular expressions. They
|
| 1050 | translate to POSIX ERE syntax, for use with tools like `egrep`, `awk`, and `sed
|
| 1051 | --regexp-extended` (GNU only).
|
| 1052 |
|
| 1053 | They're designed to be readable and composable. Example:
|
| 1054 |
|
| 1055 | var D = / digit{1,3} /
|
| 1056 | var ip_pattern = / D '.' D '.' D '.' D'.' /
|
| 1057 |
|
| 1058 | var z = '192.168.0.1'
|
| 1059 | if (z ~ ip_pattern) { # Use the ~ operator to match
|
| 1060 | echo "$z looks like an IP address"
|
| 1061 | } # => 192.168.0.1 looks like an IP address
|
| 1062 |
|
| 1063 | if (z !~ / '.255' %end /) {
|
| 1064 | echo "doesn't end with .255"
|
| 1065 | } # => doesn't end with .255"
|
| 1066 |
|
| 1067 | See the [Egg Expressions doc](eggex.html) for details.
|
| 1068 |
|
| 1069 | ## Interlude
|
| 1070 |
|
| 1071 | Let's review what we've seen before moving onto other YSH features.
|
| 1072 |
|
| 1073 | ### Three Interleaved Languages
|
| 1074 |
|
| 1075 | Here are the languages we saw in the last 3 sections:
|
| 1076 |
|
| 1077 | 1. **Words** evaluate to a string, or list of strings. This includes:
|
| 1078 | - literals like `'mystr'`
|
| 1079 | - substitutions like `${x}` and `$(hostname)`
|
| 1080 | - globs like `*.sh`
|
| 1081 | 2. **Commands** are used for
|
| 1082 | - I/O: pipelines, builtins like `read`
|
| 1083 | - control flow: `if`, `for`
|
| 1084 | - abstraction: `proc`
|
| 1085 | 3. **Expressions** on typed data are borrowed from Python, with some JavaScript
|
| 1086 | influence.
|
| 1087 | - Lists: `['ale', 'bean']` or `:| ale bean |`
|
| 1088 | - Dicts: `{name: 'bob', age: 42}`
|
| 1089 | - Functions: `split('ale bean')` and `join(['pea', 'nut'])`
|
| 1090 |
|
| 1091 | ### How Do They Work Together?
|
| 1092 |
|
| 1093 | Here are two examples:
|
| 1094 |
|
| 1095 | (1) In this this *command*, there are **four** *words*. The fourth word is an
|
| 1096 | *expression sub* `$[]`.
|
| 1097 |
|
| 1098 | write hello $name $[d['age'] + 1]
|
| 1099 | # =>
|
| 1100 | # hello
|
| 1101 | # world
|
| 1102 | # 43
|
| 1103 |
|
| 1104 | (2) In this assignment, the *expression* on the right hand side of `=`
|
| 1105 | concatenates two strings. The first string is a literal, and the second is a
|
| 1106 | *command sub*.
|
| 1107 |
|
| 1108 | var food = 'ale ' ++ $(echo bean | tr a-z A-Z)
|
| 1109 | write $food # => ale BEAN
|
| 1110 |
|
| 1111 | So words, commands, and expressions are **mutually recursive**. If you're a
|
| 1112 | conceptual person, skimming [Syntactic Concepts](syntactic-concepts.html) may
|
| 1113 | help you understand this on a deeper level.
|
| 1114 |
|
| 1115 | <!--
|
| 1116 | One way to think about these sublanguages is to note that the `|` character
|
| 1117 | means something different in each context:
|
| 1118 |
|
| 1119 | - In the command language, it's the pipeline operator, as in `ls | wc -l`
|
| 1120 | - In the word language, it's only valid in a literal string like `'|'`, `"|"`,
|
| 1121 | or `\|`. (It's also used in `${x|html}`, which formats a string.)
|
| 1122 | - In the expression language, it's the bitwise OR operator, as in Python and
|
| 1123 | JavaScript.
|
| 1124 | -->
|
| 1125 |
|
| 1126 | ## Languages for Data (Interchange Formats)
|
| 1127 |
|
| 1128 | In addition to languages for **code**, YSH also deals with languages for
|
| 1129 | **data**. [JSON]($xref) is a prominent example of the latter.
|
| 1130 |
|
| 1131 | <!-- TODO: Link to slogans, fallacies, and concepts -->
|
| 1132 |
|
| 1133 | ### UTF-8
|
| 1134 |
|
| 1135 | UTF-8 is the foundation of our textual data languages.
|
| 1136 |
|
| 1137 | <!-- TODO: there's a runes() iterator which gives integer offsets, usable for
|
| 1138 | slicing -->
|
| 1139 |
|
| 1140 | <!-- TODO: write about J8 notation -->
|
| 1141 |
|
| 1142 | ### Lines of Text (traditional), and JSON/J8 Strings
|
| 1143 |
|
| 1144 | Traditional Unix tools like `grep` and `awk` operate on streams of lines. YSH
|
| 1145 | supports this style, just like any other shell.
|
| 1146 |
|
| 1147 | But YSH also has [J8 Notation][], a data format based on [JSON][].
|
| 1148 |
|
| 1149 | [J8 Notation]: j8-notation.html
|
| 1150 |
|
| 1151 | It lets you encode arbitrary byte strings into a single (readable) line,
|
| 1152 | including those with newlines and terminal escape sequences.
|
| 1153 |
|
| 1154 | Example:
|
| 1155 |
|
| 1156 | # A line with a tab char in the middle
|
| 1157 | var mystr = u'pea\t' ++ u'42\n'
|
| 1158 |
|
| 1159 | # Print it as JSON
|
| 1160 | write $[toJson(mystr)] # => "pea\t42\n"
|
| 1161 |
|
| 1162 | # JSON8 is the same, but it's not lossy for binary data
|
| 1163 | write $[toJson8(mystr)] # => "pea\t42\n"
|
| 1164 |
|
| 1165 | ### Structured: JSON8, TSV8
|
| 1166 |
|
| 1167 | You can write and read **tree-shaped** as [JSON][]:
|
| 1168 |
|
| 1169 | var d = {key: 'value'}
|
| 1170 | json write (d) # dump variable d as JSON
|
| 1171 | # =>
|
| 1172 | # {
|
| 1173 | # "key": "value"
|
| 1174 | # }
|
| 1175 |
|
| 1176 | echo '["ale", 42]' > example.json
|
| 1177 |
|
| 1178 | json read (&d2) < example.json # parse JSON into var d2
|
| 1179 | pp cell d2 # inspect the in-memory value
|
| 1180 | # =>
|
| 1181 | # ['ale', 42]
|
| 1182 |
|
| 1183 | [JSON][] will lose information when strings have binary data, but the slight
|
| 1184 | [JSON8]($xref) upgrade won't:
|
| 1185 |
|
| 1186 | var b = {binary: $'\xff'}
|
| 1187 | json8 write (b)
|
| 1188 | # =>
|
| 1189 | # {
|
| 1190 | # "binary": b'\yff'
|
| 1191 | # }
|
| 1192 |
|
| 1193 | [JSON]: $xref
|
| 1194 |
|
| 1195 | <!--
|
| 1196 | TODO:
|
| 1197 | - Fix pp cell output
|
| 1198 | - Use json write (d) syntax
|
| 1199 | -->
|
| 1200 |
|
| 1201 | **Table-shaped** data can be read and written as [TSV8]($xref). (TODO: not yet
|
| 1202 | implemented.)
|
| 1203 |
|
| 1204 | <!-- Figure out the API. Does it work like JSON?
|
| 1205 |
|
| 1206 | Or I think we just implement
|
| 1207 | - rows: 'where' or 'filter' (dplyr)
|
| 1208 | - cols: 'select' conflicts with shell builtin; call it 'cols'?
|
| 1209 | - sort: 'sort-by' or 'arrange' (dplyr)
|
| 1210 | - TSV8 <=> sqlite conversion. Are these drivers or what?
|
| 1211 | - and then let you pipe output?
|
| 1212 |
|
| 1213 | Do we also need TSV8 space2tab or something? For writing TSV8 inline.
|
| 1214 |
|
| 1215 | More later:
|
| 1216 | - MessagePack (e.g. for shared library extension modules)
|
| 1217 | - msgpack read, write? I think user-defined function could be like this?
|
| 1218 | - SASH: Simple and Strict HTML? For easy processing
|
| 1219 | -->
|
| 1220 |
|
| 1221 | ## The Runtime Shared by OSH and YSH
|
| 1222 |
|
| 1223 | Although we describe OSH and YSH as different languages, they use the **same**
|
| 1224 | interpreter under the hood. This interpreter has various `shopt` flags that
|
| 1225 | are flipped for different behavior, e.g. with `shopt --set ysh:all`.
|
| 1226 |
|
| 1227 | Understanding this interpreter and its interface to the Unix kernel will help
|
| 1228 | you understand **both** languages!
|
| 1229 |
|
| 1230 | ### Interpreter Data Model
|
| 1231 |
|
| 1232 | The [Interpreter State](interpreter-state.html) doc is **under construction**.
|
| 1233 | It will cover:
|
| 1234 |
|
| 1235 | - Two separate namespaces (like Lisp 1 vs. 2):
|
| 1236 | - **proc** namespace for procs as the first word
|
| 1237 | - **variable** namespace
|
| 1238 | - The variable namespace has a **call stack**, for the local variables of a
|
| 1239 | proc.
|
| 1240 | - Each **stack frame** is a `{name -> cell}` mapping.
|
| 1241 | - A **cell** has one of the above data types: `Bool`, `Int`, `Str`, etc.
|
| 1242 | - A cell has `readonly`, `export`, and `nameref` **flags**.
|
| 1243 | - Boolean shell options with `shopt`: `parse_paren`, `simple_word_eval`, etc.
|
| 1244 | - String shell options with `shvar`: `IFS`, `PATH`
|
| 1245 | - **Registers** that are silently modified by the interpreter
|
| 1246 | - `$?` and `_error`
|
| 1247 | - `$!` for the last PID
|
| 1248 | - `_this_dir`
|
| 1249 | - `_reply`
|
| 1250 |
|
| 1251 | ### Process Model (the kernel)
|
| 1252 |
|
| 1253 | The [Process Model](process-model.html) doc is **under construction**. It will cover:
|
| 1254 |
|
| 1255 | - Simple Commands, `exec`
|
| 1256 | - Pipelines. #[shell-the-good-parts](#blog-tag)
|
| 1257 | - `fork`, `forkwait`
|
| 1258 | - Command and process substitution.
|
| 1259 | - Related links:
|
| 1260 | - [Tracing execution in Oils](xtrace.html) (xtrace), which divides
|
| 1261 | process-based concurrency into **synchronous** and **async** constructs.
|
| 1262 | - [Three Comics For Understanding Unix
|
| 1263 | Shell](http://www.oilshell.org/blog/2020/04/comics.html) (blog)
|
| 1264 |
|
| 1265 |
|
| 1266 | <!--
|
| 1267 | Process model additions: Capers, Headless shell
|
| 1268 |
|
| 1269 | some optimizations: See YSH starts fewer processes than other shells.
|
| 1270 | -->
|
| 1271 |
|
| 1272 | ## Summary
|
| 1273 |
|
| 1274 | YSH is a large language that evolved from Unix shell. It has shell-like
|
| 1275 | commands, Python-like expressions on typed data, and Ruby-like command blocks.
|
| 1276 |
|
| 1277 | Even though it's large, you can "forget" the bad parts of shell like `[ $x -lt
|
| 1278 | $y ]`.
|
| 1279 |
|
| 1280 | These concepts are central to YSH:
|
| 1281 |
|
| 1282 | 1. Interleaved *word*, *command*, and *expression* languages.
|
| 1283 | 2. A standard library of *shell builtins*, as well as *builtin functions*
|
| 1284 | 3. Languages for *data*: J8 Notation, including JSON8 and TSV8
|
| 1285 | 4. A *runtime* shared by OSH and YSH
|
| 1286 |
|
| 1287 | ## Related Docs
|
| 1288 |
|
| 1289 | - [YSH vs. Shell Idioms](idioms.html) - YSH side-by-side with shell.
|
| 1290 | - [YSH Language Influences](language-influences.html) - In addition to shell,
|
| 1291 | Python, and JavaScript, YSH is influenced by Ruby, Perl, Awk, PHP, and more.
|
| 1292 | - [A Feel For YSH Syntax](syntax-feelings.html) - Some thoughts that may help
|
| 1293 | you remember the syntax.
|
| 1294 | - [YSH Language Warts](warts.html) documents syntax that may be surprising.
|
| 1295 |
|
| 1296 | ## Appendix: Features Not Shown
|
| 1297 |
|
| 1298 | ### Advanced
|
| 1299 |
|
| 1300 | These shell features are part of YSH, but aren't shown for brevity.
|
| 1301 |
|
| 1302 | - The `fork` and `forkwait` builtins, for concurrent execution and subshells.
|
| 1303 | - Process Substitution: `diff <(sort left.txt) <(sort right.txt)`
|
| 1304 |
|
| 1305 | ### Deprecated Shell Constructs
|
| 1306 |
|
| 1307 | The shared interpreter supports many shell constructs that are deprecated:
|
| 1308 |
|
| 1309 | - YSH code uses shell's `||` and `&&` in limited circumstances, since `errexit`
|
| 1310 | is on by default.
|
| 1311 | - Assignment builtins like `local` and `declare`. Use YSH keywords.
|
| 1312 | - Boolean expressions like `[[ x =~ $pat ]]`. Use YSH expressions.
|
| 1313 | - Shell arithmetic like `$(( x + 1 ))` and `(( y = x ))`. Use YSH expressions.
|
| 1314 | - The `until` loop can always be replaced with a `while` loop
|
| 1315 | - Most of what's in `${}` can be written in other ways. For example
|
| 1316 | `${s#/tmp}` could be `s => removePrefix('/tmp')` (TODO).
|
| 1317 |
|
| 1318 | ### Not Yet Implemented
|
| 1319 |
|
| 1320 | This document mentions a few constructs that aren't yet implemented. Here's a
|
| 1321 | summary:
|
| 1322 |
|
| 1323 | ```none
|
| 1324 | # Unimplemented syntax:
|
| 1325 |
|
| 1326 | echo ${x|html} # formatters
|
| 1327 |
|
| 1328 | echo ${x %.2f} # statically-parsed printf
|
| 1329 |
|
| 1330 | var x = j"line\n"
|
| 1331 | echo j"line\n" # JSON-style string literal
|
| 1332 |
|
| 1333 | var x = "<p>$x</p>"html
|
| 1334 | echo "<p>$x</p>"html # tagged string
|
| 1335 |
|
| 1336 | var x = 15 Mi # units suffix
|
| 1337 | ```
|
| 1338 |
|
| 1339 | Important builtins that aren't implemented:
|
| 1340 |
|
| 1341 | - `describe` for testing
|
| 1342 | - `parseArgs()` to parse flags
|
| 1343 | - Builtins for [TSV8]($xref) - selection, projection, sorting
|
| 1344 |
|
| 1345 | <!--
|
| 1346 |
|
| 1347 | - To document: Method calls
|
| 1348 | - To implement: Capers: stateless coprocesses
|
| 1349 | -->
|
| 1350 |
|
| 1351 | ## Appendix: Example of an YSH Module
|
| 1352 |
|
| 1353 | YSH can be used to write simple "shell scripts" or longer programs. It has
|
| 1354 | *procs* and *modules* to help with the latter.
|
| 1355 |
|
| 1356 | A module is just a file, like this:
|
| 1357 |
|
| 1358 | ```
|
| 1359 | #!/usr/bin/env ysh
|
| 1360 | ### Deploy script
|
| 1361 |
|
| 1362 | source-guard main || return 0 # declaration, "include guard"
|
| 1363 |
|
| 1364 | source $_this_dir/lib/util.ysh # defines 'log' helper
|
| 1365 |
|
| 1366 | const DEST = '/tmp/ysh-tour'
|
| 1367 |
|
| 1368 | proc my-sync(...files) {
|
| 1369 | ### Sync files and show which ones
|
| 1370 |
|
| 1371 | cp --verbose @files $DEST
|
| 1372 | }
|
| 1373 |
|
| 1374 | proc main {
|
| 1375 | mkdir -p $DEST
|
| 1376 |
|
| 1377 | touch {foo,bar}.py {build,test}.sh
|
| 1378 |
|
| 1379 | log "Copying source files"
|
| 1380 | my-sync *.py *.sh
|
| 1381 |
|
| 1382 | if test --dir /tmp/logs {
|
| 1383 | cd /tmp/logs
|
| 1384 |
|
| 1385 | log "Copying logs"
|
| 1386 | my-sync *.log
|
| 1387 | }
|
| 1388 | }
|
| 1389 |
|
| 1390 | if is-main { # The only top-level statement
|
| 1391 | main @ARGV
|
| 1392 | }
|
| 1393 | ```
|
| 1394 |
|
| 1395 | <!--
|
| 1396 | TODO:
|
| 1397 | - Also show flags parsing?
|
| 1398 | - Show longer examples where it isn't boilerplate
|
| 1399 | -->
|
| 1400 |
|
| 1401 | You wouldn't bother with the boilerplate for something this small. But this
|
| 1402 | example illustrates the idea, which is that the top level often contains these
|
| 1403 | words: `proc`, `const`, `module`, `source`, and `use`.
|
| 1404 |
|