| 1 | ---
|
| 2 | default_highlighter: oils-sh
|
| 3 | ---
|
| 4 |
|
| 5 | YSH Fixes Shell's Error Handling (`errexit`)
|
| 6 | ============================================
|
| 7 |
|
| 8 | <style>
|
| 9 | .faq {
|
| 10 | font-style: italic;
|
| 11 | color: purple;
|
| 12 | }
|
| 13 |
|
| 14 | /* copied from web/blog.css */
|
| 15 | .attention {
|
| 16 | text-align: center;
|
| 17 | background-color: #DEE;
|
| 18 | padding: 1px 0.5em;
|
| 19 |
|
| 20 | /* to match p tag etc. */
|
| 21 | margin-left: 2em;
|
| 22 | }
|
| 23 | </style>
|
| 24 |
|
| 25 | YSH is unlike other shells:
|
| 26 |
|
| 27 | - It never silently ignores an error, and it never loses an exit code.
|
| 28 | - There's no reason to write an YSH script without `errexit`, which is on by
|
| 29 | default.
|
| 30 |
|
| 31 | This document explains how YSH makes these guarantees. We first review shell
|
| 32 | error handling, and discuss its fundamental problems. Then we show idiomatic
|
| 33 | YSH code, and look under the hood at the underlying mechanisms.
|
| 34 |
|
| 35 | [file a bug]: https://github.com/oilshell/oil/issues
|
| 36 |
|
| 37 | <div id="toc">
|
| 38 | </div>
|
| 39 |
|
| 40 | ## Review of Shell Error Handling Mechanisms
|
| 41 |
|
| 42 | POSIX shell has fundamental problems with error handling. With `set -e` aka
|
| 43 | `errexit`, you're [damned if you do and damned if you don't][bash-faq].
|
| 44 |
|
| 45 | GNU [bash]($xref) fixes some of the problems, but **adds its own**, e.g. with
|
| 46 | respect to process subs, command subs, and assignment builtins.
|
| 47 |
|
| 48 | YSH fixes all the problems by adding new builtin commands, special variables,
|
| 49 | and global options. But you see a simple interface with `try` and `_status`.
|
| 50 |
|
| 51 | Let's review a few concepts before discussing YSH.
|
| 52 |
|
| 53 | ### POSIX Shell
|
| 54 |
|
| 55 | - The special variable `$?` is the exit status of the "last command". It's a
|
| 56 | number between `0` and `255`.
|
| 57 | - If `errexit` is enabled, the shell will abort if `$?` is nonzero.
|
| 58 | - This is subject to the *Disabled `errexit` Quirk*, which I describe below.
|
| 59 |
|
| 60 | These mechanisms are fundamentally incomplete.
|
| 61 |
|
| 62 | ### Bash
|
| 63 |
|
| 64 | Bash improves error handling for pipelines like `ls /bad | wc`.
|
| 65 |
|
| 66 | - `${PIPESTATUS[@]}` stores the exit codes of all processes in a pipeline.
|
| 67 | - When `set -o pipefail` is enabled, `$?` takes into account every process in a
|
| 68 | pipeline.
|
| 69 | - Without this setting, the failure of `ls` would be ignored.
|
| 70 | - `shopt -s inherit_errexit` was introduced in bash 4.4 to re-introduce error
|
| 71 | handling in command sub child processes. This fixes a bash-specific bug.
|
| 72 |
|
| 73 | But there are still places where bash will lose an exit code.
|
| 74 |
|
| 75 |
|
| 76 |
|
| 77 | ## Fundamental Problems
|
| 78 |
|
| 79 | Let's look at **four** fundamental issues with shell error handling. They
|
| 80 | underlie the **nine** [shell pitfalls enumerated in the
|
| 81 | appendix](#list-of-pitfalls).
|
| 82 |
|
| 83 | ### When Is `$?` Set?
|
| 84 |
|
| 85 | Each external process and shell builtin has one exit status. But the
|
| 86 | definition of `$?` is obscure: it's tied to the `pipeline` rule in the POSIX
|
| 87 | shell grammar, which does **not** correspond to a single process or builtin.
|
| 88 |
|
| 89 | We saw that `pipefail` fixes one case:
|
| 90 |
|
| 91 | ls /nonexistent | wc # 2 processes, 2 exit codes, but just one $?
|
| 92 |
|
| 93 | But there are others:
|
| 94 |
|
| 95 | local x=$(false) # 2 exit codes, but just one $?
|
| 96 | diff <(sort left) <(sort right) # 3 exit codes, but just one $?
|
| 97 |
|
| 98 | This issue means that shell scripts fundamentally **lose errors**. The
|
| 99 | language is unreliable.
|
| 100 |
|
| 101 | ### What Does `$?` Mean?
|
| 102 |
|
| 103 | Each process or builtin decides the meaning of its exit status independently.
|
| 104 | Here are two common choices:
|
| 105 |
|
| 106 | 1. **The Failure Paradigm**
|
| 107 | - `0` for success, or non-zero for an error.
|
| 108 | - Examples: most shell builtins, `ls`, `cp`, ...
|
| 109 | 1. **The Boolean Paradigm**
|
| 110 | - `0` for true, `1` for false, or a different number like `2` for an error.
|
| 111 | - Examples: the `test` builtin, `grep`, `diff`, ...
|
| 112 |
|
| 113 | New error handling constructs in YSH deal with this fundamental inconsistency.
|
| 114 |
|
| 115 | ### The Meaning of `if`
|
| 116 |
|
| 117 | Shell's `if` statement tests whether a command exits zero or non-zero:
|
| 118 |
|
| 119 | if grep class *.py; then
|
| 120 | echo 'found class'
|
| 121 | else
|
| 122 | echo 'not found' # is this true?
|
| 123 | fi
|
| 124 |
|
| 125 | So while you'd expect `if` to work in the boolean paradigm, it's closer to
|
| 126 | the failure paradigm. This means that using `if` with certain commands can
|
| 127 | cause the *Error or False Pitfall*:
|
| 128 |
|
| 129 | if grep 'class\(' *.py; then # grep syntax error, status 2
|
| 130 | echo 'found class('
|
| 131 | else
|
| 132 | echo 'not found is a lie'
|
| 133 | fi
|
| 134 | # => grep: Unmatched ( or \(
|
| 135 | # => not found is a lie
|
| 136 |
|
| 137 | That is, the `else` clause conflates grep's **error** status 2 and **false**
|
| 138 | status 1.
|
| 139 |
|
| 140 | Strangely enough, I encountered this pitfall while trying to disallow shell's
|
| 141 | error handling pitfalls in YSH! I describe this in another appendix as the
|
| 142 | "[meta pitfall](#the-meta-pitfall)".
|
| 143 |
|
| 144 | ### Design Mistake: The Disabled `errexit` Quirk
|
| 145 |
|
| 146 | There's more bad news about the design of shell's `if` statement. It's subject
|
| 147 | to the *Disabled `errexit` Quirk*, which means when you use a **shell function**
|
| 148 | in a conditional context, errors are unexpectedly **ignored**.
|
| 149 |
|
| 150 | That is, while `if ls /tmp` is useful, `if my-ls-function /tmp` should be
|
| 151 | avoided. It yields surprising results.
|
| 152 |
|
| 153 | I call this the *`if myfunc` Pitfall*, and show an example in [the
|
| 154 | appendix](#disabled-errexit-quirk-if-myfunc-pitfall).
|
| 155 |
|
| 156 | We can't fix this decades-old bug in shell. Instead we disallow dangerous code
|
| 157 | with `strict_errexit`, and add new error handling mechanisms.
|
| 158 |
|
| 159 |
|
| 160 |
|
| 161 | ## YSH Error Handling: The Big Picture
|
| 162 |
|
| 163 | We've reviewed how POSIX shell and bash work, and showed fundamental problems
|
| 164 | with the shell language.
|
| 165 |
|
| 166 | But when you're using YSH, **you don't have to worry about any of this**!
|
| 167 |
|
| 168 | ### YSH Fails On Every Error
|
| 169 |
|
| 170 | This means you don't have to explicitly check for errors. Examples:
|
| 171 |
|
| 172 | shopt --set ysh:upgrade # Enable good error handling in bin/osh
|
| 173 | # It's the default in bin/ysh.
|
| 174 | shopt --set strict_errexit # Disallow bad shell error handling.
|
| 175 | # Also the default in bin/ysh.
|
| 176 |
|
| 177 | local date=$(date X) # 'date' failure is fatal
|
| 178 | # => date: invalid date 'X'
|
| 179 |
|
| 180 | echo $(date X) # ditto
|
| 181 |
|
| 182 | echo $(date X) $(ls > F) # 'ls' isn't executed; 'date' fails first
|
| 183 |
|
| 184 | ls /bad | wc # 'ls' failure is fatal
|
| 185 |
|
| 186 | diff <(sort A) <(sort B) # 'sort' failure is fatal
|
| 187 |
|
| 188 | On the other hand, you won't experience this problem caused by `pipefail`:
|
| 189 |
|
| 190 | yes | head # doesn't fail due to SIGPIPE
|
| 191 |
|
| 192 | The details are explained below.
|
| 193 |
|
| 194 | ### `try` Handles Command and Expression Errors
|
| 195 |
|
| 196 | You may want to **handle failure** instead of aborting the shell. In this
|
| 197 | case, use the `try` builtin and inspect the `_status` variable it sets.
|
| 198 |
|
| 199 | try { # try takes a block of commands
|
| 200 | ls /etc
|
| 201 | ls /BAD # it stops at the first failure
|
| 202 | ls /lib
|
| 203 | } # After try, $? is always 0
|
| 204 | if (_status !== 0) { # Now check _status
|
| 205 | echo 'failed'
|
| 206 | }
|
| 207 |
|
| 208 | Note that:
|
| 209 |
|
| 210 | - The `_status` variable is different than `$?`.
|
| 211 | - The leading `_` is a PHP-like convention for special variables /
|
| 212 | "registers" in YSH.
|
| 213 | - Idiomatic YSH programs don't look at `$?`.
|
| 214 |
|
| 215 | You can omit `{ }` when invoking a single command. Here's how to invoke a
|
| 216 | function without the *`if myfunc` Pitfall*:
|
| 217 |
|
| 218 | try myfunc # Unlike 'myfunc', doesn't abort on error
|
| 219 | if (_status !== 0) {
|
| 220 | echo 'failed'
|
| 221 | }
|
| 222 |
|
| 223 | You also have fine-grained control over every process in a pipeline:
|
| 224 |
|
| 225 | try {
|
| 226 | ls /bad | wc
|
| 227 | }
|
| 228 | write -- @_pipeline_status # every exit status
|
| 229 |
|
| 230 | And each process substitution:
|
| 231 |
|
| 232 | try {
|
| 233 | diff <(sort left.txt) <(sort right.txt)
|
| 234 | }
|
| 235 | write -- @_process_sub_status # every exit status
|
| 236 |
|
| 237 |
|
| 238 |
|
| 239 |
|
| 240 | <div class="attention">
|
| 241 |
|
| 242 | See [YSH vs. Shell Idioms > Error Handling](idioms.html#error-handling) for
|
| 243 | more examples.
|
| 244 |
|
| 245 | </div>
|
| 246 |
|
| 247 |
|
| 248 |
|
| 249 | Certain expressions produce fatal errors, like:
|
| 250 |
|
| 251 | var x = 42 / 0 # divide by zero will abort shell
|
| 252 |
|
| 253 | The `try` builtin also handles them:
|
| 254 |
|
| 255 | try {
|
| 256 | var x = 42 / 0
|
| 257 | }
|
| 258 | if (_status !== 0) {
|
| 259 | echo 'divide by zero'
|
| 260 | }
|
| 261 |
|
| 262 | More examples:
|
| 263 |
|
| 264 | - Index out of bounds `a[i]`
|
| 265 | - Nonexistent key `d->foo` or `d['foo']`.
|
| 266 |
|
| 267 | Such expression evaluation errors result in status `3`, which is an arbitrary non-zero
|
| 268 | status that's not used by other shells. Status `2` is generally for syntax
|
| 269 | errors and status `1` is for most runtime failures.
|
| 270 |
|
| 271 | ### `boolstatus` Enforces 0 or 1 Status
|
| 272 |
|
| 273 | The `boolstatus` builtin addresses the *Error or False Pitfall*:
|
| 274 |
|
| 275 | if boolstatus grep 'class' *.py { # may abort the program
|
| 276 | echo 'found' # status 0 means 'found'
|
| 277 | } else {
|
| 278 | echo 'not found' # status 1 means 'not found'
|
| 279 | }
|
| 280 |
|
| 281 | Rather than confusing **error** with **false**, `boolstatus` will abort the
|
| 282 | program if `grep` doesn't return 0 or 1.
|
| 283 |
|
| 284 | You can think of this as a shortcut for
|
| 285 |
|
| 286 | try grep 'class' *.py
|
| 287 | case $_status {
|
| 288 | (0) echo 'found'
|
| 289 | ;;
|
| 290 | (1) echo 'not found'
|
| 291 | ;;
|
| 292 | (*) echo 'fatal'
|
| 293 | exit $_status
|
| 294 | ;;
|
| 295 | }
|
| 296 |
|
| 297 | ### FAQ on Language Design
|
| 298 |
|
| 299 | <div class="faq">
|
| 300 |
|
| 301 | Why is there `try` but no `catch`?
|
| 302 |
|
| 303 | </div>
|
| 304 |
|
| 305 | First, it offers more flexibility:
|
| 306 |
|
| 307 | - The handler usually inspects `_status`, but it may also inspect
|
| 308 | `_pipeline_status` or `_process_sub_status`.
|
| 309 | - The handler may use `case` instead of `if`, e.g. to distinguish true / false
|
| 310 | / error.
|
| 311 |
|
| 312 | Second, it makes the language smaller:
|
| 313 |
|
| 314 | - `try` / `catch` would require specially parsed keywords. But our `try` is a
|
| 315 | shell builtin that takes a block, like `cd` or `shopt`.
|
| 316 | - The builtin also lets us write either `try ls` or `try { ls }`, which is hard
|
| 317 | with a keyword.
|
| 318 |
|
| 319 | Another way to remember this is that there are **three parts** to handling an
|
| 320 | error, each of which has independent choices:
|
| 321 |
|
| 322 | 1. Does `try` take a simple command or a block? For example, `try ls` versus
|
| 323 | `try { ls; var x = 42 / n }`
|
| 324 | 2. Which status do you want to inspect?
|
| 325 | 3. Inspect it with `if` or `case`? As mentioned, `boolstatus` is a special
|
| 326 | case of `try / case`.
|
| 327 |
|
| 328 | <div class="faq">
|
| 329 |
|
| 330 | Why is `_status` different from `$?`
|
| 331 |
|
| 332 | </div>
|
| 333 |
|
| 334 | This avoids special cases in the interpreter for `try`, which is again a
|
| 335 | builtin that takes a block.
|
| 336 |
|
| 337 | The exit status of `try` is always `0`. If it returned a non-zero status, the
|
| 338 | `errexit` rule would trigger, and you wouldn't be able to handle the error!
|
| 339 |
|
| 340 | Generally, [errors occur *inside* blocks, not
|
| 341 | outside](proc-block-func.html#errors).
|
| 342 |
|
| 343 | Again, idiomatic YSH scripts never look at `$?`, which is only used to trigger
|
| 344 | shell's `errexit` rule. Instead they invoke `try` and inspect `_status` when
|
| 345 | they want to handle errors.
|
| 346 |
|
| 347 | <div class="faq">
|
| 348 |
|
| 349 | Why `boolstatus`? Can't you just change what `if` means in YSH?
|
| 350 |
|
| 351 | </div>
|
| 352 |
|
| 353 | I've learned the hard way that when there's a shell **semantics** change, there
|
| 354 | must be a **syntax** change. In general, you should be able to read code on
|
| 355 | its own, without context.
|
| 356 |
|
| 357 | Readers shouldn't have to constantly look up whether `ysh:upgrade` is on. There
|
| 358 | are some cases where this is necessary, but it should be minimized.
|
| 359 |
|
| 360 | Also, both `if foo` and `if boolstatus foo` are useful in idiomatic YSH code.
|
| 361 |
|
| 362 |
|
| 363 |
|
| 364 | <div class="attention">
|
| 365 |
|
| 366 | **Most users can skip to [the summary](#summary).** You don't need to know all
|
| 367 | the details to use YSH.
|
| 368 |
|
| 369 | </div>
|
| 370 |
|
| 371 |
|
| 372 |
|
| 373 | ## Reference: Global Options
|
| 374 |
|
| 375 |
|
| 376 | Under the hood, we implement the `errexit` option from POSIX, bash options like
|
| 377 | `pipefail` and `inherit_errexit`, and add more options of our
|
| 378 | own. They're all hidden behind [option groups](options.html) like `strict:all`
|
| 379 | and `ysh:upgrade`.
|
| 380 |
|
| 381 | The following sections explain new YSH options.
|
| 382 |
|
| 383 | ### `command_sub_errexit` Adds More Errors
|
| 384 |
|
| 385 | In all Bourne shells, the status of command subs is lost, so errors are ignored
|
| 386 | (details in the [appendix](#quirky-behavior-of)). For example:
|
| 387 |
|
| 388 | echo $(date X) $(date Y) # 2 failures, both ignored
|
| 389 | echo # program continues
|
| 390 |
|
| 391 | The `command_sub_errexit` option makes both `date` invocations an an error.
|
| 392 | The status `$?` of the parent `echo` command will be `1`, so if `errexit` is
|
| 393 | on, the shell will abort.
|
| 394 |
|
| 395 | (Other shells should implement `command_sub_errexit`!)
|
| 396 |
|
| 397 | ### `process_sub_fail` Is Analogous to `pipefail`
|
| 398 |
|
| 399 | Similarly, in this example, `sort` will fail if the file doesn't exist.
|
| 400 |
|
| 401 | diff <(sort left.txt) <(sort right.txt) # any failures are ignored
|
| 402 |
|
| 403 | But there's no way to see this error in bash. YSH adds `process_sub_fail`,
|
| 404 | which folds the failure into `$?` so `errexit` can do its job.
|
| 405 |
|
| 406 | You can also inspect the special `_process_sub_status` array variable to
|
| 407 | implement custom error logic.
|
| 408 |
|
| 409 | ### `strict_errexit` Flags Two Problems
|
| 410 |
|
| 411 | Like other `strict_*` options, YSH `strict_errexit` improves your shell
|
| 412 | programs, even if you run them under another shell like [bash]($xref)! It's
|
| 413 | like a linter *at runtime*, so it can catch things that [ShellCheck][] can't.
|
| 414 |
|
| 415 | [ShellCheck]: https://www.shellcheck.net/
|
| 416 |
|
| 417 | `strict_errexit` disallows code that exhibits these problems:
|
| 418 |
|
| 419 | 1. The `if myfunc` Pitfall
|
| 420 | 1. The `local x=$(false)` Pitfall
|
| 421 |
|
| 422 | See the appendix for examples of each.
|
| 423 |
|
| 424 | #### Rules to Prevent the `if myfunc` Pitfall
|
| 425 |
|
| 426 | In any conditional context, `strict_errexit` disallows:
|
| 427 |
|
| 428 | 1. All commands except `((`, `[[`, and some simple commands (e.g. `echo foo`).
|
| 429 | - Detail: `! ls` is considered a pipeline in the shell grammar. We have to
|
| 430 | allow it, while disallowing `ls | grep foo`.
|
| 431 | 2. Function/proc invocations (which are a special case of simple
|
| 432 | commands.)
|
| 433 | 3. Command sub and process sub (`shopt --unset allow_csub_psub`)
|
| 434 |
|
| 435 | This means that you should check the exit status of functions and pipeline
|
| 436 | differently. See [Does a Function
|
| 437 | Succeed?](idioms.html#does-a-function-succeed), [Does a Pipeline
|
| 438 | Succeed?](idioms.html#does-a-pipeline-succeed), and other [YSH vs. Shell
|
| 439 | Idioms](idioms.html).
|
| 440 |
|
| 441 | #### Rule to Prevent the `local x=$(false)` Pitfall
|
| 442 |
|
| 443 | - Command Subs and process subs are disallowed in assignment builtins: `local`,
|
| 444 | `declare` aka `typeset`, `readonly`, and `export`.
|
| 445 |
|
| 446 | No:
|
| 447 |
|
| 448 | local x=$(false)
|
| 449 |
|
| 450 | Yes:
|
| 451 |
|
| 452 | var x = $(false) # YSH style
|
| 453 |
|
| 454 | local x # Shell style
|
| 455 | x=$(false)
|
| 456 |
|
| 457 | ### `sigpipe_status_ok` Ignores an Issue With `pipefail`
|
| 458 |
|
| 459 | When you turn on `pipefail`, you may inadvertently run into this behavior:
|
| 460 |
|
| 461 | yes | head
|
| 462 | # => y
|
| 463 | # ...
|
| 464 |
|
| 465 | echo ${PIPESTATUS[@]}
|
| 466 | # => 141 0
|
| 467 |
|
| 468 | That is, `head` closes the pipe after 10 lines, causing the `yes` command to
|
| 469 | **fail** with `SIGPIPE` status `141`.
|
| 470 |
|
| 471 | This error shouldn't be fatal, so OSH has a `sigpipe_status_ok` option, which
|
| 472 | is on by default in YSH.
|
| 473 |
|
| 474 | ### `verbose_errexit`
|
| 475 |
|
| 476 | When `verbose_errexit` is on, the shell prints errors to `stderr` when the
|
| 477 | `errexit` rule is triggered.
|
| 478 |
|
| 479 | ### FAQ on Options
|
| 480 |
|
| 481 | <div class="faq">
|
| 482 |
|
| 483 | Why is there no `_command_sub_status`? And why is `command_sub_errexit` named
|
| 484 | differently than `process_sub_fail` and `pipefail`?
|
| 485 |
|
| 486 | </div>
|
| 487 |
|
| 488 | Command subs are executed **serially**, while process subs and pipeline parts
|
| 489 | run **in parallel**.
|
| 490 |
|
| 491 | So a command sub can "abort" its parent command, setting `$?` immediately.
|
| 492 | The parallel constructs must wait until all parts are done and save statuses in
|
| 493 | an array. Afterward, they determine `$?` based on the value of `pipefail` and
|
| 494 | `process_sub_fail`.
|
| 495 |
|
| 496 | <div class="faq">
|
| 497 |
|
| 498 | Why are `strict_errexit` and `command_sub_errexit` different options?
|
| 499 |
|
| 500 | </div>
|
| 501 |
|
| 502 | Because `shopt --set strict:all` can be used to improve scripts that are run
|
| 503 | under other shells like [bash]($xref). It's like a runtime linter that
|
| 504 | disallows dangerous constructs.
|
| 505 |
|
| 506 | On the other hand, if you write code with `command_sub_errexit` on, it's
|
| 507 | impossible to get the same failures under bash. So `command_sub_errexit` is
|
| 508 | not a `strict_*` option, and it's meant for code that runs only under YSH.
|
| 509 |
|
| 510 | <div class="faq">
|
| 511 |
|
| 512 | What's the difference between bash's `inherit_errexit` and YSH
|
| 513 | `command_sub_errexit`? Don't they both relate to command subs?
|
| 514 |
|
| 515 | </div>
|
| 516 |
|
| 517 | - `inherit_errexit` enables failure in the **child** process running the
|
| 518 | command sub.
|
| 519 | - `command_sub_errexit` enables failure in the **parent** process, after the
|
| 520 | command sub has finished.
|
| 521 |
|
| 522 |
|
| 523 |
|
| 524 | ## Summary
|
| 525 |
|
| 526 | YSH uses three mechanisms to fix error handling once and for all.
|
| 527 |
|
| 528 | It has two new **builtins** that relate to errors:
|
| 529 |
|
| 530 | 1. `try` lets you explicitly handle errors when `errexit` is on.
|
| 531 | 1. `boolstatus` enforces a true/false meaning. (This builtin is less common).
|
| 532 |
|
| 533 | It has three **special variables**:
|
| 534 |
|
| 535 | 1. The `_status` integer, which is set by `try`.
|
| 536 | - Remember that it's distinct from `$?`, and that idiomatic YSH programs
|
| 537 | don't use `$?`.
|
| 538 | 1. The `_pipeline_status` array (another name for bash's `PIPESTATUS`)
|
| 539 | 1. The `_process_sub_status` array for process substitutions.
|
| 540 |
|
| 541 | Finally, it supports all of these **global options**:
|
| 542 |
|
| 543 | - From POSIX shell:
|
| 544 | - `errexit`
|
| 545 | - From [bash]($xref):
|
| 546 | - `pipefail`
|
| 547 | - `inherit_errexit` aborts the child process of a command sub.
|
| 548 | - New:
|
| 549 | - `command_sub_errexit` aborts the parent process immediately after a failed
|
| 550 | command sub.
|
| 551 | - `process_sub_fail` is analogous to `pipefail`.
|
| 552 | - `strict_errexit` flags two common problems.
|
| 553 | - `sigpipe_status_ok` ignores a spurious "broken pipe" failure.
|
| 554 | - `verbose_errexit` controls whether error messages are printed.
|
| 555 |
|
| 556 | When using `bin/osh`, set all options at once with `shopt --set ysh:upgrade
|
| 557 | strict:all`. Or use `bin/ysh`, where they're set by default.
|
| 558 |
|
| 559 | <!--
|
| 560 | Related 2020 blog post [Reliable Error
|
| 561 | Handling](https://www.oilshell.org/blog/2020/10/osh-features.html#reliable-error-handling).
|
| 562 | -->
|
| 563 |
|
| 564 |
|
| 565 | ## Related Docs
|
| 566 |
|
| 567 | - [YSH vs. Shell Idioms](idioms.html) shows more examples of `try` and `boolstatus`.
|
| 568 | - [Shell Idioms](shell-idioms.html) has a section on fixing `strict_errexit`
|
| 569 | problems in Bourne shell.
|
| 570 |
|
| 571 | Good articles on `errexit`:
|
| 572 |
|
| 573 | - Bash FAQ: [Why doesn't `set -e` do what I expected?][bash-faq]
|
| 574 | - [Bash: Error Handling](http://fvue.nl/wiki/Bash:_Error_handling) from
|
| 575 | `fvue.nl`
|
| 576 |
|
| 577 | [bash-faq]: http://mywiki.wooledge.org/BashFAQ/105
|
| 578 |
|
| 579 | Spec Test Suites:
|
| 580 |
|
| 581 | - <https://www.oilshell.org/release/latest/test/spec.wwz/survey/errexit.html>
|
| 582 | - <https://www.oilshell.org/release/latest/test/spec.wwz/survey/errexit-oil.html>
|
| 583 |
|
| 584 | These docs aren't about error handling, but they're also painstaking
|
| 585 | backward-compatible overhauls of shell!
|
| 586 |
|
| 587 | - [Simple Word Evaluation in Unix Shell](simple-word-eval.html)
|
| 588 | - [Egg Expressions (YSH Regexes)](eggex.html)
|
| 589 |
|
| 590 | For reference, this work on error handling was described in [Four Features That
|
| 591 | Justify a New Unix
|
| 592 | Shell](https://www.oilshell.org/blog/2020/10/osh-features.html) (October 2020).
|
| 593 | Since then, we changed `try` and `_status` to be more powerful and general.
|
| 594 |
|
| 595 |
|
| 596 |
|
| 597 | ## Appendices
|
| 598 |
|
| 599 | ### List Of Pitfalls
|
| 600 |
|
| 601 | We mentioned some of these pitfalls:
|
| 602 |
|
| 603 | 1. The `if myfunc` Pitfall, caused by the Disabled `errexit` Quirk (`strict_errexit`)
|
| 604 | 1. The `local x=$(false)` Pitfall (`strict_errexit`)
|
| 605 | 1. The Error or False Pitfall (`boolstatus`, `try` / `case`)
|
| 606 | - Special case: When the child process is another instance of the shell, the
|
| 607 | Meta Pitfall is possible.
|
| 608 | 1. The Process Sub Pitfall (`process_sub_fail` and `_process_sub_status`)
|
| 609 | 1. The `yes | head` Pitfall (`sigpipe_status_ok`)
|
| 610 |
|
| 611 | There are two pitfalls related to command subs:
|
| 612 |
|
| 613 | 6. The `echo $(false)` Pitfall (`command_sub_errexit`)
|
| 614 | 6. Bash's `inherit_errexit` pitfall.
|
| 615 | - As mentioned, this bash 4.4 option fixed a bug in earlier versions of
|
| 616 | bash. YSH reimplements it and turns it on by default.
|
| 617 |
|
| 618 | Here are two more pitfalls that don't require changes to YSH:
|
| 619 |
|
| 620 | 8. The Trailing `&&` Pitfall
|
| 621 | - When `test -d /bin && echo found` is at the end of a function, the exit
|
| 622 | code is surprising.
|
| 623 | - Solution: always use `if` rather than `&&`.
|
| 624 | - More reasons: the `if` is easier to read, and `&&` isn't useful when
|
| 625 | `errexit` is on.
|
| 626 | 8. The surprising return value of `(( i++ ))`, `let`, `expr`, etc.
|
| 627 | - Solution: Use `i=$((i + 1))`, which is valid POSIX shell.
|
| 628 | - In YSH, use `setvar i += 1`.
|
| 629 |
|
| 630 | #### Example of `inherit_errexit` Pitfall
|
| 631 |
|
| 632 | In bash, `errexit` is disabled in command sub child processes:
|
| 633 |
|
| 634 | set -e
|
| 635 | shopt -s inherit_errexit # needed to avoid 'touch two'
|
| 636 | echo $(touch one; false; touch two)
|
| 637 |
|
| 638 | Without the option, it will touch both files, even though there is a failure
|
| 639 | `false` after the first.
|
| 640 |
|
| 641 | #### Bash has a grammatical quirk with `set -o failglob`
|
| 642 |
|
| 643 | This isn't a pitfall, but a quirk that also relates to errors and shell's
|
| 644 | **grammar**. Recall that the definition of `$?` is tied to the grammar.
|
| 645 |
|
| 646 | Consider this program:
|
| 647 |
|
| 648 | set -o failglob
|
| 649 | echo *.ZZ # no files match
|
| 650 | echo status=$? # show failure
|
| 651 | # => status=1
|
| 652 |
|
| 653 | This is the same program with a newline replaced by a semicolon:
|
| 654 |
|
| 655 | set -o failglob
|
| 656 |
|
| 657 | # Surprisingly, bash doesn't execute what's after ;
|
| 658 | echo *.ZZ; echo status=$?
|
| 659 | # => (no output)
|
| 660 |
|
| 661 | But it behaves differently. This is because newlines and semicolons are handled
|
| 662 | in different **productions of the grammar**, and produce distinct syntax trees.
|
| 663 |
|
| 664 | (A related quirk is that this same difference can affect the number of
|
| 665 | processes that shells start!)
|
| 666 |
|
| 667 | ### Disabled `errexit` Quirk / `if myfunc` Pitfall
|
| 668 |
|
| 669 | This quirk is a bad interaction between the `if` statement, shell functions,
|
| 670 | and `errexit`. It's a **mistake** in the design of the shell language.
|
| 671 | Example:
|
| 672 |
|
| 673 | set -o errexit # don't ignore errors
|
| 674 |
|
| 675 | myfunc() {
|
| 676 | ls /bad # fails with status 1
|
| 677 | echo 'should not get here'
|
| 678 | }
|
| 679 |
|
| 680 | myfunc # Good: script aborts before echo
|
| 681 | # => ls: '/bad': no such file or directory
|
| 682 |
|
| 683 | if myfunc; then # Surprise! It behaves differently in a condition.
|
| 684 | echo OK
|
| 685 | fi
|
| 686 | # => ls: '/bad': no such file or directory
|
| 687 | # => should not get here
|
| 688 |
|
| 689 | We see "should not get here" because the shell **silently disables** `errexit`
|
| 690 | while executing the condition of `if`. This relates to the fundamental
|
| 691 | problems above:
|
| 692 |
|
| 693 | 1. Does the function use the failure paradigm or the boolean paradigm?
|
| 694 | 2. `if` tests a single exit status, but every command in a function has an exit
|
| 695 | status. Which one should we consider?
|
| 696 |
|
| 697 | This quirk occurs in all **conditional contexts**:
|
| 698 |
|
| 699 | 1. The condition of the `if`, `while`, and `until` constructs
|
| 700 | 2. A command/pipeline prefixed by `!` (negation)
|
| 701 | 3. Every clause in `||` and `&&` except the last.
|
| 702 |
|
| 703 | ### The Meta Pitfall
|
| 704 |
|
| 705 | I encountered the *Error or False Pitfall* while trying to disallow other error
|
| 706 | handling pitfalls! The *meta pitfall* arises from a combination of the issues
|
| 707 | discussed:
|
| 708 |
|
| 709 | 1. The `if` statement tests for zero or non-zero status.
|
| 710 | 1. The condition of an `if` may start child processes. For example, in `if
|
| 711 | myfunc | grep foo`, the `myfunc` invocation must be run in a subshell.
|
| 712 | 1. You may want an external process to use the **boolean paradigm**, and
|
| 713 | that includes **the shell itself**. When any of the `strict_` options
|
| 714 | encounters bad code, it aborts the shell with **error** status `1`, not
|
| 715 | boolean **false** `1`.
|
| 716 |
|
| 717 | The result of this fundamental issue is that `strict_errexit` is quite strict.
|
| 718 | On the other hand, the resulting style is straightforward and explicit.
|
| 719 | Earlier attempts allowed code that is too subtle.
|
| 720 |
|
| 721 | ### Quirky Behavior of `$?`
|
| 722 |
|
| 723 | This is a different way of summarizing the information above.
|
| 724 |
|
| 725 | Simple commands have an obvious behavior:
|
| 726 |
|
| 727 | echo hi # $? is 0
|
| 728 | false # $? is 1
|
| 729 |
|
| 730 | But the parent process loses errors from failed command subs:
|
| 731 |
|
| 732 | echo $(false) # $? is 0
|
| 733 | # YSH makes it fail with command_sub_errexit
|
| 734 |
|
| 735 | Surprisingly, bare assignments take on the value of any command subs:
|
| 736 |
|
| 737 | x=$(false) # $? is 1 -- we did NOT lose the exit code
|
| 738 |
|
| 739 | But assignment builtins have the problem again:
|
| 740 |
|
| 741 | local x=$(false) # $? is 0 -- exit code is clobbered
|
| 742 | # disallowed by YSH strict_errexit
|
| 743 |
|
| 744 | So shell is confusing and inconsistent, but YSH fixes all these problems. You
|
| 745 | never lose the exit code of `false`.
|
| 746 |
|
| 747 |
|
| 748 |
|
| 749 |
|
| 750 | ## Acknowledgments
|
| 751 |
|
| 752 | - Thank you to `ca2013` for extensive review and proofreading of this doc.
|
| 753 |
|
| 754 |
|