| 1 | ---
|
| 2 | in_progress: yes
|
| 3 | ---
|
| 4 |
|
| 5 | Interpreter State
|
| 6 | =================
|
| 7 |
|
| 8 | The Oils project has a single interpreter that supports both the OSH and YSH
|
| 9 | languages.
|
| 10 |
|
| 11 | In other words, It's useful to think of Unix shell in historical layers:
|
| 12 |
|
| 13 | - [OSH]($xref): A compatible but cleaned-up shell language.
|
| 14 | 1. Thompson Shell (pipelines, exit status)
|
| 15 | 2. Bourne Shell (variables, functions)
|
| 16 | 3. [Korn Shell]($xref:ksh) (indexed arrays)
|
| 17 | 4. [Bash]($xref:bash) (`shopt`, associative arrays)
|
| 18 | - [YSH]($xref): A new shell language that manipulates the same interpreter
|
| 19 | state in a cleaner way.
|
| 20 |
|
| 21 | <!--
|
| 22 | TODO:
|
| 23 |
|
| 24 | - New "Pulp"?
|
| 25 | - Use fenced code blocks
|
| 26 | - and run through BOTH bash and osh
|
| 27 | - and link to this doc
|
| 28 | - bash 4.4 in a sandbox?
|
| 29 | -->
|
| 30 |
|
| 31 |
|
| 32 | <div id="toc">
|
| 33 | </div>
|
| 34 |
|
| 35 | ## Example
|
| 36 |
|
| 37 | Shell has many syntaxes for the same semantics, which can be confusing. For
|
| 38 | example, in bash, these four statements do similar things:
|
| 39 |
|
| 40 | ```sh-prompt
|
| 41 | $ foo='bar'
|
| 42 | $ declare -g foo=bar
|
| 43 | $ x='foo=bar'; typeset $x
|
| 44 | $ printf -v foo bar
|
| 45 |
|
| 46 | $ echo $foo
|
| 47 | bar
|
| 48 | ```
|
| 49 |
|
| 50 | In addition, YSH adds JavaScript-like syntax:
|
| 51 |
|
| 52 | ```
|
| 53 | var foo = 'bar'
|
| 54 | ```
|
| 55 |
|
| 56 | YSH syntax can express more data types, but it may also confuse new users.
|
| 57 |
|
| 58 | So the sections below describe the shell from a **semantic** perspective, which
|
| 59 | should help users reason about their programs.
|
| 60 |
|
| 61 | Quick tip: Use the [pp]($help) builtin to inspect shell variables.
|
| 62 |
|
| 63 | ## Design Goals
|
| 64 |
|
| 65 | ### Simplify and Rationalize bash
|
| 66 |
|
| 67 | POSIX shell has a fairly simple model: everything is a string, and `"$@"` is a
|
| 68 | special case.
|
| 69 |
|
| 70 | Bash adds many features on top of POSIX, including arrays and associative
|
| 71 | arrays. Oils implements those features, and a few more.
|
| 72 |
|
| 73 | However, it also significantly simplifies the model.
|
| 74 |
|
| 75 | A primary difference is mentioned in [Known Differences](known-differences.html):
|
| 76 |
|
| 77 | - In bash, the *locations* of values are tagged with types, e.g. `declare -A
|
| 78 | unset_assoc_array`.
|
| 79 | - In Oils, *values* are tagged with types. This is how common dynamic languages
|
| 80 | like Python and JavaScript behave.
|
| 81 |
|
| 82 | In other words, Oils "salvages" the confusing semantics of bash and produces
|
| 83 | something simpler, while still being very compatible.
|
| 84 |
|
| 85 | ### Add New Features and Types
|
| 86 |
|
| 87 | TODO
|
| 88 |
|
| 89 | - eggex type
|
| 90 | - later: floating point type
|
| 91 |
|
| 92 | ## High Level Description
|
| 93 |
|
| 94 | ### Memory Is a Stack
|
| 95 |
|
| 96 | - Shell has a stack but no heap. The stack stores:
|
| 97 | - Variables that are local to a function.
|
| 98 | - The **arguments array** which is spelled `"$@"` in shell, and `@ARGV` in
|
| 99 | YSH.
|
| 100 | - Shell's memory has values and locations, but **no** references/pointers.
|
| 101 |
|
| 102 | <!--
|
| 103 | later: YSH adds references to data structures on the heap, which may be recurisve.
|
| 104 | -->
|
| 105 |
|
| 106 | ### Environment Variables Become Global Variables
|
| 107 |
|
| 108 | On initialization, environment variables like `PYTHONPATH=.` are copied into
|
| 109 | the shell's memory as global variables, with the `export` flag set.
|
| 110 |
|
| 111 | Global variables are stored in the first stack frame, i.e. the one at index
|
| 112 | `0`.
|
| 113 |
|
| 114 | ### Functions and Variables Are Separate
|
| 115 |
|
| 116 | There are two distinct namespaces. For example:
|
| 117 |
|
| 118 | ```
|
| 119 | foo() {
|
| 120 | echo 'function named foo'
|
| 121 | }
|
| 122 | foo=bar # a variable; doesn't affect the function
|
| 123 | ```
|
| 124 |
|
| 125 | ### Variable Name Lookup with "Dynamic Scope"
|
| 126 |
|
| 127 | OSH has it, but YSH limits it.
|
| 128 |
|
| 129 | ### Limitations of Arrays And Compound Data Structures
|
| 130 |
|
| 131 | Shell is a value-oriented language.
|
| 132 |
|
| 133 | - Can't Be Nested
|
| 134 | - Can't Be Passed to Functions or Returned From Functions
|
| 135 | - Can't Take References; Must be Copied
|
| 136 |
|
| 137 | Example:
|
| 138 |
|
| 139 | ```
|
| 140 | declare -a myarray=("${other_array[@]}") # shell
|
| 141 |
|
| 142 | var myarray = :| @other_array | # Oils
|
| 143 | ```
|
| 144 |
|
| 145 | Reason: There's no Garbage collection.
|
| 146 |
|
| 147 | ### Integers and Coercion
|
| 148 |
|
| 149 | - Strings are coerced to integers to do math.
|
| 150 | - What about `-i` in bash?
|
| 151 |
|
| 152 |
|
| 153 | ### Unix `fork()` Has Copy-On-Write Semantics
|
| 154 |
|
| 155 | See the [Process Model](process-model.html) document.
|
| 156 |
|
| 157 |
|
| 158 | ## Key Data Types
|
| 159 |
|
| 160 | TODO: [core/runtime.asdl]($oils-src)
|
| 161 |
|
| 162 | <!--
|
| 163 | TODO:
|
| 164 | - Make a graphviz diagram once everything is settled?
|
| 165 | -->
|
| 166 |
|
| 167 | ### `cell`
|
| 168 |
|
| 169 | TODO
|
| 170 |
|
| 171 | - [export]($help) only applies to **strings**
|
| 172 |
|
| 173 | ### `value`
|
| 174 |
|
| 175 | Undef, Str, Sequential/Indexed Arrays, Associative Array
|
| 176 |
|
| 177 | - OSH has `value.BashArray`, and YSH has `value.List`.
|
| 178 | - no integers, but there is (( ))
|
| 179 | - "$@" is an array, and "${a[@]}" too
|
| 180 | - not true in bash -- it's fuzzy there
|
| 181 | - but $@ and ${a[@]} are NOT arrays
|
| 182 | - flags: readonly and exported (but arrays/assoc arrays shouldn't be exported)
|
| 183 | - TODO: find that
|
| 184 |
|
| 185 | ### `cmd_value` for shell builtins
|
| 186 |
|
| 187 | Another important type:
|
| 188 |
|
| 189 | ```
|
| 190 | assign_arg = (lvalue lval, value? rval, int spid)
|
| 191 |
|
| 192 | cmd_value =
|
| 193 | Argv(string* argv, int* arg_spids, command__BraceGroup? block)
|
| 194 | | Assign(builtin builtin_id,
|
| 195 | string* argv, int* arg_spids,
|
| 196 | assign_arg* pairs)
|
| 197 | ```
|
| 198 |
|
| 199 |
|
| 200 | ## Printing State
|
| 201 |
|
| 202 | ### Shell Builtins
|
| 203 |
|
| 204 | Oils supports various shell and bash operations to view the interpreter state.
|
| 205 |
|
| 206 | - `set` prints variables and their values
|
| 207 | - `set -o` prints options
|
| 208 | - `declare/typeset/readonly/export -p` prints a subset of variables
|
| 209 | - `test -v` tests if a variable is defined.
|
| 210 |
|
| 211 | ### [pp]($help) in Oils
|
| 212 |
|
| 213 | Pretty prints a cell.
|
| 214 |
|
| 215 | This is cleaner!
|
| 216 |
|
| 217 | TODO: What about functions
|
| 218 |
|
| 219 |
|
| 220 |
|
| 221 |
|
| 222 | ## Modifying State
|
| 223 |
|
| 224 | ### YSH Keywords
|
| 225 |
|
| 226 | TODO: See YSH Keywords doc.
|
| 227 |
|
| 228 | ### Shell Assignment Builtins: declare/typeset, readonly, export
|
| 229 |
|
| 230 | ...
|
| 231 |
|
| 232 | ### [unset]($help)
|
| 233 |
|
| 234 | You can't unset an array in OSH? But you can in bash.
|
| 235 |
|
| 236 | ### Other Builtins
|
| 237 |
|
| 238 | - [read]($help). Sometimes sets the magic `$REPLY` variable.
|
| 239 | - [getopts]($help)
|
| 240 |
|
| 241 |
|
| 242 | ## Links
|
| 243 |
|
| 244 | - [Process Model](process-mode.html)
|
| 245 | - <https://opensource.com/article/18/5/you-dont-know-bash-intro-bash-arrays>
|
| 246 | - <https://www.thegeekstuff.com/2010/06/bash-array-tutorial>
|
| 247 |
|
| 248 |
|
| 249 | ## Appendix: Bash Issues
|
| 250 |
|
| 251 | <!--
|
| 252 | ### Surprising Parsing
|
| 253 |
|
| 254 | Parsing bash is undecidable.
|
| 255 |
|
| 256 | A[x]
|
| 257 | a[x]
|
| 258 | -->
|
| 259 |
|
| 260 | ### Strings and Arrays Are Confused
|
| 261 |
|
| 262 | Horrible
|
| 263 |
|
| 264 | a=('1 2' 3)
|
| 265 | b=(1 '2 3') # two different elements
|
| 266 |
|
| 267 | [[ $a == $b ]]
|
| 268 | [[ ${a[0]} == ${b[0]} ]]
|
| 269 |
|
| 270 | [[ ${a[@]} == ${b[@]} ]]
|
| 271 |
|
| 272 |
|
| 273 | Associative arrays and being undefined
|
| 274 |
|
| 275 | - half an array type
|
| 276 | - strict_array removes this
|
| 277 | - case $x in "$@"
|
| 278 | - half an associative array type
|
| 279 |
|
| 280 | ### Indexed Arrays and Associative Arrays Are Confused
|
| 281 |
|
| 282 | ### Empty and Unset Are Confused
|
| 283 |
|
| 284 | - empty array conflicts with `set -o nounset` (in bash 4.3). I can't recommend
|
| 285 | in good faith.
|
| 286 |
|
| 287 | <!--
|
| 288 | test -v (???) Was there a bug here?
|
| 289 | -->
|
| 290 |
|
| 291 |
|
| 292 |
|
| 293 | <!--
|
| 294 |
|
| 295 | ## Quirky Syntax and Semantics in Shell Sublanguages
|
| 296 |
|
| 297 | ### Command
|
| 298 |
|
| 299 | Mentioned above:
|
| 300 |
|
| 301 | a[x+1]+=x
|
| 302 | a[x+1]+=$x
|
| 303 |
|
| 304 | s+='foo'
|
| 305 |
|
| 306 | ### Word
|
| 307 |
|
| 308 | Mentioned above:
|
| 309 |
|
| 310 | echo ${a[0]}
|
| 311 | echo "${a[0]}"
|
| 312 | echo ${a[i+1]}
|
| 313 |
|
| 314 | ### Arithmetic Does Integer Coercion
|
| 315 |
|
| 316 | SURPRISING! Avoid if you can!!!
|
| 317 |
|
| 318 | (( a[ x+1 ] += s )) #
|
| 319 |
|
| 320 |
|
| 321 | ### Boolean: [[ $a = $b ]]
|
| 322 |
|
| 323 | Operates on strings only. Can't compare
|
| 324 |
|
| 325 | -->
|
| 326 |
|