| 1 | ---
|
| 2 | in_progress: true
|
| 3 | ---
|
| 4 |
|
| 5 | Notes on Oils Architecture
|
| 6 | ===========================
|
| 7 |
|
| 8 | This doc is for contributors or users who want to understand the Oils codebase.
|
| 9 | These internal details are subject to change.
|
| 10 |
|
| 11 | <div id="toc">
|
| 12 | </div>
|
| 13 |
|
| 14 | ## Links
|
| 15 |
|
| 16 | - [Contributing][] (wiki) helps you change the code for the first time.
|
| 17 | - [README](oils-repo/README.html) describes how the code is organized.
|
| 18 | - [Interpreter State](interpreter-state.html) describes the interpreter's user-facing data
|
| 19 | structures.
|
| 20 | - [Parser Architecture](parser-architecture.html)
|
| 21 | - [OSH Word Evaluation Algorithm][word-eval] (wiki) describes shell's complex
|
| 22 | word evaluation. Oils uses [Simple Word Evaluation](simple-word-eval.html)
|
| 23 | instead.
|
| 24 |
|
| 25 | [Contributing]: https://github.com/oilshell/oil/wiki/Contributing
|
| 26 | [word-eval]: https://github.com/oilshell/oil/wiki/OSH-Word-Evaluation-Algorithm
|
| 27 |
|
| 28 | ## Source Code
|
| 29 |
|
| 30 | ### Build Dependencies
|
| 31 |
|
| 32 | - Essential: [libc]($xref)
|
| 33 | - Optional: GNU [readline]($xref) (TODO: other line editing libraries).
|
| 34 | - Only in the OVM build (as of March 2020): [yajl]($xref)
|
| 35 |
|
| 36 | ### Borrowed Code
|
| 37 |
|
| 38 | - [ASDL]($oils-src:asdl/) front end from [CPython]($xref:cpython) (heavily
|
| 39 | refactored)
|
| 40 | - [frontend/tdop.py]($oils-src): Adapted from tinypy, but almost no original code
|
| 41 | remains
|
| 42 | - [pgen2]($oils-src:pgen2/)
|
| 43 | - All of OPy (will be obsolete)
|
| 44 | - compiler2 from stdlib
|
| 45 | - byterun
|
| 46 | - Build Dependency: [MyPy]($xref:mypy)
|
| 47 |
|
| 48 | ### Metaprogramming / Generated Code
|
| 49 |
|
| 50 | - Try `ls */*_def.py */*_gen.py`
|
| 51 | - The `def.py` files are abstract definitions. They're not translated by
|
| 52 | [mycpp]($xref).
|
| 53 | - The `gen.py` files generate source code in Python and C++ from these
|
| 54 | definitions.
|
| 55 | - For example, we define the core `Id` type and the lexing rules abstractly.
|
| 56 | - *TODO: Details on each `def` / `gen` pair*.
|
| 57 | - See [build/dev.sh]($oils-src) and [build/codegen.sh]($oils-src)
|
| 58 |
|
| 59 |
|
| 60 | ## Other Cross-Cutting Observations
|
| 61 |
|
| 62 | ### Where $IFS is Used
|
| 63 |
|
| 64 | - Splitting of unquoted substitutions
|
| 65 | - The [read]($help) builtin
|
| 66 | - To split words in `compgen -W` (bash only)
|
| 67 |
|
| 68 | ### Shell Function Callbacks
|
| 69 |
|
| 70 | - Completion hooks registered by `complete -F ls_complete_func ls`
|
| 71 | - bash has a `command_not_found` hook; OSH doesn't yet
|
| 72 |
|
| 73 | ### Where Unicode is Respected
|
| 74 |
|
| 75 | See the doc on [Unicode](unicode.html).
|
| 76 |
|
| 77 | ### Parse-time and Runtime Pairs
|
| 78 |
|
| 79 | In OSH:
|
| 80 |
|
| 81 | - `echo -e '\x00\n'` and `echo $'\x00\n'` (OSH shares lexer rules between them)
|
| 82 | - `test` / `[` and `[[` (OSH shares the parser and evaluator)
|
| 83 | - Static vs. Dynamic Assignment. `local x=$y` vs. `s='x=$y'; local $s`.
|
| 84 | - All shells have both notions!
|
| 85 |
|
| 86 | Other Pairs:
|
| 87 |
|
| 88 | - `expr` and `$(( ))` (`expr` not in shell)
|
| 89 | - Later:
|
| 90 | - [printf]($help) can have a static variant like `${myfloat %.3f}`
|
| 91 | - `find` and our own language (although this may be done with blocks)
|
| 92 |
|
| 93 | ## State Machines
|
| 94 |
|
| 95 | - `$IFS` splitting in `osh/split.py`
|
| 96 | - [compadjust]($help) needs to split partial `argv` by user-defined delimiters,
|
| 97 | e.g. `:=`
|
| 98 |
|
| 99 | The point of a state machine is to make sure all cases are handled!
|
| 100 |
|
| 101 | <!--
|
| 102 | Idea:
|
| 103 | - Model the prompt state and completion as a state machine (?)
|
| 104 | - vtparse is another good example
|
| 105 | -->
|
| 106 |
|
| 107 | ## Error Locations and Fallbacks
|
| 108 |
|
| 109 | - `ExecuteAndCatch` uses mem.CurrentLocation()
|
| 110 | - `Failglob` uses mem.CurrentLocation()
|
| 111 | - `mem.GetVar $LINENO` uses `current_tok`, but it can be buggy
|
| 112 | - this is difference than `BASH_LINENO`, which is for the call stack!
|
| 113 |
|
| 114 | Other:
|
| 115 |
|
| 116 | - `ui.ctx_Location`
|
| 117 | - So builtins can call `errfmt.Print_()` without locations
|
| 118 | - `alloc.ctx_SourceCode` for assigning `source_t` to tokens
|
| 119 |
|
| 120 | ## Other Topics
|
| 121 |
|
| 122 | - [Dependency Injection]($xref:dependency-injection)
|
| 123 |
|