1 | Oils Source Code
|
2 | ================
|
3 |
|
4 | [![Build
|
5 | Status](https://github.com/oils-for-unix/oils/actions/workflows/all-builds.yml/badge.svg)](https://github.com/oils-for-unix/oils/actions/workflows/all-builds.yml) <a href="https://gitpod.io/from-referrer/">
|
6 | <img src="https://img.shields.io/badge/Contribute%20with-Gitpod-908a85?logo=gitpod" alt="Contribute with Gitpod" />
|
7 | </a>
|
8 |
|
9 | [Oils][] is our upgrade path from bash to a better language and runtime!
|
10 |
|
11 | - [OSH][] runs your existing shell scripts.
|
12 | - [YSH][] is for Python and JavaScript users who avoid shell.
|
13 |
|
14 | (The project was [slightly renamed][rename] in March 2023, so there are still
|
15 | old references to "Oil". Feel free to send pull requests with corrections!)
|
16 |
|
17 | [Oils]: https://www.oilshell.org/
|
18 |
|
19 | [OSH]: https://www.oilshell.org/cross-ref.html#OSH
|
20 | [YSH]: https://www.oilshell.org/cross-ref.html#YSH
|
21 |
|
22 | [rename]: https://www.oilshell.org/blog/2023/03/rename.html
|
23 |
|
24 | [Oils 2023 FAQ][faq-2023] / [Why Create a New Unix Shell?][why]
|
25 |
|
26 | [faq-2023]: https://www.oilshell.org/blog/2023/03/faq.html
|
27 | [why]: https://www.oilshell.org/blog/2021/01/why-a-new-shell.html
|
28 |
|
29 | It's written in Python, so the code is short and easy to change. But we
|
30 | automatically translate it to C++ with custom tools, to make it fast and small.
|
31 | The deployed executable doesn't depend on Python.
|
32 |
|
33 | This README is at the root of the [git repo][git-repo].
|
34 |
|
35 | [git-repo]: https://github.com/oils-for-unix/oils
|
36 |
|
37 | <div id="toc">
|
38 | </div>
|
39 |
|
40 | ## Contributing
|
41 |
|
42 | * Try making the **dev build** of Oils with the instructions on the
|
43 | [Contributing][] page. This should take 1 to 5 minutes if you have a Linux
|
44 | machine.
|
45 | * If it doesn't, let us know. You can post on the `#oil-dev` channel of
|
46 | [oilshell.zulipchat.com][], or file an issue on Github.
|
47 | * Feel free to grab an [issue from
|
48 | Github](https://github.com/oils-for-unix/oils/issues?q=is%3Aissue+is%3Aopen+label%3A%22good+first+issue%22).
|
49 | Let us know what you're thinking before you get too far.
|
50 |
|
51 | [Contributing]: https://github.com/oils-for-unix/oils/wiki/Contributing
|
52 | [oilshell.zulipchat.com]: https://oilshell.zulipchat.com/
|
53 | [blog]: https://www.oilshell.org/blog/
|
54 |
|
55 | ### Quick Start on Linux
|
56 |
|
57 | After following the instructions on the [Contributing][] page, you'll have a
|
58 | Python program that you can quickly run and change! Try it interactively:
|
59 |
|
60 | bash$ bin/osh
|
61 |
|
62 | osh$ name=world
|
63 | osh$ echo "hello $name"
|
64 | hello world
|
65 |
|
66 | - Try running a shell script you wrote with `bin/osh myscript.sh`.
|
67 | - Try [YSH][] with `bin/ysh`.
|
68 |
|
69 | Let us know if any of these things don't work! [The continuous
|
70 | build](http://travis-ci.oilshell.org/) tests them at every commit.
|
71 |
|
72 | ### Dev Build vs. Release Build
|
73 |
|
74 | Again, note that the **developer build** is **very different** from the release
|
75 | tarball. The [Contributing][] page describes this difference in detail.
|
76 |
|
77 | The release tarballs are linked from the [home
|
78 | page](https://www.oilshell.org/). (Developer builds don't work on OS X, so use
|
79 | the release tarballs on OS X.)
|
80 |
|
81 | ### Important: We Accept Small Contributions!
|
82 |
|
83 | Oils is full of [many ideas](https://www.oilshell.org/blog/), which may be
|
84 | intimidating at first.
|
85 |
|
86 | But the bar to contribution is very low. It's basically a medium size Python
|
87 | program with many tests, and many programmers know how to change such programs.
|
88 | It's great for prototyping.
|
89 |
|
90 | - For OSH compatibility, I often merge **failing [spec
|
91 | tests](https://www.oilshell.org/cross-ref.html#spec-test)**. You don't even
|
92 | have to write code! The tests alone help. I search for related tests with
|
93 | `grep xtrace spec/*.test.sh`, where `xtrace` is a shell feature.
|
94 | - You only have to make your code work **in Python**. Plain Python programs
|
95 | are easy to modify. The semi-automated translation to C++ is a separate
|
96 | step, although it often just works.
|
97 | - You can **influence the design** of [YSH][]. If you have an itch to
|
98 | scratch, be ambitious. For example, you might want to show us how to
|
99 | implement [nonlinear pipelines](https://github.com/oils-for-unix/oils/issues/843).
|
100 |
|
101 | ### I aim for 24 hour response time
|
102 |
|
103 | Please feel free to ping `andychu` on Zulip or Github if you're **waiting** for
|
104 | a pull request review! (or to ask questions)
|
105 |
|
106 | Usually I can respond in 24 hours. I might be traveling, in which case I'll
|
107 | respond with something like *I hope to look at this by Tuesday*.
|
108 |
|
109 | I might have also **missed** your Github message, so it doesn't hurt to ping
|
110 | me.
|
111 |
|
112 | Thank you for the contributions!
|
113 |
|
114 | ### Docs
|
115 |
|
116 | The [Wiki](https://github.com/oils-for-unix/oils/wiki) has many developer docs. Feel
|
117 | free to edit them. If you make a major change, let us know on Zulip!
|
118 |
|
119 | There are also READMEs in some subdirectories, like `opy/` and `mycpp/`.
|
120 |
|
121 | If you're confused, the best thing to do is to ask on Zulip and someone should
|
122 | produce a pointer and/or improve the docs.
|
123 |
|
124 | Docs for **end users** are linked from each [release
|
125 | page](https://www.oilshell.org/releases.html).
|
126 |
|
127 | ## Repository Structure
|
128 |
|
129 | Try this to show a summary of what's in the repo and their line counts:
|
130 |
|
131 | $ metrics/source-code.sh overview
|
132 |
|
133 | (Other functions in this file may be useful as well.)
|
134 |
|
135 | ### A Collection of Interpreters
|
136 |
|
137 | Oils is naturally structured as a set of mutually recursive parsers and
|
138 | evaluators. These interpreters are specified at a high-level: with regular
|
139 | languages, Zephyr ASDL, and a statically-typed subset of Python.
|
140 |
|
141 | bin/ # Main entry points like bin/osh (source in bin/oils_for_unix.py)
|
142 | frontend/ # Input and lexing common to OSH and YSH
|
143 | osh/ # OSH parsers and evaluators (cmd, word, sh_expr)
|
144 | ysh/ # YSH parser and evaluator
|
145 | data_lang/ # Languages based on JSON
|
146 | builtin/ # Builtin commands and functions
|
147 | core/ # Other code shared between OSH and YSH
|
148 | pyext/ # Python extension modules, e.g. libc.c
|
149 | pylib/ # Borrowed from the Python standard library.
|
150 | tools/ # User-facing tools, e.g. the osh2oil translator
|
151 |
|
152 | ### DSLs / Code Generators
|
153 |
|
154 | Here are the tools that transform that high-level code to efficient code:
|
155 |
|
156 | asdl/ # ASDL implementation, derived from CPython
|
157 | pgen2/ # Parser Generator, borrowed from CPython
|
158 | mycpp/ # Experimental translator from typed Python to C++.
|
159 | # Depends on MyPy. See mycpp/README.md
|
160 | pea/ # Perhaps a cleaner version of mycpp
|
161 | opy/ # Python compiler in Python (mycpp/ will replace it)
|
162 |
|
163 | ### Native Code and Build System
|
164 |
|
165 | We have native code to support both the dev build (running under CPython) and
|
166 | the `oils-for-unix` build (pure C++):
|
167 |
|
168 | NINJA-config.sh # Generates build.ninja
|
169 |
|
170 | build/ # High level build
|
171 | NINJA-steps.sh
|
172 | NINJA_main.py # invoked by NINJA-config.sh
|
173 | NINJA_subgraph.py
|
174 | oil-defs/ # Files that define our slice of CPython.
|
175 | py.sh # For development builds, running CPython
|
176 | cpp/ # C++ code which complements the mycpp translation
|
177 | NINJA-steps.sh
|
178 | NINJA_subgraph.py
|
179 | mycpp/ # Runtime for the translator
|
180 | NINJA-steps.sh
|
181 | NINJA_subgraph.py
|
182 |
|
183 | prebuilt/ # Prebuilt files committed to git, instead of in _gen/
|
184 |
|
185 | Python-2.7.13/ # For the slow Python build
|
186 |
|
187 | # Temp dirs (see below)
|
188 | _bin/
|
189 | _build/
|
190 | _gen/
|
191 | _test/
|
192 |
|
193 | ### Several Kinds of Tests
|
194 |
|
195 | Unit tests are named `foo_test.py` and live next to `foo.py`.
|
196 |
|
197 | test/ # Test automation
|
198 | gold/ # Gold Test cases
|
199 | gold.sh
|
200 | sh_spec.py # shell spec test framework
|
201 | spec.sh # Types of test runner: spec, unit, gold, wild
|
202 | unit.sh
|
203 | wild.sh
|
204 | testdata/
|
205 | spec/ # Spec test cases
|
206 | bin/ # tools used in many spec tests
|
207 | testdata/ # scripts for specific test cases
|
208 | stateful/ # Tests that use pexpect
|
209 |
|
210 | ### Dev Tools and Scripts
|
211 |
|
212 | We use a lot of automation to improve the dev process. It's largely written in
|
213 | shell, of course!
|
214 |
|
215 | benchmarks/ # Benchmarks should be run on multiple machines.
|
216 | metrics/ # Metrics don't change between machines (e.g. code size)
|
217 | client/ # Demonstration of OSH as a headless server.
|
218 | deps/ # Dev dependencies and Docker images
|
219 | devtools/ # For Oils developers (not end users)
|
220 | release.sh # The (large) release process.
|
221 | services/ # talk to cloud services
|
222 | demo/ # Demonstrations of bash/shell features. Could be
|
223 | # moved to tests/ if automated.
|
224 | old/ # A junk drawer.
|
225 | web/ # HTML/JS/CSS for tests and tools
|
226 | soil/ # Multi-cloud continuous build (e.g. sourcehut, Github)
|
227 |
|
228 | ### Temp Dirs
|
229 |
|
230 | Directories that begin with `_` are **not** stored in `git`. The dev tools
|
231 | above create and use these dirs.
|
232 |
|
233 | _bin/ # Native executables are put here
|
234 | cxx-dbg/
|
235 | _build/ # Temporary build files
|
236 | _cache/ # Dev dependency tarballs
|
237 | _devbuild/ # Generated Python code, etc.
|
238 | _gen/ # Generated C++ code that mirrors the repo
|
239 | frontend/
|
240 | _release/ # Source release tarballs are put here
|
241 | VERSION/ # Published at oilshell.org/release/$VERSION/
|
242 | benchmarks/
|
243 | doc/
|
244 | metrics/
|
245 | test/
|
246 | spec.wwz
|
247 | wild.wwz
|
248 | ...
|
249 | web/ # Static files, copy of $REPO_ROOT/web
|
250 | table/
|
251 | _test/ # Unit tests, mycpp examples
|
252 | tasks/
|
253 | _tmp/ # Output of other test suites; temp files
|
254 | spec/
|
255 | wild/
|
256 | raw/
|
257 | www/
|
258 | osh-parser/
|
259 | osh-runtime/
|
260 | vm-baseline/
|
261 | oheap/
|
262 | startup/
|
263 | ...
|
264 |
|
265 | ### Build Dependencies in `../oil_DEPS`
|
266 |
|
267 | These tools are built from shell scripts in `soil/`. The `oil_DEPS` dir is
|
268 | "parallel" to Oils because it works better with container bind mounds.
|
269 |
|
270 | ../oil_DEPS/
|
271 | re2c/ # to build the lexer
|
272 | cmark/ # for building docs
|
273 | spec-bin/ # shells to run spec tests against
|
274 | mypy/ # MyPy repo
|
275 | mycpp-venv/ # MyPy binaries deps in a VirtualEnv
|
276 |
|
277 | py3/ # for mycpp and pea/
|
278 | cpython-full/ # for bootstrapping Oils-CPython
|
279 |
|
280 |
|
281 | ### Build System for End Users version.
|
282 |
|
283 | These files make the slow "Oils Python" build, which is very different than the
|
284 | **developer build** of Oils.
|
285 |
|
286 | Makefile
|
287 | configure
|
288 | install
|
289 |
|
290 | These files are for the C++ `oils-for-unix` tarball (in progress):
|
291 |
|
292 | _build/
|
293 | oils.sh
|
294 |
|
295 | ### Doc Sources
|
296 |
|
297 | doc/ # A mix of docs
|
298 | doctools/ # Tools that use lazylex/ to transform Markdown/HTML
|
299 | lazylex/ # An HTML lexer which doctools/ builds upon.
|
300 | README.md # This page, which is For Oils developers
|
301 |
|
302 | LICENSE.txt # For end users
|
303 | INSTALL.txt
|
304 |
|
305 | ## More info
|
306 |
|
307 | There are README files in many subdirectories, like
|
308 | [mycpp/README.md](mycpp/README.md).
|
309 |
|
310 | * [The blog][blog] has updates on the project status.
|
311 | * [Oils Home Page](https://www.oilshell.org/)
|
312 | * [oilshell.zulipchat.com][] is for any kind of discussion
|
313 | * Subscribe for updates:
|
314 | * [/r/oilshell on Reddit](https://www.reddit.com/r/oilshell/)
|
315 | * [@oilsforunix on Twitter](https://twitter.com/oilsforunix)
|
316 |
|
317 |
|