OILS / README.md View on Github | oilshell.org

317 lines, 234 significant
1Oils Source Code
2================
3
4[![Build
5Status](https://github.com/oils-for-unix/oils/actions/workflows/all-builds.yml/badge.svg)](https://github.com/oils-for-unix/oils/actions/workflows/all-builds.yml) <a href="https://gitpod.io/from-referrer/">
6 <img src="https://img.shields.io/badge/Contribute%20with-Gitpod-908a85?logo=gitpod" alt="Contribute with Gitpod" />
7</a>
8
9[Oils][] is our upgrade path from bash to a better language and runtime!
10
11- [OSH][] runs your existing shell scripts.
12- [YSH][] is for Python and JavaScript users who avoid shell.
13
14(The project was [slightly renamed][rename] in March 2023, so there are still
15old references to "Oil". Feel free to send pull requests with corrections!)
16
17[Oils]: https://www.oilshell.org/
18
19[OSH]: https://www.oilshell.org/cross-ref.html#OSH
20[YSH]: https://www.oilshell.org/cross-ref.html#YSH
21
22[rename]: https://www.oilshell.org/blog/2023/03/rename.html
23
24[Oils 2023 FAQ][faq-2023] / [Why Create a New Unix Shell?][why]
25
26[faq-2023]: https://www.oilshell.org/blog/2023/03/faq.html
27[why]: https://www.oilshell.org/blog/2021/01/why-a-new-shell.html
28
29It's written in Python, so the code is short and easy to change. But we
30automatically translate it to C++ with custom tools, to make it fast and small.
31The deployed executable doesn't depend on Python.
32
33This README is at the root of the [git repo][git-repo].
34
35[git-repo]: https://github.com/oils-for-unix/oils
36
37<div id="toc">
38</div>
39
40## Contributing
41
42* Try making the **dev build** of Oils with the instructions on the
43 [Contributing][] page. This should take 1 to 5 minutes if you have a Linux
44 machine.
45* If it doesn't, let us know. You can post on the `#oil-dev` channel of
46 [oilshell.zulipchat.com][], or file an issue on Github.
47* Feel free to grab an [issue from
48 Github](https://github.com/oils-for-unix/oils/issues?q=is%3Aissue+is%3Aopen+label%3A%22good+first+issue%22).
49 Let us know what you're thinking before you get too far.
50
51[Contributing]: https://github.com/oils-for-unix/oils/wiki/Contributing
52[oilshell.zulipchat.com]: https://oilshell.zulipchat.com/
53[blog]: https://www.oilshell.org/blog/
54
55### Quick Start on Linux
56
57After following the instructions on the [Contributing][] page, you'll have a
58Python program that you can quickly run and change! Try it interactively:
59
60 bash$ bin/osh
61
62 osh$ name=world
63 osh$ echo "hello $name"
64 hello world
65
66- Try running a shell script you wrote with `bin/osh myscript.sh`.
67- Try [YSH][] with `bin/ysh`.
68
69Let us know if any of these things don't work! [The continuous
70build](http://travis-ci.oilshell.org/) tests them at every commit.
71
72### Dev Build vs. Release Build
73
74Again, note that the **developer build** is **very different** from the release
75tarball. The [Contributing][] page describes this difference in detail.
76
77The release tarballs are linked from the [home
78page](https://www.oilshell.org/). (Developer builds don't work on OS X, so use
79the release tarballs on OS X.)
80
81### Important: We Accept Small Contributions!
82
83Oils is full of [many ideas](https://www.oilshell.org/blog/), which may be
84intimidating at first.
85
86But the bar to contribution is very low. It's basically a medium size Python
87program with many tests, and many programmers know how to change such programs.
88It's great for prototyping.
89
90- For OSH compatibility, I often merge **failing [spec
91 tests](https://www.oilshell.org/cross-ref.html#spec-test)**. You don't even
92 have to write code! The tests alone help. I search for related tests with
93 `grep xtrace spec/*.test.sh`, where `xtrace` is a shell feature.
94- You only have to make your code work **in Python**. Plain Python programs
95 are easy to modify. The semi-automated translation to C++ is a separate
96 step, although it often just works.
97- You can **influence the design** of [YSH][]. If you have an itch to
98 scratch, be ambitious. For example, you might want to show us how to
99 implement [nonlinear pipelines](https://github.com/oils-for-unix/oils/issues/843).
100
101### I aim for 24 hour response time
102
103Please feel free to ping `andychu` on Zulip or Github if you're **waiting** for
104a pull request review! (or to ask questions)
105
106Usually I can respond in 24 hours. I might be traveling, in which case I'll
107respond with something like *I hope to look at this by Tuesday*.
108
109I might have also **missed** your Github message, so it doesn't hurt to ping
110me.
111
112Thank you for the contributions!
113
114### Docs
115
116The [Wiki](https://github.com/oils-for-unix/oils/wiki) has many developer docs. Feel
117free to edit them. If you make a major change, let us know on Zulip!
118
119There are also READMEs in some subdirectories, like `opy/` and `mycpp/`.
120
121If you're confused, the best thing to do is to ask on Zulip and someone should
122produce a pointer and/or improve the docs.
123
124Docs for **end users** are linked from each [release
125page](https://www.oilshell.org/releases.html).
126
127## Repository Structure
128
129Try this to show a summary of what's in the repo and their line counts:
130
131 $ metrics/source-code.sh overview
132
133(Other functions in this file may be useful as well.)
134
135### A Collection of Interpreters
136
137Oils is naturally structured as a set of mutually recursive parsers and
138evaluators. These interpreters are specified at a high-level: with regular
139languages, Zephyr ASDL, and a statically-typed subset of Python.
140
141 bin/ # Main entry points like bin/osh (source in bin/oils_for_unix.py)
142 frontend/ # Input and lexing common to OSH and YSH
143 osh/ # OSH parsers and evaluators (cmd, word, sh_expr)
144 ysh/ # YSH parser and evaluator
145 data_lang/ # Languages based on JSON
146 builtin/ # Builtin commands and functions
147 core/ # Other code shared between OSH and YSH
148 pyext/ # Python extension modules, e.g. libc.c
149 pylib/ # Borrowed from the Python standard library.
150 tools/ # User-facing tools, e.g. the osh2oil translator
151
152### DSLs / Code Generators
153
154Here are the tools that transform that high-level code to efficient code:
155
156 asdl/ # ASDL implementation, derived from CPython
157 pgen2/ # Parser Generator, borrowed from CPython
158 mycpp/ # Experimental translator from typed Python to C++.
159 # Depends on MyPy. See mycpp/README.md
160 pea/ # Perhaps a cleaner version of mycpp
161 opy/ # Python compiler in Python (mycpp/ will replace it)
162
163### Native Code and Build System
164
165We have native code to support both the dev build (running under CPython) and
166the `oils-for-unix` build (pure C++):
167
168 NINJA-config.sh # Generates build.ninja
169
170 build/ # High level build
171 NINJA-steps.sh
172 NINJA_main.py # invoked by NINJA-config.sh
173 NINJA_subgraph.py
174 oil-defs/ # Files that define our slice of CPython.
175 py.sh # For development builds, running CPython
176 cpp/ # C++ code which complements the mycpp translation
177 NINJA-steps.sh
178 NINJA_subgraph.py
179 mycpp/ # Runtime for the translator
180 NINJA-steps.sh
181 NINJA_subgraph.py
182
183 prebuilt/ # Prebuilt files committed to git, instead of in _gen/
184
185 Python-2.7.13/ # For the slow Python build
186
187 # Temp dirs (see below)
188 _bin/
189 _build/
190 _gen/
191 _test/
192
193### Several Kinds of Tests
194
195Unit tests are named `foo_test.py` and live next to `foo.py`.
196
197 test/ # Test automation
198 gold/ # Gold Test cases
199 gold.sh
200 sh_spec.py # shell spec test framework
201 spec.sh # Types of test runner: spec, unit, gold, wild
202 unit.sh
203 wild.sh
204 testdata/
205 spec/ # Spec test cases
206 bin/ # tools used in many spec tests
207 testdata/ # scripts for specific test cases
208 stateful/ # Tests that use pexpect
209
210### Dev Tools and Scripts
211
212We use a lot of automation to improve the dev process. It's largely written in
213shell, of course!
214
215 benchmarks/ # Benchmarks should be run on multiple machines.
216 metrics/ # Metrics don't change between machines (e.g. code size)
217 client/ # Demonstration of OSH as a headless server.
218 deps/ # Dev dependencies and Docker images
219 devtools/ # For Oils developers (not end users)
220 release.sh # The (large) release process.
221 services/ # talk to cloud services
222 demo/ # Demonstrations of bash/shell features. Could be
223 # moved to tests/ if automated.
224 old/ # A junk drawer.
225 web/ # HTML/JS/CSS for tests and tools
226 soil/ # Multi-cloud continuous build (e.g. sourcehut, Github)
227
228### Temp Dirs
229
230Directories that begin with `_` are **not** stored in `git`. The dev tools
231above create and use these dirs.
232
233 _bin/ # Native executables are put here
234 cxx-dbg/
235 _build/ # Temporary build files
236 _cache/ # Dev dependency tarballs
237 _devbuild/ # Generated Python code, etc.
238 _gen/ # Generated C++ code that mirrors the repo
239 frontend/
240 _release/ # Source release tarballs are put here
241 VERSION/ # Published at oilshell.org/release/$VERSION/
242 benchmarks/
243 doc/
244 metrics/
245 test/
246 spec.wwz
247 wild.wwz
248 ...
249 web/ # Static files, copy of $REPO_ROOT/web
250 table/
251 _test/ # Unit tests, mycpp examples
252 tasks/
253 _tmp/ # Output of other test suites; temp files
254 spec/
255 wild/
256 raw/
257 www/
258 osh-parser/
259 osh-runtime/
260 vm-baseline/
261 oheap/
262 startup/
263 ...
264
265### Build Dependencies in `../oil_DEPS`
266
267These tools are built from shell scripts in `soil/`. The `oil_DEPS` dir is
268"parallel" to Oils because it works better with container bind mounds.
269
270 ../oil_DEPS/
271 re2c/ # to build the lexer
272 cmark/ # for building docs
273 spec-bin/ # shells to run spec tests against
274 mypy/ # MyPy repo
275 mycpp-venv/ # MyPy binaries deps in a VirtualEnv
276
277 py3/ # for mycpp and pea/
278 cpython-full/ # for bootstrapping Oils-CPython
279
280
281### Build System for End Users version.
282
283These files make the slow "Oils Python" build, which is very different than the
284**developer build** of Oils.
285
286 Makefile
287 configure
288 install
289
290These files are for the C++ `oils-for-unix` tarball (in progress):
291
292 _build/
293 oils.sh
294
295### Doc Sources
296
297 doc/ # A mix of docs
298 doctools/ # Tools that use lazylex/ to transform Markdown/HTML
299 lazylex/ # An HTML lexer which doctools/ builds upon.
300 README.md # This page, which is For Oils developers
301
302 LICENSE.txt # For end users
303 INSTALL.txt
304
305## More info
306
307There are README files in many subdirectories, like
308[mycpp/README.md](mycpp/README.md).
309
310* [The blog][blog] has updates on the project status.
311* [Oils Home Page](https://www.oilshell.org/)
312* [oilshell.zulipchat.com][] is for any kind of discussion
313* Subscribe for updates:
314 * [/r/oilshell on Reddit](https://www.reddit.com/r/oilshell/)
315 * [@oilsforunix on Twitter](https://twitter.com/oilsforunix)
316
317