| 1 | OPy Compiler and Byterun
|
| 2 | ========================
|
| 3 |
|
| 4 | The OPy compiler is a Python bytecode compiler written in Python. See
|
| 5 | [Building Oil with the OPy Bytecode Compiler][oil-with-opy]. It's currently
|
| 6 | used to translate Python source code in Oil to `.pyc` files.
|
| 7 |
|
| 8 | The `byterun/` directory is a fork of [byterun][]. It's an experiment for
|
| 9 | learning what it will take to write a minimal interpreter for Oil. It can
|
| 10 | currently run all Oil unit tests, but isn't otherwise used.
|
| 11 |
|
| 12 | [oil-with-opy]: http://www.oilshell.org/blog/2018/03/04.html
|
| 13 |
|
| 14 | [byterun]: http://aosabook.org/en/500L/a-python-interpreter-written-in-python.html
|
| 15 |
|
| 16 | ## 2022 Update: OPy Will be "Replaced" By mycpp / Pea
|
| 17 |
|
| 18 | A bytecode interpreter isn't fast enough to run Oil. We still have the
|
| 19 | double-interpretation problem.
|
| 20 |
|
| 21 | ## Getting started
|
| 22 |
|
| 23 | Do the "Quick Start" in "in https://github.com/oilshell/oil/wiki/Contributing .
|
| 24 |
|
| 25 | Then build the `py27.grammar` file:
|
| 26 |
|
| 27 | $ make _build/opy/py27.grammar.pickle
|
| 28 |
|
| 29 | After Oil is setup, we can try out OPy. Run these commands (and let me know if
|
| 30 | any of them doesn't work):
|
| 31 |
|
| 32 | oil$ cd opy
|
| 33 | opy$ ../bin/opyc run gold/hello_py2.py # basic test of compiler and runtime
|
| 34 |
|
| 35 | Compile Oil with the OPy compiler:
|
| 36 |
|
| 37 | $ ./build.sh oil-repo # makes _tmp/repo-with-opy and _tmp/repo-with-cpython
|
| 38 |
|
| 39 | Run Oil unit tests, compiled with OPy, under **CPython**:
|
| 40 |
|
| 41 | $ ./test.sh oil-unit
|
| 42 |
|
| 43 | Run Oil unit tests, compiled with OPy, under **byterun**:
|
| 44 |
|
| 45 | $ ./test.sh oil-unit-byterun # Run Oil unit tests, compiled with OPy, under CPython
|
| 46 |
|
| 47 | Gold tests in `gold/` compare the output of CPython vs. byterun:
|
| 48 |
|
| 49 | $ ./test.sh gold
|
| 50 |
|
| 51 | Oil spec tests under byterun (slow):
|
| 52 |
|
| 53 | opy$ ./test.sh spec smoke # like $REPO_ROOT/test/spec.sh smoke
|
| 54 | opy$ ./test.sh spec all # like $REPO_ROOT/test/spec.sh all
|
| 55 |
|
| 56 | FYI, they can be run manually like this:
|
| 57 |
|
| 58 | $ gold/regex_compile.py # run with CPython
|
| 59 | $ ../bin/opyc run gold/regex_compile.py
|
| 60 |
|
| 61 | Demo of the speed difference between OSH under CPython and OSH under byterun:
|
| 62 |
|
| 63 | ./demo.sh osh-byterun-speed
|
| 64 |
|
| 65 | ## OPy Compiler Regtest
|
| 66 |
|
| 67 | This uses an old snapshot of the repo in `_regtest/`.
|
| 68 |
|
| 69 | ./regtest.sh compile
|
| 70 | ./regtest.sh verify-golden
|
| 71 |
|
| 72 | ## Notes on Three OPy Builds
|
| 73 |
|
| 74 | - `$REPO_ROOT/_build/oil/bytecode-opy`: Bytecode for the release binary. Built
|
| 75 | by `Makefile`.
|
| 76 | - `$REPO_ROOT/opy/_tmp/repo-with-opy`: The entire repo with OPy. For running
|
| 77 | Oil unit/spec tests under byterun, etc. Built by `./build.sh oil-repo`.
|
| 78 | - `$REPO_ROOT/opy/_tmp/regtest`: The snapshot of Python files in `opy/_regtest`
|
| 79 | are compiled, so we are insensitive to repo changes. Built by `./regtest.sh
|
| 80 | compile`.
|
| 81 |
|
| 82 | ## OPy Compiler Divergences from CPython
|
| 83 |
|
| 84 | ### Lexer
|
| 85 |
|
| 86 | - I don't remember where exactly, but I ran into a bug lexing the CPython test
|
| 87 | suite. IIRC, CPython's lexer was more lenient about adjacent tokens without
|
| 88 | spaces than `tokenize.py`.
|
| 89 | - `heapq.py` had `-*- coding: latin-1 -*-`, which causes problems. OPy
|
| 90 | should require `utf-8` source anyway.
|
| 91 |
|
| 92 | ### Parser
|
| 93 |
|
| 94 | - I ran into a bug where a file like `d = {}`, without a trailing newline,
|
| 95 | gives a parse error. Adding the newline fixes it.
|
| 96 | - print statements aren't allowed; we force Python 3-style `print(x, y,
|
| 97 | file=sys.stderr)`. I think this is because the parser doesn't know about
|
| 98 | `__future__` statements, so it can't change the parsing mode on the fly.
|
| 99 |
|
| 100 | ### Bytecode Compiler
|
| 101 |
|
| 102 | - I think there are no `LOAD_FAST` bytecodes generated? TODO: Make a bytecode
|
| 103 | histogram using `opy/misc/inspect_pyc`.
|
| 104 | - The OPy bytecode is bigger than the CPython bytecode! Why is that?
|
| 105 |
|