1 | OPy Compiler and Byterun
|
2 | ========================
|
3 |
|
4 | The OPy compiler is a Python bytecode compiler written in Python. See
|
5 | [Building Oil with the OPy Bytecode Compiler][oil-with-opy]. It's currently
|
6 | used to translate Python source code in Oil to `.pyc` files.
|
7 |
|
8 | The `byterun/` directory is a fork of [byterun][]. It's an experiment for
|
9 | learning what it will take to write a minimal interpreter for Oil. It can
|
10 | currently run all Oil unit tests, but isn't otherwise used.
|
11 |
|
12 | [oil-with-opy]: http://www.oilshell.org/blog/2018/03/04.html
|
13 |
|
14 | [byterun]: http://aosabook.org/en/500L/a-python-interpreter-written-in-python.html
|
15 |
|
16 | ## 2022 Update: OPy Will be "Replaced" By mycpp / Pea
|
17 |
|
18 | A bytecode interpreter isn't fast enough to run Oil. We still have the
|
19 | double-interpretation problem.
|
20 |
|
21 | ## Getting started
|
22 |
|
23 | Do the "Quick Start" in "in https://github.com/oilshell/oil/wiki/Contributing .
|
24 |
|
25 | Then build the `py27.grammar` file:
|
26 |
|
27 | $ make _build/opy/py27.grammar.pickle
|
28 |
|
29 | After Oil is setup, we can try out OPy. Run these commands (and let me know if
|
30 | any of them doesn't work):
|
31 |
|
32 | oil$ cd opy
|
33 | opy$ ../bin/opyc run gold/hello_py2.py # basic test of compiler and runtime
|
34 |
|
35 | Compile Oil with the OPy compiler:
|
36 |
|
37 | $ ./build.sh oil-repo # makes _tmp/repo-with-opy and _tmp/repo-with-cpython
|
38 |
|
39 | Run Oil unit tests, compiled with OPy, under **CPython**:
|
40 |
|
41 | $ ./test.sh oil-unit
|
42 |
|
43 | Run Oil unit tests, compiled with OPy, under **byterun**:
|
44 |
|
45 | $ ./test.sh oil-unit-byterun # Run Oil unit tests, compiled with OPy, under CPython
|
46 |
|
47 | Gold tests in `gold/` compare the output of CPython vs. byterun:
|
48 |
|
49 | $ ./test.sh gold
|
50 |
|
51 | Oil spec tests under byterun (slow):
|
52 |
|
53 | opy$ ./test.sh spec smoke # like $REPO_ROOT/test/spec.sh smoke
|
54 | opy$ ./test.sh spec all # like $REPO_ROOT/test/spec.sh all
|
55 |
|
56 | FYI, they can be run manually like this:
|
57 |
|
58 | $ gold/regex_compile.py # run with CPython
|
59 | $ ../bin/opyc run gold/regex_compile.py
|
60 |
|
61 | Demo of the speed difference between OSH under CPython and OSH under byterun:
|
62 |
|
63 | ./demo.sh osh-byterun-speed
|
64 |
|
65 | ## OPy Compiler Regtest
|
66 |
|
67 | This uses an old snapshot of the repo in `_regtest/`.
|
68 |
|
69 | ./regtest.sh compile
|
70 | ./regtest.sh verify-golden
|
71 |
|
72 | ## Notes on Three OPy Builds
|
73 |
|
74 | - `$REPO_ROOT/_build/oil/bytecode-opy`: Bytecode for the release binary. Built
|
75 | by `Makefile`.
|
76 | - `$REPO_ROOT/opy/_tmp/repo-with-opy`: The entire repo with OPy. For running
|
77 | Oil unit/spec tests under byterun, etc. Built by `./build.sh oil-repo`.
|
78 | - `$REPO_ROOT/opy/_tmp/regtest`: The snapshot of Python files in `opy/_regtest`
|
79 | are compiled, so we are insensitive to repo changes. Built by `./regtest.sh
|
80 | compile`.
|
81 |
|
82 | ## OPy Compiler Divergences from CPython
|
83 |
|
84 | ### Lexer
|
85 |
|
86 | - I don't remember where exactly, but I ran into a bug lexing the CPython test
|
87 | suite. IIRC, CPython's lexer was more lenient about adjacent tokens without
|
88 | spaces than `tokenize.py`.
|
89 | - `heapq.py` had `-*- coding: latin-1 -*-`, which causes problems. OPy
|
90 | should require `utf-8` source anyway.
|
91 |
|
92 | ### Parser
|
93 |
|
94 | - I ran into a bug where a file like `d = {}`, without a trailing newline,
|
95 | gives a parse error. Adding the newline fixes it.
|
96 | - print statements aren't allowed; we force Python 3-style `print(x, y,
|
97 | file=sys.stderr)`. I think this is because the parser doesn't know about
|
98 | `__future__` statements, so it can't change the parsing mode on the fly.
|
99 |
|
100 | ### Bytecode Compiler
|
101 |
|
102 | - I think there are no `LOAD_FAST` bytecodes generated? TODO: Make a bytecode
|
103 | histogram using `opy/misc/inspect_pyc`.
|
104 | - The OPy bytecode is bigger than the CPython bytecode! Why is that?
|
105 |
|