1 | Doctools
|
2 | ========
|
3 |
|
4 | Tools we use to generate the [Oils documentation](../doc/). Some of this code
|
5 | is used to build the [the blog](//www.oilshell.org/blog/) as well.
|
6 |
|
7 | See [doc/doc-toolchain.md](../doc/doc-toolchain.md) for details.
|
8 |
|
9 | Tools shared with the blog:
|
10 |
|
11 | - `cmark.py`: Our wrapper around CommonMark.
|
12 | - `spelling.py`: spell checker
|
13 | - `split_doc.py`: Split "front matter" from Markdown.
|
14 |
|
15 | More tools:
|
16 |
|
17 | - `html_head.py`: Common HTML fragments.
|
18 | - `oil_doc.py`: HTML filters.
|
19 | - `help_gen.py`: For `doc/ref/index-{osh,ysh}.md`.
|
20 |
|
21 | ## Micro Syntax
|
22 |
|
23 | - `src_tree.py` is a fast and minimal source viewer.
|
24 | - It uses polyglot syntax analysis called "micro syntax". See
|
25 | [micro-syntax.md](micro-syntax.md).
|
26 |
|
27 | ## TODO
|
28 |
|
29 | Immediate:
|
30 |
|
31 | - iPhone CSS
|
32 | - font sizes are wrong when the line wraps
|
33 | - need to debug this directly
|
34 |
|
35 | - Experiment with shared library -- narrow waist that eliminates process overhead
|
36 | - main(argv)
|
37 | - stdin / stdout are buffers?
|
38 | - for the coprocess protocol, I wanted file descriptors. But this is a
|
39 | portable interface
|
40 | - returns status code
|
41 | - what about errors?
|
42 | - I think they can go to stderr to start
|
43 |
|
44 | - Ninja makes it faster
|
45 | - CLI syntax
|
46 | - modes: wc and cat
|
47 | - `--col` and `--omit-col`
|
48 | - `mtax cat --no-comments --no-strs` or `--empty-strs`
|
49 | - replace with spaces
|
50 | - reminds me of "garfield minus garfield"
|
51 |
|
52 | - output: 4 columns
|
53 | - which I guess are selectable
|
54 | - should tokens be binary data though?
|
55 |
|
56 | - Detect UTF-8
|
57 | - lexing doesn't work without UTF-8
|
58 |
|
59 | - Analyze TSV for function names in Python parser combinators
|
60 | - add () {} -- this is all you need really
|
61 | - oh and : for Python
|
62 |
|
63 | - C++ multi-line
|
64 | - comes up in `_gen/bin/text_files.cc`
|
65 | - this is an architecture issues, will allow Rust/Lua as well
|
66 |
|
67 | - Max color mode for debugging?
|
68 | - detail: blank lines in re2c blocks shouldn't be significant
|
69 | - I guess you detect whitespace in re2c blocks then
|
70 |
|
71 | - SLOC
|
72 | - add to index.html, with attrs
|
73 | - light grey monospace?
|
74 | - Subsumes these tools:
|
75 | - <https://github.com/AlDanial/cloc> - this is a 17K line Perl script!
|
76 | - <https://dwheeler.com/sloccount/> - no release since 2004 ?
|
77 |
|
78 | - Maybe add language for `*.test.sh`
|
79 | - the `####` and `##` lines are special
|
80 |
|
81 | src-tree:
|
82 |
|
83 | - should README.md be inserted in index.html ?
|
84 | - probably, sourcehut has this too
|
85 | - use cmark
|
86 | - also use our TOC plugin
|
87 | - line counts in metrics/source-code.sh could link to src-tree
|
88 | - combine the CI jobs
|
89 | - should use `micro_syntax --wc` to count SLOC
|
90 | - Also `micro_syntax` --print --format ansi
|
91 | - this is the default?
|
92 |
|
93 | Later:
|
94 |
|
95 | - Recast as TSV-Netstring, which is different than TSV8
|
96 | - select one of these cols:
|
97 | - Path, HTML, `num_lines`, `num_sig_lines`, and I guess ANSI?
|
98 | - line/tokens binary? You have line number and so forth
|
99 | - 3:foo\t 5:foo\n # last cell has to end with newline ?
|
100 | - preserves wc -l when data has no newlines?
|
101 | - or is this too easily confused with TSV8 itself? You don't want it to be valid TSV8?
|
102 | - it can start out as text
|
103 | - !tsv8 Str
|
104 | - !type
|
105 | - !nets-row ?
|
106 | - another thing you can do is have a prefix of cells
|
107 | - netstring is 3:foo,
|
108 | - prefix is 5; 3:foo,
|
109 | - that's a command to read 5 net strings I guess?
|
110 |
|
111 | - Parsing, jump to definition
|
112 |
|