| 1 | Soil
|
| 2 | ====
|
| 3 |
|
| 4 | Continuous testing on many platforms.
|
| 5 |
|
| 6 | ## Server Setup
|
| 7 |
|
| 8 | soil/web-init.sh deploy # deploy static assets and tools
|
| 9 |
|
| 10 | ## Data
|
| 11 |
|
| 12 | https://test.oils-for-unix.org/ # not Soil, Coil could be a better name
|
| 13 |
|
| 14 | static-api/ # used by the maybe-merge task
|
| 15 | github/
|
| 16 | 1234/ GITHUB_RUN_ID (or NUMBER)
|
| 17 | dev-minimal # file with status 0
|
| 18 | cpp-small
|
| 19 |
|
| 20 | github-jobs/
|
| 21 | index.html # pretty list of runs, links to commits/ and foo.wwz/index
|
| 22 | raw.html # flat
|
| 23 | 1234/ # GITHUB_RUN_NUMBER
|
| 24 | dev-minimal.wwz/
|
| 25 | index.html # worker runs format-wwz-index for TSV -> HTML
|
| 26 | dev-minimal.{tsv,json}
|
| 27 | cpp-small.wwz/
|
| 28 | index.html
|
| 29 | cpp-small.{tsv,json}
|
| 30 | commits/
|
| 31 |
|
| 32 | srht-jobs/
|
| 33 | index.html
|
| 34 | raw.html
|
| 35 | 345/ # JOB_ID
|
| 36 | dev-minimal.wwz/
|
| 37 | index.html
|
| 38 | dev-minimal.{tsv,json}
|
| 39 | 344/
|
| 40 | cpp-small.wwz/
|
| 41 | index.html
|
| 42 | cpp-small.{tsv,json}
|
| 43 | commits/ # index grouped by commit?
|
| 44 | 09ab09ab.html # links to ../345/dev-minimal.wwz/
|
| 45 | 1010abab.html
|
| 46 |
|
| 47 | ## Code
|
| 48 |
|
| 49 | Running a job starts at either:
|
| 50 |
|
| 51 | - `.github/workflows/all-builds.yml` for Github Actions
|
| 52 | - `.builds/worker{1,2,3,4}.yml` for sourcehut
|
| 53 |
|
| 54 | The YAML files either:
|
| 55 |
|
| 56 | - Directly invoke `soil/worker.sh` for a raw VM job.
|
| 57 | - Invoke wrappers `soil/github-actions.sh` / `soil/sourcehut.sh`, which in turn
|
| 58 | uses `soil/host-shim.sh` to run a job in a container.
|
| 59 | - They wrappers also publish via SSH with `soil/web-worker.sh`
|
| 60 |
|
| 61 | `soil/host-shim.sh` pulls and start an OCI container each node, and then runs
|
| 62 | `soil/worker.sh` inside the container.
|
| 63 |
|
| 64 | `soil/worker.sh` runs the job, and publishes 3 files to `travis-ci.oilshell.org`:
|
| 65 |
|
| 66 | 1. JSON metadata about the commit and build environment
|
| 67 | 1. TSV metadata for each "toil" step
|
| 68 | 1. A `.wwz` file (servable zip file) of logs
|
| 69 |
|
| 70 | `soil/web.sh` is a wrapper around `soil/web.py`, and it runs on the SERVER
|
| 71 | (`travis-ci.oilshell.org`). It reads and joins metadata from every job to
|
| 72 | construct `index.html` and `$RUN_NUMBER/index.html`.
|
| 73 |
|
| 74 | The server-side components are deployed by `soil/web-init.sh`.
|
| 75 |
|
| 76 | ## Terminology
|
| 77 |
|
| 78 | Three level hierarchy of executables:
|
| 79 |
|
| 80 | - Run - A group of jobs run per commit, usually on different machines.
|
| 81 | Github has `GITHUB_RUN_{NUMBER,ID}`, but sourcehut lacks this concept.
|
| 82 | - Job - A group of tasks run on the same machine, in the same container, or on a raw VM.
|
| 83 | - Task - A shell command that's run within the Oil repo. Tasks are currently
|
| 84 | run sequentially.
|
| 85 |
|
| 86 | TODO:
|
| 87 |
|
| 88 | - Tasks need explicit dependencies
|
| 89 | - Dynamically schedule tasks, and remove the idea of a job. It should just be
|
| 90 | a flat list of tasks run per commit, on various machines, with
|
| 91 | dynamically-scheduled and resolved dependencies.
|
| 92 |
|
| 93 | ## Events
|
| 94 |
|
| 95 | - End Job.
|
| 96 | - First upload wwz, tsv, and JSON last
|
| 97 | - web.py can rewrites the following files in one pass, based on list-json
|
| 98 | state
|
| 99 | - github-jobs/tmp-$$.{index,raw}.html - shell script does mv
|
| 100 | - github-jobs/commits/tmp-$$.$HASH.html - shell script does mv
|
| 101 | - this is based on github-jobs/$RUN/*.tsv -- similar to format-wwz-index
|
| 102 | - or srht-jobs/*/*.tsv and filtered by commit
|
| 103 | - github-jobs/tmp-$$.remove.txt - shell script does rm
|
| 104 | - status-api/github-jobs/$RUN/$job -- PUT this
|
| 105 |
|
| 106 | - Start job. TODO: This doesn't exist.
|
| 107 | - github-jobs/index.html should show Started / Pass / Fail
|
| 108 |
|
| 109 | ## Web Interface / Security
|
| 110 |
|
| 111 | This one is a pure uploader, which you can borrow from picdir.
|
| 112 |
|
| 113 | POST https://test.oils-for-unix.org/results
|
| 114 | ?jobName=dev-minimal
|
| 115 | &workerHost=github
|
| 116 |
|
| 117 | It does multi-part file upload of wwz, TSV, JSON, and saves the files. Start
|
| 118 | with basic auth over HTTPS? See picdir.
|
| 119 |
|
| 120 |
|
| 121 | POST https://test.oils-for-unix.org/event
|
| 122 |
|
| 123 | ?eventName=start
|
| 124 | &workerHost=github
|
| 125 | &runId=1234
|
| 126 | &jobName=dev-minimal
|
| 127 |
|
| 128 | ?eventName=start
|
| 129 | &workerHost=sourcehut
|
| 130 | &jobId=345
|
| 131 | &jobName=dev-minimal
|
| 132 |
|
| 133 | # whether to update the status-api
|
| 134 | ?eventName=done
|
| 135 | &workerHost=github
|
| 136 | &runId=1234
|
| 137 | &jobName=dev-minimal
|
| 138 | &status=124
|
| 139 | &updateStatusApi=1
|
| 140 |
|
| 141 | This PHP script can just run a shell script synchronously? Events start and
|
| 142 | done are meant to be quick.
|
| 143 |
|
| 144 | Other events:
|
| 145 |
|
| 146 | - begin pulling container
|
| 147 | - done pulling container
|
| 148 | - done with task, although we want to get rid of this abstraction
|
| 149 |
|
| 150 |
|
| 151 | ## Notes
|
| 152 |
|
| 153 | ### Idea For Tuple Space?
|
| 154 |
|
| 155 | Listening to `oilshell/oil`:
|
| 156 |
|
| 157 | - `dev-minimal`
|
| 158 | - Can a contributor quickly get started with the Oil repo?
|
| 159 | - They just want to run bin/osh without installing much
|
| 160 | - `ovm-tarball`
|
| 161 | - `cpp`
|
| 162 | - `other-tests`
|
| 163 |
|
| 164 | Listening to `oilshell/tarballs`:
|
| 165 |
|
| 166 | - Travis
|
| 167 | - OS X (bin-darwin)
|
| 168 | - Sourcehut
|
| 169 | - Alpine (bin-alpine)
|
| 170 | - OpenBSD (bin-openbsd)
|
| 171 | - Alternate architectures like ARM
|
| 172 |
|
| 173 | All of these build both oil.ovm and oil-native. Need maintainers. We build
|
| 174 | them as a "start".
|
| 175 |
|
| 176 | It would nice to implement this as SQL.
|
| 177 |
|
| 178 |
|
| 179 |
|