Testing#
The suite has three layers, and each catches what the one before cannot.
Layer |
Where |
What it checks |
|---|---|---|
Examples |
|
One input, one expected answer |
Properties |
|
A rule, against every input a generator can draw |
Invariants |
|
What a correct panel is, shared by both layers above |
Coverage says which lines ran, not whether anything checked what they did. The bugs
that prompted these tests were in code the suite already ran, at 95% coverage; nobody had
written the input that broke it. The properties exist for that gap: the checks that used to run on five hand-written panels now
run on every panel tests/strategies.py can draw.
Coverage, branches included, is measured by uv run pytest and must reach 90%. In CI only
the Linux Python 3.14 job measures it; the 3.12, 3.13 and Windows jobs run the same tests
with --no-cov. Before 3.14, coverage cannot use sys.monitoring for branches and falls
back to a tracer, which made the 3.12 job, a required check, the slowest of every pull
request. Pass --no-cov locally too when the number is not what you are after.
The properties#
tests/properties/test_panel_properties.py compiles generated single-panel documents and
checks four kinds of property, the kinds that have found real bugs in numerical code
(Falsify your Software, SciPy 2020):
Determinism. Compiling twice gives byte-identical Core JSON and SVG, with and without live text, and the debug overlay.
Invariants. A compiled panel satisfies everything in
tests/invariants.py:balloons and captions stay inside the margin, off every face, out from under any exclusion, and clear of each other;
each one reads after every box before it, not only the last;
every tail runs from its balloon to its speaker;
the SVG parses, its ids are unique and every reference resolves, no number in it is
nanorinf, itsviewBoxis the panel’s size, and every glyph is drawn at the scale it was measured at.
Round trip. Panel Core reads back equal, and writes back the same bytes.
Oracle.
scenet check --deepis clean exactly when the panel compiles, and otherwise reports one finding whose rule names how it failed. A compile may fail only withBalloonPlacementErrororLayoutError; anything else escaping is a bug.
tests/properties/test_page_properties.py does the same for books:
Reading order. Frames come out tier by tier, left to right, down each column, each inset before or after its parent as
readsays. The expected order is worked out from the authored pages bytests.strategies.scenes, never by asking the solver.Frames. Every frame stays inside the page margins, and no two overlap, compared as the shapes they are drawn as, so a slanted tier’s shared cuts count. The exception is an inset, which lies wholly inside its parent. Each panel is compiled at its frame’s size.
Round trip and determinism. Page Core reads back equal and writes back the same bytes, and compiling twice gives the same pages.
SVG. Pages, a book of pages and a strip all pass the SVG checks, ids included: unique across every page, not only within one.
Comic scripts lose nothing. As many panels as were written, each holding as many balloons and captions as it had lines, whether panel numbers run on or start again on each page.
tests/properties/test_broken_documents.py holds scenet check to its purpose:
One mistake, one finding. A valid generated document with exactly one mistake from the catalogue in
tests.strategies.MISTAKES– a misspelled key, a missing field, a wrong type,.nan, an unknown actor, a cycle, a duplicate key and the rest – gives exactly one finding, under the right rule, at the right path, on a line that holds the mistake, in a SARIF document that serialises. Every entry runs against its own examples. Scenes add a page naming a missing panel, a panel placed twice, two stacks side by side and a missingover:parent; scripts add a repeated PANEL heading.Two mistakes, two findings. Independent mistakes in fields are reported independently.
Never a traceback. Arbitrary documents, arbitrary text and arbitrary scripts get findings, never an exception and never the
internalrule, andscenet checkexits 0 or 1.
A cross-reference check, such as an unknown actor, runs only once every field is valid, so a document with a broken field and an unknown actor reports the field first. That is by design: the reference cannot be checked against a cast that did not validate.
Golden outputs#
tests/test_golden.py checks the promise in the reference that two machines produce files
cmp calls equal, which compiling twice in one process cannot see. It builds the whole
gallery through scenet build, in every CI job – Linux on 3.12, 3.13 and 3.14, and Windows – and
compares the bytes on disk with what is committed:
tests/golden/core/holds every Panel Core and Page Core, so a layout change is reviewed coordinate by coordinate in the diff;tests/golden/digests.jsonholds the SHA-256 of every SVG: panels, overlays, live text, strips and pages; and of every print PDF;docs/images/gallery/holds what the documentation shows of each gallery example: its pages, or else its strip, or else its one panel, written whole. An example lettered over a picture of its own is built once more with--embed-imagesfor it, since an SVG shown as an image loads nothing from outside itself.tests/test_docs_gallery.pyholds each picture to its digest.
It also builds the gallery in two processes with different PYTHONHASHSEEDs, and one scene
and one script from a relative path, an absolute one and another directory; each must
write the same bytes, and none may hold the path it was built from.
After a change that is meant to move the output, regenerate the goldens and read the diff:
uv run python scripts/update_golden.py
--check exits 1 and names every stale golden without writing anything.
The print plates#
tests/test_print_separations.py renders the print PDF (#80) to its plates with
Ghostscript’s tiffsep device, one per ink, and checks the black-plate rule in ink: lettering
on K alone, an overprinting line leaving the colour under it, the paper and reversed captions
knocking out, the night foreground rich black. qpdf --check reads every gallery PDF. Both
are external tools, installed in CI’s Python 3.14 job, where SCENET_REQUIRE_PRINT_TOOLS makes
a missing one fail; elsewhere the tests skip without them. To run them locally, install
Ghostscript (gs, or gswin64c on Windows) and qpdf, then:
SCENET_PLATES_DIR=print-plates uv run pytest tests/test_print_separations.py
SCENET_PLATES_DIR keeps the plates, which CI also uploads as the print-plates artifact.
They are looked at, never kept as goldens: the determinism contract is on the PDF’s bytes.
The reference as a contract#
docs/reference/cli.md is the specification, so tests/test_reference_contract.py
holds the code to it, failing in both directions – documented and missing, or present and
undocumented:
each command’s options, their choices and the values marked as defaults, against
scenet.cli.build_parser();each command’s exit-status table, against a scenario that produces every code in it;
each command’s and each option’s release, against
docs/reference/cli.since.toml, whichscripts/mine_cli_since.pymines from the tags, and each option against abashexample in its command’s section that uses it;the extension table, against the frontends
buildreads with;every rule except
internal, against a document intests/rule_corpus/named after it that must produce exactly one finding of that rule, read besideexamples/galleryas the playground reads a document, and the rule objects SARIF emits against the catalogue;the behavioural sentences – output naming, clashes,
--quiet, whatcheckwrites, SARIF on stdout – one test each.
docs/reference/diagnostics.md is generated from the catalogue and the same corpus by
scripts/build_diagnostics.py, and tests/test_diagnostics_reference.py holds it to them:
an entry for every rule and every compile note, each document giving exactly its diagnostic,
each fixed twin in tests/rule_corpus/fixed/ giving nothing, and the cards and the entries
linking each other.
The scenet lines of every bash block in the documentation run as well, in-process in a
copy of examples/ (tests/shell_examples.py). pip, uv, git and the like never run.
A block that needs a file the reader creates along the way is marked <!--- skip: next -->.
A new rule needs a corpus document; a new option needs a definition in the reference. The tests say which.
Examples a reader can paste#
A reader copies an example into a fresh file and runs it, and it must work. The suite runs every example that way (#317), and fails on each way it could not:
A missing import.
--doctest-modulesruns a docstring’s examples in a copy of its module’s globals, where an example could call the function it documents without importing it. A hook in the rootconftest.pyempties them first: each docstring runs with builtins only, as a pasted script does.A block that leans on an earlier one. Sybil runs every example of a page in one namespace. The root
conftest.pygives each Python block a namespace of its own; the>>>lines of one block still share it, as the lines of one session do.A public name imported from a submodule.
from scenet.ir import Massworks, and teaches an import path the API does not promise. A name inscenet.__all__is imported fromscenet; any other class or function from the module that defines it, not one that only imports it.A document that does not compile. Every
yamlandscenet-scriptblock indocs/andREADME.mdis compiled as the kind of document it shows: a scene when it holdspanels:orpages:, a script when it is ascenet-scriptblock, and a panel otherwise. A comic script in atextor untagged block fails, since it would never be compiled.
A block that is only a part of a document says so where the reader sees it, in a caption:
```{code-block} yaml
:caption: Part of story.scene.yaml
pages:
- tiers: [{panels: [arrival, door]}]
```
A caption of Part of <document> is not compiled, on purpose; a YAML part is still read as
YAML. A caption that is a file name says what the block is when its contents cannot: a
*.puppet.yaml is loaded as a puppet, a *.art.yaml as an artwork contract, and a file
Scenet does not read, such as .github/workflows/comic.yml, is only read as YAML. Any other
caption is an error. The README holds no caption, since GitHub and PyPI render no MyST: every
block of it is complete. A block that is an example file word for word, as each one on the
gallery page is, is compiled by tests/test_gallery.py beside the files it names.
tests/doc_examples.py holds the mechanisms and tests/test_doc_examples.py the tests,
including one per mistake above, on an example written to make it.
Profiles#
A property failure has to replay identically, so the default is derandomized
everywhere, on a laptop as in CI. The profiles are in tests/conftest.py, chosen with
HYPOTHESIS_PROFILE:
Profile |
Examples |
Seeds |
Use |
|---|---|---|---|
|
20 |
fixed |
every run, locally and in CI |
|
20 |
fixed |
the same as |
|
300 |
random, remembered in |
hunting for new failures |
uv run pytest tests/properties --hypothesis-show-statistics
HYPOTHESIS_PROFILE=explore uv run pytest tests/properties --no-cov
--hypothesis-show-statistics prints, for each property, how many examples compiled, how
many failed for lack of room, and how many balloons and captions they held. A generator
whose examples mostly fail, or mostly hold no lettering, tests far less than its example
count suggests, so check those numbers after changing a strategy.
When a property fails#
Hypothesis shrinks the failure to the smallest document it can and prints it, along with
a @reproduce_failure line. Under fixed the same example comes back on every run, so
re-running the test is enough to replay it.
A failure found under explore is different: its seed was random. Pin the document it
printed as an @example(...) on the property, so the fixed run covers it from then on,
and then fix the bug in its own pull request with its own failing test.
Adding a strategy#
Strategies live in tests/strategies.py and return a Drawn, which keeps the authored
document as a dict. Tests derive their expectations from what was authored, such as the
margin, never from what the compiler made of it.
Draw the vocabulary from the code. Enums, puppet poses and expressions, places and font coverage are all read from the package, so a new shot type is generated the day it lands. Never copy a list of values into a strategy.
Build valid documents by construction rather than drawing and filtering. Pydantic v2 has no Hypothesis plugin, and the model validators would reject most of what
st.builds(PanelIR)drew.Keep the failure rate low. A panel that is valid but has no room for its lettering tests only the failure path.
panels()holds the tightest shots to one line or none for that reason.
How sensitive the properties are#
A property that cannot fail is worth nothing, so each was checked against a deliberate break in the code it guards. Each break below made a property fail, and the failure replayed on the next run:
the solver’s reading-order rule disabled, which a caption placed before a balloon caught;
the glyph scale written to two places again, the bug fixed in #99, which the SVG check caught at the first panel with lettering.
What the properties found#
Each was fixed in its own pull request, with a failing test first, before the property that trips on it landed:
Found |
Fixed in |
|---|---|
A control character in a panel wrote an SVG no XML parser accepts |
|
A document nested a few hundred levels deep ended in |
|
An inset in a narrow panel was drawn outside it |
|
|
|
A misspelled required key was reported twice |
|
A broken |
|
A syntax error at the end of a file pointed past its last line; a block value at its first field |
|
An unknown speaker was located one step short of its |
Mutation testing#
Coverage shows which lines ran, not whether any assertion would notice them changing.
mutmut does notice: it makes one small edit at a time to
the solver and the emitters – < becomes <=, a constant is nudged, an argument dropped –
and reports every edit the suite still passes. Each survivor is a behaviour nothing checks.
It runs weekly, and on demand, in .github/workflows/mutation.yml. The only pull request
it runs on is one that changes the workflow itself: a full run takes far longer than a review
should wait. The job summary gives the score and lists the survivors. The mutmut-results
artifact holds the same list, and survivors.diff: every survivor’s change, as mutmut show
prints it, which is what triage starts from. A run on one file is a workflow_dispatch with
path set, on whichever branch holds the tests to try. It is not a required check, and there is no threshold until the baseline
below has been worked down.
Why mutmut. Our imports are slow – numpy, shapely, kiwisolver, fontTools – and the suite takes about 40 seconds. mutmut forks each mutant from a process that has already imported everything, runs only the tests that reach the mutated function, and resumes where it left off, re-testing only functions whose code changed. A tool that ran the whole suite once per mutant would take days.
Running it. It is in its own dependency group, so a plain uv sync does not install it.
Install every group: its clean run is the whole suite, and tests/test_site.py needs docs:
uv sync --all-groups
ulimit -v 2097152
uv run mutmut run --max-children 4
uv run mutmut results
uv run mutmut show <mutant>
The ulimit caps each process at 2 GiB. A few mutants allocate without bound – / turned
into * in the rain or snow density asks for millions of streaks – and, uncapped, one of
them can fill a 16 GB machine before mutmut’s timeout fires. Capped, it ends in a
MemoryError and counts as killed. A test process needs under 1 GB.
[tool.mutmut] in pyproject.toml limits it to src/scenet/solve/ and src/scenet/emit/,
mutates only lines the tests cover, and runs mutants without the coverage floor, doctests or
the documentation examples. To try one file, run the workflow by hand with its path input.
On Windows, mutmut needs fork(), so run it under WSL with a Linux distribution:
wsl --install -d Ubuntu
then, inside WSL, clone the repository and run the commands above. A checkout on the Windows
side, under /mnt/c/, works but is several times slower.
Known blind spot. mutmut 3 does not mutate decorated functions. In solve/ and emit/
those are nine @property accessors and one cached loader, and none of them is ever mutated:
Function |
Decorator |
Lines |
|---|---|---|
|
|
8 |
|
|
3 |
|
|
7 |
|
|
7 |
|
|
5 |
|
|
3 |
|
|
3 |
|
|
3 |
|
|
3 |
|
|
2 |
The undecorated methods of the same classes are mutated as usual. was_pulled_back and
head_top_y are the ones that matter: they are part of the framing geometry, and until
mutmut reaches them the shot-type tests in tests/test_camera.py and the golden Cores are
what guard them.
Baseline#
The first full run with the golden outputs, made locally on four cores in about 57 minutes, produced 4,094 mutants: 3,761 killed (91.9%) and 333 survivors. The three that ran out of memory, described above, are among the killed. By module:
Module |
Survivors |
Triage |
|---|---|---|
|
123 |
done: 16 accepted, below |
|
50 |
done: 14 accepted, below |
|
49 |
done: 23 accepted, below |
|
32 |
done: 7 accepted, below |
|
22 |
done: 9 accepted, below |
|
18 |
done: none accepted |
|
16 |
done: 8 accepted, below |
|
9 |
done: none accepted |
|
7 |
done: 1 accepted, below |
|
7 |
done: 6 accepted, below |
Before the golden outputs, the same run left 845 survivors (79.4% killed), 188 of them in
emit/svg.py: comparing every emitted byte took most of the emitters’ share.
Triage#
Every survivor ends in one of two states:
Killed, by a new assertion in the relevant
tests/test_*.py.Accepted, recorded below with a reason: an equivalent mutant that cannot change any output, or one whose effect is below what the format can show.
# pragma: no mutateis used only where nothing else works, with a comment saying why.
A triaged module’s survivors in the weekly summary should be exactly its accepted list below. Anything else is new, and is triaged the same way.
Accepted survivors#
solve/balloons.py – triaged: of the 123 survivors, 107 are killed by tests in
tests/test_balloons.py, and these 16 are accepted. Mutant numbers are mutmut’s, from
uv run mutmut show <name>, and shift when the function they are in changes.
Mutant |
Change |
Why it cannot change an output |
|---|---|---|
|
|
The mouth is strictly inside the face, so the line through it always crosses the outline twice: the discriminant is never zero. |
|
the second root mutated |
The tail starts outside the face – a balloon may not overlap one – so the first crossing is always the smaller root, in (0, 1). The second is never chosen. |
|
|
|
|
|
Only a tail that starts inside the face has its first crossing past 1, and none does, for the reason above. |
|
|
|
|
|
Used only for a zero-length chord, whose normal is |
|
|
|
|
|
The default applies only when the panel has no actors, and then the loop the reach is for runs zero times. |
|
|
Differs only when ink and paper contrast exactly equally with the fill. Contrast is measured on neutral greys, so there are 256 fills to try, and none ties. |
|
sampling starts at step 0 |
Step 0 adds a zero-length segment at the start; the next segment begins there anyway. |
|
sample count 16 to 17, first sample skipped, 12 to 13 |
Sampling resolution, below what the format can show: the chord between samples is within a fraction of a unit of the curve, and faces and boxes are tens of units across. |
Off-panel speech and small type (#75) added 27 survivors. 16 are killed by TestOffPanel
and TestTheNearerSide in tests/test_balloons.py and by tests/test_off_panel.py, and
these 11 are accepted, numbered as in run 38642945474:
Mutant |
Change |
Why it cannot change an output |
|---|---|---|
|
|
|
|
the line out doubled to tripled |
Any end past the rectangle crosses the outline at the same point. |
|
|
For left and right the line’s height is the centre’s either way. For top and bottom the line collapses to a point, crosses nothing, and falls back to the rectangle’s edge – which is where it would have crossed, because a slant leans only the gutters: a panel’s top and bottom stay level. |
|
the handling of several crossings, or none |
The line runs from the balloon, inside the outline, to beyond it, and a panel’s outline is convex, so it crosses exactly once: a single point, which has |
solve/text.py – triaged twice. The first pass, before kerning, killed 36 of 50
survivors; most were in FontMetrics’s constructor, and survived for a reason worth knowing:
load_metrics is cached, so a test that measures only through it uses the instance built at
import and never runs the constructor under a mutant. The tests build one of their own.
Kerning, an author’s own face and dashes that keep to their word brought it to 81 survivors
in run 39038961474, whose numbers are the ones below. 37 are killed by tests in
tests/test_kerning.py and tests/test_text.py. Kerning is the reason most of those exist:
a font in every shape a reader must handle – Apple’s kern table, a skipped subtable before
a kept one, a latn script beside DFLT, two lookups kerning one pair, a class row shorter
than its classes, a font with no name table – is built in the test, since the shipped
faces are none of them. These 44 are accepted:
Mutant |
Change |
Why it cannot change an output |
|---|---|---|
|
|
Laziness decides when fontTools parses a table, not what it reads from it. |
|
the default for a missing |
Every TrueType font has a |
|
the default for a glyph with no advance changed, or always looked up |
Every glyph the character map names has an advance in |
|
the cache never hit, or written as |
The cache only saves work: a miss computes the same sum again. |
|
|
Every glyph the character map names is in the glyph set, so the second test only ever agrees with the first. |
|
the regular face loaded through its file path |
|
|
|
A |
|
the default |
Followed by |
|
the default for a table with no |
fontTools always gives a |
|
the defaults for a subtable’s |
Every subtable fontTools reads has a |
|
the Apple mask |
Bit 0 of Apple’s coverage flags is reserved, and zero. |
|
the first gap’s face |
A gap is recorded only after whitespace has set its face. |
|
the last word sliced to |
The same slice. |
|
runs that end before they start, or a gap too many |
An empty run measures the one word |
|
the line height taken from the regular face |
Line height is the size times a fixed factor, the same for every face. |
|
|
|
|
|
An exact tie between two different wrappings needs two different blocks to score the same float; equal wrappings tie, and either is the same block. |
|
the fallback line built with no gaps |
The fallback is reached only when every block is 0 wide. With two words or more, the space between them has width, so the fallback only ever sets one word – and one word has no gap to read. |
|
|
|
solve/page.py – triaged: of the 49 survivors, 26 are killed by tests in
tests/test_page_solver.py, which call the solver’s pieces directly at the edges each one
holds – an inset that exactly fits, two insets exactly a gutter apart, a lean that leaves
half a unit – and these 23 are accepted.
Mutant |
Change |
Why it cannot change an output |
|---|---|---|
|
|
It is read only for an inset’s clearance, and every inset passes its gutter. |
|
|
|
|
|
Each zip pairs sequences built from the same list of weights, so they are always the same length, and |
|
|
|
|
|
Differs only for a width of exactly 0, which needs the tangent of the slant times a height to land exactly on a gutter’s edge; a width half a unit either side is tested. |
|
|
Both are top corners, at the same |
|
||
|
||
horizon is half a unit from the top, where a clamp anywhere but the panel’s edge shows. |
||
These 7 are accepted. |
Mutant |
Change |
Why it cannot change an output |
|---|---|---|
|
|
Big-endian is the default since Python 3.11, and the floor is 3.12. |
|
|
The same codec; Python’s codec names are case-insensitive. |
|
|
No 8-bit channel is exactly 0.04045, and the sRGB curve is continuous there anyway. |
|
|
The sky under rain cloud is 0.69, 0.83, 0.55 and 0.37 at dawn, day, dusk and night: never exactly 0.5. |
|
||
|
||
sum and a difference from it agree, so the tests also frame a puppet measured from lower |
||
down, and one drawn a thousand times smaller. This one is accepted. |
Mutant |
Change |
Why it cannot change an output |
|---|---|---|
|
|
At an equal scale it builds a new solution equal in every field – pullback 1.0 – and |
|
||
|
||
times smaller: a puppet’s units are its own, so the composition must come out the same, |
||
even with the whole cast less than one unit wide. These 8 are accepted. A ninth, |
||
|
||
order ( |
Mutant |
Change |
Why it cannot change an output |
|---|---|---|
|
widths measured with |
A mirrored silhouette is exactly as wide. |
|
widths measured from an origin of |
Moving a silhouette does not change its width. |
|
|
Every part has a positive width or radius, so a cast’s total width is never 0. |
|
|
Only equal for an actor looking at itself, which the IR refuses as |
|
extents measured with the root at |
An extent is measured from the root wherever it stands; |
|
the expression left out when measuring extents |
An expression moves the features inside the head, which the silhouette does not include. |
|
||
|
||
mutants and left 2, |
||
shipped expression has. |
||
kills both. None accepted. |
The emitters – triaged together: of their 50 survivors, 36 are killed by
tests/test_emit_edges.py. The goldens pin every byte the gallery produces, so what
survived is what the gallery never produces: several pages side by side, a curved tail, a
gaze that is not level, a limb thinner than its outline. A Core is a format people
hand-edit, so the tests edit a compiled one to hold them.
Accessible output (#76) changed svg.py, strip.py, page.py and debug_svg.py: render
and render_debug became wrappers round panel_body and debug_body, which placed_panel
places instead of cutting text, and every renderer took aria. A fresh run of emit/*,
39124932431, tried 1438
mutants and left 15. Seven are killed in tests/test_emit_edges.py,
tests/test_accessible.py and tests/test_lettering_face.py, each applied by hand to see
its test fail. These 8 are accepted, numbered as in that run – six of them the same
mutants as before, one renumbered, one new.
Mutant |
Change |
Why it cannot change an output |
|---|---|---|
|
|
|
|
|
A number formatted to fixed places always keeps the digit before its point, so stripping never leaves |
|
the id for a missing veil changed |
Both callers return before asking when there is no veil. |
|
|
A Core compiled with those |
|
|
Used only for a zero-length chord, whose normal is |
Supplied pictures (#70) changed solve/backdrop.py, solve/describe.py and every emitter.
Fresh runs from the stack’s emitter branch:
solve/describe.py,39154486553: 190 mutants, all killed.solve/backdrop.py,39154483015: 1001 mutants and 8 survivors – the 7 accepted above, under the same numbers, andsolve_backdrop6, a changed message, now killed intests/test_image_backdrop.py.emit/*,39154489426: 1520 mutants and 13 survivors – the 8 accepted above,panel_body39 now numbered 49;image_href11 andrender_pages76, killed intests/test_emit_image.py; and these 3, accepted, numbered as in that run.
Each mutant killed was applied by hand to see its test fail.
Mutant |
Change |
Why it cannot change an output |
|---|---|---|
|
|
|
|
|
|
|
|
The same codec; Python’s codec names are case-insensitive. |
Figures a picture draws (#71) changed solve/balloons.py and solve/sfx.py – the speaker
interface, bystanders, keep-outs, a tail to a cropped mouth – added solve/speakers.py and
solve/figures.py, and drew figures in the debug overlay. Fresh runs from the stack’s top
branch. The first batch never tried a mutant: two @given tests written as methods failed
Hypothesis’s differing_executors health check in mutmut’s clean run, which runs the suite twice
in one process; every property test is a module function now, as the rest of the suite’s are.
solve/speakers.py,39483572899: 26 mutants, all killed.solve/figures.py,39483587638: 52 mutants, all killed.solve/describe.py,39483600321: 195 mutants, all killed.emit/*,39483612696: 1589 mutants and 11 survivors, exactly the 11 accepted above under the same numbers. The figures in the debug overlay left none.solve/balloons.py,39483547250: 1335 mutants and 51 survivors.27 are the 27 accepted above. Two are renumbered:
_score_caption55 is now 59, and_place_off_panel87 is now 89.Killed in
tests/test_balloons.py(TestTriageForFigures,TestASoundInAPanelOfFigures):bystanders’ bodies dropped from
_score(23) and_score_caption(21, 22), or overwritten rather than summed in_occlusion_cost(33);bystanders not handed to an off-panel balloon (
_place_off_panel54,place_script163) or to the sound effects (place_script78);the sound effects’
inside,marks– which hold the keep-outs – andblockeddropped (place_script67, 75, 76, 77);the bottom-right corner candidate (
_candidate_positions62-65) and the fallback tail’s start (route_tail83). These two had been killed only by golden panels whose balloons sat there before #71’s speaker fix moved them.
Removed:
_within_frame15, 30, 36, 37, 40 and 43 checked whether a mouth was inside the frame first, and walked shapely’s result by hand. A convex frame is crossed once by a line to a mouth beyond it and never by one to a mouth inside, so the function now only intersects, and reads the crossing withshapely.get_coordinates. Every golden is unchanged.
solve/sfx.py,39483560230: 489 mutants and 37 survivors.solve/sfx.pyarrived after the baseline above and had not been triaged. #71 changed_costandplace_effects, and their 21 survivors are triaged here:18 killed in
tests/test_sfx.py:TestTheCostOfAPositionpins each term of_costexactly, on a footprint smaller than a mutant’s area floor;TestWhereAnEffectMayGoholdsplace_effectsto an inset elsewhere, a sliver of one, and a shaped panel.3 accepted, below.
The other 16, in
layout_effect(9),_ring(6) and_candidates(1), are in code #71 did not change, and were triaged after it.TestTheShapeOfAnEffectandTestWhereAnEffectIsLookedForintests/test_sfx.pykill 9: the last pair of letters kerned as every other is (layout_effect10), an arc of one degree bowing the same way as one of thirty (41), the rings round a source at its reach times each factor (_candidates16), and every ring starting straight up and going clockwise (_ring5, 8, 10, 16, 21). The other 7 are accepted, below.solve/sfx.pyis triaged: run39497617662, from that branch, left 11 survivors of 489 mutants, exactly the 11 accepted below.
Both files were run again from the triage branch, after the tests and the change to
_within_frame: solve/balloons.py,
39491758453, 1317 mutants and 29
survivors – the 27 accepted above and the two below; solve/sfx.py,
39491774024, 489 mutants and 19
survivors – the three below and the 16 still to triage. The numbers below are those runs’.
Each mutant killed was applied by hand to see its test fail.
Mutant |
Change |
Why it cannot change an output |
|---|---|---|
|
|
A convex frame is crossed at most once, so there is never a second point to choose between: the same argument as |
|
the source term |
It is the first term, added to the |
|
|
Differs only on an exact tie, which a later candidate would then win. Candidates are placed by trigonometry, so two that look equally good differ in their last bits – and which way depends on the platform: a test pinning one, with an inset over the middle and its two mirror points left, passed on Windows and failed on Linux. No test can hold a tie that the arithmetic never makes exact. |
|
|
The bow is read only when the arc is not zero. |
|
|
The steps are built one per character of the same text, so the two are always the same length. |
|
every glyph’s height shifted by 1, or by the arc’s radius |
The shift is the same for every glyph and corner, and the footprint is re-centred on its hull afterwards, which takes it back out: below what the format can show. |
|
the rotation about the hull’s centre instead of |
The hull has just been centred on |
|
|
Both are |
|
|
Differs only for a footprint exactly tangent to a face. Footprints are glyph outlines placed at candidate centres a ring step apart, which never land at a face’s radius exactly; a balloon’s own face test is strict for the same reason. |
Right-to-left reading (#68) changed solve/balloons.py, solve/page.py, solve/describe.py,
emit/strip.py, emit/page.py and emit/debug_svg.py. Fresh runs from the stack’s top branch:
solve/balloons.py,39836238615: 1356 mutants and 40 survivors.29 are the 29 accepted above. One more is renumbered:
_place_off_panel89 is now 96.11 killed in
tests/test_direction.py:which way caption zones are tried (
_caption_positions41). The mirror test alone could not tell the two directions apart, since flipping both still mirrors, soTestTheRuleItselfpins the first zone to where reading starts;the tolerance of the right-to-left rule (
_reading_order_allows4, 5), mirrored from the left-to-right tests;the direction dropped from the zones, the order and the nearer side of an off-panel balloon (
_place_off_panel27, 30, 34, 37, 70, 73) and from a caption’s zones (_place_caption24, 27). Most of these change a layout only on an exact tie between mirrored candidates, which falls differently by platform and so is never pinned.TestEveryReadingDecisionIsAskedTheWayThePanelReadswatches every reading decision in a panel read right to left, and requires each to be asked right to left.
solve/page.py,39836251923: 542 mutants and 26 survivors, all accepted above:_frame1,_fmt5 and 6, and_slanted30, 33, 35, 41, 87, 92, 142 and 157, under the same numbers;the
zip(..., strict=True)mutants, renumbered toresolve_frames28, 32, 36, 90, 93, 98, 137, 140, 145, 159, 163 and 167;three new ones of the same kind in
_across(5, 8, 12), which pairs the starts and sizes of one_split.
solve/describe.py,39836265688: 203 mutants and 2 survivors, accepted below.emit/*,39836278068: 1643 mutants and 20 survivors.11 are the 11 accepted above, under the same numbers.
9 killed in
tests/test_reading_pages.py(TestTwoDirectionsAreRefusedByName): the refusal of panels or pages read two ways, word for word (reading_direction8, 9;render_strip7, 10, 11;render_pages7, 10, 11), and the default for nothing to lay out (reading_direction11).
Each mutant killed was applied by hand to see its test fail.
Mutant |
Change |
Why it cannot change an output |
|---|---|---|
|
the sort key scaled by -2 instead of -1, or 2 instead of 1 |
Scaling every face’s x by one factor of the same sign keeps their order, and keeps a tie a tie for the id to break. |
The second job of each of those runs, exploring with fresh seeds, found the right-to-left page property compiling the reversed scene in full to compare its frames. That scene’s lettering is read the other way and can find no room where the original did. The property now lays the reversed pages out without compiling a panel, since frames cannot depend on lettering, and passes 300 examples.
The exports for image models (#72) added emit/conditioning.py, emit/pose.py,
emit/boxes.py, emit/semantics.py, emit/depth.py, emit/mask.py and emit/export.py, moved
the painter’s order of emit/svg.py into in_painters_order (#221: masses past the tenth were
drawn out of order), and took traits out of solve/describe.py.
A first triage ran before the stack’s review: emit/*
40008532976, 2177 mutants and 17
survivors, which killed emit/depth.py _plane 2 and 3 (a figure’s plane merged with an
actor’s, by test_a_hand_edited_core_with_figures_and_actors_puts_the_figures_in_front) and
emit/conditioning.py _fraction 1 (+ 0.0 to - 0.0, by
test_a_negative_zero_is_written_as_zero). The review then changed the box clipping, the maps’
canvas and edges, the mask colours, and the semantics, so emit/* was run again from the top of
the stack. A cached run of it (40031036460) reported 46 mutants and was discarded; the branch’s
mutants/ caches were deleted and it ran again from nothing:
emit/*,40032456101: 2263 mutants and 27 survivors.10 are the
emit/svg.pysurvivors accepted above, under the same numbers.panel_body39, the painter’s sort keyed on the whole tuple, is gone with the sort it mutated, and its row with it:in_painters_ordersorts on depth alone.2 killed:
emit/mask.pycolours9, a character stepping aside tested atslot - stepbut placed atslot + step. The two agree unless a third character holds the slot ahead, sotest_a_character_stepping_aside_never_lands_on_a_colour_already_takenbuilds that case: under the mutant two characters share a colour;emit/conditioning.pyfraction_box21,area <= 0.0to<= 1.0, bytest_a_box_smaller_than_a_unit_is_still_a_box.
15 accepted below.
solve/describe.py,40008537043: 210 mutants and 2 survivors,describe8 and 10, accepted above under the same numbers. The review changed nothing in it.
Each mutant killed was applied by hand to see its test fail.
Mutant |
Change |
Why it cannot change an output |
|---|---|---|
|
|
|
|
|
|
|
|
|
|
|
The same codec; Python’s codec names are case-insensitive. |
|
|
|
|
|
|
|
|
|
The second job of every run, exploring with fresh seeds, passed every property, the new ones for the exports included, at 300 examples.
The depth export’s receding surfaces (#227) added recedes and a ramped render_depth to
emit/depth.py, and taught flat_map in emit/conditioning.py a Ramp. Both modules were run
from the pull request’s branch, each from nothing (the totals are every mutant of the module):
emit/depth.py,40952714050: 81 mutants and 1 survivor, killed:recedes25, a ceiling’s run checked against1.0instead of the top edge, so a horizon less than a unit below the top left it flat.test_a_ceiling_with_a_horizon_just_below_the_top_still_recedessets one at0.5; the mutant, applied by hand, fails it.emit/conditioning.py,40952725859: 188 mutants and 1 survivor,to_json5, accepted above under the same number.
The colour policy (#244) changed solve/backdrop.py (the tinted ladder, colour_for, the
twins in _atmosphere, and _luminance refusing a colour), solve/balloons.py (the warm
caption’s colour), and in emit/, svg.py, page.py, strip.py and semantics.py (drawing the
colour, mono=, and the semantics’ palette). Fresh runs from the stack’s top branch, each from
nothing (the totals are every mutant of the path):
solve/backdrop.py,41293701148: 1068 mutants and 12 survivors.7 are the 7 accepted above:
seed_for7 and 14 and_rng6 and 13 under the same numbers, and three renumbered:_luminance15 is now 24,_atmosphere22 is now 38, and_atmosphere63 is now 55.1 killed:
_atmosphere106, the snowy sky’s colour left out. Rain’s was pinned and snow’s was not, sotest_the_air_it_falls_through_is_tintedchecks the sky and the cloud of a snowy panel at every hour.3 removed:
_atmosphere107, 112 and 114 mutatedfall_colour=colour_for(SNOW_TONE, ...), which is always"", since white snow is not a rung of the ladder. The line is gone and snow’sfall_colourtakes its default, which says the same thing.test_snow_stays_whitestill holds it.1 accepted below.
solve/balloons.py,41293703313: 1371 mutants and 29 survivors, the 29 accepted above under the same numbers. None is in the caption’s colour.emit/*,41293705447: 2349 mutants and 28 survivors.25 are the 25 accepted above, under the same numbers: the 10 in
emit/svg.pyand the 15 in the exports.3 killed:
render_page4,render_pages4 andrender_strip4,monoon by default. The gallery test always passesmono=Trueand the command always passesmonoitself, so nothing called them without it.test_without_it_a_page_and_a_strip_draw_the_colour_toodoes.
Each mutant killed was applied by hand to see its test fail. core.py, where as_mono lives, is
outside what the workflow mutates; as_mono is held by the gallery test, which compares it with
the mono Core on every example.
Mutant |
Change |
Why it cannot change an output |
|---|---|---|
|
|
The channels are only compared with each other and read as hexadecimal, and both are case-insensitive. |
The print PDF (#80) added four modules to emit/: geometry.py and marks.py, which the
SVG now draws from, and pdf.py and pdf_images.py. A run of all of emit/,
41777653921, reached the
job’s time limit with 748 of 4767 mutants untried, all in geometry.py and marks.py, so each
was run on its own, and pdf*.py again once the new tests were in:
emit/geometry.py,41791026492: 199 mutants and 1 survivor,pointer20: theor 1.0accepted above as_tail_shape47.emit/marks.py,41795875604: 590 mutants and 22 survivors. 14 were aroundedor awidthset on a mark with no outline, where caps, joins and width mean nothing; the parameters are gone, and with them the mutants.test_the_marks_are_what_the_svg_paints_in_ordernow compares every mark’s fill, stroke, width, caps, opacity and dash with the SVG’s, which kills the colour mutants of the first run, and the paper caption and the snow have tests of their own.emit/pdf*.py,41792992306: 1603 mutants and 52 survivors. 10 are killed since – a JPEG’s single fill byte, and the page boxes’ fourth decimal place – each applied by hand to see its test fail.
Accepted, by what makes them equivalent:
Mutants |
Change |
Why it cannot change an output |
|---|---|---|
|
|
The three sequences zipped come from one run’s text, so they are always as long. |
|
|
The owner names a mark for tests; the PDF writer never reads it. |
|
the default opacity changed |
Every call passes a fill or a stroke, whose opacity replaces it. |
|
|
Both are read as false, as |
|
|
Both sides of the knee give the same value there, and no |
|
13 digits of the digest in the message |
A message, which says the same thing. |
|
|
Read only for truth, and |
|
a loop’s bound or a slice one byte wider |
A frame, a chunk header and an IHDR body are longer than the slack, so each still ends where the format says. |
|
unsigned fields read as signed |
No chunk is 2 GB, and no width, depth or type in a header reaches a sign bit. |
|
bit-depth tests moved by one, or a default that is never read |
At depth 8 the scale is 1 and the sub-byte path packs one sample a byte; alpha is only written when there is a mask; a palette with no |
|
|
Python’s default byte order is big-endian; depth 8 takes the one-sample-a-byte path either way; samples past a row’s count are never read. |
|
a row one byte longer |
The extra byte is past every index a filter reads. |
|
|
They tie only when they are equal, so which is returned cannot be told apart. |
|
the grid’s fill, or a pass’s empty test changed |
Every cell is written by some pass, and a pass that starts past the edge counts 0 either way. |
The print follow-ups (#280–#284) added emit/pdf_fonts.py and emit/turbulence.py, changed
emit/pdf.py and emit/geometry.py, and made solve/page.py bleed. Each file was run on its own
from the stack’s top (#310):
File |
Run |
Mutants |
Survivors |
Killed since |
Accepted |
|---|---|---|---|---|---|
|
1380 |
52 |
27 |
25: 13 above, 12 below |
|
|
306 |
51 |
35 |
16, below |
|
|
143 |
5 |
3 |
2, below |
|
|
715 |
52 |
17 |
35: 26 above, 9 below |
|
|
237 |
5 |
1 |
4: 1 above, 3 below |
Each mutant killed was applied by hand to see its test fail. Two lessons from it:
A cache outlives a mutant. mutmut forks each mutant from a process that has already run the suite, so a function behind an
lru_cache–lattice,_pdf_font– answers from what the unmutated code put there.park_miller7, 9 and 11 survived that way, andtest_the_lattice_is_the_reference_codescould not see them. They are killed by tests ofpark_milleritself: every number stays between 1 and M - 1, and the step that gives 1 gives 1. Schrage’s step is right modulo M whichever way it corrects, so the 10,000th number alone cannot tell.A run’s test selection can miss a test that kills.
render_pdf54, which prints every panel’s pictures rather than the printed ones, failstest_a_picture_on_no_page_is_not_carriedwhen applied by hand.
Killed since, by what the tests now hold:
Faces unlike the gallery’s, which are all 1000 units per em, upright, proportional and fully named. A 2048-unit face scales its box, ascent, descent, cap height and widths to thousandths (
_descriptor66, 81, 83, 85;_widths15, 20). A kern in it is written to the millionth (_shown29, 45). Other cases:italic by its angle or its selection, and fixed pitch (
_descriptor25, 30, 32, 33, 35, 36, 40, 78, 79);a cap height stated at
OS/2version 2, or of 0 or 1 (_descriptor52-55);no
OS/2, which takes a regular stem (_descriptor61);names that need cutting, missing, or absent (
_postscript_name1, 2, 13, 16, 26-30);a CFF font’s CIDFont, which maps no glyphs (
objects31);a face that may not be subset, which goes in byte for byte, or, from a WOFF, unwrapped with its date (
_sfnt2, 6, 7, 11, 14).
Faces print may not embed, in two files and two runs of a line: each glyph outlined exactly as outline mode outlines it, in every run, and each face said (
_outlined_run9, 22, 24, 25, 27-30;_text5, 11;_faces_lettered13).Hand-edited colours in the rain, the veil and a caption are said, and air with no veil says nothing (
_colours_not_owned4, 6, 8-13).The ends of the greys:
#050505on CIELAB’s straight part, a grey lighter than black’s first step, and one between its last two (_linear4,_star9,black_for12, 16).A picture with alpha: its soft mask is an object, and the profile’s number counts it (
_shared8).Where a bleed mistake is said – a column, a slanted tier, a panel that reaches no edge – and a slanted outline whose every corner moves only on the sides that bleed (
resolve_frames75, 76, 199-203;_bled28, 31, 34, 35, 86, 87, 89, 92, 96, 99).A border round three sides is one line, wherever the open side is listed (
border17).
Accepted above, under new numbers: in pdf.py, _linear 5, ink 22 (was 7), _embedded 6,
_page_content 3, 9, 10, 29, 35 and 36, _state 2 and 3, _path 23 and _Document.write 131
(was 126); in solve/page.py, _frame 1, _fmt 5 and 6, _slanted 30, 33, 35, 41, 87, 92, 142
and 157, the zip(..., strict=True) mutants now resolve_frames 30, 34, 38, 96, 99, 104, 146,
149, 154, 168, 172 and 176, and _across 5, 8 and 12; in geometry.py, pointer 20. Accepted
here:
Mutants |
Change |
Why it cannot change an output |
|---|---|---|
|
|
Both sides give the same value there: the two parts of CIELAB’s curve meet at the knee, and interpolating at a step gives that step’s percent from either segment. |
|
the CMYK table read for a grey too |
No key of the table is a grey, so a grey finds no twin either way. |
|
a type size written to 2 places, not 6 |
Core rounds every type size to 2 places ( |
|
|
A run has one origin per character. |
|
a glyph’s metrics or character |
|
|
|
Read only for truth. |
|
|
Read only for truth; lazy loading only decides when a table is decompiled. |
|
|
|
|
boxes recalculated when a WOFF is unwrapped |
Read lazily, |
|
|
fontTools reads the flavour for truth, so an empty one writes a plain font file. |
|
the italic angle when there is no |
|
|
|
Python’s codec names are not case-sensitive. |
|
|
A result is 0 only for a seed that is a multiple of M, and no seed between 1 and M - 1 is. |
|
|
A 0-d zero, which broadcasts to the same sums; the alpha is cast to bytes either way. |
|
|
An edge on the margin is within float noise of it; one that is not is a gutter or more away, never exactly a hundredth. |
|
every corner not on the top moved to the bottom line |
A slanted tier’s corners all lie on its top or its bottom edge. |
|
|
A panel whose border is drawn open has a side open and a side kept, so a start is found before the fifth. |
|
the current line |
|
Exploring with fresh seeds#
The same workflow’s second job runs the property tests under the explore profile: random
seeds, 300 examples each. A failure prints the shrunk document and a @reproduce_failure
line. Pin the document as an @example(...) on the property, so the fixed run covers it from
then on, and fix the bug in its own pull request.