Building a continuously updated manual
Source:vignettes/building-a-continuously-updated-manual.Rmd
building-a-continuously-updated-manual.RmdA manual either pins the modules it documents or tracks them, and that choice determines the rest of its setup.
A project manual documents the modules as a
particular project used them, so it pins them and the manual agrees
with the results it describes. See
vignette("building-a-project-manual"); LandWeb is the
example.
A continuously updated manual documents the modules as they are now, so it tracks them and a documentation fix reaches the published site without anyone republishing the manual. This vignette covers that arrangement, as used by LandR-Manual and fireSenseManual.
A git submodule records a commit, so keeping it current means someone bumping a pointer for every module change. Checking the modules out by branch in CI records the branch instead, and each build resolves it to whatever that branch points at that day.
Setting one up means a repository holding the book, a workflow that fetches the modules and deploys the rendered site, a build script that assembles the chapters and bibliography, and – if the manual has releases – an archive of released PDFs.
The shape
The manual is its own repository, and the module sources are build output:
<manual>/
index.Rmd
_bookdown.yml
_output.yml
build.R
install.R
DESCRIPTION # so install.R can resolve the manual's own deps
modules/ # fetched by CI; gitignored
_manual_rmds/ # generated chapters; gitignored
docs/ # rendered output, deployed to the published branch (gh-pages)
.github/workflows/
Fetching the modules
Check each module out by branch, with the standard action. The workflow is then the manifest: every entry names a repository and a ref, and there is no custom code to maintain.
env:
## one edit moves every module; workflow_dispatch can override it for a trial
MODULE_REF: development
jobs:
build-manual-site:
steps:
- uses: actions/checkout@v7
- uses: actions/checkout@v7
with:
repository: PredictiveEcology/Biomass_core
ref: ${{ github.event.inputs.module_ref || env.MODULE_REF }}
path: modules/Biomass_coreUse github.event.inputs, not the inputs
context, so the expression is null rather than an error on
push and schedule events.
A module whose branches differ from the rest can override the default
in its own step – but prefer fixing the module.
Biomass_validationKNN was checked out at main
for a year because work there had resumed directly on main,
leaving its development 12 commits behind; the exception
then had to be remembered every time MODULE_REF was
considered. Merging main back into development
let the exception go, and MODULE_REF covers every module
again.
Nothing detects the situation that made the exception necessary: a
module whose development has quietly fallen behind its
main builds perfectly, from stale sources.
Because nothing in the repository records which commits a build used, have the workflow write them out:
Installing what the chapters need
The manual’s own dependencies come from its DESCRIPTION.
The modules declare their own, which cannot be resolved until the
modules are on disk:
remotes::install_deps()
SpaDES.docs::installModulePkgs("modules")The build script
paths <- SpaDES.docs::manualPaths()
SpaDES.docs::manualVersion() # index.Rmd's subtitle reads it from the environment
SpaDES.docs::writePkgBib(file.path(paths$citations, "packages.bib"))
SpaDES.docs::downloadCSL("ecology-letters", paths$citations)
## references.bib is OUTPUT ONLY -- see the note below
SpaDES.docs::collapseModuleBibs(
modulePath = file.path(paths$prj, "modules"),
extraBibs = file.path(paths$citations, c("packages.bib", "references_manual.bib")),
outFile = file.path(paths$citations, "references.bib")
)
chapters <- SpaDES.docs::prepManualRmds("./modules", rebuildCache = FALSE)
SpaDES.docs::renderManual(outputDir = paths$docs, output_format = "all")
## these must be inside the published directory, not at the repository root
SpaDES.docs::stagePagesFiles(paths$docs, cname = "example.predictiveecology.org")
unlink(unique(dirname(chapters)), recursive = TRUE)Use renderManual() rather than
bookdown::render_book() directly. bookdown knits once and
then writes each declared format in turn, so the website is already on
disk when a later format fails, and letting that failure propagate
discards it along with the deploy that would have published it.
renderManual() reports the failure as a GitHub Actions
warning and still requires the site itself, so a format that cannot be
built costs only that format.
Declare the fragile formats last in
_output.yml, for the same reason: the formats are written
in declaration order and a failure stops the rest, so a PDF in the
middle costs the EPUB after it too.
Keep the output bibliography out of its own extraBibs:
listing it there merges the file into itself on every build,
accumulating RefManageR’s :1-suffixed duplicates of every
entry. Put curated entries in a tracked file of their own and treat the
assembled one as disposable, which it has to be –
citations/ is generated and starts empty on CI.
Deploying
The deploy publishes the contents of the rendered directory
to a branch, so anything written at the repository root never reaches
the site. .nojekyll and CNAME both have to be
inside it, which is what stagePagesFiles() is for.
Writing them at the root appears to work for as long as an earlier
deploy left copies behind and the action deploys with
clean: false. A branch published for the first time has
neither, and the failure is quiet: a green build, and a site that either
loses its custom domain or drops the _-prefixed directories
bookdown emits.
Caching
Cache the directories bookdown actually writes. With
output_dir set it writes knitr’s cache to
<book_filename>_cache/ and the figures to
<book_filename>_files/, both at the repository root.
There is no _bookdown_files directory, and caching a path
of that name fails silently – the action warns that the path does not
exist and saves nothing, on every build.
- uses: actions/cache@v6
with:
path: |
*_cache
*_files
key: bookdown-${{ hashFiles('**/*Rmd') }}
restore-keys: bookdown-Cache the R library as well. Installing the SpaDES stack from source dominates the job, and a daily schedule would otherwise pay it every day.
Rebuild cadence
schedule, plus workflow_dispatch, plus push
to the manual’s own default branch. A daily rebuild keeps the site
within a day of the modules without anyone touching this repository.
Per-push rebuilding, where each module repository notifies the manual, needs a token with write access distributed to every module repository, and it rebuilds a whole book because one typo was fixed. The cost is not worth the freshness.
Releases, and keeping the old PDFs
A manual that tracks a branch has no version of its own: it describes whatever the modules say today. A release is the other build, with pinned refs, a version number, and a PDF that stays available after the site has moved on.
Keep the released PDFs in version control, under
archive/pdf/, and publish them with the site rather than
leaving them on the published branch. That branch is rebuilt by every
deploy, and an old PDF cannot be regenerated from current sources – so
it is the one artifact that must not live only there.
paths <- SpaDES.docs::manualPaths()
## after render_book(), and only when a PDF was produced
SpaDES.docs::archiveManualPDF(
file.path(paths$docs, "MyManual.pdf"),
version = read.dcf("DESCRIPTION", fields = "Version")[1],
prefix = "my-manual"
)
## publish every archived release with the site, and index them
SpaDES.docs::publishManualArchive(
archiveDir = file.path(paths$prj, "archive", "pdf"),
docsDir = paths$docs,
manualName = "My Manual"
)Take the version from DESCRIPTION rather than setting it
in the build script. Two places to edit is one place to forget, and
archiveManualPDF() refuses an empty version rather than
writing my-manual-v.pdf over the previous release.
Link the archive from the book’s preface with a
static link to archive/. The page behind
it is regenerated every build from the files actually present, so
nothing has to be edited when a release is cut.
Committing the new PDF
Have the workflow do it, so a release does not depend on someone remembering:
## after the deploy, never before it -- see below
- name: Archive the release PDF
if: github.event_name != 'pull_request'
env:
GH_TOKEN: ${{ secrets.GITHUB_TOKEN }}
run: |
set -euo pipefail
## '??' is untracked: a version not in the repository before. A
## modified tracked file is a re-render of a version already archived
if [ -z "$(git status --porcelain archive/pdf | grep '^??' || true)" ]; then
echo "No newly archived PDF; nothing to commit."
git checkout -- archive/pdf 2>/dev/null || true
exit 0
fi
git checkout -- archive/pdf 2>/dev/null || true
version="$(Rscript -e 'cat(read.dcf("DESCRIPTION", fields = "Version")[1])')"
branch="archive/pdf-v${version}"
if gh pr list --head "${branch}" --state open --json number --jq 'length' \
| grep -qv '^0$'; then
echo "A pull request for ${branch} is already open."
exit 0
fi
git config --local user.name "github-actions[bot]"
git config --local user.email "41898282+github-actions[bot]@users.noreply.github.com"
git checkout -b "${branch}"
git add archive/pdf
git commit -m "Archive the manual PDF for v${version}"
git push --force \
"https://x-access-token:${GH_TOKEN}@github.com/${GITHUB_REPOSITORY}.git" \
"HEAD:refs/heads/${branch}"
gh pr create --base "${GITHUB_REF_NAME}" --head "${branch}" \
--title "Archive the manual PDF for v${version}"Three parts of that need to stay as they are.
Open a pull request; do not push. A ruleset on the
default branch refuses a direct push, with
GH006: Changes must be made through a pull request. Note
also that the step is gated
if: github.event_name != 'pull_request', so no pull request
build runs it: whatever it does is first exercised by a merge, and
should be read carefully rather than trusted to a green check.
Test for an untracked file, not for any change. A
PDF is not reproducible byte for byte – xelatex stamps a creation date
and a document ID into each run – so
git status --porcelain archive/pdf reports a modification
after every build, and a check for any change opens a pull request every
night. Requires SpaDES.docs >= 0.5.1, which leaves an
already-archived version alone; this check covers the case where
something else rewrites the file.
Put it after the deploy. A failure here must not stop the site from updating, which is what happens when it runs first: nothing handling a secondary artifact should be able to withhold the primary one.
Linking the versions
bs4_book has no download menu:
download: ["pdf", "epub"] is a gitbook option,
accepted and ignored under bookdown::bs4_book, so setting
it publishes the files with nothing pointing at them. Link them from the
landing page:
## in index.Rmd, in a `results = "asis"` chunk
cat(SpaDES.docs::manualArchiveLinks(manualName = "My Manual"), sep = "")manualArchiveLinks() lists the archived releases newest
first, from the files present, excluding the version being built. For
the current build’s own PDF and EPUB, take the formats from
_output.yml rather than testing for the files:
bs4_book renders first, so neither file exists yet when
index.Rmd is knitted, and a test for them drops the links
from every build.
Do not use clean-exclude for this
Keeping the archive only on the published branch, preserved with the
deploy action’s clean-exclude, looks simpler and is worse.
It reintroduces files that linger on that branch without any build
producing them – which is how a stale PDF from a previous era ends up
published beside a current one. Deploying with clean: true,
from a directory the build fully populates, keeps the branch an exact
image of what was built.
Adding a module
Three things change together, and two of them fail quietly:
- a checkout step in the workflow;
- a chapter entry in
_bookdown.yml; - the module list in the manual’s landing page, if it has one.
Miss the second and the chapter is fetched, staged and silently
absent from the book. prepManualRmds() warns in both
directions for this reason. Miss the third and it is in the book but not
on the landing page, which nothing checks.