Executes all pending steps in order. On repeated runs, steps that are already
"done" are skipped and only steps that are still "new" or were marked
"outdated" (because one of their dependencies changed) are executed. Use
force = TRUE to re-execute every step.
Arguments
- x
A pipeflow pip or view
- lgr
A logging function with the arguments
level("info","warn"or"error") andmsg, which is called aslgr(level = level, msg = msg). The default is the optionpipeflow.lgrif set, otherwisepipeflow_lgr(), which can also be used to build custom loggers. To suppress logging, you can setlgr = NULL, or for all runsoptions(pipeflow.lgr = function(level, msg) NULL). Setting the option toNULLremoves it, so thatpipeflow_lgr()is used again.- force
Logical indicating if all steps should be forced to run, regardless of whether they are outdated or not.
- progress
Optional callback of the form
function(value, detail)called before each step, with the position of the step in the run and its name. If it returnsFALSE, the run is halted before the step, as by.self$halt()(see section 'Runtime control flow via restart and halt' below).- on_error
What to do when a step fails:
"stop"(the default) aborts the run,"continue"goes on with all steps that do not depend on a failed step (see section 'Continuing after errors' below).- map
Optional function like
lapply()that runs the calls of a step mapped over partitions (exec = "auto", seepip_add()), for exampleparallel::mclapplyorfuture.apply::future_lapply. It is called asmap(X, FUN)and must return a list of the results ofFUNfor the elements ofX. The defaultNULLcalls them one after another (see section 'Mapping in parallel' below).
Details
Step states
A step can take the following states:
new: the step has been added but not yet executed, or was reset via
pip_reset().outdated: a step can get outdated in the following scenarios:
one of the step's parameters were changed (e.g. via
pip_set_params())the step's execution mode was changed (
p[[step, "exec"]] <- mode)a step it depends on was re-executed, replaced or reset, got a new execution mode, or its output was set (
p[[step, "out"]] <- value)a step it depends on was executed, but the run did not reach the step itself, either on purpose (see section 'Running views' below) or because the run was aborted early due to a failed step
done: the step was executed successfully with its current inputs
failed: the step raised an error during its last execution
Only steps that have run before can become outdated: a step in state "new" stays
"new" until it is executed. A "done" step is skipped unless force = TRUE was set.
For all other states, the step will be re-executed in the next pip_run(). A step
whose input is still "new" (and not locked), for example a step that an earlier
step added via .self during the run, is skipped and keeps its state, as the input
has no output yet.
Time stamps
The step table has two time stamps and a duration per step:
time_start: the start of the step's last run. It is set each time the step is run, also if the run fails. It isNAfor a step that has not run since it was added, replaced (seepip_replace()) or reset (seepip_reset()). A skipped step (because it is"done", locked or has a failed or new input) keeps its value.duration: how long the step's last run took (adifftimein seconds), also if the run failed. It isNAand kept in the same cases astime_start.time: the time the step was added or replaced, or the end of its last successful run. A failed run does not change it. It is the time stamp thatpip_output_value()gets inctx.
A step ran in a run if its time_start is not before a Sys.time() taken before the
run, and it succeeded if its state is "done" and time >= time_start.
Running views
When x is a view, the requested rows are run together with their upstream
dependencies, so the steps covered by the view are brought up to date even if their
inputs come from steps outside the view. The rest of the pipeline is not executed;
downstream steps of executed steps are marked "outdated". Steps whose inputs were
not re-executed (for example, because they were already "done") keep their state.
Runtime errors
By default, a failing step stops the run: the step gets state "failed" and its
error is stored as its output, the pipeline run state becomes "failed", and steps
after it that were not reached are marked "outdated", so the next run retries them.
The error message names the step and, for a mapped step, the partition key. The error
has class pipeflow_step_error and keeps the original error in its field parent:
tryCatch(pip_run(p), pipeflow_step_error = function(e) e$parent).
Continuing after errors
With on_error = "continue", the run does not stop at a failed step:
The failed step gets state
"failed", and its error is stored as its output (as above).Steps that depend on a failed step are not run, and neither are their own dependents. All other steps are run.
To run a step despite a failed input, refer to that input via
try(), e.g.x = ~try(fit)(seepip_add()). The argument then receives a failure object (an error condition) whose message tells which step failed, e.g."step 'fit' failed: negative".Locked steps are never run, so a failed input does not affect them.
A step mapped over partitions fails only for the keys whose call failed:
Its output keeps the results of the other keys and holds a failure object for each failed key, e.g.
"step 'fit' failed: key 'b': negative". The step gets state"failed", and printing the pipeline shows its output as e.g.<partitioned: 1/3 failed>.A mapped step after it is not run for the failed keys, whose output is then a failure object too (
"step 'show' was not run: step 'fit' failed: key 'b': negative"), unless it takes the input viatry().A step that is not mapped (e.g.
exec = "reduce") is not run, unless it takes the input viatry(). Then it receives the results of all keys, including the failure objects.
At the end of the run, the pipeline run state is "continued", and a warning of
class pipeflow_run_failed lists the failed steps (with the number of failed keys)
and the steps that were not run.
The next run retries the failed steps and the steps that were not run. Once a failed
step succeeds, the steps after it are run again, including those that received
failure objects. A mapped step is only run again for its failed keys, unless
something changed that affects all keys (its params or exec mode, an input step, or
force = TRUE). See vignette("v05c-error-handling", package = "pipeflow").
Mapping in parallel
With map, the keys of a mapped step are run by map, for example in parallel
processes. This applies to all mapped steps of the run, but not to inner pipelines of
functions made by as.function.pipeflow(), which run in parallel only if made with
cache = FALSE (the cached inner pipelines would be lost in another process). For
backends that need more arguments, pass a wrapper, e.g.
\(X, FUN) parallel::mclapply(X, FUN, mc.cores = 4).
Errors are handled as without map, but all keys of a step run before the error of
the first failed key is raised. Warnings, messages and random numbers are up to the
backend, e.g. future.apply::future_lapply relays warnings and messages and needs
future.seed = TRUE, while parallel::mclapply does not relay them.
Runtime control flow via restart and halt
A running pipeline can be interrupted via the restart() and halt() virtual
methods. They are intended for advanced, self-modifying pipelines and are most often
called from within a step function via .self (see section 'The pipeline as .self'
in pip_add()).
.self$restart(force = TRUE, times = 1L): aborts the current run after the current step has finished and restarts it from the first step. The default parameters areforce = TRUEandtimes = 1L, that is, the above call is the same as just invoking .self$restart(). To skip steps that are already in state"done", setforce = FALSE. Thetimesparameter limits the number of consecutive restarts within a singlepip_run()call. If a view is being run, a restart covers the view steps together with their upstream dependencies.p$halt(): aborts the current run after the current step has finished. This is a controlled halt and deliberately distinct frombase::stop(): no error is raised and the pipeline is not marked as"failed". The run simply ends with run state"halted"(seepip_run_state()) and a subsequentpip_run()will continue where the run left off.
A run can also be halted from outside: if the progress callback returns FALSE,
the run ends before the next step, e.g. when a user cancels the run in a Shiny app.
In all cases, steps downstream of executed steps that have not been executed until
the restart or halt happens are marked as "outdated".
See also
vignette("v06-self-modify-pipeline", package = "pipeflow") for an advanced
example of dynamic pipelines.
Examples
p <- pip_new() |>
pip_add("load", \(n = 3) seq_len(n)) |>
pip_add("prep", \(x = ~load, weight = 1) x * weight) |>
pip_add("square", \(x = ~prep) x^2) |>
pip_add("total", \(x = ~square) sum(x))
pip_run(p)
#> info [2026-11-15 19:18:50.246 UTC]: Starting run of pipeflow 'pipe'
#> info [2026-11-15 19:18:50.246 UTC]: Step 1/4 load
#> info [2026-11-15 19:18:50.247 UTC]: Step 2/4 prep
#> info [2026-11-15 19:18:50.247 UTC]: Step 3/4 square
#> info [2026-11-15 19:18:50.248 UTC]: Step 4/4 total
#> info [2026-11-15 19:18:50.248 UTC]: Finished in 0.003 s (4 done, 0 skipped, 0 failed)
p
#> <pipeflow> pipe (4 steps)
#> -------------------------
#> step params depends state out
#> 1: load n done 1,2,3
#> 2: prep x,weight load done 1,2,3
#> 3: square x prep done 1,4,9
#> 4: total x square done 14
#> -------------------------
#> <ready> last run: 2026-11-15 19:18:50 (0.003 s)
#> steps: 4 done
pip_set_params(p, list(weight = 2))
p
#> <pipeflow> pipe (4 steps)
#> -------------------------
#> step params depends state out
#> 1: load n done 1,2,3
#> 2: prep x,weight load outdated 1,2,3
#> 3: square x prep outdated 1,4,9
#> 4: total x square outdated 14
#> -------------------------
#> <ready> last run: 2026-11-15 19:18:50 (0.003 s)
#> steps: 1 done, 3 outdated
# Already-done steps are skipped on a second run
pip_run(p) # first step skipped
#> info [2026-11-15 19:18:50.254 UTC]: Starting run of pipeflow 'pipe'
#> info [2026-11-15 19:18:50.254 UTC]: Step 1/4 load - skipping done step
#> info [2026-11-15 19:18:50.254 UTC]: Step 2/4 prep
#> info [2026-11-15 19:18:50.255 UTC]: Step 3/4 square
#> info [2026-11-15 19:18:50.255 UTC]: Step 4/4 total
#> info [2026-11-15 19:18:50.256 UTC]: Finished in 0.002 s (3 done, 1 skipped, 0 failed)
# lgr = NULL suppresses log output
pip_run(p, lgr = NULL)
# force = TRUE re-executes every step regardless of state
pip_run(p, force = TRUE)
#> info [2026-11-15 19:18:50.258 UTC]: Starting run of pipeflow 'pipe'
#> info [2026-11-15 19:18:50.258 UTC]: Step 1/4 load
#> info [2026-11-15 19:18:50.259 UTC]: Step 2/4 prep
#> info [2026-11-15 19:18:50.259 UTC]: Step 3/4 square
#> info [2026-11-15 19:18:50.260 UTC]: Step 4/4 total
#> info [2026-11-15 19:18:50.260 UTC]: Finished in 0.002 s (4 done, 0 skipped, 0 failed)
p
#> <pipeflow> pipe (4 steps)
#> -------------------------
#> step params depends state out
#> 1: load n done 1,2,3
#> 2: prep x,weight load done 2,4,6
#> 3: square x prep done 4,16,36
#> 4: total x square done 56
#> -------------------------
#> <ready> last run: 2026-11-15 19:18:50 (0.002 s)
#> steps: 4 done
# Run only a subset of steps via a view;
# upstream dependencies are automatically included
v <- pip_view(p, step = "total")
pip_run(v)
#> info [2026-11-15 19:18:50.264 UTC]: Starting run of pipeflow 'pipe'
#> info [2026-11-15 19:18:50.264 UTC]: Step 1/4 [upstream] load - skipping done step
#> info [2026-11-15 19:18:50.264 UTC]: Step 2/4 [upstream] prep - skipping done step
#> info [2026-11-15 19:18:50.264 UTC]: Step 3/4 [upstream] square - skipping done step
#> info [2026-11-15 19:18:50.265 UTC]: Step 4/4 [view] total - skipping done step
#> info [2026-11-15 19:18:50.265 UTC]: Finished in 0.001 s (0 done, 4 skipped, 0 failed)
# Halt or restart pipeline at runtime (for advanced usage)
p <- pip_new("restart") |>
pip_add("load", \(n = 3) seq_len(n)) |>
pip_add("check", \(x = ~load) {
if (length(x) > 10L) .self$halt()
}) |>
pip_add("model", \(x = ~load) {
if (length(x) == 3L) {
.self$set_params(list(n = 5))
.self$restart()
}
x * 2
})
pip_run(p)
#> info [2026-11-15 19:18:50.267 UTC]: Starting run of pipeflow 'restart'
#> info [2026-11-15 19:18:50.268 UTC]: Step 1/3 load
#> info [2026-11-15 19:18:50.269 UTC]: Step 2/3 check
#> info [2026-11-15 19:18:50.269 UTC]: Step 3/3 model
#> info [2026-11-15 19:18:50.270 UTC]: Restarting pipeline execution.
#> info [2026-11-15 19:18:50.271 UTC]: Restarting run of pipeflow 'restart'
#> info [2026-11-15 19:18:50.271 UTC]: Step 1/3 load
#> info [2026-11-15 19:18:50.271 UTC]: Step 2/3 check
#> info [2026-11-15 19:18:50.272 UTC]: Step 3/3 model
#> info [2026-11-15 19:18:50.272 UTC]: Finished in 0.005 s (3 done, 0 skipped, 0 failed)
p
#> <pipeflow> restart (3 steps)
#> ----------------------------
#> step params depends state out
#> 1: load n done 1,2,3,4,5
#> 2: check x load done [NULL]
#> 3: model x load done 2, 4, 6, 8,10
#> ----------------------------
#> <ready> last run: 2026-11-15 19:18:50 (0.005 s)
#> steps: 3 done
pip_set_params(p, list(n = 15)) # now halt() in 'check' step is triggered
pip_run(p)
#> info [2026-11-15 19:18:50.276 UTC]: Starting run of pipeflow 'restart'
#> info [2026-11-15 19:18:50.276 UTC]: Step 1/3 load
#> info [2026-11-15 19:18:50.277 UTC]: Step 2/3 check
#> info [2026-11-15 19:18:50.277 UTC]: Aborting pipeline execution on manual halt.
#> info [2026-11-15 19:18:50.277 UTC]: Finished in 0.001 s (2 done, 1 skipped, 0 failed)
# Continue after errors and pass failures to the steps that allow them
p <- pip_new("continue") |>
pip_add("data", \(n = 5) seq_len(n)) |>
pip_add("check", \(x = ~data) stop("check failed")) |>
pip_add("count", \(x = ~check) length(x)) |>
pip_add("total", \(x = ~data) sum(x)) |>
pip_add("report", \(total = ~total, n = ~try(count)) {
if (inherits(n, "pipeflow_failure")) conditionMessage(n) else n
})
withCallingHandlers(
pip_run(p, lgr = NULL, on_error = "continue"),
pipeflow_run_failed = \(w) {
message("failed: ", toString(names(w$failed)))
invokeRestart("muffleWarning")
}
)
#> failed: check
p
#> <pipeflow> continue (5 steps)
#> -----------------------------
#> step params depends state
#> 1: data n done
#> 2: check x data failed
#> 3: count x check new
#> 4: total x data done
#> 5: report total,n total,count done
#> out
#> 1: 1,2,3,4,5
#> 2: <error/simpleError>
#> 3: [NULL]
#> 4: 15
#> 5: step 'count' was not run: step 'check' f...
#> -----------------------------
#> <continued> last run: 2026-11-15 19:18:50 (0.003 s)
#> steps: 3 done, 1 failed, 1 new
p[["report", "out"]]
#> [1] "step 'count' was not run: step 'check' failed: check failed"
# Run the keys of a mapped step with a map function, e.g. in parallel
p <- pip_new("map") |>
pip_add("parts", \(x = list(a = 1, b = 2, c = 3)) x, exec = "split") |>
pip_add("square", \(x = ~parts) x^2) |>
pip_add("total", \(x = ~square) sum(unlist(x)), exec = "reduce")
pip_run(p, map = lapply)
#> info [2026-11-15 19:18:50.290 UTC]: Starting run of pipeflow 'map'
#> info [2026-11-15 19:18:50.290 UTC]: Step 1/3 parts
#> info [2026-11-15 19:18:50.291 UTC]: Step 2/3 square
#> info [2026-11-15 19:18:50.292 UTC]: Step 3/3 total
#> info [2026-11-15 19:18:50.293 UTC]: Finished in 0.003 s (3 done, 0 skipped, 0 failed)
p[["total", "out"]]
#> [1] 14
if (.Platform$OS.type == "unix") {
mc2 <- \(X, FUN) parallel::mclapply(X, FUN, mc.cores = 2)
pip_run(p, force = TRUE, map = mc2)
}
#> info [2026-11-15 19:18:50.294 UTC]: Starting run of pipeflow 'map'
#> info [2026-11-15 19:18:50.294 UTC]: Step 1/3 parts
#> info [2026-11-15 19:18:50.295 UTC]: Step 2/3 square
#> info [2026-11-15 19:18:50.317 UTC]: Step 3/3 total
#> info [2026-11-15 19:18:50.319 UTC]: Finished in 0.025 s (3 done, 0 skipped, 0 failed)
