Adds a named step to the pipeline. Each step is a function whose parameters either
hold constant defaults or reference the output of a prior step using formula notation
(~step_name). Dependencies are validated when the step is added.
Arguments
- x
A pipeflow pipeline object.
- step
Unique step name, or a pipeline or view whose steps are copied into
x(see 'Copying steps' below), in which casefun,tags,paramsandexecmust not be given. A step name must not start with-, which marks a relative reference like~-1.- fun
Function to execute for the step. Each function parameter needs a value: a default value in
funor an entry inparams. Default values that are simple constants are resolved immediately. Default values that are formulas like~other_stepare treated as dependencies to those steps and resolved to the respective output values at runtime once the step is executed. A reference wrapped intry(), like~try(other_step), accepts a failed input when the pipeline was run withon_error = "continue"(see section 'Continuing after errors' inpip_run()).Optional character vector of tags belonging to the step. Can also be adjusted later using
[pip_tag()].- params
Optional named list of parameter values. They override the defaults of
funand can be used to give a value to parameters offunthat have no default. As infun, a formula like~other_steprefers to the output of another step. Names that are no parameters offunare an error, unlessfunhas a...argument, which then takes them. Use cases forparamsare:Set param values programmatically, e.g. when adding steps in a loop or at runtime.
Use existing functions whose parameters have no defaults, e.g.
pip_add(p, "avg", mean, params = list(x = ~data, na.rm = TRUE)).Provide extra param values to "mark" dependencies that are defined in pipelines nested in a step, which ensures that it is re-executed when one of the nested param values change. See
vignette("v05b-nested-pipeline", package = "pipeflow").
- exec
Execution mode for this step, which controls how partitioned input is handled. One of:
"auto": map over the partitions if an input is partitioned, otherwise a single call. Several partitioned inputs must have the same keys in the same order.
"split": single call, then mark the output as partitioned.
"reduce": single call, only valid with partitioned input (or, via
~try(), a failure object)."plain": single call, only valid with non-partitioned input.
A mapped step that fails for some partition keys in a run with
on_error = "continue"keeps the outputs of the other keys (see section 'Continuing after errors' inpip_run()).- after
Optional position after which the new step should be inserted (defaults to last position). Can be a step name or an integer index. If set to 0, the new step will be inserted at the beginning of the pipeline.
Details
If after was specified, the new step will be inserted after the given step or
position. Be aware that in contrast to adding a step at the end, inserting a step in
the middle is a rather expensive operation as it requires re-wiring parts of the
internal pipeline structure, especially if the new step is inserted at an early
position.
Copying steps
If an entire pipeline or view is provided as step, its steps are added in their
order, with their functions, current params, tags and exec modes. Upstream steps of
views are not copied, which means that references (including relative references
like ~-1) are resolved (and must be valid) within x. If this blocks a step from
being added (e.g., its name already exists in x, or it refers to a step not
existing in x), x is left unchanged. Outputs and locks are not copied either. To
copy a view v with its upstream steps, use pip_add(x, v[, view = FALSE]) (see
[), to bind whole pipelines, rbind().
Params with the same name
A param name is meant to have one value in the whole pipeline.Therefore,
pip_add(), pip_replace() and rbind()
signal a warning if the default param value of a new step differs from the value
of the same param in the other steps.
To give steps different values on purpose, give the params different names, silence
the warning with suppressWarnings(..., classes = "pipeflow_param_conflict"), or
turn it off with options(pipeflow.warn_param_conflicts = FALSE).
The pipeline as .self
A step function can refer to the pipeline that is being run as .self without
declaring it as an argument, for example to set parameters or add steps at runtime,
or to call .self$restart() or .self$halt() (see pip_run()). .self is always
the full pipeline, also when a view is run. pipeflow provides it each time the step
runs, but only if the body of the step function contains the name .self (also
within functions defined in the body). Indirect access, such as get(".self"), is
not detected.
A step that changes its own function, params or exec mode via .self, ends the run
as "outdated". The same is true for other steps changed via .self. As these
steps and their downstream steps run again in the next run, pip_run() warns (class
pipeflow_outdated_before) if a step changes itself or a done step before it, unless
the step halts the run or the run is restarted. Lastly, .self is only valid while
the step runs and released once the step was returned. For this reason, a step that
returns a function referring to .self fails with object '.self' not found when
called later.
Outputs that keep an environment
An output that keeps the environment it was made in, such as a function or a ggplot made in the step, also keeps the environment of the step function. If that function was defined where the pipeline is (e.g. in a function that builds it), saving the output saves the whole pipeline too. Define step functions in a package or at top level, or return a form without an environment (e.g. an image instead of a ggplot).
Examples
# --- Tags, and view filtering ---
p <- pip_new("analysis") |>
pip_add("load", \(n = 5) seq_len(n), tags = c("io", "raw")) |>
pip_add("clean", \(x = ~load) x * 2, tags = c("io", "process")) |>
pip_add("fit", \(x = ~clean) sum(x), tags = c("model", "core", "daily")) |>
pip_add("report", \(x = ~fit) paste("result:", x), tags = "report")
pip_run(p)
#> info [2026-11-15 19:18:46.647 UTC]: Starting run of pipeflow 'analysis'
#> info [2026-11-15 19:18:46.647 UTC]: Step 1/4 load
#> info [2026-11-15 19:18:46.648 UTC]: Step 2/4 clean
#> info [2026-11-15 19:18:46.649 UTC]: Step 3/4 fit
#> info [2026-11-15 19:18:46.649 UTC]: Step 4/4 report
#> info [2026-11-15 19:18:46.650 UTC]: Finished in 0.003 s (4 done, 0 skipped, 0 failed)
p
#> <pipeflow> analysis (4 steps)
#> -----------------------------
#> step params depends state out tags
#> 1: load n done 1,2,3,4,5 io,raw
#> 2: clean x load done 2, 4, 6, 8,10 io,process
#> 3: fit x clean done 30 model,core,daily
#> 4: report x fit done result: 30 report
#> -----------------------------
#> <ready> last run: 2026-11-15 19:18:46 (0.003 s)
#> steps: 4 done
# Filter by tag using pip_view — keeps steps with any matching tag
pip_view(p, tags = "daily")
#> <pipeflow_view> analysis (1 of 4 steps)
#> ---------------------------------------
#> step params depends state out tags
#> 1: fit x clean done 30 model,core,daily
#> ---------------------------------------
#> <ready> last run: 2026-11-15 19:18:46 (0.003 s)
#> steps: 1 done
pip_view(p, tags = "core")
#> <pipeflow_view> analysis (1 of 4 steps)
#> ---------------------------------------
#> step params depends state out tags
#> 1: fit x clean done 30 model,core,daily
#> ---------------------------------------
#> <ready> last run: 2026-11-15 19:18:46 (0.003 s)
#> steps: 1 done
pip_view(p, tags = c("raw", "report"))
#> <pipeflow_view> analysis (2 of 4 steps)
#> ---------------------------------------
#> step params depends state out tags
#> 1: load n done 1,2,3,4,5 io,raw
#> 2: report x fit done result: 30 report
#> ---------------------------------------
#> <ready> last run: 2026-11-15 19:18:46 (0.003 s)
#> steps: 2 done
# --- Split / reduce execution modes ---
q <- pip_new("split-demo") |>
pip_add("data", \(x = iris) x) |>
pip_add("split", \(x = ~data) split(x, x$Species),
exec = "split"
) |>
pip_add("stats", \(x = ~split) summary(x)) |>
pip_add("combine", \(x = ~stats) do.call(rbind, x),
exec = "reduce"
)
pip_run(q)
#> info [2026-11-15 19:18:46.661 UTC]: Starting run of pipeflow 'split-demo'
#> info [2026-11-15 19:18:46.661 UTC]: Step 1/4 data
#> info [2026-11-15 19:18:46.662 UTC]: Step 2/4 split
#> info [2026-11-15 19:18:46.663 UTC]: Step 3/4 stats
#> info [2026-11-15 19:18:46.667 UTC]: Step 4/4 combine
#> info [2026-11-15 19:18:46.667 UTC]: Finished in 0.007 s (4 done, 0 skipped, 0 failed)
q[["stats", "out"]] # partitioned list — one summary per species
#> $setosa
#> Sepal.Length Sepal.Width Petal.Length Petal.Width
#> Min. :4.300 Min. :2.300 Min. :1.000 Min. :0.100
#> 1st Qu.:4.800 1st Qu.:3.200 1st Qu.:1.400 1st Qu.:0.200
#> Median :5.000 Median :3.400 Median :1.500 Median :0.200
#> Mean :5.006 Mean :3.428 Mean :1.462 Mean :0.246
#> 3rd Qu.:5.200 3rd Qu.:3.675 3rd Qu.:1.575 3rd Qu.:0.300
#> Max. :5.800 Max. :4.400 Max. :1.900 Max. :0.600
#> Species
#> setosa :50
#> versicolor: 0
#> virginica : 0
#>
#>
#>
#>
#> $versicolor
#> Sepal.Length Sepal.Width Petal.Length Petal.Width Species
#> Min. :4.900 Min. :2.000 Min. :3.00 Min. :1.000 setosa : 0
#> 1st Qu.:5.600 1st Qu.:2.525 1st Qu.:4.00 1st Qu.:1.200 versicolor:50
#> Median :5.900 Median :2.800 Median :4.35 Median :1.300 virginica : 0
#> Mean :5.936 Mean :2.770 Mean :4.26 Mean :1.326
#> 3rd Qu.:6.300 3rd Qu.:3.000 3rd Qu.:4.60 3rd Qu.:1.500
#> Max. :7.000 Max. :3.400 Max. :5.10 Max. :1.800
#>
#> $virginica
#> Sepal.Length Sepal.Width Petal.Length Petal.Width
#> Min. :4.900 Min. :2.200 Min. :4.500 Min. :1.400
#> 1st Qu.:6.225 1st Qu.:2.800 1st Qu.:5.100 1st Qu.:1.800
#> Median :6.500 Median :3.000 Median :5.550 Median :2.000
#> Mean :6.588 Mean :2.974 Mean :5.552 Mean :2.026
#> 3rd Qu.:6.900 3rd Qu.:3.175 3rd Qu.:5.875 3rd Qu.:2.300
#> Max. :7.900 Max. :3.800 Max. :6.900 Max. :2.500
#> Species
#> setosa : 0
#> versicolor: 0
#> virginica :50
#>
#>
#>
#>
#> attr(,"class")
#> [1] "list" "pipeflow_partitioned"
q[["combine", "out"]] # combined table
#> Sepal.Length Sepal.Width Petal.Length Petal.Width
#> "Min. :4.300 " "Min. :2.300 " "Min. :1.000 " "Min. :0.100 "
#> "1st Qu.:4.800 " "1st Qu.:3.200 " "1st Qu.:1.400 " "1st Qu.:0.200 "
#> "Median :5.000 " "Median :3.400 " "Median :1.500 " "Median :0.200 "
#> "Mean :5.006 " "Mean :3.428 " "Mean :1.462 " "Mean :0.246 "
#> "3rd Qu.:5.200 " "3rd Qu.:3.675 " "3rd Qu.:1.575 " "3rd Qu.:0.300 "
#> "Max. :5.800 " "Max. :4.400 " "Max. :1.900 " "Max. :0.600 "
#> "Min. :4.900 " "Min. :2.000 " "Min. :3.00 " "Min. :1.000 "
#> "1st Qu.:5.600 " "1st Qu.:2.525 " "1st Qu.:4.00 " "1st Qu.:1.200 "
#> "Median :5.900 " "Median :2.800 " "Median :4.35 " "Median :1.300 "
#> "Mean :5.936 " "Mean :2.770 " "Mean :4.26 " "Mean :1.326 "
#> "3rd Qu.:6.300 " "3rd Qu.:3.000 " "3rd Qu.:4.60 " "3rd Qu.:1.500 "
#> "Max. :7.000 " "Max. :3.400 " "Max. :5.10 " "Max. :1.800 "
#> "Min. :4.900 " "Min. :2.200 " "Min. :4.500 " "Min. :1.400 "
#> "1st Qu.:6.225 " "1st Qu.:2.800 " "1st Qu.:5.100 " "1st Qu.:1.800 "
#> "Median :6.500 " "Median :3.000 " "Median :5.550 " "Median :2.000 "
#> "Mean :6.588 " "Mean :2.974 " "Mean :5.552 " "Mean :2.026 "
#> "3rd Qu.:6.900 " "3rd Qu.:3.175 " "3rd Qu.:5.875 " "3rd Qu.:2.300 "
#> "Max. :7.900 " "Max. :3.800 " "Max. :6.900 " "Max. :2.500 "
#> Species
#> "setosa :50 "
#> "versicolor: 0 "
#> "virginica : 0 "
#> NA
#> NA
#> NA
#> "setosa : 0 "
#> "versicolor:50 "
#> "virginica : 0 "
#> NA
#> NA
#> NA
#> "setosa : 0 "
#> "versicolor: 0 "
#> "virginica :50 "
#> NA
#> NA
#> NA
# --- Insert a step at a specific position with 'after' ---
p2 <- pip_new("insert-demo") |>
pip_add("load", \(x = 1) x) |>
pip_add("fit", \(x = ~load) x + 1)
p2
#> <pipeflow> insert-demo (2 steps)
#> --------------------------------
#> step params depends state
#> 1: load x new
#> 2: fit x load new
#> --------------------------------
#> <ready> last run: never
#> steps: 2 new
pip_add(p2, "clean", \(x = ~load) x * 2, after = "load")
p2 # "load", "clean", "fit" — inserted after "load"
#> <pipeflow> insert-demo (3 steps)
#> --------------------------------
#> step params depends state
#> 1: load x new
#> 2: clean x load new
#> 3: fit x load new
#> --------------------------------
#> <ready> last run: never
#> steps: 3 new
# after = 0 inserts the new step at the very beginning
pip_add(p2, "preload", \(x = 1) x, after = 0)
p2
#> <pipeflow> insert-demo (4 steps)
#> --------------------------------
#> step params depends state
#> 1: preload x new
#> 2: load x new
#> 3: clean x load new
#> 4: fit x load new
#> --------------------------------
#> <ready> last run: never
#> steps: 4 new
# --- Copy steps of another pipeline, without their upstream steps ---
other <- pip_new("other-data") |> pip_add("load", \(x = 10) x)
pip_add(other, p2["fit"]) # "fit" now refers to "load" of `other`
pip_run(other)
#> info [2026-11-15 19:18:46.688 UTC]: Starting run of pipeflow 'other-data'
#> info [2026-11-15 19:18:46.688 UTC]: Step 1/2 load
#> info [2026-11-15 19:18:46.689 UTC]: Step 2/2 fit
#> info [2026-11-15 19:18:46.689 UTC]: Finished in 0.002 s (2 done, 0 skipped, 0 failed)
other[["fit", "out"]] # 11
#> [1] 11
# --- Provide parameter values with 'params' ---
# 'params' override the defaults of the function ...
p3 <- pip_new("params-demo") |>
pip_add("data", \(x = c(1, NA, 3)) x) |>
pip_add("scale", \(x = ~data, factor = 1) x * factor,
params = list(factor = 10)
) |>
# ... and give values to parameters without default, e.g. of mean()
pip_add("avg", mean, params = list(x = ~scale, na.rm = TRUE))
pip_run(p3)
#> info [2026-11-15 19:18:46.693 UTC]: Starting run of pipeflow 'params-demo'
#> info [2026-11-15 19:18:46.693 UTC]: Step 1/3 data
#> info [2026-11-15 19:18:46.694 UTC]: Step 2/3 scale
#> info [2026-11-15 19:18:46.694 UTC]: Step 3/3 avg
#> info [2026-11-15 19:18:46.695 UTC]: Finished in 0.002 s (3 done, 0 skipped, 0 failed)
p3
#> <pipeflow> params-demo (3 steps)
#> --------------------------------
#> step params depends state out
#> 1: data x done 1,NA, 3
#> 2: scale x,factor data done 10,NA,30
#> 3: avg x,na.rm scale done 20
#> --------------------------------
#> <ready> last run: 2026-11-15 19:18:46 (0.002 s)
#> steps: 3 done
p3[["avg", "out"]] # 20
#> [1] 20
# Extra params are passed via `...`
p4 <- pip_new("dots-demo") |>
pip_add("load", \(x = 1, ...) c(x, ...), params = list(size = 42))
pip_run(p4)
#> info [2026-11-15 19:18:46.699 UTC]: Starting run of pipeflow 'dots-demo'
#> info [2026-11-15 19:18:46.699 UTC]: Step 1/1 load
#> info [2026-11-15 19:18:46.700 UTC]: Finished in 0.001 s (1 done, 0 skipped, 0 failed)
p4[["load", "out"]] # c(1, size = 42)
#> size
#> 1 42
