Articles • pipeflow

All vignettes

Get started with pipeflow: Start here if this is your first time using pipeflow.
Modify existing pipelines: How to insert, replace, and remove steps in a pipeline.
Combine pipelines: How to combine different pipelines to a single pipeline.
Pipeline views: How to filter pipelines with pip_view() and the [ operator, compose views, and run only a subset of steps.
Collect and group output: How to collect and group pipeline output.
Split, map, and reduce: Shows how to split data, apply the pipeline to each subset, and then reduce the results back into a combined output.
Nested pipelines: Shows how to embed pipelines within pipeline steps and how to mark their parameters so that changing them re-executes the outer step.
Handling failures: Shows what happens when a step fails, how to keep a pipeline running with on_error = “continue”, how to react to failed inputs with ~try() references, and how to recover once the cause is fixed.
Annotated parameter values: Shows how to store annotated values (e.g. values with units or labels) as step parameters and how to unwrap them right before a step function is called, using the pip_param_value() generic.
Self-modifying pipelines: Shows how you can setup pipelines to modify themselves at runtime, which, for example, allows for changing pipeline parameters based on intermediate results or even dynamically modify the pipeline’s own structure during a pipeline run.
pipeflow vs targets: Comparing general properties and performance of {pipeflow} vs {targets} in a scoped latency comparison.