Composing Multiple Notebooks¶
A single notebook is not always the right unit of work. Preprocessing that several projects share, a training notebook someone else owns, an evaluation step you want to keep separate: these are naturally different notebooks, and copying cells between them is how they drift apart.
A notebook: cell lets one notebook reference another. The referenced notebook
is compiled into the same pipeline as a nested sub-pipeline, keeping each
of its cells visible as its own step in the KFP UI.
A first composition¶
Take a notebook that prepares data and a notebook that trains on it. Neither references the other; each is a normal Kale notebook.
preprocessing.ipynb:
# tag: step:load
dataset = pd.read_csv("data.csv")
training.ipynb:
# tag: step:train
model = RandomForestClassifier().fit(dataset.drop("y", axis=1), dataset["y"])
Now a third notebook composes them. It contains nothing but two reference cells:
cell tags |
cell source |
cell metadata |
|---|---|---|
|
(empty) |
|
|
(empty) |
|
Compiling that notebook produces one pipeline containing both:
root pipeline
├── preprocessing (sub-DAG)
│ └── load-step
└── training (sub-DAG)
└── train-step
We call the notebook holding the references the root notebook.
Where the path lives¶
The tag names the reference; it does not contain the path. The path lives in
the cell’s metadata, under notebook_path, and is resolved relative to the
notebook that contains the cell.
{
"metadata": {
"tags": ["notebook:preprocessing"],
"notebook_path": "./preprocessing.ipynb"
}
}
The Kale panel writes both for you: choose the Notebook cell type and enter a path. A reference cell holds no code of its own, so when the cell is empty Kale places a comment there explaining that.
Note
The Notebook cell type is hidden by default in the JupyterLab panel. See
Enabling the cell type below, which also explains
what that setting does and does not control.
How data crosses a notebook boundary¶
It crosses the same way it crosses a cell boundary: Kale finds the variables one unit produces and another consumes, and marshals them between the two. See Data Passing & Marshalling for the mechanism.
The difference is only the level. Inside a notebook, Kale matches variables between cells. In a composition, it matches them between whole notebooks, and then hands each variable to the specific step inside the referenced notebook that actually produces or consumes it.
So in the example above, training reads dataset and never defines it, while
preprocessing defines it. Kale therefore:
orders
preprocessingbeforetraining, with noprev:tag neededmarshals
datasetout ofload-stepand intotrain-stepas a KFP artifact
Two referenced notebooks that share no variables get no edge between them, so unrelated work still runs in parallel.
Mixing steps and references¶
A root notebook can also carry its own step: cells alongside its references.
Its own steps become top-level components in the pipeline, next to the sub-DAGs.
cell tags |
source |
|---|---|
|
(empty) |
|
(empty) |
|
|
A root step runs after everything above it in the notebook and before everything below it, so a composition runs the way it reads. Variables cross in both directions: a root step can feed a referenced notebook, and a referenced notebook can feed a root step.
What you get out¶
Each referenced notebook is compiled into its own importable module, written
next to the root notebook’s generated DSL in .kale/. The root notebook’s DSL
imports those modules and wires everything together.
Each module is a complete pipeline definition of its own, so you can read it and compile it independently of the composition that imports it.
Enabling the cell type¶
The Notebook cell type is gated behind the Enable composable notebooks
setting in the Kale panel’s settings (JupyterLab Settings → Settings Editor →
Kale). It is off by default while the feature settles.
Important
The setting controls only whether the cell type is offered in the cell-metadata editor. It does not gate compilation.
A notebook that already contains a notebook: cell compiles as a composition
whether or not the setting is on, and kale --nb never sees the setting at
all. “Off by default” means the cell type is hidden, not that composition is
inert.
Current limitations¶
A notebook that reads another’s variables is not self-contained. It runs only as part of the composition; compiled alone, nothing provides those variables and the step fails at runtime. Stand-in values do not help: the notebook then produces the variable itself, so it either stops receiving the upstream value or conflicts with the real producer. This is a known limitation of inferring data flow from names (KEP-0812, caveat 6).
The following are reported at compile time rather than at pipeline runtime.
Nested references are not supported. A referenced notebook cannot itself reference a third notebook. Reference every notebook from the root notebook instead.
A reference cell holds no code. Putting code in a notebook: cell raises,
rather than silently dropping it:
`notebook:training` cell contains code. A cell that references a notebook holds no code of its own. Move the code to its own `step:` cell.
Comments are fine, which is why the panel can leave an explanatory one there.
A notebook cannot reference itself. That raises as a reference cycle.
A variable needs a single producer. If two notebooks or steps both define a variable that another one reads, Kale raises rather than guessing which to use.
A reference needs a path. A notebook: cell with no notebook_path in its
metadata raises, rather than compiling to nothing.
Untagged code after a reference needs an owner. A reference breaks the cell
merge chain, so an untagged cell after one attaches to the next step: cell.
If there is no following step, it raises rather than being dropped.
Two references cannot share a name. Reference names have to be unique within a root notebook, since each names a node in the same pipeline.
Dive deeper¶
Cell Types & Annotations, the full tag vocabulary
Data Passing & Marshalling, how variables move between steps
Pipeline Compilation, how a notebook becomes KFP DSL
examples/composition/, five example notebooks demonstrating both shapes