Kubeflow Pipeline Using Python Based Function Components

Introduction 

In our last blog we learned about running Kubeflow on Kubernetes in Minikube. We will learn more about creating Kubeflow Pipelines, compiling these pipelines and running them on minikube in this blog.

Kubeflow Platform

KFP Platform consists of:

• A UI for managing and tracking pipelines and their execution
• An engine for scheduling a pipeline’s execution
• An SDK for defining, building, and deploying pipelines in Python

Kubeflow SDK

We will create a virtual environment  kubeflow and install kfp sdk in it.

$ mkvirtualenv kubeflow
$ pip install kfp

Check if the KFP installed correctly

$ python -c "import kfp; print('KFP SDK version: {}'.format(kfp.__version__))"
KFP SDK version: 1.8.16

Custom Kubeflow Pipeline 

We will create a custom KF pipeline using python function based components. Read more about KFP here.

import os
import kfp
from kfp import compiler
from kfp.components import func_to_container_op

@func_to_container_op
def hello_world(text: str) -> str:
    print(text)
    return text

def pipeline_hello_world(text: str = 'hi there'):
    """Pipeline that passes small pipeline parameter string to consumer op."""

    consume_task = hello_world(
        text=text)  # Passing pipeline parameter as argument to consumer op

def compile():
    print("Compiling Pipeline")
    compiler.Compiler().compile(
      pipeline_func=pipeline_hello_world,
      package_path='hello_world_pipeline.yaml')

def run():
    print("Running Pipeline")
    client = kfp.Client(host="http://localhost:8080")
    client.create_run_from_pipeline_func(pipeline_hello_world, arguments={})

def main():
    # compile()
    # run()
    pass

if __name__ == "__main__":
    main()

Save the above code to hello_world_pipeline.py. In this code main function has two functions:

• compile - Compiles the pipeline code and creates a hello_world_pipeline.yaml
• run - Runs the pipeline directly on local KFP Platform http://localhost:8080/

You can un-comment the function based on your need. 

Compile Kubeflow Pipeline 

I will first un-comment  compile function.

$ python hello_world_pipeline.py
Compiling Pipeline

The above creates hello_world_pipeline.yaml in the above path. This file can be imported on KFP UI as below:

Fill the Name and Description, Choose Upload File. Navigate to path.

Once the file is upload, click the create.

This will create a new Pipeline. You can run the pipeline by creating an experiment by clicking the "Create an Experiment" on the UI, give it a name, and then you should end up on a page to start a run.

Click start on below screenshot.

After few minutes the pipeline will complete with Green Tick Symbol ✅

Pipeline Executed Successfully. 

Running Kubeflow Pipeline 

I will first un-comment  run function only. This directly launch the pipeline in KFP UI.

$ python hello_world_pipeline.py
Running Pipeline

We can check the status on KFP UI. After few minutes the pipeline will complete with Green Tick Symbol ✅

Pipeline Executed Successfully. 

Please post your queries below.

Happy Coding