OptiFlow

Appearance

Getting Started ​

Prerequisites ​

  • .NET 8+
  • Visual Studio

Overview ​

OptiFlow aims for ETL pattern agnosticism, providing a platform where your workflows, essentially any compiled .NET code, grant you the complete freedom to articulate almost any process necessary for your tasks.

With a focus on minimal overhead and the utilization of source generators to produce boilerplate code, you can define clear and succinct Workflows. This approach eliminates the need for repetitive and cumbersome setup, allowing you to delve straight into the core objectives of your workflow without unnecessary complexity.

Setup ​

Create a new dotnet console project and add the TzDigital.OptiFlow NuGet package to your project.

bash
dotnet new console -n MyWorkflowProject
dotnet add MyWorkflowProject package TzDigital.OptiFlow
dotnet new console -n MyWorkflowProject
dotnet add MyWorkflowProject package TzDigital.OptiFlow

Entry Point ​

In the program.cs, annotate your entry point with it's Group Name and create your application with the OptiFlow Host Builder.

INFO

The OptiFlowBuilder uses the same technologies and builder pattern as AspNetCore. If you are familar with that pattern then this should be a pretty straight forward setup.

Create your application inside the entry point method and configure your Dependency Injection

csharp
using Microsoft.Extensions.Configuration;
using Microsoft.Extensions.Hosting;
using TzDigital.OptiFlow.Entry;
using TzDigital.OptiFlow.Entry.Abstractions;
using TzDigital.OptiFlow.Entry.Attributes;
namespace TzDigital.OptiFlow.Workflows;

// Annotate your entry point so it can be found during execution.
[OptiFlowEntryPoint(GroupName = "digital-internal")]
public class Program
{
    public static async Task Main(string[] args)
    {
        IOptiFlowHostBuilder builder = OptiFlowEntryPoint.CreateBuilder(args);

        // When you set the `DOTNET_ENVIRONMENT` environment variable
        // to `LocalIsolated` you can run your workflow
        // using local state stores and a mock OptiFlow server.
        // This allows you to run in isolation for development purpose.
        if (builder.Environment.IsLocalIsolated())
        {
           builder.ConfigureIsolatedLocalTesting();
        }

        // Here you have access to:
        //
        // .NET Configuration (appsettings.json, Environment Variables, ect..)
        IConfiguration configuration = builder.Configuration;

        // .NET HostEnvironment with environment name
        IHostEnvironment environment = builder.Environment;
        string environmentName = environment.EnvironmentName;

        if (builder.Environment.IsProduction())
        {
            // Set prod only stuff
        }

        if (builder.Environment.IsQA())
        {
            // Set QA only stuff
        }

        // Trigger the OptiFlow Host to Build and Run
        IOptiFlowHost host = builder.Build();

        await host.StartAsync();
    }
}
using Microsoft.Extensions.Configuration;
using Microsoft.Extensions.Hosting;
using TzDigital.OptiFlow.Entry;
using TzDigital.OptiFlow.Entry.Abstractions;
using TzDigital.OptiFlow.Entry.Attributes;
namespace TzDigital.OptiFlow.Workflows;

// Annotate your entry point so it can be found during execution.
[OptiFlowEntryPoint(GroupName = "digital-internal")]
public class Program
{
    public static async Task Main(string[] args)
    {
        IOptiFlowHostBuilder builder = OptiFlowEntryPoint.CreateBuilder(args);

        // When you set the `DOTNET_ENVIRONMENT` environment variable
        // to `LocalIsolated` you can run your workflow
        // using local state stores and a mock OptiFlow server.
        // This allows you to run in isolation for development purpose.
        if (builder.Environment.IsLocalIsolated())
        {
           builder.ConfigureIsolatedLocalTesting();
        }

        // Here you have access to:
        //
        // .NET Configuration (appsettings.json, Environment Variables, ect..)
        IConfiguration configuration = builder.Configuration;

        // .NET HostEnvironment with environment name
        IHostEnvironment environment = builder.Environment;
        string environmentName = environment.EnvironmentName;

        if (builder.Environment.IsProduction())
        {
            // Set prod only stuff
        }

        if (builder.Environment.IsQA())
        {
            // Set QA only stuff
        }

        // Trigger the OptiFlow Host to Build and Run
        IOptiFlowHost host = builder.Build();

        await host.StartAsync();
    }
}

Basic Workflow Definition ​

To establish a recurring OptiFlow Workflow, you need a class outlining your process:

  • Workflows should include an OptiFlowWorkflow attribute for discovery and execution.

  • Optional OptiFlowTag attributes within Workflows serve as visual tags in the OptiFlow UI.

  • Workflow Operations must feature an OptiFlowOperation attribute, specifying a Step value to determine the execution order.

  • Workflow Operations can be designed for synchronous or asynchronous operation.

  • The return value from Workflow Operations is stored as a local state item, accessible to subsequent Operations.

Using a Raw Cron Schedule ​

You can define a Raw Cron Schedule in your Workflow Attribute.

INFO

Rest assured that your cron schedule is validated for accuracy. In the event of an invalid or malformed cron expression, your OptiFlow workflow will not compile, providing an immediate warning.

For added convenience, you can use the Cron Generator online tool to facilitate the construction of your cron expressions.

csharp
[OptiFlowWorkflow("third-workflow", "Third Test Workflow", "0 0/2 * * * ?")] // Execute every 2 minutes.
[OptiFlowTag("owner", "digital")]
[OptiFlowTag("scope", "internal")]
internal partial class ThirdWorkflow : IWorkflow
{
    [OptiFlowOperation(step: 1, continueOnError: true)]
    public string ThisIsPrettyCool()
    {
        // Do some process to get a string and save it to local state for the next process to access it.
        return "I am from step 1";
    }

    [OptiFlowOperation(step: 2, continueOnError: true)]
    public async Task FinalStepAsync()
    {
        string? step1State = Step1_State;

        // Do some Async stuff here.
    }
}
[OptiFlowWorkflow("third-workflow", "Third Test Workflow", "0 0/2 * * * ?")] // Execute every 2 minutes.
[OptiFlowTag("owner", "digital")]
[OptiFlowTag("scope", "internal")]
internal partial class ThirdWorkflow : IWorkflow
{
    [OptiFlowOperation(step: 1, continueOnError: true)]
    public string ThisIsPrettyCool()
    {
        // Do some process to get a string and save it to local state for the next process to access it.
        return "I am from step 1";
    }

    [OptiFlowOperation(step: 2, continueOnError: true)]
    public async Task FinalStepAsync()
    {
        string? step1State = Step1_State;

        // Do some Async stuff here.
    }
}

Using the Schedule Builder ​

If you prefer not to define a Raw Cron Schedule, you have the option to utilize the OptiFlowSchedule class, which streamlines the process by automatically constructing a cron schedule for you.

INFO

The Raw Cron and Schedule Builder options share the same robust validation framework. This ensures that, regardless of the method chosen for schedule creation, be it through Raw Cron or the Schedule Builder, you can trust in the accuracy of your schedule with peace of mind.

csharp
[OptiFlowWorkflow("second-workflow", "Second Test Workflow")]
[OptiFlowTag("owner", "digital-dev")]
[OptiFlowTag("scope", "metrics")]
internal partial class SecondWorkflow : IWorkflow
{
    // Create a schedule using the builder.
    public IOptiFlowSchedule CreateSchedule() => OptiFlowSchedule.EveryNMinutes(1);

    [OptiFlowOperation(1, true)]
    public async Task DoSomeCoolStuff(IWorkflowExecutionContext context)
    {
        await Task.CompletedTask;
    }
}
[OptiFlowWorkflow("second-workflow", "Second Test Workflow")]
[OptiFlowTag("owner", "digital-dev")]
[OptiFlowTag("scope", "metrics")]
internal partial class SecondWorkflow : IWorkflow
{
    // Create a schedule using the builder.
    public IOptiFlowSchedule CreateSchedule() => OptiFlowSchedule.EveryNMinutes(1);

    [OptiFlowOperation(1, true)]
    public async Task DoSomeCoolStuff(IWorkflowExecutionContext context)
    {
        await Task.CompletedTask;
    }
}