Building a Workflow System with HTMX and ASP.NET Core

Introduction

Workflow systems are everywhere in modern applications - from simple approval processes to complex multi-step automations. While there are excellent commercial solutions like Temporal, Airflow, and n8n, sometimes you need something tailored to your specific needs. In this series, I'll walk you through building a complete workflow system from scratch using ASP.NET Core, HTMX, Alpine.js, and Hangfire.

By the end of this series, you'll have a fully functional workflow system with:

A visual node-based editor for building workflows
JSON-defined workflow nodes with inputs, outputs, and conditions
A custom workflow execution engine
Background job processing with Hangfire for polling remote APIs
State management and persistence
A dashboard for monitoring workflow execution

Why Build Your Own Workflow System?

Before diving in, let's address the elephant in the room: why not use an existing solution?

When to Build Your Own:

You need deep integration with your existing application
You have specific business logic that doesn't fit into generic workflow tools
You want full control over the execution model and scaling
You need to minimize external dependencies
You want to learn how workflow systems work under the hood

When to Use Existing Tools:

You need a proven, battle-tested solution immediately
You require features like distributed execution across many machines
You don't have time to build and maintain custom infrastructure

For this series, we're choosing to build our own because it gives us complete flexibility and is an excellent learning opportunity.

What We're Building

Our workflow system will support several key concepts:

1. Nodes

Nodes are the building blocks of workflows. Each node represents a discrete action or decision point. Examples:

HTTP Request Node: Makes API calls to external services
Condition Node: Evaluates expressions and branches execution
Transform Node: Processes data (JSON manipulation, filtering, etc.)
Delay Node: Waits for a specified duration
Trigger Node: Starts a workflow based on events or schedules

2. Workflows

A workflow is a directed graph of connected nodes. Data flows from one node to another through connections, with each node processing and transforming the data as needed.

3. Execution Engine

The engine that interprets the workflow definition and executes nodes in the correct order, handling:

Sequential execution
Parallel execution (when nodes don't depend on each other)
Conditional branching
Error handling and retries
State persistence

4. Visual Editor

A web-based UI where users can:

Drag and drop nodes onto a canvas
Connect nodes by drawing edges between them
Configure node properties
Test workflows
View execution history

5. Background Processing

Using Hangfire, we'll implement:

Scheduled workflow execution
API polling for external data
Retry logic for failed executions
Dashboard for monitoring jobs

Technology Stack

Here's what we'll be using:

Backend

ASP.NET Core 9.0: Our web framework
Entity Framework Core: For data persistence
PostgreSQL: Database for storing workflow definitions and execution state
Hangfire: Background job processing
System.Text.Json: For workflow JSON serialization

Frontend

HTMX 2.0: For dynamic, server-driven UI updates without writing JavaScript
Alpine.js: Lightweight JavaScript framework for interactive components
TailwindCSS + DaisyUI: Styling and UI components
LeaderLine: For drawing connections between nodes (we'll explore alternatives)

Infrastructure

Docker: For containerization
Seq: For structured logging
Prometheus + Grafana: For monitoring

Architecture Overview

Our system will be organized into several projects:

Mostlylucid.Workflow/
├── Mostlylucid.Workflow.Engine/      # Core workflow execution engine
│   ├── Models/                        # Node, Workflow, Connection models
│   ├── Execution/                     # Workflow executor and runtime
│   └── Nodes/                         # Built-in node implementations
├── Mostlylucid.Workflow.Shared/       # Shared models and DTOs
├── Mostlylucid.Workflow.DbContext/    # Entity Framework context
└── Mostlylucid/                       # Main web app (existing)
    ├── Controllers/WorkflowController.cs
    ├── Services/WorkflowService.cs
    └── Views/Workflow/                # Workflow UI views

Key Design Decisions

1. JSON-Based Node Definitions Nodes will be defined in JSON, making them easy to serialize, version, and share:

{
  "id": "node-1",
  "type": "HttpRequest",
  "name": "Fetch User Data",
  "inputs": {
    "url": "https://api.example.com/users/{{userId}}",
    "method": "GET",
    "headers": {
      "Authorization": "Bearer {{apiToken}}"
    }
  },
  "outputs": {
    "response": "{{result.body}}",
    "statusCode": "{{result.statusCode}}"
  },
  "conditions": {
    "onSuccess": "next-node-id",
    "onError": "error-handler-node-id"
  }
}

2. Graph-Based Execution Workflows are directed acyclic graphs (DAGs). The execution engine will:

Perform topological sorting to determine execution order
Execute nodes in parallel where possible
Handle conditional branching based on node outputs

3. State Persistence Every workflow execution will be tracked:

Workflow instance state (Running, Completed, Failed)
Node execution results
Trigger states for scheduled workflows
Execution history and logs

4. Extensibility The node system will be plugin-based:

Built-in nodes for common operations
Custom node types can be registered
Node validation and schema enforcement

What's Coming in This Series

Part 2: Architecture and Core Workflow Engine

Setting up the project structure
Building the core workflow models (Node, Workflow, Connection)
Implementing the graph execution engine
Topological sorting and dependency resolution

Part 3: Building the Visual Editor

Creating the workflow canvas with HTMX and Alpine.js
Drag-and-drop node placement
Visual connection drawing
Node configuration panel
Workflow serialization

Part 4: Hangfire Integration and State Management

Setting up Hangfire for background processing
Implementing API polling nodes
Trigger state management
Retry logic and error handling
Dashboard integration

Part 5: Advanced Features and Deployment

Implementing custom nodes
Workflow versioning
Testing strategies
Performance optimization
Docker deployment

Prerequisites

To follow along with this series, you should have:

.NET 9.0 SDK installed
Basic knowledge of ASP.NET Core MVC
Familiarity with Entity Framework Core
Understanding of HTML, CSS, and basic JavaScript
Docker (optional, for running dependencies)

Getting Started

In the next post, we'll dive straight into building the core workflow engine. We'll create the models, implement the execution engine, and write our first workflow that can be executed programmatically.

This is going to be a fun journey! We'll build something genuinely useful while learning about graph algorithms, background processing, and modern web development patterns.

Conclusion

Building a workflow system from scratch is an ambitious project, but it's also incredibly rewarding. You'll gain deep insights into how modern automation platforms work, and you'll have a system that's perfectly tailored to your needs.

In Part 2, we'll start coding! We'll set up the project structure and build the core workflow engine that can execute our node-based workflows.

Stay tuned!