Architecture
Every workflow you design passes through a six-stage compiler pipeline that transforms visual nodes into a production-grade distributed DAG.
Kahn's algorithm catches cyclic dependencies at compile time, not runtime. Invalid topologies are rejected before any resources are consumed.
Spring auto-discovers NodeCompiler implementations. Adding a new node type means writing one class and zero configuration changes.
User expressions are compiled once and serialized into the production DAG. Per-record evaluation uses the pre-compiled form for maximum throughput.
Dual-Mode Builder
Business analysts use Canvas mode with physics-based interactions. Data engineers use Studio mode with precision DAG editing. Both compile to the same production pipeline.
Drag, drop, and connect nodes with Matter.js physics. Nodes have weight, connections have tension. Designed for business analysts who think visually.
ReactFlow-powered directed graph editor with node palette, property inspector, and real-time validation. Built for data engineers who need precision.
Node Catalog
Every node is a compiled, serializable pipeline citizen. Drag it onto the canvas, configure it, and deploy.
Magister AI Assistant
Natural language to production pipeline in seconds. The Magister AI Assistant uses MCP tools to generate, explain, and optimize WorkflowDefinitions directly.
"Build a fraud detection pipeline that reads from Kafka, classifies with GPT-4o, and routes high-risk to an alert queue."
Ask "What does this pipeline do?" and get a structured breakdown of every node, connection, and data transformation.
The AI Assistant analyzes your flow and suggests optimizations: better window sizes, expression caching, parallel execution paths.
Build me a sentiment analysis pipeline. Read customer reviews from a CSV file, analyze sentiment with AI, and store results in an IMap grouped by sentiment score.
I've generated a 4-node pipeline:
The pipeline is ready on your canvas. Shall I deploy it?
Yes, deploy it.
Job ID: 7f3a1b2c | Status: RUNNING
Monitoring & Observability
Live dashboards, DAG visualization, and IMap data inspector. Monitor pipeline health in real time via SSE streaming.
Real-time job status, throughput, and error rates. SSE-powered updates without polling. See every job state change the moment it happens.
Visualize the compiled DAG as an interactive SVG. See how your visual nodes translate to distributed execution vertices.
Browse the contents of any IMap directly from the Studio. Search, sort, and export key-value data produced by your pipelines.
Connectors
First-class connectors for streaming, databases, APIs, and file systems. Each compiled natively into the production DAG.
Source & sink. Topics, consumer groups, Redpanda compatible.
JDBC source & sink. pgvector for embeddings.
Document source & sink. Change streams supported.
HTTP polling source & webhook sink. Any endpoint.
CSV, JSON, XML. File source, file watcher, file sink.
AWS S3, MinIO. Read and write objects to buckets.
Email inbox source via Camel IMAP connector.
In-memory key-value store. Source, sink, and journal.
Under the Hood
Production-grade technology stack designed for real-time AI at enterprise scale.
Distributed stream processing engine. Sub-10ms latency, 100K+ events/sec per node, exactly-once semantics. Runs embedded in the platform.
Records for immutable schemas, virtual threads for concurrent operations, pattern matching for cleaner code. Spring Boot 3 for dependency injection and lifecycle management.
Kahn's algorithm for topological sorting. Pluggable NodeCompiler registry. Visual workflows compile to distributed in-memory DAGs with cycle detection and validation.
Compile-once, execute-per-record expression evaluation. Sandboxed execution with configurable timeouts. Safe user-defined predicates without raw Java compilation.
Zustand for state management with Immer middleware. ReactFlow for DAG visualization. Dual-mode interface: engineering workbench and physics-based canvas.
Built-in AI assistant for pipeline generation via natural language. Model Context Protocol server enables integration with any AI client. 14 purpose-built AI nodes.
See the full platform in action with your data. Book a 30-minute demo or jump straight into the live Studio.