Quick Reference
You've walked through nine pipeline stages, fourteen HKJ features, and enough concurrent
plumbing to make a thread pool blush. This page is the one you bookmark. Come back here
when you're adapting the pipeline to your own domain, when you can't remember whether you
need parEvalMap or parEvalMapUnordered, or when you just want to find the right source
file without scrolling through three chapters.
HKJ Features Used
| Stage | HKJ Feature | What It Does |
|---|---|---|
| Generate | VStream.unfold | Stateful infinite stream from a seed |
| Merge | VStreamPar.merge | Concurrent multi-source merging with virtual threads |
| Enrich | Par.map2 | Run two VTasks in parallel, combine results |
| Enrich | VStreamPar.parEvalMap | Bounded parallel map preserving order |
| Risk | VStreamPar.parEvalMapUnordered | Bounded parallel map in completion order |
| Window | VStream.chunk | Group elements into fixed-size batches |
| Aggregate | VStream.map | Transform each chunk into a summary |
| Detect | VStream.flatMap | Each element produces zero or more results |
| Dispatch | VStream.mapTask | Apply effectful function per element |
| Dispatch | Scope.allSucceed | Fork/join with all-must-succeed semantics |
| Throttle | VStreamThrottle.throttle | Rate-limit emissions |
| Recover | VStream.recoverWith | Switch to fallback stream on failure |
| Protect | CircuitBreaker | Trip open after repeated failures |
| Limit | VStream.take | Safety valve: caps total elements |
Which Operator Should I Pick?
The feature table tells you what's available. This section tells you when to reach for each one.
I need to run two independent tasks and combine their results.
Use Par.map2. It forks both into virtual threads and joins. If either fails, the scope
cancels the other. This is what the enrichment stage does with its reference-data and
FX-rate lookups.
I need to apply an effectful function to every element in a stream.
If downstream order matters (e.g. enrichment feeds into risk assessment), use
parEvalMap. It processes elements concurrently but emits them in the original order.
If order does not matter and you want maximum throughput (e.g. independent risk
calculations), use parEvalMapUnordered. It emits results as they complete.
I need to fan out to multiple channels and all must succeed.
Use Scope.allSucceed with fork for each channel and join to wait. If any channel
fails, the scope cancels the rest. This is the alert dispatch pattern.
I need to handle a source that might fail.
Use recoverWith to switch to a fallback stream on error. Wrap the source in a
CircuitBreaker if you want to fail fast after repeated errors rather than retrying
a known-broken source.
I need to combine multiple live sources into a single stream.
Use VStreamPar.merge. It forks one virtual thread per source and interleaves elements
by arrival order. Do not use concat; that would drain the first source completely before
starting the second.
Architecture Decision Summary
| Decision | Choice | Rationale |
|---|---|---|
| Tick generation | unfold over iterate | Need S → (A, S) where state differs from output |
| Feed combination | merge over concat | Live feeds must interleave by arrival time |
| Lookup parallelism | Par.map2 over flatMap | Two independent lookups: 50ms vs 100ms |
| Stream parallelism | parEvalMap for enrichment | Order matters for downstream processing |
| Risk parallelism | parEvalMapUnordered | Order irrelevant, maximise throughput |
| Windowing | chunk (fixed-size) | Deterministic, simple; time-based is a config change |
| Detection | flatMap over map + filter | Natural 0-to-many mapping per view |
| Alert dispatch | Scope.allSucceed | All-or-nothing: every channel must confirm |
| Feed failover | recoverWith over retry | Outages are sustained, not transient |
Source Files
Running
Demo (all 9 steps with output)
./gradlew :hkj-examples:run \
-PmainClass=org.higherkindedj.example.market.runner.MarketDataDemo
Tests
./gradlew :hkj-examples:test --tests "*.MarketDataPipelineTest"
Related Documentation
| Topic | Link |
|---|---|
| VStream | Core pull-based streaming type |
| VStream Parallel | parEvalMap, merge, and concurrency |
| VStream Advanced | recoverWith, recover, onFinalize |
| VTask | Virtual thread effect type |
| Structured Concurrency | Scope, Par, StructuredTaskScope |
| Resilience | Retry, Circuit Breaker, Bulkhead, Saga |
| Order Processing | Another complete application example |
XKCD 2347 — Dependency. Now you know why Step 8 exists.
The entire pipeline (nine stages, fourteen features, three exchange feeds) composes into a single expression that a new team member can read top to bottom. Every design decision is explicit in the code and documented in the table above. When you adapt this to your own domain, you will not be copying a framework; you will be reusing a vocabulary of composable operations that each do one thing well.
That's the point of higher-kinded Java: not the type theory, but the fact that your Monday morning is a little less terrifying.
Previous: Alerts and Resilience | Up: Market Data Pipeline