# Quick Reference
You've walked through the nine stages of the [Market Data Pipeline](examples_market_data.md), fourteen HKJ features, and enough concurrent
plumbing to make a thread pool blush. This page is the one you bookmark. Come back here
when you're adapting the pipeline to your own domain, when you can't remember whether you
need `parEvalMap` or `parEvalMapUnordered`, or when you just want to find the right source
file without scrolling through three chapters.
---
## HKJ Features Used
| Stage | HKJ Feature | What It Does |
|-------|-------------|--------------|
| **Generate** | `VStream.unfold` | Stateful infinite stream from a seed |
| **Merge** | `VStreamPar.merge` | Concurrent multi-source merging with virtual threads |
| **Enrich** | `Par.map2` | Run two VTasks in parallel, combine results |
| **Enrich** | `VStreamPar.parEvalMap` | Bounded parallel map preserving order |
| **Risk** | `VStreamPar.parEvalMap` | Bounded parallel map preserving order (windowing needs it) |
| **Window** | `VStream.chunk` | Group elements into fixed-size batches |
| **Aggregate** | `VStream.map` | Transform each chunk into a summary |
| **Detect** | `VStream.flatMap` | Each element produces zero or more results |
| **Dispatch** | `VStream.mapTask` | Apply effectful function per element |
| **Dispatch** | `Scope.allSucceed` | Fork/join with all-must-succeed semantics |
| **Throttle** | `VStreamThrottle.throttle` | Rate-limit emissions |
| **Recover** | `VStream.recoverWith` | Switch to fallback stream on failure |
| **Protect** | `CircuitBreaker` | Trip open after repeated failures |
| **Limit** | `VStream.take` | Safety valve: caps total elements |
---
## Which Operator Should I Pick?
The feature table tells you *what's available*. This section tells you *when to reach
for each one*.
**I need to run two independent tasks and combine their results.**
Use `Par.map2`. It forks both into virtual threads and joins. If either fails, the scope
cancels the other. This is what the enrichment stage does with its reference-data and
FX-rate lookups.
**I need to apply an effectful function to every element in a stream.**
If downstream order matters, use `parEvalMap`. It processes elements concurrently but emits
them in the original order: the pipeline uses it for both enrichment *and* risk assessment,
because the risk results feed a count-based windower (`chunk`) that groups ticks by position.
If order genuinely does not matter and you want maximum throughput, use
`parEvalMapUnordered`. It emits results as they complete.
**I need to fan out to multiple channels and all must succeed.**
Use `Scope.allSucceed` with `fork` for each channel and `join` to wait. If any channel
fails, the scope cancels the rest. This is the alert dispatch pattern.
**I need to handle a source that might fail.**
Use `recoverWith` to switch to a fallback stream on error. Wrap the source in a
`CircuitBreaker` if you want to fail fast after repeated errors rather than retrying
a known-broken source.
**I need to combine multiple live sources into a single stream.**
Use `VStreamPar.merge`. It forks one virtual thread per source and interleaves elements
by arrival order. Do not use `concat`; that would drain the first source completely before
starting the second.
---
## Architecture Decision Summary
| Decision | Choice | Rationale |
|----------|--------|-----------|
| Tick generation | `unfold` over `iterate` | Need `S → (A, S)` where state differs from output |
| Feed combination | `merge` over `concat` | Live feeds must interleave by arrival time |
| Lookup parallelism | `Par.map2` over `flatMap` | Two independent lookups: 50ms vs 100ms |
| Stream parallelism | `parEvalMap` for enrichment | Order matters for downstream processing |
| Risk parallelism | `parEvalMap` | Windowing groups ticks by position, so order matters |
| Windowing | `chunk` (fixed-size) | Deterministic, simple; time-based is a config change |
| Detection | `flatMap` over `map` + `filter` | Natural 0-to-many mapping per view |
| Alert dispatch | `Scope.allSucceed` | All-or-nothing: every channel must confirm |
| Feed failover | `recoverWith` over retry | Outages are sustained, not transient |
---
## Source Files
---
## Running
### Demo (all 9 steps with output)
```bash
./gradlew :hkj-examples:run \
-PmainClass=org.higherkindedj.example.market.runner.MarketDataDemo
```
### Tests
```bash
./gradlew :hkj-examples:test --tests "*.MarketDataPipelineTest"
```
---
## Related Documentation
| Topic | Link |
|-------|------|
| **VStream** | [Core pull-based streaming type](../monads/vstream.md) |
| **VStream Parallel** | [parEvalMap, merge, and concurrency](../monads/vstream_parallel.md) |
| **VStream Advanced** | [recoverWith, recover, onFinalize](../monads/vstream_advanced.md) |
| **VTask** | [Virtual thread effect type](../monads/vtask_monad.md) |
| **Structured Concurrency** | [Scope, Par, StructuredTaskScope](../monads/vtask_scope.md) |
| **Resilience** | [Retry, Circuit Breaker, Bulkhead, Saga](../resilience/ch_intro.md) |
| **Order Processing** | [Another complete application example](examples_order.md) |
---
XKCD 2347: Dependency. Now you know why Step 8 exists.
---
The entire pipeline (nine stages, fourteen features, three exchange feeds) composes
into a single expression that a new team member can read top to bottom. Every design
decision is explicit in the code and documented in the table above. When you adapt this
to your own domain, you will not be copying a framework; you will be reusing a vocabulary
of composable operations that each do one thing well.
That's the point of higher-kinded Java: not the type theory, but the fact that your
Monday morning is a little less terrifying.
---
**Previous:** [Alerts and Resilience](market_alerts.md)
**Next:** [Portfolio Risk Analysis](examples_portfolio_risk.md)