For-Comprehensions

Endless nested callbacks and unreadable chains of flatMap calls can be tiresome. The higher-kinded-j library brings the elegance and power of Scala-style for-comprehensions to Java, allowing you to write complex asynchronous and sequential logic in a way that is clean, declarative, and easy to follow.

Let's see how to transform "callback hell" into a readable, sequential script.

The "Pyramid of Doom" Problem

In functional programming, monads are a powerful tool for sequencing operations, especially those with a context like Optional, List, or CompletableFuture. However, chaining these operations with flatMap can quickly become hard to read.

Consider combining three Maybe values:

// The "nested" way
Kind<MaybeKind.Witness, Integer> result = maybeMonad.flatMap(a ->
    maybeMonad.flatMap(b ->
        maybeMonad.map(c -> a + b + c, maybeC),
    maybeB),
maybeA);

This code works, but the logic is buried inside nested lambdas. The intent (to simply get values from maybeA, maybeB, and maybeC and add them) is obscured. This is often called the "pyramid of doom."

For A Fluent, Sequential Builder

The For comprehension builder provides a much more intuitive way to write the same logic. It lets you express the sequence of operations as if they were simple, imperative steps.

Here’s the same example rewritten with the For builder:

import static org.higherkindedj.hkt.maybe.MaybeKindHelper.MAYBE;
import org.higherkindedj.hkt.expression.For;
// ... other imports

var maybeMonad = MaybeMonad.INSTANCE;
var maybeA = MAYBE.just(5);
var maybeB = MAYBE.just(10);
var maybeC = MAYBE.just(20);

// The clean, sequential way
var result = For.from(maybeMonad, maybeA)    // Get a from maybeA
    .from(a -> maybeB)                       // Then, get b from maybeB
    .from(t -> maybeC)                       // Then, get c from maybeC
    .yield((a, b, c) -> a + b + c);          // Finally, combine them

System.out.println(MAYBE.narrow(result)); // Prints: Just(35)

This version is flat, readable, and directly expresses the intended sequence of operations. The For builder automatically handles the flatMap and map calls behind the scenes.

Extended Arity Example (6+ Bindings)

The generated Steps2-Steps12 classes provide the same fluent API seamlessly:

var idMonad = IdMonad.instance();

Kind<IdKind.Witness, String> result =
    For.from(idMonad, Id.of("Alice"))       // a = "Alice"
        .let(name -> name.length())          // b = 5
        .from(t -> Id.of(t._1().toUpperCase())) // c = "ALICE"
        .let(t -> t._2() * 10)              // d = 50
        .let(t -> t._3() + "!")             // e = "ALICE!"
        .let(t -> t._1() + " has " + t._2() + " letters")  // f = summary
        .yield((name, len, upper, score, exclaimed, summary) ->
            summary + " (score: " + score + ")");

// Result: "Alice has 5 letters (score: 50)"

At higher arities, the tuple-style yield is especially convenient for accessing accumulated values by position:

.yield(t -> t._6() + " (score: " + t._4() + ")")

@GenerateLenses
record NameInfo(String name, int len, String upper, int score, String exclaimed, String summary) {}

Kind<IdKind.Witness, String> result =
    For.from(idMonad, Id.of("Alice"))
        .let(name -> name.length())
        .toState((name, len) -> new NameInfo(name, len, "", 0, "", ""))
        .fromThen(ctx -> Id.of(ctx.name().toUpperCase()),               upperLens)
        .fromThen(ctx -> Id.of(ctx.len() * 10),                        scoreLens)
        .fromThen(ctx -> Id.of(ctx.upper() + "!"),                     exclaimedLens)
        .fromThen(ctx -> Id.of(ctx.name() + " has " + ctx.len() + " letters"), summaryLens)
        .yield(ctx -> ctx.summary() + " (score: " + ctx.score() + ")");

// Same result: "Alice has 5 letters (score: 50)"

Core Operations of the For Builder

A for-comprehension is built by chaining four types of operations:

1. Generators: .from()

A generator is the workhorse of the comprehension. It takes a value from a previous step, uses it to produce a new monadic value (like another Maybe or List), and extracts the result for the next step. This is a direct equivalent of flatMap.

Each .from() adds a new variable to the scope of the comprehension.

// Generates all combinations of userLogin IDs and roles
var userRoles = For.from(listMonad, LIST.widen(List.of("userLogin-1", "userLogin-2"))) // a: "userLogin-1", "userLogin-2"
    .from(a -> LIST.widen(List.of("viewer", "editor")))       // b: "viewer", "editor"
    .yield((a, b) -> a + " is a " + b);

// Result: ["userLogin-1 is a viewer", "userLogin-1 is a editor", "userLogin-2 is a viewer", "userLogin-2 is a editor"]

2. Value Bindings: .let()

A .let() binding allows you to compute a pure, simple value from the results you've gathered so far and add it to the scope. It does not involve a monad. This is equivalent to a map operation that carries the new value forward.

var idMonad = IdMonad.instance();

var result = For.from(idMonad, Id.of(10))        // a = 10
    .let(a -> a * 2)                          // b = 20 (a pure calculation)
    .yield((a, b) -> "Value: " + a + ", Doubled: " + b);

// Result: "Value: 10, Doubled: 20"
System.out.println(ID.unwrap(result));

3. Guards: .when()

For monads that can represent failure or emptiness (like List, Maybe, or Optional), you can use .when() to filter results. If the condition is false, the current computational path is stopped by returning the monad's "zero" value (e.g., an empty list or Maybe.nothing()).

This feature requires a MonadZero instance. See the MonadZero documentation for more details.

var evens = For.from(listMonad, LIST.widen(List.of(1, 2, 3, 4, 5, 6)))
    .when(i -> i % 2 == 0) // Guard: only keep even numbers
    .yield(i -> i);

// Result: [2, 4, 6]

4. Projection: .yield()

Every comprehension ends with .yield(). This is the final map operation where you take all the values you've gathered from the generators and bindings and produce your final result. You can access the bound values as individual lambda parameters or as a single Tuple.

Turn the power up: StateT Example

• ForComprehensionExample.java

The true power of for-comprehensions becomes apparent when working with complex structures like monad transformers. A StateT over Optional represents a stateful computation that can fail. Writing this with nested flatMap calls would be extremely complex. With the For builder, it becomes a simple, readable script.

import static org.higherkindedj.hkt.optional.OptionalKindHelper.OPTIONAL;
import static org.higherkindedj.hkt.state_t.StateTKindHelper.STATE_T;
// ... other imports

private static void stateTExample() {
    final var optionalMonad = OptionalMonad.INSTANCE;
    final var stateTMonad = StateTMonad.<Integer, OptionalKind.Witness>instance(optionalMonad);

    // Helper: adds a value to the state (an integer)
    final Function<Integer, Kind<StateTKind.Witness<Integer, OptionalKind.Witness>, Unit>> add =
        n -> StateT.create(s -> optionalMonad.of(StateTuple.of(s + n, Unit.INSTANCE)), optionalMonad);

    // Helper: gets the current state as the value
    final var get = StateT.<Integer, OptionalKind.Witness, Integer>create(s -> optionalMonad.of(StateTuple.of(s, s)), optionalMonad);

    // This workflow looks like a simple script, but it's a fully-typed, purely functional composition!
    final var statefulComputation =
        For.from(stateTMonad, add.apply(10))      // Add 10 to state
            .from(a -> add.apply(5))              // Then, add 5 more
            .from(b -> get)                       // Then, get the current state (15)
            .let(t -> "The state is " + t._3())   // Compute a string from it
            .yield((a, b, c, d) -> d + ", original value was " + c); // Produce the final string

    // Run the computation with an initial state of 0
    final var resultOptional = STATE_T.runStateT(statefulComputation, 0);
    final Optional<StateTuple<Integer, String>> result = OPTIONAL.narrow(resultOptional);

    result.ifPresent(res -> {
        System.out.println("Final value: " + res.value());
        System.out.println("Final state: " + res.state());
    });
    // Expected Output:
    // Final value: The state is 15, original value was 15
    // Final state: 15
}

In this example, Using the For comprehension really helps hide the complexity of threading the state (Integer) and handling potential failures (Optional), making the logic clear and maintainable.

For a more extensive example of using the full power of the For comprehension head over to the Order Workflow

Similarities to Scala

If you're familiar with Scala, you'll recognise the pattern. In Scala, a for-comprehension looks like this:

for {
 a <- maybeA
 b <- maybeB
 if (a + b > 10)
 c = a + b
} yield c * 2

This is built in syntactic sugar that the compiler translates into a series of flatMap, map, and withFilter calls. The For builder in higher-kinded-j provides the same expressive power through a method-chaining API.

Working with Optics

The For builder provides two optic-aware operations that extend the comprehension capabilities:

Extracting Values with focus()

The focus() method extracts a value using a function and adds it to the accumulated tuple. This is particularly useful when working with nested data structures.

record User(String name, Address address) {}
record Address(String city, String street) {}

var users = List.of(
    new User("Alice", new Address("London", "Baker St")),
    new User("Bob", new Address("Paris", "Champs-Élysées"))
);

Kind<ListKind.Witness, String> result =
    For.from(listMonad, LIST.widen(users))
        .focus(user -> user.address().city())
        .yield((user, city) -> user.name() + " lives in " + city);

// Result: ["Alice lives in London", "Bob lives in Paris"]

The extracted value is accumulated alongside the original value, making both available in subsequent steps and in the final yield().

Pattern Matching with match()

The match() method provides prism-like pattern matching within comprehensions. When used with a MonadZero (such as List or Maybe), it short-circuits the computation if the match fails.

sealed interface Result permits Success, Failure {}
record Success(String value) implements Result {}
record Failure(String error) implements Result {}

Prism<Result, Success> successPrism = Prism.of(
    r -> r instanceof Success s ? Optional.of(s) : Optional.empty(),
    s -> s
);

List<Result> results = List.of(
    new Success("data1"),
    new Failure("error"),
    new Success("data2")
);

Kind<ListKind.Witness, String> successes =
    For.from(listMonad, LIST.widen(results))
        .match(successPrism)
        .yield((result, success) -> success.value().toUpperCase());

// Result: ["DATA1", "DATA2"] - failures are filtered out

With Maybe, failed matches short-circuit to Nothing:

Kind<MaybeKind.Witness, String> result =
    For.from(maybeMonad, MAYBE.just((Result) new Failure("error")))
        .match(successPrism)
        .yield((r, s) -> s.value());

// Result: Nothing - the match failed

For.from(listMonad, LIST.widen(items))
    .focus(item -> item.category())
    .match(premiumCategoryPrism)
    .when(t -> t._3().discount() > 0.1)
    .yield((item, category, premium) -> item.name());

Bulk Operations with ForTraversal

For operations over multiple elements within a structure, use ForTraversal. This provides a fluent API for applying transformations to all elements focused by a Traversal.

record Player(String name, int score) {}

List<Player> players = List.of(
    new Player("Alice", 100),
    new Player("Bob", 200)
);

Traversal<List<Player>, Player> playersTraversal = Traversals.forList();
Lens<Player, Integer> scoreLens = Lens.of(
    Player::score,
    (p, s) -> new Player(p.name(), s)
);

// Add bonus points to all players
Kind<IdKind.Witness, List<Player>> result =
    ForTraversal.over(playersTraversal, players, IdMonad.instance())
        .modify(scoreLens, score -> score + 50)
        .run();

List<Player> updated = IdKindHelper.ID.unwrap(result);
// Result: [Player("Alice", 150), Player("Bob", 250)]

Filtering Within Traversals

The filter() method preserves non-matching elements unchanged whilst applying transformations only to matching elements:

Kind<IdKind.Witness, List<Player>> result =
    ForTraversal.over(playersTraversal, players, IdMonad.instance())
        .filter(p -> p.score() >= 150)
        .modify(scoreLens, score -> score * 2)
        .run();

// Alice (100) unchanged, Bob (200) doubled to 400

Collecting Results

Use toList() to collect all focused elements:

Kind<IdKind.Witness, List<Player>> allPlayers =
    ForTraversal.over(playersTraversal, players, IdMonad.instance())
        .toList();

Bridging to ForState with toState()

When a workflow starts with a few monadic steps (fetching data, computing values) and then needs to thread named state through a series of updates, toState() lets you transition seamlessly from For into ForState mid-comprehension. The accumulated values become the constructor arguments for your state record, and from that point on you work with named fields and lenses instead of tuple positions.

record Dashboard(String user, int count, boolean ready) {}

Lens<Dashboard, Boolean> readyLens = Lens.of(
    Dashboard::ready, (d, v) -> new Dashboard(d.user(), d.count(), v));
Lens<Dashboard, Integer> countLens = Lens.of(
    Dashboard::count, (d, v) -> new Dashboard(d.user(), v, d.ready()));

// Start with For (value accumulation), then switch to ForState (named state)
Kind<IdKind.Witness, Dashboard> result =
    For.from(idMonad, Id.of("Alice"))               // a = "Alice"
        .from(name -> Id.of(name.length()))          // b = 5
        .toState((name, count) ->                    // bridge: construct record
            new Dashboard(name, count, false))
        .modify(countLens, c -> c * 10)              // named lens operation
        .update(readyLens, true)
        .yield();

// Dashboard("Alice", 50, true)

The toState() method is available at every arity (1 through 12) in both spread-style and tuple-style:

// Spread-style: arguments unpacked
.toState((name, count) -> new Dashboard(name, count, false))

// Tuple-style: single tuple argument
.toState(t -> new Dashboard(t._1(), t._2(), false))

When the comprehension uses a MonadZero (like Maybe or List), the returned builder is a ForState.FilterableSteps, preserving access to when() and matchThen() guards:

Kind<MaybeKind.Witness, Dashboard> result =
    For.from(maybeMonad, MAYBE.just("Alice"))
        .toState(name -> new Dashboard(name, 0, false))
        .when(d -> d.user().length() > 3)   // guard still available
        .update(readyLens, true)
        .yield();

// Just(Dashboard("Alice", 0, true))

Stateful Updates with ForState

For workflows with more than a few steps, tuple-based access becomes fragile. ForState solves this by threading a named record through each step, with lenses providing type-safe field access. Every intermediate value has a name, not a position.

// ForState: named fields instead of tuple positions
ForState.withState(monad, monad.of(initialContext))
    .fromThen(ctx -> validateOrder(ctx.orderId()),   validatedLens)
    .fromThen(ctx -> processPayment(ctx),            confirmationLens)
    .when(ctx -> ctx.totalCents() > 0)               // guard (MonadZero)
    .zoom(addressLens)                                // narrow scope
        .update(cityLens, "SPRINGFIELD")
    .endZoom()
    .yield(ctx -> buildReceipt(ctx.user(), ctx.confirmationId()));

ForState supports the full range of comprehension operations: pure updates (update, modify), effectful operations (from, fromThen), guards (when), pattern matching (matchThen), bulk traversal (traverse), and scope narrowing (zoom/endZoom).

Position-Aware Traversals with ForIndexed

ForIndexed extends traversal comprehensions to include index/position awareness, enabling transformations that depend on element position within the structure.

IndexedTraversal<Integer, List<Player>, Player> indexedPlayers =
    IndexedTraversals.forList();

List<Player> players = List.of(
    new Player("Alice", 100),
    new Player("Bob", 200),
    new Player("Charlie", 150)
);

// Add position-based bonus (first place gets more)
Kind<IdKind.Witness, List<Player>> result =
    ForIndexed.overIndexed(indexedPlayers, players, IdMonad.instance())
        .modify(scoreLens, (index, score) -> score + (100 - index * 10))
        .run();

// Alice: 100 + 100 = 200
// Bob: 200 + 90 = 290
// Charlie: 150 + 80 = 230

Filtering by Position

Use filterIndex() to focus only on specific positions:

// Only modify top 3 players
ForIndexed.overIndexed(indexedPlayers, players, idApplicative)
    .filterIndex(i -> i < 3)
    .modify(scoreLens, (i, s) -> s * 2)
    .run();

Combined Index and Value Filtering

Use filter() with a BiPredicate to filter based on both position and value:

ForIndexed.overIndexed(indexedPlayers, players, idApplicative)
    .filter((index, player) -> index < 5 && player.score() > 100)
    .modify(scoreLens, (i, s) -> s + 50)
    .run();

Collecting with Indices

Use toIndexedList() to collect elements along with their indices:

Kind<IdKind.Witness, List<Pair<Integer, Player>>> indexed =
    ForIndexed.overIndexed(indexedPlayers, players, idApplicative)
        .toIndexedList();

// Result: [Pair(0, Player("Alice", 100)), Pair(1, Player("Bob", 200)), ...]

Previous: Natural Transformation | Next: ForState: Named State Comprehensions