For-Comprehensions

Write a chain of flatMap calls as a list of steps, with filters, independent steps and traversals.

What You'll Learn

  • Turn nested flatMap calls into steps with For.from, from, let and yield
  • Filter a chain with when, given a MonadZero
  • Mark independent steps with par, and say when they run at the same time
  • Traverse a collection inside a chain with traverse, sequence and flatTraverse
  • Choose between For, ForPath and ForState

See Example Code

The code on this page is ForBook.java: the page includes it, so the build compiles and runs it. ForBookTest.java holds what the page says each step does.


When nested flatMap calls get deep

A receipt needs three things in turn: the order, its customer's verified email, and a confirmation from charging the order's total. Each step needs what an earlier step found, so Monad's flatMap chains them. With one Optional you would nest flatMap calls inside one another, and the same happens over Kind.

For one container, an Effect Path chains the same steps with ForPath, such as ForPath.from(eitherPath). Application code stays on the Paths. This page is for For, a chain written once for any container that has a Monad. Written as nested calls, the receipt is:

  // One receipt for any container F, as nested flatMap calls
  public static <F extends WitnessArity<TypeArity.Unary>> Kind<F, String> receiptNested(
      Monad<F> monad,
      Kind<F, Order> found,
      Function<Customer, Kind<F, EmailAddress>> verify,
      Function<BigDecimal, Kind<F, String>> charge) {
    return monad.flatMap(
        order -> {
          BigDecimal total = total(order);
          return monad.flatMap(
              email ->
                  monad.map(
                      confirmation ->
                          "receipt " + confirmation + " for " + total + " to " + email.value(),
                      charge.apply(total)),
              verify.apply(order.customer()));
        },
        found);
  }

The receipt is in the innermost lambda, and total has to be a local variable so that a later step can still see it. A For comprehension writes the same chain as a list of steps.

The code on this page uses these imports, with Order from the order records the Optics chapter introduces:

import static org.higherkindedj.hkt.either.EitherKindHelper.EITHER;
import static org.higherkindedj.hkt.instances.Witnesses.either;
import static org.higherkindedj.hkt.instances.Witnesses.list;
import static org.higherkindedj.hkt.instances.Witnesses.maybe;
import static org.higherkindedj.hkt.list.ListKindHelper.LIST;
import static org.higherkindedj.hkt.maybe.MaybeKindHelper.MAYBE;
import static org.higherkindedj.hkt.validated.ValidatedKindHelper.VALIDATED;

import java.math.BigDecimal;
import java.util.List;
import java.util.UUID;
import java.util.function.Function;
import org.higherkindedj.example.book.optics.cast.Customer;
import org.higherkindedj.example.book.optics.cast.EmailAddress;
import org.higherkindedj.example.book.optics.cast.LineItem;
import org.higherkindedj.example.book.optics.cast.Order;
import org.higherkindedj.example.book.optics.cast.OrderStatus;
import org.higherkindedj.hkt.Kind;
import org.higherkindedj.hkt.Monad;
import org.higherkindedj.hkt.MonadZero;
import org.higherkindedj.hkt.Semigroups;
import org.higherkindedj.hkt.TypeArity;
import org.higherkindedj.hkt.WitnessArity;
import org.higherkindedj.hkt.either.Either;
import org.higherkindedj.hkt.either.EitherKind;
import org.higherkindedj.hkt.expression.For;
import org.higherkindedj.hkt.instances.Instances;
import org.higherkindedj.hkt.list.ListKind;
import org.higherkindedj.hkt.list.ListTraverse;
import org.higherkindedj.hkt.maybe.Maybe;
import org.higherkindedj.hkt.maybe.MaybeKind;
import org.higherkindedj.hkt.validated.Validated;
import org.higherkindedj.hkt.validated.ValidatedKind;

Steps: from, let and yield

For.from(monad, source) starts the chain, and each later step adds values, filters the chain, or ends it:

StepWhat it doesBuilt on
from(f)Adds the value inside the container f gives backflatMap
let(f)Adds a plain value worked out from the values so farmap
when(test)Gives the instance's empty value when test failsMonadZero's filter
par(f1, f2)Adds two to four values that do not need one anotherflatMap, then map2 to map4
yield(f)Ends the chain with a result made from every valuemap

The first step's function receives the value bound so far. From the second value on, a step receives a tuple holding the values in the order they were bound, read as t._1() for the first, t._2() for the second, and so on. yield takes either the tuple or one parameter for each value. Written with For, the receipt reads top to bottom, and in it t._1() is the order and t._2() the total that let added:

  // The same receipt, as a For comprehension: each step reads what the steps before it bound
  public static <F extends WitnessArity<TypeArity.Unary>> Kind<F, String> receipt(
      Monad<F> monad,
      Kind<F, Order> found,
      Function<Customer, Kind<F, EmailAddress>> verify,
      Function<BigDecimal, Kind<F, String>> charge) {
    return For.from(monad, found)
        .let(order -> total(order))
        .from(t -> verify.apply(t._1().customer()))
        .from(t -> charge.apply(t._2()))
        .yield(
            (order, total, email, confirmation) ->
                "receipt " + confirmation + " for " + total + " to " + email.value());
  }

Run it with Either's Monad, once with an email that is verified and once with one that is not. ORDER is any order of two lines, totalling 50.00:

    Monad<EitherKind.Witness<String>> eitherMonad = Instances.monad(either());
    Function<BigDecimal, Kind<EitherKind.Witness<String>, String>> charge =
        amount -> EITHER.widen(Either.right("PAY-1"));

    Either<String, String> sent =
        EITHER.narrow(
            receipt(
                eitherMonad,
                EITHER.widen(Either.right(ORDER)),
                customer -> EITHER.widen(Either.right(customer.email())),
                charge));
    // Right(receipt PAY-1 for 50.00 to ada@example.com)

    Either<String, String> unverified =
        EITHER.narrow(
            receipt(
                eitherMonad,
                EITHER.widen(Either.right(ORDER)),
                customer -> EITHER.widen(Either.left("email not verified")),
                charge));
    // Left(email not verified)

What happens when a step has no value is the instance's, as on the Monad page: Either's stops at the Left, and the charge never runs.

Filtering with when

when needs a MonadZero: a Monad with an empty value, such as an empty list or Nothing, to give back when a test fails. The type you start the chain from decides. Instances.monadZero(list()) gives a chain with when, and Instances.monad(list()) gives one without:

For.from(Instances.monad(list()), orders).when(order -> order.status() == OrderStatus.SHIPPED);

javac refuses it with "cannot find symbol", because a chain started from a plain Monad has no when. The instance decides what the empty value is: List's drops the element, and Maybe's gives Nothing:

    MonadZero<ListKind.Witness> listMonad = Instances.monadZero(list());
    List<UUID> shippedIds =
        LIST.narrow(
            For.from(listMonad, LIST.widen(List.of(ORDER, SHIPPED)))
                .when(order -> order.status() == OrderStatus.SHIPPED)
                .yield(Order::id));
    // [00000000-0000-0000-0000-000000000002]

    MonadZero<MaybeKind.Witness> maybeMonad = Instances.monadZero(maybe());
    Maybe<UUID> notShipped =
        MAYBE.narrow(
            For.from(maybeMonad, MAYBE.just(ORDER))
                .when(order -> order.status() == OrderStatus.SHIPPED)
                .yield(Order::id));
    // Nothing

Independent steps with par

par binds values that do not need one another. For.par(monad, …) starts a chain with two to five of them, and a step's par(f1, …) fans out two to four from the values bound before it. Its result is an ordinary step, so from, let, when or another par can follow.

par combines its values with the instance's map2 to map5, where from uses flatMap. So it does what the instance's Applicative does: with Validated's instance, par reports both errors, and from stops at the first. The step's par fans out from the order:

    Monad<ValidatedKind.Witness<List<String>>> validatedMonad =
        Instances.validated(Semigroups.list());
    Kind<ValidatedKind.Witness<List<String>>, String> blankSku =
        VALIDATED.widen(Validated.invalid(List.of("SKU is blank")));
    Kind<ValidatedKind.Witness<List<String>>, Integer> noQuantity =
        VALIDATED.widen(Validated.invalid(List.of("quantity must be positive")));

    Validated<List<String>, String> independent =
        VALIDATED.narrow(
            For.par(validatedMonad, blankSku, noQuantity)
                .yield((sku, quantity) -> quantity + " x " + sku));
    // Invalid([SKU is blank, quantity must be positive])

    Validated<List<String>, String> inSequence =
        VALIDATED.narrow(
            For.from(validatedMonad, blankSku)
                .from(sku -> noQuantity)
                .yield((sku, quantity) -> quantity + " x " + sku));
    // Invalid([SKU is blank])

    Either<String, String> fannedOut =
        EITHER.narrow(
            For.from(eitherMonad, EITHER.widen(Either.right(ORDER)))
                .par(
                    order -> EITHER.widen(Either.right(order.customer().email())),
                    order -> EITHER.widen(Either.right(total(order))))
                .yield((order, email, total) -> total + " to " + email.value()));
    // Right(50.00 to ada@example.com)

par says the values are independent, and it runs them at the same time only where the instance's map2 does. No instance's map2 starts its values together: IO's and VTask's run them one after the other, and CompletableFuture's waits for futures that may already be running. To run two VTasks at the same time, use ForPath.par on two VTaskPaths, which calls Par.map2, or call Par.map2 yourself.

Traversing inside a chain

A step can run an effect on every element of a collection the chain holds, and bind the collected results:

StepWhat it bindsBuilt on
traverse(traversable, extractor, f)f run on every element, the results collectedTraverse's traverse
sequence(traversable, extractor)A collection of containers, turned into one container of a collectionTraverse's sequenceA, which is traverse with the identity function
flatTraverse(traversable, innerMonad, extractor, f)f run on every element, the nested collections flattenedtraverse, then the collection's own flatMap

traversable is the collection's Traverse, such as ListTraverse.INSTANCE, and innerMonad the collection's own Monad. extractor picks the collection out of the values bound so far; here the chain starts from it, so it is lines -> lines. LIST.widen puts the list in, and .map(LIST::narrow) takes the result out. priceInStock prices a line, or gives Nothing for a line with no stock, so one missing line makes the whole result Nothing. With maybeMonad from the when example:

    Function<LineItem, Kind<MaybeKind.Witness, BigDecimal>> priceInStock =
        line -> line.quantity() > 0 ? MAYBE.just(lineTotal(line)) : MAYBE.nothing();

    Maybe<List<BigDecimal>> priced =
        MAYBE
            .narrow(
                For.from(maybeMonad, MAYBE.just(LIST.widen(ORDER.lines())))
                    .traverse(ListTraverse.INSTANCE, lines -> lines, priceInStock)
                    .yield((lines, prices) -> prices))
            .map(LIST::narrow);
    // Just([40.00, 10.00])

    Maybe<List<BigDecimal>> backordered =
        MAYBE
            .narrow(
                For.from(maybeMonad, MAYBE.just(LIST.widen(BACKORDERED.lines())))
                    .traverse(ListTraverse.INSTANCE, lines -> lines, priceInStock)
                    .yield((lines, prices) -> prices))
            .map(LIST::narrow);
    // Nothing

    Maybe<List<Integer>> quantities =
        MAYBE
            .narrow(
                For.from(maybeMonad, MAYBE.just(LIST.widen(List.of(MAYBE.just(1), MAYBE.just(4)))))
                    .sequence(ListTraverse.INSTANCE, items -> items)
                    .yield((items, values) -> values))
            .map(LIST::narrow);
    // Just([1, 4])

    Maybe<List<Integer>> flattened =
        MAYBE
            .narrow(
                For.from(maybeMonad, MAYBE.just(LIST.widen(List.of(1, 2))))
                    .flatTraverse(
                        ListTraverse.INSTANCE,
                        Instances.monad(list()),
                        numbers -> numbers,
                        n -> MAYBE.just(LIST.widen(List.of(n, n * 10))))
                    .yield((numbers, flat) -> flat))
            .map(LIST::narrow);
    // Just([1, 10, 2, 20])

Which comprehension should you use?

You are writingUse
Application code over one Effect PathForPath, which takes and gives back Paths
A method for any container, over KindFor, on this page
A chain whose values are easier to name than to numberForState, with named fields in a record, or toState from For, as ForState's two ways in shows
Steps that read a field, convert a value or match a casefocus, through and match, as For with Optics shows: through works on the first step only, and match needs a MonadZero

A monad transformer, such as StateT, is a Monad over Kind, so For chains it as it does any other: the StateT page shows one.

Things people get wrong

Common Misunderstandings

  • "par runs the steps at the same time." It marks them as independent, and the instance's map2 decides how they run. ForPath.par on VTaskPath is the one that runs them at once.
  • "List has when, so my chain does." Only a chain started from a MonadZero, such as Instances.monadZero(list()), has when. Started from Instances.monad(list()), it has none.
  • "For is Java's version of Scala's for." It is a library builder, with no compiler support. Each step is a method call you can read in the source.

You can ship now

You can now write a chain of dependent steps once for any container, as receipt is, with plain values between the steps. You can filter it with when, mark independent steps with par knowing what each instance does with them, and traverse a collection inside it. The rest of this page is for how many values a chain holds, and for readers who know Scala's for.

Checkpoint: when on a MonadZero held as a Monad

Maybe's instance is a MonadZero. Why does this chain not compile, and what one change gives it when?

Monad<MaybeKind.Witness> monad = Instances.monadZero(maybe());
For.from(monad, found).when(order -> order.status() == OrderStatus.SHIPPED);

Answer and why

The declared type decides, and monad is declared a Monad, so the chain has no when and javac reports "cannot find symbol". Declare it MonadZero<MaybeKind.Witness>, and For.from starts the chain that has when.

Where this lives: Filtering with when.

Checkpoint: par over two failures

With Either's instance, For.par starts a chain from Left("SKU is blank") and Left("quantity must be positive"). What does yield give?

Answer and why

Left(SKU is blank). par combines its values with the instance's map2, and Either's keeps the first failure. Only an instance whose map2 joins errors, such as Validated's, reports both.

Where this lives: Independent steps with par.


The fine print

Twelve values, and their tuples

A chain holds up to twelve values, from any mix of from, let, par and the optic steps. The first step is written by hand, and the steps that hold two to twelve values are generated when Higher-Kinded-J is built, so your build needs no processor for them. The twelfth has no room for another value, so it can only yield, call toState, or filter with when in a chain started from a MonadZero. Past four or five values, positions such as t._4() get hard to follow, and toState moves the values into a record with named fields.

Coming from Scala

Scala builds the same chain into its syntax:

for {
  order        <- found
  total         = order.total
  email        <- verify(order.customer)
  confirmation <- charge(total)
} yield s"receipt $confirmation for $total to $email"

The compiler turns every <- but the last into flatMap and the last into map, an if line into withFilter, and = into a map that carries the value. For gives the same steps as methods: from, when, let and yield.


See Also

Further Reading

Hands-On Learning

Practise par on Paths in the ForPath Parallel journey (9 exercises), and traverse in Tutorial 03: For Traverse Comprehension (14 exercises).


Previous: MonadError Next: Check Your Understanding