Invasion of the Monad Transformers

In functional programming, a monad transformer is a type constructor which takes a monad as an argument and returns a monad as a result. Monad transformers can be used to compose features encapsulated by monads - such as state, exception handling, and I/O - in a modular way. Wikipedia

When writing real-world Scala programs you quite often have to deal with services and APIs returning different instances of a monad. For example a service method which reads from a database quite commonly returns an Option[T]. Other service methods may return a Future[T] or a nested container like a Future[Option[T]] because they are running asynchronously.

At some point you may want to combine these methods to do something useful. So we most commonly use for comprehensions on our containers which are syntactic sugar for flatMap and map. Unfortunately, those for comprehensions are not enough in cases where we have to deal with nested containers or containers of different types. The result is ugly nested code.

Within this article I will show you how to get rid of such nested for comprehensions by using monad transformers and combining them with syntactic sugar to make them even more readable.

Some of the code snippets presented here are a somewhat simplified version of Erik Bakker’s great hands-on tutorial on monad transformers on github which go along with his slides on speakerdeck.

If you are not familiar with typeclasses I recommend reading the article The Typeclass Pattern in Scala first.

So let’s get started!

The problem

We elaborate on a small cutout of a ficticious program which reads a user from a database and sends an email to a given address.

Now fire up a Scala REPL and paste in the following code:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
// Imports Future and ExecutionContext
import scala.concurrent.Future
import scala.concurrent.ExecutionContext.Implicits.global

// Our user type
case class User(id: Long, name: String)

// Reads a username from a fictitious database
def getUser(id: Long): Option[User] = ???
def getEmail(user: User): String = ???
// This method send an email asynchronously
def sendEmail(email: String): Future[Option[Boolean]] = ???

// This will not typecheck!
for {
  user <- getUser(42)
  email = getEmail(user)
  success <- sendEmail(email)
} yield success

Note that ??? is just a function from scala.Predef throwing a NotImplementedError when called. Here we just rely on the typechecker so there will be no problem. However, the code above will yield a type mismatch error since those methods return different container types/monads which do not compose:

1
2
3
4
5
<console>:18: error: type mismatch;
 found   : scala.concurrent.Future[Option[Boolean]]
 required: Option[?]
                success <- sendEmail(email)
                        ^

But what does - do not compose - mean? Just to get an idea, let’s have a look at the signature of a monad’s flatMap (often also called bind or >>=) and map signatures:

1
2
def flatMap[A, B](fa: F[A])(f: A => F[B]): F[B]
def map[A, B](fa: F[A])(f: A => B): F[B]

What’s important here is the (higher-kinded) type parameter F[_]. Now, when we create instances of that monad typeclass, say for Option or Future, the flatMap methods get the following shapes:

1
2
3
4
// An Option's map and flatMap implementations
def flatMap[A, B](fa: Option[A])(f: A => Option[B]): Option[B] =
  fa.flatMap(f)
def map[A, B](fa: Option[A])(f: A => B): Option[B]
1
2
3
4
// A Future's map and flatMap implementations
def flatMap[A, B](fa: Future[A])(f: A => Future[B]): Future[B] =
  fa.flatMap(f)
def map[A, B](fa: Future[A])(f: A => B): Future[B]

So, as already mentioned, for comprehensions are just a sugared version of flatMap and map. To make things even more clear, here is the desugared version of the for comprehension from above:

1
2
3
4
5
6
7
8
getUser(42) flatMap { user =>
  val email = getEmail(user)
  sendEmail(email) map { success =>
    // Actually there is no need to map here.
    // It's just for the sake of clarity.
    success
  }
}

Of course this produces exactly the same error as the for comprehension when pasted into a REPL.

So let’s find the root cause of the problem by stepping through the desugared version. First we call flatMap on an Option[User] (the getUser method’s return type). Remember that flatMap on that type requires as parameter f, a function of type A => Option[B]. Let’s ignore the call to getEmail for now and step over to the next line since this is the crucial part. Here sendEmail is called, which returns a Future[Option[Boolean]] where the innermost success (which we put back unchanged into the Future) is of type Option[Boolean]. So the point here is that getUser expects a function of type User => Option[X] but gets a User => Future[Option[Boolean]] instead, which does not compile!

Let’s try to find out how we can solve this problem.

Dipping your toes…

To bypass this problem people quite often try to come up with a solution like this:

1
2
3
4
5
6
// Does not compile!
for {
  user <- Future.successful(getUser(42))
  email = getEmail(user)
  success <- sendEmail(email)
} yield success

This does not compile since for comprehensions unwrap only a single container. So user has now an improper type and the typechecker complains that getEmail requires a User type but found Option[User] for the user parameter.

Admittedly, in our simple example we could do the following:

1
2
3
4
5
for {
  Some(user) <- Future.successful(getUser(42))
  email = getEmail(user)
  success <- sendEmail(email)
} yield success

But most things in the real world are not that easy and will mess up very quickly.

Stair-stepping

To make to compiler happy people start stair-stepping like this:

1
2
3
4
5
6
7
8
for {
  user <- getUser(42)
  email = getEmail(user)
} yield {
  for {
    success <- sendEmail(email)
  } yield success
}

What we did was nesting the service method returning the Future[Option[Boolean]] into the yield’s body. The resulting type for our really simple example will now be Option[scala.concurrent.Future[Option[Boolean]]]. As you can imagine real world programs can get nested even deeper.

Sure you could come up with a little trick using an implicit conversion flattening your nested types as Will Sargent pointed out in his article on Composing Dependent Futures:

1
2
implicit def flatten[A](ofoa: Option[Future[Option[A]]]): Future[Option[A]] =
  ofoa.getOrElse(Future.successful(None))

But still you have this deeply nested for comprehension which can quickly turn into a maintenance nightmare.

Breaking up the code

Another approach would be to break the code into very small blocks to avoid nesting:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
def userEmail(optionUser: Option[User]): Future[Option[String]] =
  Future.successful(optionUser map { getEmail })

def sendEmail(optionEmail: Option[String]): Future[Option[Boolean]] =
  optionEmail match {
    case Some(email) => sendEmail(email)
    case _ => Future.successful(None)
  }

for {
  userOption <- Future.successful(getUser(42))
  email <- userEmail(userOption)
  success <- sendEmail(email)
} yield success

That’s a pretty neat solution but requires us to write quite much glue code which bloats our software unneccessarily.

Setting the stage

So what we actually want is a monadic type, i.e. a type that implements map and flatMap that operates on values inside an Option which itself is nested in a Future, so that for comprehensions treat such types like a single container.

And this is where monad transformers come into play. But before we get to some off-the-shelf monad transformers, let’s reinvent the wheel a bit to better understand how they work.

Crafting your own monad transformer

The best way to understand monad transformers is to create one for yourself. As it turns out this is surprisingly easy:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
case class FutureOption[A](contents: Future[Option[A]]) {
  def flatMap[B](fn: A => FutureOption[B]): FutureOption[B] =
    FutureOption {
      contents.flatMap {
        case Some(a) => fn(a).contents
        case None => Future.successful(None)
      }
    }
  def map[B](fn: A => B): FutureOption[B] =
    FutureOption {
      contents.map { option =>
        option.map(fn)
      }
    }
}

The case class is all what you need to extract a value from an Option inside a Future, apply a function to it and throw it back into the containers. So we basically just implemented yet another monadic typeclass instance serving as a wrapper for Future[Option[T]] types, which can be used as follows:

1
2
3
4
5
val result = for {
  user <- FutureOption(Future.successful(getUser(42)))
  email = getEmail(user)
  success <- FutureOption(sendEmail(email))
} yield success

The typechecker is now happy though it still does not compile because we have not implemented our service methods yet. To try that out at your Scala REPL just replace the ??? to return some static values. You will see that your first monad transformer will return something similar to this:

1
2
result: FutureOption[Boolean] =
  FutureOption(scala.concurrent.impl.Promise$DefaultPromise@7b79f9d)

Note that the extracted parameters within our for comprehension (user, success) are now the fully unpacked values. Of course the final result is now of type FutureOption and to do some useful stuff with it you can access the actual values and types “from outside” by the contents field as follows:

1
2
scala> result.contents map { println }
Some(true)

Sure, in real world programs you will not be bothered writing your own monad transformers for each possible combination of nested monadic containers. Fortunately Scalaz has already created a bunch of monad transformers lurking to be awakened!

Scalaz - the ultimate monad transformer storehouse

When using monad transformers from Scalaz you choose them according to the inner container. So for our fictitious program, we choose a monad transformer for Option. In Scalaz this type is called OptionT. As you might already have guessed the general notation for monad transformers in Scalaz is appending a T to the corresponding type name.

But what about the outer container type, you may ask? Well, let’s have a look the definition of OptionT:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
final case class OptionT[F[_], A](run: F[Option[A]]) { self =>

  def map[B](f: A => B)(implicit F: Functor[F]): OptionT[F, B] =
    new OptionT[F, B](mapO(_ map f))

  def flatMap[B](f: A => OptionT[F, B])(implicit F: Monad[F]) =
    new OptionT[F, B](
      F.bind(self.run) {
        case None    => F.point(None: Option[B])
        case Some(z) => f(z).run
      }
    )
    // Some more cool stuff left out intentionally
}

Scalaz makes heavy use of the typeclass pattern and implicits, so from the signatures of map and flatMap you can see that using OptionT say for Future requires having a typeclass instance of Functor[Future] and Monad[Future] in scope. Take your time and think about it, all we need to ensure about our enclosing type/container is to be a monad! As with monad transformers you can be quite confident that for most standard types Scalaz has already implemented appropriate typeclass instances for you. If not, creating such typeclass instances for yourself is very easy. Note that in Scalaz your refer to the contents of the monad transformers via the run field whereas we used contents in our self-made version.

So let’s get back to our initial problem and rewrite our for comprehension using OptionT:

1
2
3
4
5
6
7
8
import scalaz._
import Scalaz._

val result = for {
  user <- OptionT(Future.successful(getUser(42)))
  email = getEmail(user)
  success <- OptionT(sendEmail(email))
} yield success

When you want to try this at your Scala REPL make sure that you have Scalaz in your classpath (please refer to Getting Scalaz on github).

Now the final result is of type OptionT[Future, Boolean] and to do something useful with it we can access the actual values and types “from outside” by the run field as follows:

1
2
scala> result.run map { println }
Some(true)

Thrush - a permuting combinator

We can finally compose different types of monads within our appreciated for comprehensions, however they now look a little bit clunky. But we can do better!

Let’s apply a little syntactic sugar provided by Scalaz called the thrush combinator |> which reverses the order of evaluation. In other words it shifts operators/constructors and their parameter like this:

1
2
3
4
5
val result = for {
  user <- getUser(42) |> Future.successful |> OptionT.apply
  email = getEmail(user)
  success <- sendEmail(email) |> OptionT.apply
} yield success

We have now improved readability. Having service methods wrapped into monad transformers and futures is a technical detail which is not important in order to understand what the for comprehension actually does. So the trush combinator shifted these parts to the right.

And that’s it. We have now beautifully lined up all our service methods within one for comprehension by using monad transformers and added a little syntactic sugar to it.

But there is actually a lot more so say about monad transformers and we were really just scratching the surface.

I highly recommend checking out Erik Bakker’s hands-on tutorial or his Activator template where he also showcases using monad transformers with Play which is really awesome.

Stay tuned!

References

« The Typeclass Pattern in Scala