In functional programming, a monad transformer is a type constructor which takes a monad as an argument and returns a monad as a result. Monad transformers can be used to compose features encapsulated by monads  such as state, exception handling, and I/O  in a modular way. Wikipedia
When writing realworld Scala programs you quite often have to deal with services and APIs returning different instances of a monad. For example a service method which reads from a database quite commonly returns an Option[T]
. Other service methods may return a Future[T]
or a nested container like a Future[Option[T]]
because they are running asynchronously.
At some point you may want to combine these methods to do something useful. So we most commonly use for comprehensions on our containers which are syntactic sugar for flatMap
and map
. Unfortunately, those for comprehensions are not enough in cases where we have to deal with nested containers or containers of different types. The result is ugly nested code.
Within this article I will show you how to get rid of such nested for comprehensions by using monad transformers and combining them with syntactic sugar to make them even more readable.
Some of the code snippets presented here are a somewhat simplified version of Erik Bakker’s great handson tutorial on monad transformers on github which go along with his slides on speakerdeck.
If you are not familiar with typeclasses I recommend reading the article The Typeclass Pattern in Scala first.
So let’s get started!
The problem
We elaborate on a small cutout of a ficticious program which reads a user from a database and sends an email to a given address.
Now fire up a Scala REPL and paste in the following code:
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 

Note that ???
is just a function from scala.Predef
throwing a NotImplementedError
when called. Here we just rely on the typechecker so there will be no problem. However, the code above will yield a type mismatch error since those methods return different container types/monads which do not compose:
1 2 3 4 5 

But what does  do not compose  mean? Just to get an idea, let’s have a look at the signature of a monad’s flatMap
(often also called bind
or >>=
) and map
signatures:
1 2 

What’s important here is the (higherkinded) type parameter F[_]
. Now, when we create instances of that monad typeclass, say for Option
or Future
, the flatMap
methods get the following shapes:
1 2 3 4 

1 2 3 4 

So, as already mentioned, for comprehensions are just a sugared version of flatMap
and map
. To make things even more clear, here is the desugared version of the for comprehension from above:
1 2 3 4 5 6 7 8 

Of course this produces exactly the same error as the for comprehension when pasted into a REPL.
So let’s find the root cause of the problem by stepping through the desugared version. First we call flatMap
on an Option[User]
(the getUser
method’s return type). Remember that flatMap
on that type requires as parameter f
, a function of type A => Option[B]
. Let’s ignore the call to getEmail
for now and step over to the next line since this is the crucial part. Here sendEmail
is called, which returns a Future[Option[Boolean]]
where the innermost success
(which we put back unchanged into the Future
) is of type Option[Boolean]
. So the point here is that getUser
expects a function of type User => Option[X]
but gets a User => Future[Option[Boolean]]
instead, which does not compile!
Let’s try to find out how we can solve this problem.
Dipping your toes…
To bypass this problem people quite often try to come up with a solution like this:
1 2 3 4 5 6 

This does not compile since for comprehensions unwrap only a single container. So user
has now an improper type and the typechecker complains that getEmail
requires a User
type but found Option[User]
for the user
parameter.
Admittedly, in our simple example we could do the following:
1 2 3 4 5 

But most things in the real world are not that easy and will mess up very quickly.
Stairstepping
To make to compiler happy people start stairstepping like this:
1 2 3 4 5 6 7 8 

What we did was nesting the service method returning the Future[Option[Boolean]]
into the yield
’s body. The resulting type for our really simple example will now be Option[scala.concurrent.Future[Option[Boolean]]]
. As you can imagine real world programs can get nested even deeper.
Sure you could come up with a little trick using an implicit conversion flattening your nested types as Will Sargent pointed out in his article on Composing Dependent Futures:
1 2 

But still you have this deeply nested for comprehension which can quickly turn into a maintenance nightmare.
Breaking up the code
Another approach would be to break the code into very small blocks to avoid nesting:
1 2 3 4 5 6 7 8 9 10 11 12 13 14 

That’s a pretty neat solution but requires us to write quite much glue code which bloats our software unneccessarily.
Setting the stage
So what we actually want is a monadic type, i.e. a type that implements map
and flatMap
that operates on values inside an Option
which itself is nested in a Future
, so that for comprehensions treat such types like a single container.
And this is where monad transformers come into play. But before we get to some offtheshelf monad transformers, let’s reinvent the wheel a bit to better understand how they work.
Crafting your own monad transformer
The best way to understand monad transformers is to create one for yourself. As it turns out this is surprisingly easy:
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 

The case class is all what you need to extract a value from an Option
inside a Future
, apply a function to it and throw it back into the containers. So we basically just implemented yet another monadic typeclass instance serving as a wrapper for Future[Option[T]]
types, which can be used as follows:
1 2 3 4 5 

The typechecker is now happy though it still does not compile because we have not implemented our service methods yet. To try that out at your Scala REPL just replace the ???
to return some static values. You will see that your first monad transformer will return something similar to this:
1 2 

Note that the extracted parameters within our for comprehension (user
, success
) are now the fully unpacked values. Of course the final result
is now of type FutureOption
and to do some useful stuff with it you can access the actual values and types “from outside” by the contents
field as follows:
1 2 

Sure, in real world programs you will not be bothered writing your own monad transformers for each possible combination of nested monadic containers. Fortunately Scalaz has already created a bunch of monad transformers lurking to be awakened!
Scalaz  the ultimate monad transformer storehouse
When using monad transformers from Scalaz you choose them according to the inner container. So for our fictitious program, we choose a monad transformer for Option
. In Scalaz this type is called OptionT
. As you might already have guessed the general notation for monad transformers in Scalaz is appending a T
to the corresponding type name.
But what about the outer container type, you may ask? Well, let’s have a look the definition of OptionT:
1 2 3 4 5 6 7 8 9 10 11 12 13 14 

Scalaz makes heavy use of the typeclass pattern and implicits, so from the signatures of map
and flatMap
you can see that using OptionT
say for Future
requires having a typeclass instance of Functor[Future]
and Monad[Future]
in scope. Take your time and think about it, all we need to ensure about our enclosing type/container is to be a monad! As with monad transformers you can be quite confident that for most standard types Scalaz has already implemented appropriate typeclass instances for you. If not, creating such typeclass instances for yourself is very easy. Note that in Scalaz your refer to the contents of the monad transformers via the run
field whereas we used contents
in our selfmade version.
So let’s get back to our initial problem and rewrite our for comprehension using OptionT
:
1 2 3 4 5 6 7 8 

When you want to try this at your Scala REPL make sure that you have Scalaz in your classpath (please refer to Getting Scalaz on github).
Now the final result
is of type OptionT[Future, Boolean]
and to do something useful with it we can access the actual values and types “from outside” by the run
field as follows:
1 2 

Thrush  a permuting combinator
We can finally compose different types of monads within our appreciated for comprehensions, however they now look a little bit clunky. But we can do better!
Let’s apply a little syntactic sugar provided by Scalaz called the thrush combinator >
which reverses the order of evaluation. In other words it shifts operators/constructors and their parameter like this:
1 2 3 4 5 

We have now improved readability. Having service methods wrapped into monad transformers and futures is a technical detail which is not important in order to understand what the for comprehension actually does. So the trush combinator shifted these parts to the right.
And that’s it. We have now beautifully lined up all our service methods within one for comprehension by using monad transformers and added a little syntactic sugar to it.
But there is actually a lot more so say about monad transformers and we were really just scratching the surface.
I highly recommend checking out Erik Bakker’s handson tutorial or his Activator template where he also showcases using monad transformers with Play which is really awesome.
Stay tuned!
References
 Flatten your code (slides) by Erik Bakker
 Flatten your code (handson tutorial on github) by Erik Bakker
 Activator template using monad transformers with Play by Erik Bakker
 Scalaz Monad Transformers (blog) by Noel Welsh
 Monad Transformers in Scalamachine & Scaliak (video) by Jordan West
 Composing Dependent Futures (blog) by Will Sargent