1+-|
2`-"
Links
EIP seems to be a popular paper to cover among Scala fp people.
[Eric Torreborre (@etorreborre)](https://twitter.com/etorreborre)’s The Essence of the Iterator Pattern is the most thorough study of the paper. It also covers lots of ground works, so it’s worth digging in.
[Debasish Ghosh (@debasishg)](https://twitter.com/debasishg)’s Iteration in Scala - effectful yet functionalis shorter but covering the good part by focusing on Scalaz.
[Marc-Daniel Ortega (@patterngazer)](https://twitter.com/patterngazer)’s Where we traverse, accumulate and collect in Scala also covers sequence and collect using Scalaz.
We’ll pick it up from here later.
day 13 (import guide)
e.e d3si9n
Yesterday we skimmed two papers by Jeremy Gibbons and quickly looked at origami programming and applicative traversal. Instead of reading something, why don’t we focus on using Scalaz today.
implicits review
Scalaz makes heavy use of implicits. Both as a user and an extender of the library, it’s important to have general idea on where things are coming from.
Let’s quickly review Scala’s imports and implicits!
In Scala, imports are used for two purposes: 1. To include names of values and types into the scope. 2. To include implicits into the scope.
Implicits are for 4 purposes that I can think of: 1. To provide typeclass instances.
2. To inject methods and operators. (static monkey patching) 3. To declare type constraints. 4. To retrieve type information from compiler.
Implicits are selected in the following precedence: 1. Values and converters accessible without prefix via local declaration, imports, outer scope, inheritance, and current package object. Inner scope can shadow values when they are named the same. 2. Implicit scope. Values and converters declared in companion objects and package object of the type, its parts, or super types.
import scalaz._
Now let’s see what gets imported with import scalaz._.
First, the names. Typeclasses like Equal[A] and Functor[F[_]] are imple-mented as trait, and are defined under scalaz package. So instead of writing scalaz.Equal[A] we can write Equal[A].
Next, also the names, but type aliases. scalaz’s package object declares most of the major type aliases like @@[T, Tag] and Reader[E, A], which is treated as a specialization of ReaderT transformer. Again, these can also be accessed as scalaz.Reader[E, A] if you want.
Finally, idInstance is defined as typeclass instance of Id[A] for Traverse[F[_]], Monad[F[_]] etc, but it’s not relevant. By virtue of declaring an instance within its package object it will be available, so importing doesn’t add much.
Let’s check this:
scala> scalaz.Monad[scalaz.Id.Id]
res1: scalaz.Monad[scalaz.Id.Id] = scalaz.IdInstances$$anon$1@fc98c94 No import needed, which is a good thing. So, the merit of import scalaz._ is for convenience, and it’s optional.
import Scalaz._
What then is import Scalaz._ doing? Here’s the definition ofScalaz object:
package scalaz object Scalaz
extends StateFunctions // Functions related to the state monad with syntax.ToTypeClassOps // syntax associated with type classes
with syntax.ToDataOps // syntax associated with Scalaz data structures with std.AllInstances // Type class instances for the standard library types with std.AllFunctions // Functions related to standard library types
with syntax.std.ToAllStdOps // syntax associated with standard library types with IdInstances // Identity type and instances
This is quite a nice way of organizing the imports. Scalaz object itself doesn’t define anythig and it just mixes in the traits. We are going to look at each traits in detail, but they can also be imported a la carte, dim sum style. Back to the full course.
StateFunctions Remember, import brings in names and implicits. First, the names. StateFunctions defines several functions:
package scalaz
trait StateFunctions {
def constantState[S, A](a: A, s: => S): State[S, A] = ...
def state[S, A](a: A): State[S, A] = ...
def init[S]: State[S, S] = ...
def get[S]: State[S, S] = ...
def gets[S, T](f: S => T): State[S, T] = ...
def put[S](s: S): State[S, Unit] = ...
def modify[S](f: S => S): State[S, Unit] = ...
def delta[A](a: A)(implicit A: Group[A]): State[A, A] = ...
}
By bringing these functions in we can treat get and put like a global function.
Why? This enables DSL we saw onday 7:
for {
xs <- get[List[Int]]
_ <- put(xs.tail) } yield xs.head
std.AllFunctions Second, the names again. std.AllFunctionsis actually a mixin of traits itself:
package scalaz package std
trait AllFunctions extends ListFunctions with OptionFunctions with StreamFunctions
with math.OrderingFunctions with StringFunctions
object AllFunctions extends AllFunctions
Each of the above trait bring in various functions into the scope that acts as a global function. For example, ListFunctions bring in intersperse function that puts a given element in ever other position:
scala> intersperse(List(1, 2, 3), 7) res3: List[Int] = List(1, 7, 2, 7, 3)
It’s ok. Since I personally use injected methods, I don’t have much use to these functions.
IdInstances Although it’s named IdInstances, it also defines the type alias Id[A] as follows:
type Id[+X] = X
That’s it for the names. Imports can bring in implicits, and I said there are four uses for the implicits. We mostly care about the first two: typeclass instances and injected methods and operators.
std.AllInstances Thus far, I have been intentionally conflating the concept of typeclass instances and method injection (aka enrich my library). But the fact that List is a Monad and that Monad introduces >>= operator are two different things.
One of the most interesting design of Scalaz 7 is that it rigorously separates the two concepts into “instance” and “syntax.” Even if it makes logical sense to some users, the choice of symbolic operators can often be a point of contention with any libraries. Libraries and tools such as sbt, dispatch, and specs introduce its
own DSL, and their effectiveness have been hotly debated. To make the matter complicated, injected methods may conflict with each other when more than one DSLs are used together.
std.AllInstancesis a mixin of typeclass instances for built-in (std) data struc-tures:
package scalaz.std trait AllInstances
extends AnyValInstances with FunctionInstances with ListInstances with MapInstances
with OptionInstances with SetInstances with StringInstances with StreamInstances with TupleInstances with EitherInstances with PartialFunctionInstances with TypeConstraintInstances
with scalaz.std.math.BigDecimalInstances with scalaz.std.math.BigInts with scalaz.std.math.OrderingInstances
with scalaz.std.util.parsing.combinator.Parsers with scalaz.std.java.util.MapInstances
with scalaz.std.java.math.BigIntegerInstances
with scalaz.std.java.util.concurrent.CallableInstances with NodeSeqInstances
// Intentionally omitted: IterableInstances object AllInstances extends AllInstances
syntax.ToTypeClassOps Next are the injected methods and operators. All of them are defined under scalaz.syntax package. syntax.ToTypeClassOps introduces all the injected methods for typeclasses:
package scalaz package syntax trait ToTypeClassOps
extends ToSemigroupOps with ToMonoidOps with ToGroupOps with ToEqualOps with ToLengthOps with ToShowOps with ToOrderOps with ToEnumOps with ToMetricSpaceOps with ToPlusEmptyOps with ToEachOps with ToIndexOps with ToFunctorOps with ToPointedOps with ToContravariantOps with ToCopointedOps with ToApplyOps
with ToApplicativeOps with ToBindOps with ToMonadOps with ToCojoinOps with ToComonadOps with ToBifoldableOps with ToCozipOps
with ToPlusOps with ToApplicativePlusOps with ToMonadPlusOps with ToTraverseOps with ToBifunctorOps with ToBitraverseOps with ToArrIdOps with ToComposeOps with ToCategoryOps
with ToArrowOps with ToFoldableOps with ToChoiceOps with ToSplitOps with ToZipOps with ToUnzipOps with ToMonadWriterOps with ToListenableMonadWriterOps For example, [syntax.ToBindOps] implicitly converts F[A] where [F: Bind]
into BindOps[F, A] that implements >>= operator.
syntax.ToDataOps syntax.ToDataOps introduces injected methods for data structures defined in Scalaz:
trait ToDataOps extends ToIdOps with ToTreeOps with ToWriterOps with ToValidationOps with ToReducerOps with ToKleisliOps IdOpsmethods are injected to all types, and are mostly there for convenience:
package scalaz.syntax
trait IdOps[A] extends Ops[A] {
final def ??(d: => A)(implicit ev: Null <:< A): A = ...
final def |>[B](f: A => B): B = ...
final def squared: (A, A) = ...
def left[B]: (A \/ B) = ...
def right[B]: (B \/ A) = ...
final def wrapNel: NonEmptyList[A] = ...
def matchOrZero[B: Monoid](pf: PartialFunction[A, B]): B = ...
final def doWhile(f: A => A, p: A => Boolean): A = ...
final def whileDo(f: A => A, p: A => Boolean): A = ...
def visit[F[_] : Pointed](p: PartialFunction[A, F[A]]): F[A] = ...
}
trait ToIdOps {
implicit def ToIdOps[A](a: A): IdOps[A] = new IdOps[A] { def self: A = a
} }
Interestingly, ToTreeOps converts all data types to TreeOps[A] injecting two methods:
package scalaz package syntax
trait TreeOps[A] extends Ops[A] {
def node(subForest: Tree[A]*): Tree[A] = ...
def leaf: Tree[A] = ...
}
trait ToTreeOps {
implicit def ToTreeOps[A](a: A) = new TreeOps[A]{ def self = a } }
So these are injected methods to create Tree.
scala> 1.node(2.leaf)
res7: scalaz.Tree[Int] = <tree>
The same goes for WriterOps[A], ValidationOps[A], ReducerOps[A], and Kleis-liIdOps[A]:
scala> 1.set("log1")
res8: scalaz.Writer[String,Int] = scalaz.WriterTFunctions$$anon$26@2375d245 scala> "log2".tell
res9: scalaz.Writer[String,Unit] = scalaz.WriterTFunctions$$anon$26@699289fb scala> 1.success[String]
res11: scalaz.Validation[String,Int] = Success(1) scala> "boom".failureNel[Int]
res12: scalaz.ValidationNEL[String,Int] = Failure(NonEmptyList(boom)) So most of the mixins under syntax.ToDataOps introduces methods to all types to create Scalaz data structure.
syntax.std.ToAllStdOps Finally, we have syntax.std.ToAllStdOps, which introduces methods and operators to Scala’s standard types.
package scalaz package syntax package std trait ToAllStdOps
extends ToBooleanOps with ToOptionOps with ToOptionIdOps with ToListOps with ToStreamOps
with ToFunction2Ops with ToFunction1Ops with ToStringOps with ToTupleOps with ToMapOps with ToEitherOps This is the fun stuff. BooleanOpsintroduces shorthands for all sorts of things:
scala> false /\ true res14: Boolean = false scala> false \/ true res15: Boolean = true scala> true option "foo"
res16: Option[String] = Some(foo)
scala> (1 > 10)? "foo" | "bar"
res17: String = bar
scala> (1 > 10)?? {List("foo")}
res18: List[String] = List()
The option operator is very useful. The ternary operator looks like a shorter notation than if-else.
OptionOpsalso introduces something similar:
scala> 1.some? "foo" | "bar"
res28: String = foo scala> 1.some | 2 res30: Int = 1
On the other handListOpsintroduced traditional Monad related things:
scala> List(1, 2) filterM {_ => List(true, false)}
res37: List[List[Int]] = List(List(1, 2), List(1), List(2), List())
a la carte style
Or, I’d like to call dim sum style, where they bring in a cart load of chinese dishes and you pick what you want.
If for whatever reason if you do not wish to import the entire Scalaz._, you can pick and choose.
typeclass instances and functions Typeclass instances are broken down by the data structures. Here’s how to get all typeclass instances for Option:
// fresh REPL
scala> import scalaz.std.option._
import scalaz.std.option._
scala> scalaz.Monad[Option].point(0) res0: Option[Int] = Some(0)
This also brings in the “global” helper functions related to Option. Scala stan-dard data structures are found under scalaz.std package.
If you just want all instances, here’s how to load them all:
scala> import scalaz.std.AllInstances._
import scalaz.std.AllInstances._
scala> scalaz.Monoid[Int]
res2: scalaz.Monoid[Int] = scalaz.std.AnyValInstances$$anon$3@784e6f7c Because we have not injected any operators, you would have to work more with helper functions and functions under typeclass instances, which could be exactly what you want.
Scalaz typeclass syntax Typeclass syntax are broken down by the typeclass.
Here’s how to get injected methods and operators for Monads:
scala> import scalaz.syntax.monad._
import scalaz.syntax.monad._
scala> import scalaz.std.option._
import scalaz.std.option._
scala> 0.point[Option]
res0: Option[Int] = Some(0)
As you can see, not only Monad method was injected but also Pointed methods got in too.
Scalaz data structure syntax like Tree are also available under scalaz.syntax package. Here’s how to load all syntax for both the typeclasses and Scalaz’s data structure:
scala> import scalaz.syntax.all._
import scalaz.syntax.all._
scala> 1.leaf
res0: scalaz.Tree[Int] = <tree>
standard data structure syntax Standard data structure syntax are broken down by the data structure. Here’s how to get injected methods and operators for Boolean:
// fresh REPL
scala> import scalaz.syntax.std.boolean._
import scalaz.syntax.std.boolean._
scala> (1 > 10)? "foo" | "bar"
res0: String = bar
To load all the standard data structure syntax in:
// fresh REPL
scala> import scalaz.syntax.std.all._
import scalaz.syntax.std.all._
scala> 1.some | 2 res1: Int = 1
I thought this would be a quick thing, but it turned out to be an entire post.
We’ll pick it up from here.
day 14
bman ojel for openphoto.net
Yesterday we looked at what import scalaz._ and Scalaz._ bring into the scope, and also talked about a la carte style import. Knowing how instances and syntax are organized prepares us for the next step, which is to hack on Scalaz.
mailing list
Before we start hacking on a project, it’s probably good idea to joinits Google Group.
git clone
$ git clone -b scalaz-seven git://github.com/scalaz/scalaz.git scalaz-seven The above should clone scalaz-seven branch into ./scalaz-seven directory.
Next I edited the .git/config as follows:
[core]
repositoryformatversion = 0 filemode = true
bare = false
logallrefupdates = true ignorecase = true [remote "upstream"]
fetch = +refs/heads/*:refs/remotes/origin/*
url = git://github.com/scalaz/scalaz.git [branch "scalaz-seven"]
remote = upstream
merge = refs/heads/scalaz-seven
This way, scalaz/scalaz is referenced using the name upstream instead of origin. To track the changes, run:
$ git pull --rebase
Current branch scalaz-seven is up to date.
sbt
Next, launch sbt 0.12.0, set scala version to 2.10.1, switch to core project, and compile:
$ sbt
scalaz> ++ 2.10.1
Setting version to 2.10.1
[info] Set current project to scalaz (in build file:/Users/eed3si9n/work/scalaz-seven/) scalaz> project core
[info] Set current project to scalaz-core (in build file:/Users/eed3si9n/work/scalaz-seven/) scalaz-core> compile
This might take a few minutes. Let’s make sure this builds a snapshot version:
scalaz-core> version [info] 7.0-SNAPSHOT
To try out the locally compiled Scalaz, just get into the REPL as usual using console:
scalaz-core> console
[info] Starting scala interpreter...
[info]
Welcome to Scala version 2.10.1 (Java HotSpot(TM) 64-Bit Server VM, Java 1.6.0_33).
Type in expressions to have them evaluated.
Type :help for more information.
scala> [Ctrl + D to exit]
including Vector
Let’s address some of the things we’ve noticed in the last few weeks. For example, I think Vector instances should be part of import Scalaz._. This should be easy while my memory is fresh from yesterday’s import review. Let’s make a topic branch topic/vectorinstance:
$ git branch topic/vectorinstance
$ git co topic/vectorinstance
Switched to branch 'topic/vectorinstance'
To confirm that Vector instances and methods are not loaded in by import Scalaz._, let’s check it from sbt console:
$ sbt
scalaz> ++ 2.10.1 scalaz> project core scalaz-core> console scala> import scalaz._
import scalaz._
scala> import Scalaz._
import Scalaz._
scala> Vector(1, 2) >>= { x => Vector(x + 1)}
<console>:14: error: could not find implicit value for parameter F0: scalaz.Bind[scala.collection.immutable.Vector]
Vector(1, 2) >>= { x => Vector(x + 1)}
^
scala> Vector(1, 2) filterM { x => Vector(true, false) }
<console>:14: error: value filterM is not a member of scala.collection.immutable.Vector[Int]
Vector(1, 2) filterM { x => Vector(true, false) }
^ Failed as expected.
Updatestd.AllInstancesby mixing in VectorInstances:
trait AllInstances
extends AnyValInstances with FunctionInstances with ListInstances with MapInstances with OptionInstances with SetInstances with StringInstances with StreamInstances with TupleInstances with VectorInstances
...
Updatesyntax.std.ToAllStdOpsand add ToVectorOps:
trait ToAllStdOps
extends ToBooleanOps with ToOptionOps with ToOptionIdOps with ToListOps with ToStreamOps with ToVectorOps ...
That’s it. Let’s try it from REPL.
scala> Vector(1, 2) >>= { x => Vector(x + 1)}
res0: scala.collection.immutable.Vector[Int] = Vector(2, 3) scala> Vector(1, 2) filterM { x => Vector(true, false) }
res1: scala.collection.immutable.Vector[Vector[Int]] = Vector(Vector(1, 2), Vector(1), Vector(2), Vector()) It works. I didn’t see tests written for these type of things, so we’ll go without
one. I committed it as “include VectorInstances and ToVectorOps to import Scalaz._.” Next, fork scalaz project on github.
$ git remote add fork [email protected]:yourname/scalaz.git
$ git push fork topic/vectorinstance ...
* [new branch] topic/vectorinstance -> topic/vectorinstance Senda pull requestwith some comments, and let’s see what happens. To work on a next feature, we want to rewind back to scalaz-seven branch. For using locally, let’s create a snapshot branch:
snapshot
$ git co scalaz-seven
Switched to branch 'scalaz-seven'
$ git branch snapshot
$ git co snapshot
$ git merge topic/vectorinstance
We can use this branch as a sandbox to play around with Scalaz.
<*> operator
Next, I’d really like to roll back <*> operator for Apply back to M2/Haskell behavior. I’veasked thison the mailing list and the author seems to be ok with rolling back.
$ git co scalaz-seven
Switched to branch 'scalaz-seven'
$ git branch topic/applyops
$ git co topic/applyops
Switched to branch 'topic/applyops'
This one we really should write a test first. Let’s add an example inApplyTest:
"<*>" in {
some(9) <*> some({(_: Int) + 3}) must be_===(some(12)) }
The specs used in build.scala works for Scala 2.9.2.
$ sbt
scalaz> ++ 2.9.2
Setting version to 2.9.2 scalaz> project tests
scalaz-tests> test-only scalaz.ApplyTest
[error] /Users/eed3si9n/work/scalaz-seven/tests/src/test/scala/scalaz/ApplyTest.scala:38: type mismatch;
[error] found : org.specs2.matcher.Matcher[Option[Int]]
[error] required: org.specs2.matcher.Matcher[Option[(Int, Int => Int)]]
[error] some(9) <*> some({(_: Int) + 3}) must be_===(some(12))
[error] ^
[error] one error found
[error] (tests/test:compile) Compilation failed It didn’t even compile because of ===. Nice.
The <*> is inApplyOps, so let’s change it back to F.ap:
final def <*>[B](f: F[A => B]): F[B] = F.ap(self)(f) Now let’s run the test again:
scalaz-tests> test-only scalaz.ApplyTest [info] ApplyTest
[info]
[info] + mapN [info] + apN [info] + <*>
[info]
[info] Total for specification ApplyTest [info] Finished in 5 seconds, 27 ms [info] 3 examples, 0 failure, 0 error [info]
[info] Passed: : Total 3, Failed 0, Errors 0, Passed 3, Skipped 0 [success] Total time: 9 s, completed Sep 19, 2012 1:57:29 AM I am committing this as “roll back <*> as infix of ap” and pushing it out.
$ git push fork topic/applyops ...
* [new branch] topic/applyops -> topic/applyops
Send a pull request with some comments. Let’s apply this to our snapshot branch:
$ git co snapshot
$ git merge topic/applyops
So now it has both of the changes we created.
applicative functions
The changed we made were so far simple fixes. From here starts an experiment.
It’s about applicative functions.
The Essence of the Iterator Patternpresents an interesting idea of combining applicative functors. What’s actually going on is not just the combination of applicative functors (m � n), but the combination of applicative functions:
(�)::(Functor m,Functor n) � (a → m b) → (a → n b) → (a → (m � n) b) (f � g) x = Prod (f x) (g x)
Int is a Monoid, and any Monoid can be treated as an applicative functor, which is called monoidal applicatives. The problem is that when we make that into a function, it’s not distinguishable from Int => Int, but we need Int => [�]Int.
My first idea was to use type tags named Tags.Monoidal, so the idea is to make it:
scala> { (x: Int) => Tags.Monoidal(x + 1) }
This requires all A @@ Tags.Monoidal where [A:Monoid] to be recognized as an applicative. I got stuck on that step.
Next idea was to make Monoidal an alias of Kleisli with the following com-panion:
object Monoidal {
def apply[A: Monoid](f: A => A): Kleisli[({type �[+�]=A})#�, A, A] = Kleisli[({type �[+�]=A})#�, A, A](f)
}
This let’s me write monoidal functions as follows:
scala> Monoidal { x: Int => x + 1 }
res4: scalaz.Kleisli[[+�]Int,Int,Int] = scalaz.KleisliFunctions$$anon$18@1a0ceb34
But the compiler did not find Applicative automatically from [+�]Int:
scala> List(1, 2, 3) traverseKTrampoline { x => Monoidal { _: Int => x + 1 } }
<console>:14: error: no type parameters for method traverseKTrampoline: (f: Int => scalaz.Kleisli[G,S,B])(implicit evidence$2: scalaz.Applicative[G])scalaz.Kleisli[G,S,List[B]] exist so that it can be applied to arguments (Int => scalaz.Kleisli[[+�]Int,Int,Int]) because
---argument expression's type is not compatible with formal parameter type;
found : Int => scalaz.Kleisli[[+�]Int,Int,Int]
required: Int => scalaz.Kleisli[?G,?S,?B]
List(1, 2, 3) traverseKTrampoline { x => Monoidal { _: Int => x + 1 } }
^
Is this the infamousSI-2712? Then I thought, ok I’ll turn this into an actual type:
trait MonoidApplicative[F] extends Applicative[({type �[�]=F})#�] { self =>
implicit def M: Monoid[F]
def point[A](a: => A) = M.zero
def ap[A, B](fa: => F)(f: => F) = M.append(f, fa) override def map[A, B](fa: F)(f: (A) => B) = fa }
This does not work because now we have to convert x + 1 into MonoidApplicative.
Next I thought about giving Unapply a shot:
scala> List(1, 2, 3) traverseU {_ + 1}
<console>:14: error: Unable to unapply type `Int` into a type constructor of kind `M[_]` that is classified by the type class `scalaz.Applicative`
1) Check that the type class is defined by compiling `implicitly[scalaz.Applicative[<type constructor>]]`.
2) Review the implicits in object Unapply, which only cover common type 'shapes' (implicit not found: scalaz.Unapply[scalaz.Applicative, Int])
List(1, 2, 3) traverseU {_ + 1}
^
This could work. All we have to do is unpack Int as ({type �[�]=Int})#� in Unapply:
trait Unapply_3 {
/** Unpack a value of type `A0` into type `[a]A0`, given a instance of `TC` */
implicit def unapplyA[TC[_[_]], A0](implicit TC0: TC[({type �[�] = A0})#�]): Unapply[TC, A0] { type M[X] = A0
type A = A0
} = new Unapply[TC, A0] { type M[X] = A0
type A = A0 def TC = TC0
def apply(ma: M[A0]) = ma }
}
Let’s try:
scala> List(1, 2, 3) traverseU {_ + 1}
res0: Int = 9
This actually worked! Can we combine this?
scala> val f = { (x: Int) => x + 1 } f: Int => Int = <function1>
scala> val g = { (x: Int) => List(x, 5) } g: Int => List[Int] = <function1>
scala> val h = f &&& g
h: Int => (Int, List[Int]) = <function1>
scala> List(1, 2, 3) traverseU f res0: Int = 9
scala> List(1, 2, 3) traverseU g
res1: List[List[Int]] = List(List(1, 2, 3), List(1, 2, 5), List(1, 5, 3), List(1, 5, 5), List(5, 2, 3), List(5, 2, 5), List(5, 5, 3), List(5, 5, 5)) scala> List(1, 2, 3) traverseU h
res1: List[List[Int]] = List(List(1, 2, 3), List(1, 2, 5), List(1, 5, 3), List(1, 5, 5), List(5, 2, 3), List(5, 2, 5), List(5, 5, 3), List(5, 5, 5)) scala> List(1, 2, 3) traverseU h