Friday, November 01, 2013

Add/Remove fields to Play's default case class JSON Writes

Play's JSON Macro Inception is great. For most of the time all you need to write for JSON parsing/serialization is something like:
implicit val personFormat = Json.format[Person]
In this example, Person is a case class model in your app.
case class Person(name: String, age: Int)
The implicit personFormat will silently help you whenever you need to either parse a piece of JSON to an instance of Person or serialize an instance of Person to a JSON string.
Now, let's add a boolean val inside the class constructor: isAdult
case class Person(name: String, age: Int) {
  val isAdult: Boolean = age >= 18
}
Since Play!'s JSON Macro Inception only works with the default apply and unapply methods of the case class, it won't include the isAdult in the JSON writes (reads doesn't make much sense here). There isn't much documentation on how to include this new field while still take advantage of the default Writes generated by Play's JSON Macro. Here is how you can achieve it using writes composition.
val personWrites = Json.writes[Person]
val writesWithNewField: Writes[Person] = (personWrites ~ (__ \ "isAdult").write[Boolean])((p: Person) => (p, p.isAdult))
This is not bad, but we can also introduce some generalized helper to minimize the boilerplate. Introducing the OWritesOpts class:
class OWritesOps[A](writes: OWrites[A]) {
  def addField[T: Writes](fieldName: String, field: A => T): OWrites[A] = 
    (writes ~ (__ \ fieldName).write[T])((a: A) => (a, field(a)))


  def removeField(fieldName: String): OWrites[A] = OWrites { a: A =>
    val transformer = (__ \ fieldName).json.prune
    Json.toJson(a)(writes).validate(transformer).get
  }
}

object OWritesOps {
  implicit def from[A](writes: OWrites[A]): OWritesOps[A] = new OWritesOps(writes)
}

With this class, you can add/remove field to default JSON writes like this:
//writes that replaces the age field with the isAdult field in the JSON output. 
val customWrites: Writes[Person] = Json.Writes[Person].
                                     addField("isAdult", _.isAdult).
                                     removeField("age")

That's it, enjoy. I am still new to Play!, if you have a better alternative, please suggest it in the comment. Will appreciate it, that's partly why I wrote this little blog post.

Tuesday, October 29, 2013

Functional vs Imperative - a small example from the real world.

Recently I ported a small piece of Ruby(on Rails) code into Scala. Nothing fancy, I just want to share with developers who are interested but not yet start coding in functional programming paradigm.

The code is on the server side and the business logic is simple: from the incoming request, read some session information from the cookie; send it to an external service which validates it and returns a username; find the user with the username from the DB and finally return the user information in Json response.

First, the imperative Ruby code:

Action in the controller
 def validate_user
      user_name = UserSession.new(cookie).validate_username
      user = User.find_by_username(username) if user_name
      if user
        render json: {user: user.to_json }
      else
        render json: {error: "Cannot find user"}
      end 
    rescue => e
      render json: { error: e.message }
    end
  end
Library class that does the validation
class UserSession
  def validate_username
    return nil if [@user,@user_id,@session_key].include?(nil)
    make_http_request
    unless @response.nil? && @response.body.nil?
      json_resp = MultiJson.load(@response.body)
      if user_json["response"] && json_resp["response"]["auth_status"] == "Success"
        json_resp["response"]["user_name"] 
      end
    end
  end

  def initialize(cookies)
    @user_id = cookies["user_id"] 
    @user = cookies["user"] 
    @session_key = cookies["s"]
  end

  def make_http_request
    url = $API_URL + '?t=user&action=vrf'
    url += '&api_client_token=' + $USERAPI_TOKEN
    url += "&user_id=#{@user_id}&user=#{@user}&s=#{CGI.escape(@session_key)}"
    escaped_url = URI.escape(url)
    uri = URI.parse(escaped_url)
    http = Net::HTTP.new(uri.host, uri.port)
    request = Net::HTTP::Get.new(uri.request_uri)
    @response = http.request(request)
  end

end

It's typical imperative paradigm code that thanks to ruby is reasonably concise and expressive. The process keeps updating a set of mutable states until it gets the final result.

Now let's look at the scala one. There is a difference in functionality that the scala code is asynchronous so that the process is not blocked while waiting for the external service validating the session info. Also the Scala code provides more specific error message for each exception scenario.

Action in controller

  def validateUser = Action.async { implicit request =>
    UserSession.from(request.cookies).map { si =>
      si.validateUsername.right.map(User.findByUsername(_)).map { 
        case Left(errors) =>  Unauthorized(Json.toJson("error" -> errors))
        case Right(Some(u)) => Ok(Json.toJson(u))
        case Right(None) => NotFound
      }
    }.getOrElse(Future.successful(Unauthorized(Json.toJson("error" -> "Not Session In Cookie")))
  }

Library class that does the validation.

case class UserSession(userId: String, user: String, sessionKey: String) {
  lazy val apiUrl = WS.url(API_URL).withQueryString("api_client_token"->API_TOKEN)

  def validateUsername: Future[Either[ValidationErrors,String]] = {
    apiUrl.withQueryString(
        "t" -> "user",
        "action" -> "vrf_sess",
        "user_id" -> userId,
        "user" -> user,
        "s" -> sessionKey
    ).get.map { resp =>
      Json.parse(resp.body).validate(resultReads).fold[Either[ValidationErrors, String]](
        valid = result => Right(result._2),
        invalid = errs => Left(errs)
      )
    }
  }

  lazy val resultReads: Reads[(String, String)] = 
    (__ \ "response" ).read (
      (__ \ "auth_status" ).read[String](equalReads("Success")) ~
      (__ \ "username" ).read[String]
      tupled
    )
}

object UserSession {
  def from(cookies: Cookies): Option[UserSession] = {
    for {
      userId <- cookies.get("user_id")
      user <- cookies.get("user")
      sessionKey <- cookies.get("s")
    } yield UserSession(userId.value, user.value, sessionKey.value)
  }
}

In the functional programming paradigm, in stead of having a set of intermediate mutable states, computation is more often carried forward through a series of data transformation. In the controller, it first uses the SessionInfo.from method to create a SessionInfo out of cookie, then use the validateUsername to transform (map) it into a Future of Either. A Future represents the result of an asynchronous process, it was due the fact that the call to the external service is asynchronous. Either is scala's way to return either the result when all things go well or an error when something is wrong. So in our code, it's an Either between the validated username and the validation errors. The code then transform the right branch of it from a username into an Option of User by querying the DB. Finally, the code used a pattern match to map different possible value of the Either into different types of Http response to be returned to the client.

One of the contrasts between the Scala and Ruby code is how they handle null differently. In Ruby, if statements of null checks are everywhere while in the Scala it's mostly handled using Option. One example is how SessionInfo is generated from cookie. SessionInfo requires all three cookies present. In Ruby this is implemented using the following check.
  return nil if [@user,@user_id,@session_key].include?(nil)
It returns nil as a result if any of the cookies is missing. In Scala, it uses the for syntax sugar to map 3 cookie options into an option of SessionInfo.
  def from(cookies: Cookies): Option[UserSession] = {
    for {
      userId <- cookies.get("user_id")
      user <- cookies.get("user")
      sessionKey <- cookies.get("s")
    } yield UserSession(userId.value, user.value, sessionKey.value)
  }
cookies.get(key) returns an Option of cookie. If the key exists, it returns a Some(cookie) otherwise it returns a None, the for structure yields a Some(UserSession) only when all these cookie Options presents, a None when any of these cookies is a None. When the controller uses this Option of SessionInfo, it transform it into an Option of a http response to be returned to the client (actually, a Future of the response), and finally calls getOrElse on the Option to get the inner http response out of it. The getOrElse requires a default value when the Option is None, in our case, we used Future.successful(Unauthorized(Json.toJson("error" -> "Not Session In Cookie"))), which is a json response saying the session info is not in the cookie.

Another interesting piece of difference between the two examples is their implementation of the Json validation. In the Ruby code, the validation is performed through several if statements, such as

if user_json["response"] && json_resp["response"]["auth_status"] == "Success"
In scala, we created a structured Reads that both read and validate the json response from the external validation service.
  val resultReads: Reads[(String, String)] = 
    (__ \ "response" ).read (
      (__ \ "auth_status" ).read[String](equalReads("Success")) ~
      (__ \ "username" ).read[String]
      tupled
    )
This Reads presents an expressive way to specify the Json format we expect from the response. The code then use a fold method to handle the regular read results and when external service returned a unexpected format of Json.
   Json.parse(resp.body).validate(resultReads).fold[Either[ValidationErrors, String]](
        valid = result => Right(result._2),  //when reads succeeds
        invalid = errs => Left(errs)         //when the response json doesn't conform to the structure specified in the reads
      )

This reflects the difference between the two philosophies behind Scala and Ruby. Scala leans towards structured statically checked computation while Ruby is more about being fast, flexible and dynamic.

Conclusion

It is very easy to see that functional programming takes a very different approach towards programming from the more traditional imperative way. I hope this tiny real world example gives you some taste of how it feels like to apply functional programming in your day to day coding.

Thursday, August 29, 2013

CSRF protection in a Play+AngularJS application

From Wikipedia:
Cross-site request forgery, also known as a one-click attack or session riding and abbreviated as CSRF (sometimes pronounced sea-surf[1]) or XSRF, is a type of malicious exploit of a website whereby unauthorized commands are transmitted from a user that the website trusts.[2] Unlike cross-site scripting (XSS), which exploits the trust a user has for a particular site, CSRF exploits the trust that a site has in a user's browser.
There are plenty of articles online explaining such attacks and measures against them. The basic idea of such protection is to generate a CSRF token per session and send that token in every following request through means other than the Cookie request header (because it's the one that browser automatically set and hence the vulnerability). Typically the token can be sent back in a custom request header or in the request query string or an hidden input value in forms. Play has a built-in filter that implements such token based CSFR protection(here is a great blog article about it). However it's based on html form(with automatic query string generation). In a single-page application that relies on Play as a pure server-side API, most requests are performed by AJAX without involving html form. In that case CSRF token has to be sent through a custom request header. Our application uses AngularJS as our front-end framework. AngularJS also provides some facility to guard against CSRF (from AngularJS' documentation):
Angular provides a mechanism to counter XSRF. When performing XHR requests, the $http service reads a token from a cookie (by default, XSRF-TOKEN) and sets it as an HTTP header (X-XSRF-TOKEN). Since only JavaScript that runs on your domain could read the cookie, your server can be assured that the XHR came from JavaScript running on your domain. The header will not be set for cross-domain requests. To take advantage of this, your server needs to set a token in a JavaScript readable session cookie called XSRF-TOKEN on the first HTTP GET request. On subsequent XHR requests the server can verify that the cookie matches X-XSRF-TOKEN HTTP header, and therefore be sure that only JavaScript running on your domain could have sent the request.
So putting pieces together here is our implementation for CSFR protection in an AngularJS + Play application. The AngularJS side is automatic, so the work is at the play side to set token in the cookie and validate it.

When login succeeds, set the cookie that AngularJS consumers:
    Ok.withSession(("username" -> username).
       withCookies(Cookie("XSRF-TOKEN", createCSFRToken(username), httpOnly = false))
createCSFRToken is a function that hashes the username to a token so that the token is not reconstructible by attackers. httpOnly = true is there so that the cookie is readable by javascript (from the same domain of course).

For the following requests, here is the code to validate the token sent in the custom header (automatically set by AngularJS):
  def validateSessionUser(implicit request: RequestHeader): Option[String] = {
    request.session.get("username").filter { username =>
      request.headers.get("X-XSRF-TOKEN") == Some(createCSFRToken(username))
    }
  }
That's it. Feel free to ask any question in the comments.

Friday, June 14, 2013

Static type system over dynamic language - short stories.

About 5 years ago, I was still a "pure" enterprise developer - meaning I only knew Java and C#. One of my teammates on our Java project was a Ruby veteran who just switched to Java. He kept complaining that he underestimated how much less agile Java is in comparison with Ruby. I just couldn't understand why he had such feelings against Java - "one of the most beautiful Object Oriented programming language in the world."

Fast forward to two months ago, Ruby/Coffeescript has been my primary language for about 4 years. I got the opportunity to switch to a scala project, and boy I was excited.

But not before soon that excitement was replaced by the frustration of having to work with all the limitations imposed by the static type system, especially when it came to some generic library I tried to write. I admit that scala's complex type system wasn't helpful either. My memory of my Ruby veteran co-worker complaining about Java became really vivid. I now understand exactly how he felt!

Fast forward again to now, two months later, I reached the point that I started to see advantages of static type system over dynamic language other than performance.

I just wrote a web socket library with play and akka. It was not a large one - 5,6 classes that took me about 5 hours to design and write. I don't normally do this but because it was the 3rd time I rewrite this library, I took the "write it all in one run and throw it against the wall" approach. It took me quite some time to get all the code compiles. The turning point was after that, it only took me 5 minutes to debug the whole socket to work perfectly with my tests and my front-end application. This was unimaginable when I was still writing dynamic language and is consistent with my recent experience with scala - compilation check found more errors (including real-time check in IDE) than tests.

In a static type language, if you design carefully (and follow some functional programming principles), chances are that the majority of the human errors will be detected during compilation time. Whoever said that compilation check is close to useless is clearly wrong this time.

Plus, designing in a static type language is a lot more fun - you have more structures to design as well as more limitation to work with.

Tuesday, May 28, 2013

Why write good code (in a simple way)

Recently I saw a trend on Hacker News (like this article and this one) that people start to question why write good code, and whether we should let all the principles for "good code" prevent us from being productive. I would like to try answer that question in a simple way.

First thing first, forget about all the coding principles that people (through the Internet) try to shove into your head - SOLID, DRY, Code smells, Design Patterns, etc. You can write good code without them. One principle is enough: have sympathy towards other developers in the team (including yourself after 6 months). Good code is code that is easy to work with, namely:
  • Easy for other developers to read the code and understand the business logic
  • Easy for the other developer to locate and change the code when they need to change/enhance the business logic
All other principles can be helpful guidance (sometimes), but these two are what really matters. (I explained why in this post.)

Now, with the definition of good code set, let's answer why we should try write good code rather than code that just works, or in a more appealing term, focuses on delivering business value.

Sympathy (or indifference) is contagious.

Yes, it might be a valid strategy to write just-works code to get things started and fix them later. However that's only valid when you are in a solo project in which you have complete control. You will lost that control quickly if it's a team effort. We all know the broken window theory. In my career I've seen very few software engineering theories more true than it. If you don't make a sincere effort to avoid giving your team members some hard times, chances are they won't either. It won't take very long for the code base to become nothing but nightmares - and never ever underestimate the capability of human producing worse and worse nightmares. On the other hand, if you show them good examples of code that are easier to work with, they will follow, very quickly. It's proven to me again and again.

Writing good code isn't that intimidating.

If you were scared by the numerous coding principles, don't be, just forget about them. The two criterion of good code mentioned above are also the two most common tasks you do everyday, so it shouldn't be hard for you to assess your own code based on them. Just think as if you were another developer, that is, be considerate. It's not intimidating, it's rather basic.
What about all those other principles? Are they just BS? Of course not, it's just that you don't need to remember them. In most cases, if you follow the two easy-to-work-with criterion, your code will naturally comply with them. For example, if you want your code to be easy to find and change given the logic (criteria #2), you would avoid writing duplicated code(or logic) - you know how hard to change business logic implemented with duplication - and that's the DRY principle.

Writing good code makes it fun

It adds a new layer of challenges. Again, back to the point 2, it's not that many more challenges - remembering and following dozens of code principles might be, but thinking as if you were another team member isn't. The right amount of challenges is necessary for me to have fun at work, if you do too, that is another good reason to write good code.

Join the discussion on the Hacker News.

Saturday, May 25, 2013

Case class enumeration in Play 2.1 Application (on top of Salat/MongoDB)

Recently in our Play 2.1(Scala) application I needed to implement a model that's like Java enum. Since scala's Enumeration class doesn't have a fantastic reputation, I decided to go with the case class approach. Our Play 2.1 application is a JSON REST API that sits on top of MongoDB, so I need to make sure that this model can have both JSON and MongoDBObject serialization wired up.
Let's start from Enum base trait
trait Enum[A] {
  trait Value { self: A => }
  val values: List[A]
  def parse(v: String) : Option[A] = values.find(_.toString() == v)
}
We'll see the use of the parse function later.

In our system we have a flag system for content entries. Here is the flag case class enum.
import com.novus.salat.annotations._

@Salat
sealed trait Flag extends Flag.Value

case object DEPRESSING extends Flag
case object VIOLENCE extends Flag
case object NSFW extends Flag

object Flag extends Enum[Flag] with SimpleEnumJson[Flag] {
  val values = List(DEPRESSING, VIOLENCE, NSFW)
}
The @Salat is needed for salat to map abstract class inheritance tree. SimpleEnumJson[T] is the one that provides Play 2.1 JSON API formats which you will need for JSON serialization. Here is its implementation.
import play.api.libs.json._

trait SimpleEnumJson[A] {
  self: Enum[A] =>

  implicit def reads: Reads[A] = new Reads[A] {
    def reads(json: JsValue): JsResult[A] = json match {
      case JsString(v) => parse(v) match {
        case Some(a) => JsSuccess(a)
        case _ => JsError(s"String value ($v) is not a valid enum item ")
      }
      case _ => JsError("String value expected")
    }
  }

  implicit def writes[A]: Writes[A] = new Writes[A] {
    def writes(v: A): JsValue = JsString(v.toString)
  }
}
As the name suggests, SimpleEnumJson uses the simple toString() of the enum item for JSON representation.
Note that we could place the items definitions inside the companion object, but unfortunately salat couldn't map the class that way.
This should be good enough for most cases, a simple case class based enum that can be used in any models.
Now let's make it a little more complex, let's say we need one more attribute in our Flag class: age rating - each flag indicates that the content is only appropriate for users above a certain age.
@Salat
sealed trait Flag extends Flag.Value {
  val ageRating : Int
}
case object DEPRESSING extends Flag { val ageRating = 18 }
case object VIOLENCE extends Flag { val ageRating = 16 }
case object NSFW extends Flag { val ageRating = 21 }
Now we need a bit more work for our JSON reads/writes, the SimpleEnumJSON[T] is no longer sufficient. Let's write a CompositeEnumJSON[T]
trait CompositeEnumJson[A] {
  self: Enum[A] =>
  implicit val reads = (__ \ 'name).read[String].map( parse(_).get )
  val nameWrite = (__ \ 'name).write[String]
  implicit val writes : OWrites[A]
}
Here we map a composite case class enum into a JsObject with a name field that holds the toString() value of the item. When we read from JSON, we only need the "name" field. We need concrete inheriting case class to implement the Json write:
object Flag extends Enum[Flag] with CompositeEnumJson[Flag] {
  val values = List(DEPRESSING, VIOLENCE, NSFW)
  implicit val writes : OWrites[Flag] = (
    nameWrite ~
    (__ \ 'ageRating).write[Int]
  )(unlift(unapply))

  def unapply(flag: Flag) = {
    Some(( flag.toString(), flag.ageRating ))
  }
}
There you have it - a composite case class based enum that can have attributes.
I hope you find this helpful and let me know if you have any suggestion for improvements.

Tuesday, May 07, 2013

Define arbitrary tasks in the Play 2.1 Framework (Rakes like)

Recently with the help from James Roper from Typesafe, I added some arbitrary tasks to our application so that we can seed some data from command line. These tasks are basically like rake tasks in which you can run fire up the application environment and perform code written for you application.

The idea is to create SBT tasks and then, quoted from James, "get a hold of the dependencies, create a new URLClassLoader with them, load the class you want to use from that classloader using reflection, and then invoke it."

Here is the code in the SBT build file Build.scala
import java.net.URLClassLoader
object ApplicationBuild extends Build {
//Other project settings such as appVersion and appDependencies

def registerTask(name: String, taskClass: String, description: String) = {
  val sbtTask = (dependencyClasspath in Runtime) map { (deps) =>
    val depURLs = deps.map(_.data.toURI.toURL).toArray
    val classLoader = new URLClassLoader(depURLs, null)
    val task = classLoader.
                 loadClass(taskClass).
                 newInstance().
                 asInstanceOf[Runnable]
    task.run()
  }
  TaskKey[Unit](name, description) <<= sbtTask.dependsOn(compile in Compile)
}

val main = play.Project(appName, appVersion, appDependencies).settings(
  registerTask("seed-data-book","tasks.SeedBooks", "seed for book table" ),
  registerTask("seed-data-user","tasks.SeedUsers", "seed for user table" )
)

Here is our task classes (I placed it in app/tasks)
package tasks
import play.core.StaticApplication

abstract class Task extends Runnable { 
  val application = new StaticApplication(new java.io.File("."))
}
    
class SeedBooks extends Task{
  def run {
    Book.dao.save(new Book("Introduction to Scala", "Martin Odersky"))
  }
}
    
class SeedUsers extends Task{
  def run {
    User.dao.save(new User("jRoper", "password"))
  }
}
I hope this helps. Feel free to ask any questions in the comments.