Showing posts with label ruby. Show all posts
Showing posts with label ruby. Show all posts

Tuesday, October 29, 2013

Functional vs Imperative - a small example from the real world.

Recently I ported a small piece of Ruby(on Rails) code into Scala. Nothing fancy, I just want to share with developers who are interested but not yet start coding in functional programming paradigm.

The code is on the server side and the business logic is simple: from the incoming request, read some session information from the cookie; send it to an external service which validates it and returns a username; find the user with the username from the DB and finally return the user information in Json response.

First, the imperative Ruby code:

Action in the controller
 def validate_user
      user_name = UserSession.new(cookie).validate_username
      user = User.find_by_username(username) if user_name
      if user
        render json: {user: user.to_json }
      else
        render json: {error: "Cannot find user"}
      end 
    rescue => e
      render json: { error: e.message }
    end
  end
Library class that does the validation
class UserSession
  def validate_username
    return nil if [@user,@user_id,@session_key].include?(nil)
    make_http_request
    unless @response.nil? && @response.body.nil?
      json_resp = MultiJson.load(@response.body)
      if user_json["response"] && json_resp["response"]["auth_status"] == "Success"
        json_resp["response"]["user_name"] 
      end
    end
  end

  def initialize(cookies)
    @user_id = cookies["user_id"] 
    @user = cookies["user"] 
    @session_key = cookies["s"]
  end

  def make_http_request
    url = $API_URL + '?t=user&action=vrf'
    url += '&api_client_token=' + $USERAPI_TOKEN
    url += "&user_id=#{@user_id}&user=#{@user}&s=#{CGI.escape(@session_key)}"
    escaped_url = URI.escape(url)
    uri = URI.parse(escaped_url)
    http = Net::HTTP.new(uri.host, uri.port)
    request = Net::HTTP::Get.new(uri.request_uri)
    @response = http.request(request)
  end

end

It's typical imperative paradigm code that thanks to ruby is reasonably concise and expressive. The process keeps updating a set of mutable states until it gets the final result.

Now let's look at the scala one. There is a difference in functionality that the scala code is asynchronous so that the process is not blocked while waiting for the external service validating the session info. Also the Scala code provides more specific error message for each exception scenario.

Action in controller

  def validateUser = Action.async { implicit request =>
    UserSession.from(request.cookies).map { si =>
      si.validateUsername.right.map(User.findByUsername(_)).map { 
        case Left(errors) =>  Unauthorized(Json.toJson("error" -> errors))
        case Right(Some(u)) => Ok(Json.toJson(u))
        case Right(None) => NotFound
      }
    }.getOrElse(Future.successful(Unauthorized(Json.toJson("error" -> "Not Session In Cookie")))
  }

Library class that does the validation.

case class UserSession(userId: String, user: String, sessionKey: String) {
  lazy val apiUrl = WS.url(API_URL).withQueryString("api_client_token"->API_TOKEN)

  def validateUsername: Future[Either[ValidationErrors,String]] = {
    apiUrl.withQueryString(
        "t" -> "user",
        "action" -> "vrf_sess",
        "user_id" -> userId,
        "user" -> user,
        "s" -> sessionKey
    ).get.map { resp =>
      Json.parse(resp.body).validate(resultReads).fold[Either[ValidationErrors, String]](
        valid = result => Right(result._2),
        invalid = errs => Left(errs)
      )
    }
  }

  lazy val resultReads: Reads[(String, String)] = 
    (__ \ "response" ).read (
      (__ \ "auth_status" ).read[String](equalReads("Success")) ~
      (__ \ "username" ).read[String]
      tupled
    )
}

object UserSession {
  def from(cookies: Cookies): Option[UserSession] = {
    for {
      userId <- cookies.get("user_id")
      user <- cookies.get("user")
      sessionKey <- cookies.get("s")
    } yield UserSession(userId.value, user.value, sessionKey.value)
  }
}

In the functional programming paradigm, in stead of having a set of intermediate mutable states, computation is more often carried forward through a series of data transformation. In the controller, it first uses the SessionInfo.from method to create a SessionInfo out of cookie, then use the validateUsername to transform (map) it into a Future of Either. A Future represents the result of an asynchronous process, it was due the fact that the call to the external service is asynchronous. Either is scala's way to return either the result when all things go well or an error when something is wrong. So in our code, it's an Either between the validated username and the validation errors. The code then transform the right branch of it from a username into an Option of User by querying the DB. Finally, the code used a pattern match to map different possible value of the Either into different types of Http response to be returned to the client.

One of the contrasts between the Scala and Ruby code is how they handle null differently. In Ruby, if statements of null checks are everywhere while in the Scala it's mostly handled using Option. One example is how SessionInfo is generated from cookie. SessionInfo requires all three cookies present. In Ruby this is implemented using the following check.
  return nil if [@user,@user_id,@session_key].include?(nil)
It returns nil as a result if any of the cookies is missing. In Scala, it uses the for syntax sugar to map 3 cookie options into an option of SessionInfo.
  def from(cookies: Cookies): Option[UserSession] = {
    for {
      userId <- cookies.get("user_id")
      user <- cookies.get("user")
      sessionKey <- cookies.get("s")
    } yield UserSession(userId.value, user.value, sessionKey.value)
  }
cookies.get(key) returns an Option of cookie. If the key exists, it returns a Some(cookie) otherwise it returns a None, the for structure yields a Some(UserSession) only when all these cookie Options presents, a None when any of these cookies is a None. When the controller uses this Option of SessionInfo, it transform it into an Option of a http response to be returned to the client (actually, a Future of the response), and finally calls getOrElse on the Option to get the inner http response out of it. The getOrElse requires a default value when the Option is None, in our case, we used Future.successful(Unauthorized(Json.toJson("error" -> "Not Session In Cookie"))), which is a json response saying the session info is not in the cookie.

Another interesting piece of difference between the two examples is their implementation of the Json validation. In the Ruby code, the validation is performed through several if statements, such as

if user_json["response"] && json_resp["response"]["auth_status"] == "Success"
In scala, we created a structured Reads that both read and validate the json response from the external validation service.
  val resultReads: Reads[(String, String)] = 
    (__ \ "response" ).read (
      (__ \ "auth_status" ).read[String](equalReads("Success")) ~
      (__ \ "username" ).read[String]
      tupled
    )
This Reads presents an expressive way to specify the Json format we expect from the response. The code then use a fold method to handle the regular read results and when external service returned a unexpected format of Json.
   Json.parse(resp.body).validate(resultReads).fold[Either[ValidationErrors, String]](
        valid = result => Right(result._2),  //when reads succeeds
        invalid = errs => Left(errs)         //when the response json doesn't conform to the structure specified in the reads
      )

This reflects the difference between the two philosophies behind Scala and Ruby. Scala leans towards structured statically checked computation while Ruby is more about being fast, flexible and dynamic.

Conclusion

It is very easy to see that functional programming takes a very different approach towards programming from the more traditional imperative way. I hope this tiny real world example gives you some taste of how it feels like to apply functional programming in your day to day coding.

Friday, June 14, 2013

Static type system over dynamic language - short stories.

About 5 years ago, I was still a "pure" enterprise developer - meaning I only knew Java and C#. One of my teammates on our Java project was a Ruby veteran who just switched to Java. He kept complaining that he underestimated how much less agile Java is in comparison with Ruby. I just couldn't understand why he had such feelings against Java - "one of the most beautiful Object Oriented programming language in the world."

Fast forward to two months ago, Ruby/Coffeescript has been my primary language for about 4 years. I got the opportunity to switch to a scala project, and boy I was excited.

But not before soon that excitement was replaced by the frustration of having to work with all the limitations imposed by the static type system, especially when it came to some generic library I tried to write. I admit that scala's complex type system wasn't helpful either. My memory of my Ruby veteran co-worker complaining about Java became really vivid. I now understand exactly how he felt!

Fast forward again to now, two months later, I reached the point that I started to see advantages of static type system over dynamic language other than performance.

I just wrote a web socket library with play and akka. It was not a large one - 5,6 classes that took me about 5 hours to design and write. I don't normally do this but because it was the 3rd time I rewrite this library, I took the "write it all in one run and throw it against the wall" approach. It took me quite some time to get all the code compiles. The turning point was after that, it only took me 5 minutes to debug the whole socket to work perfectly with my tests and my front-end application. This was unimaginable when I was still writing dynamic language and is consistent with my recent experience with scala - compilation check found more errors (including real-time check in IDE) than tests.

In a static type language, if you design carefully (and follow some functional programming principles), chances are that the majority of the human errors will be detected during compilation time. Whoever said that compilation check is close to useless is clearly wrong this time.

Plus, designing in a static type language is a lot more fun - you have more structures to design as well as more limitation to work with.

Thursday, May 03, 2012

Some selenium tips

UPDATES
  • May 31, 2012: Added the tip to capture the screen when test fails.
  • May 11, 2012: Added the tip that about keeping the browser window displaying (or hidden) during the tests.
  • May 11, 2012: Rephrase the scroll to a button before clicking it tip.

Watch out the outdated articles on the internet.

Selenium 2.0 is completely different from Selenium 1.x. Selenium 2.0 is also called the selenium webdriver. So always add the keyword webdriver when googling for answers to your selenium related questions.

Implement the web UI in a modular way so it's more selenium testable.

Modularize your view logic so that you only update the part of DOM that is needed to change when your models change. If you tend to re-create a bigger part of the DOM than necessary, it's not only a waste but also could introduce risk to your functional tests written in Selenium.

Reduce unnecessary dependency on DOM structure, make element locating logic as simple as possible.

When you need to locate an element, try not rely on the DOM structure too much - for example, using code logic to locate element is the most risky one. The best approach is probably to always use a scoped CSS selector with 1 or 2 levels of IDs, And if you can locate it in one selector, don't do it in two. For example
  label = driver.find_element("#info-panel #name-label")
is more robust than
  panel = driver.find_element("#info-panel")
  label = panel.find_element("#name-label")

Do waiting in selenium the smart way.

Don't use implicit wait blindly. Implicit wait makes sense when you use find_element to find one element. But when you try to locate multiple elements by driver.find_elements, the driver will always wait the whole timeout period if implicit wait is set. That might not be what you always want. I usually write my own safe find_element method. Here is an example in the base class of my page objects:
    def s selector
      wait_until { @driver.find_element css: selector }
    end

    def wait_until(&block)
      wait = Selenium::WebDriver::Wait.new(timeout: 10, interval: INTERVAL)
      wait.until &block
    end
So that I can write the following code in my page object
   def submit_order
     s('button#submit').click
   end
The short method name "s" is inspired by jQuery. Here it will keep polling the DOM for 10 seconds until it finds the button with id "submit". It's like implicit wait but only for finding one element. When you really need to wait for multiple elements, you can use an explicit wait, which, to me, makes more sense than a hidden implicit one.

Set the initial browser window size when using Chromedriver.

Ruby code:
  profile = Selenium::WebDriver::Chrome::Profile.new
  profile['browser.window_placement.top'] = 0
  profile['browser.window_placement.left'] = 0
  profile['browser.window_placement.right'] = 1024
  profile['browser.window_placement.bottom'] = 768
  driver = Selenium::WebDriver.for :chrome, profile: profile
This works in both Windows and OSX (will try Linux and update here)
Bad news for Java, C# and Python coders though, it seems that as of now setting chrome preference is not supported in the java version of Webdrive. Your best chance could be creating a ChromeProfile class based on the exiting FirefoxProfile class.

Scroll to a button before clicking it.

Clicking buttons sometimes randomly fail. It could be caused by the fact that the button has to be the view area to be clickable and somehow the selenium auto scroll failed. In this case, add a scroll to button will improve the robustness of your suite.

When running the test using Firefox, it matters whether the browser window is displayed on the screen or hidden behind other windows.

From my experience, my guess is that selenium interacts with the browser in slightly different ways depending on if the browser is displaying in the front or not. There is rare case that certain selenium operations only work when the Firefox is displaying in the front. When running the selenium suite, changing the z position of the browser window (and thus either show or hide the browser window from other applications) can affect the tests. So you get more consistent results by keeping the browser either showing in the front(or hidden in the back) during the course of full suite.

Capture screenshots when test fails (RSpec)

In your spec_helper.rb
  RSpec.configure do |config|
    config.after(:each) do
      capture_screen_when_fails(example, @page) 
    end

    def capture_screen_when_fails example, page
      if(example.exception.present? and page.present?)
        page.capture_screen(example.description) 
      end
    end
  end
Note here that you need to keep your page object instance in an instance variable(in my case @page) in your spec. I used a naive way to name the screenshot after the example's description.
Now in your base class for your page object.
    def capture_screen filename
      path = "PATH_TO_TMP/#{filename}.png"
      @driver.save_screenshot(path) 
    end
Note that @driver is the instance variable holding the selenium webdriver currently running.

Tuesday, April 26, 2011

Introducing Collectr - my first flickr App

Alpha test: http://collectr.kailuowang.com/

What is Collectr?
Collectr is a web app that helps you, a flickr addicted, subscribe and explore flickr photos in a much more powerful and personalized way.
What's the vison of Collectr?
To achieve what flickr explore failed to achieve - an easy and free way to see more interesting pictures every day.
Why use Collectr?
  • Centralized slide show
  • New photos from multiple sources will be displayed at a single slide show
  • Personalized slide show
  • Collectr remembers your preference by recording your 'fave' action, so that when you have too many new photos from sources,
    it will display first the photos from the stream swhose photos you fave the most in the past.
  • Expendable Sources of Photos
  • With Collectr, not only can you subscribe to someone's upload stream, you can also subscribe to her favorites stream.
    This way you can discover flickr artists that are discovered by your favorite flickr artists.
    This is very important because it allows you to expand your list of sources of good photos.
  • Slide show with the best possible image quality
  • Unlike many of the third party flickr websites, when the collectr slide show display photos, it will try find the version of the photos whose reslution fits your screen the most. Also, this slide show is tablet friendly with navigation keys on the sides and links using larger font.
  • Easy share
  • You can share/backup your collections by exporting them into a backup file so that later you or other Collectr users can import it.

Friday, November 12, 2010

Inheritance and class variable

Yesterday I had a great lunch & learn presented by Brian Guthrie titled "Advanced Ruby Idioms So Clean You Can Eat Off Of Them"
It was a great talk and we all learned something interesting about ruby. After the talk Brian and I had a quick discussion about one of the design problems solved in the talk. It is about the inheritance problem in the custom validation framework. That is the subclass (Captain) didn't inherit the validators from the super class(Pirate).
Here is the code


Model classes
class Pirate < BrianRecord::Base
validates_presence_of :parrot
attr_reader :parrot
def initialize(parrot=nil)
@parrot = parrot
end
end

class Captain < Pirate
validates_presence_of :peg_leg
attr_reader :peg_leg
def initialize(parrot, peg_leg)
@parrot, @peg_leg = parrot, peg_leg
end
end




Framework base class
module BrianRecord
class Base

class << self
def validators
@validators ||= []
end

def validates_presence_of(attribute, opts={})
validators << BrianRecord::PresenceOfValidator.new(attribute, opts)
end
end

def valid?
validators.each do |validator|
is_valid = validator.validate(self)
raise validator.message unless is_valid
end
end
end

class PresenceOfValidator
def initialize(attribute, opts={})
@attribute = attribute
end

def message
"expected #{@attribute} to not be nil"
end

def validate(object)
return !object.send(@attribute).nil?
end
end
end


The problem here is that the Captain class lost the validation on parrot attribute which means that you can do the following:

failed scenario
Captain.new(nil, "wood").valid? #=>true, not as intended

The root cause here is that the @validator is only available to a certain class, both Captain and Pirate have their own @validators, so the validate method only iterate thru the @validator available to its own class.
The quick fix Brian provided in the talk was to duplicate the superclass's @validators when initialized the @validators for the subclass, as the following

Framework base class
module BrianRecord
class Base
class << self
def validators
@validators ||= if(superclass.respond_to(:validators)
superclass.validators.dup
else [] end
end
...

This is fine and concise but with only one small glitch, if an validator is added to the superclass after the subclass is loaded, that validator will not be available to the subclass. I brought up another possible solution and we were not sure if it will work. So I didn't a bit coding practice and it looks that the following code also works.


Framework base class
module BrianRecord
class Base

class << self


def validates_presence_of(attribute, opts={})
validators << BrianRecord::PresenceOfValidator.new(attribute, opts)
end

def validate obj
superclass.validate obj if superclass.respond_to?(:validate)
validators.each do |validator|
is_valid = validator.validate(obj)
raise validator.message unless is_valid
end
end

private
def validators
@validators ||= []
end

end

def valid?
self.class.validate self
end
end

In this solution, I move the validate logic to the class, and recursively calls the superclass' validate method if its available. This removed the duplication of the @validators and thus avoid the problem where dynamically added validators will be missing in the subclass. Note here that now that we have a public "validate" class method instead of the public "validators" accessor, which is private now. I would argue that exposing the validate method is better than exposing the validators.

Sunday, August 29, 2010

Testing private methods in RSpec

Why testing private methods? Well, it's not really about private vs public, if you want really fine granular unit tests that always only test no more than a couple of lines of code, you will need to do partial mocking and test private methods.

I heard argument that testing private methods exposes too much implementation and thus makes later refactoring harder. My argument is that unit test is part of the implementation. Fine grained "real unit" tests is very easy to read and understand. They help clarify the intent of that couple of lines of code in your target class. If you change implementation code, it should be perfect normal if you also need to change that simple unit test. On the other hand if your tests are in a larger granularity, then in each test, either you test a lot of code or your use a lot of mocking. Either case, chances are whenever you change implementation, you would need to change even more test code.

Another common practice is to extract private methods to another class and make them public and test from there. To me, there are only a few valid reasons to introduce a new class (or in general, to design), being able to test private methods isn't one of them.

Alright, with excuses all said (your argument is welcome), here is how I test private methods in ruby with rspec. I defined a global method in a file called describe_internally.rb in my test folder

def describe_internally *args, &block
example = describe *args, &block
klass = args[0]
if klass.is_a? Class
saved_private_instance_methods = klass.private_instance_methods
example.before do
klass.class_eval { public *saved_private_instance_methods }
end
example.after do
klass.class_eval { private *saved_private_instance_methods }
end
end
end

then whenever I need to test private methods for a class (say Foo), I use "describe_internally Foo" instead of "describe Foo". If you prefer, you can organize these two types of tests in the same file as the below example (say Foo is a class with two methods-a public one: "kick" and a private one: "aim" which returns the target to be kicked)

describe Foo do
describe "kick" do
it "should kick at where aim is" do
foo = Foo.new
foo.should_receive(:aim).with(steve_jobs).and_return :a_place
foo.kick steve_jobs
steve_jobs.should be_kicked_at :a_place
end
end
end
describe_internally Foo do
describe "aim" do
it "should aim at where the butt is" do
Foo.new.aim(steve_jobs).should be steve_jobs.butt
end
end
end

(Although in this case, I would just put all specs under the describe_internally block, because the first spec also requires knowledge of the private method and from my experience there is seldom any problem caused by that)
If you need an even more fine control of the scope of where you want private methods exposed, Jay Fields wrote a blog long ago giving another approach to achieve it http://blog.jayfields.com/2007/11/ruby-testing-private-methods.html