Showing posts with label software. Show all posts
Showing posts with label software. Show all posts

Thursday, 15 January 2015

Api Development in Ruby On Rails

Recently I wrote a series of blog posts about the best practices of API development in RoR and how to develop API in general.

Let me summarize those posts:


Happy API development :)

API development in Rails error path


In my series of of API development I haven't covered the error handling.

Let me share my experience with you about what I have learned about proper error handling.
I call error path the case when the user can't achieve what he wants.

This case can happen by the following reasons:

  • The user calls an invalid url
  • The user addresses a missing resource
  • The user misses a mandatory parameter
  • The uses passes a wrong parameter
  • Some other internal error occurs
  • ...

Since thousands of calls can be made daily or hourly against the API there is no way to stop the server and debug it. So all the information related to the erroneous call must be recorded. Basically all the information to reproduce the error must be recorded. I call this ApiCallAudit.

Such an ApiCallAudit must contain:

  • All the incoming parameters
  • The type of call (GET, POST, DELETE, ...)
  • cookies
  • backtrace
  • created_at
I added some additional fields to it for filtering purposes:

  • level. It serves to quickly determine the possible source of errors. It could be parameter_error, unexpected_error, parameter_missing_error, entity_missing_error
  • status. It is a default error message.
  • code. An error code which uniquely identifies the error.
In the beginning the error code was missing from my design and the mobile clients were using the default error message. This has some disadvantages:
  • The mobile UI is usually developed in a different codebase. And the server side error message modification is not possible. Specially if the same chunk of server side code is serving many applications.
  • -The language used on the mobile UI can vary. For example the UX developer can decide to use:
    1. "You haven't provided the group" - First person complaining style.
    2. "The group is missing" - Passive objective style
    3. "Please provide a group" - Proactive gentle style
    4. "Select a group!" - Imperative style
    5. ...
In order to formulate the sentences which reflects the application mood the mobile developer needs to interpret the returned error based on its error_code and formulate its own corresponding message.

Happy coding :)



API Development Restfull vs Facebook style


In the last months I developed mainly API's using grape gem and their related gems.

In this blog post I will express my opinion how to organise the API which manages the entities.

Generally speaking I like the REST concept and I organize my API to conform that way. For cases when there are no association among the entities this philosophy is good enough and clear.

However I like the API to reflect the associations when we are dealing with has_many or has_and_belongs_to_many associations. Just like in case of  Facebook Graph API. In these cases I find that style more intuitive and I am following that style whenever I deal with associations.

Examples:

Lets suppose that cars and manufacturers can be reviewed.

class Car < ActiveRecord::Base
  has_many :reviews, as: :reviewable

class Manufacturer < ActiveRecord::Base
  has_many :reviews, as: :reviewable

class Review < ActiveRecord::Base
  belongs_to :user
  belongs_to :reviewable, polymorphic: true

In this case the review API would be:
  • creation of reviews:
    POST /api/cars/{id}/reviews
    POST /api/manufacturers/{id}/reviews
  • retrieval of reviews:
    GET /api/cars/{id}/reviews
    GET /api/manufacturers/{id}/reviews
  • deletion of reviews:
    DELETE /api/reviews/{id}


In the classic restful style the review creation would be:

    params do
      requires :comment, type: String, desc: "The comment"
      requires :reviewable_id, type: Integer, desc: "The id of a reviewable entity"
      requires :reviewable_type, type: String, desc: "The type of a reviewable entity"
    end
    POST /api/reviews

Both solutions have pros and cons. The restful style is more DRY. The Facebook Graph Api style relieves more information about the associations ergo it is more intuitive and easier to use.

Happy coding :)



Friday, 2 January 2015

Specing the API


When you cover your API with specs the first rule is to cover everything. 
Since the API can be called by third parties you need to be sure what is happening in every case. You must be able to reproduce every scenario any time and you must have the same results. :) Does it seems scientific? Well... Indeed it is.

Let me show you some examples:
-You might need to check that only the calls with developer key can access the API. In the opposite case the response is 401, it is still a JSON and maybe an audit log is created.
-You might need to check that the user is authenticated. If not then the response is 401, it is still a JSON and again an audit log is created.
-If some of the parameters are missing or wrong then the response is 400, it is still a JSON and an audit log is created by logging all the incoming parameters.
-If everything is alright then the response is 200/201, the response is a JSON and the entities are massaged as needed by that specific case.

You also need to keep your specs DRY
This will reduce maintenance effort of the test code and it will keep your specs more readable and you will have easy time to add new features and you will feel more happy and in control.

As you can see the above steps like checking the response code, checking if the response format is JSON, checking that the audit log is created are repetitive tasks and can be DRY-ed with rspecs shared examples. The only specific thing which changes from one api endpoint to another is "the entities are massaged as needed by that specific case".

RSpec.shared_examples "returning 401" do
  specify "returns 401" do
    api_call params, developer_header
    expect(response.status).to eq(401)
  end
end
RSpec.shared_examples "being JSON" do
  specify 'returns JSON' do
    api_call params, developer_header
    expect { JSON.parse(response.body) }.not_to raise_error
  end
end
...

Then in your specs you can use these shared examples in particular cases:

context '/API/learn_by_playing/' do
  def api_call *params
    get "/api/learn_by_playing", *params
  end
  let(:api_key) { create :apikey }
  let(:developer_header) { {'Authorization' => api_key.token} }
  context 'GET' do
    let(:required_params) do
      {
        :first_name => 'Botond',
        :last_name => "Orban"
      }
    end
    let(:params) { required_params }

    it_behaves_like 'restricted for developers'

    context 'wrong parameters' do
      required_params.keys.each do |s|
        context 'when the #{s} parameter is blank' do
          let(:params) { required_params.merge({s => ''}) }
          it_behaves_like 'returning 400'
          it_behaves_like 'being JSON'
          it_behaves_like 'creating an audit log'
        end
        context 'when the #{s} parameter is missing' do
          let(:params) { required_params.except(s) }
          it_behaves_like 'returning 400'
          it_behaves_like 'being JSON'
          it_behaves_like 'creating an audit log'
        end
      end
    end
    context 'valid params' do
      specify '...whatever you need to really assert for in this particular case...' do
        api_call params, developer_header
        expect ...
      end
    end
  end
end

As you can see from the above example using Rspec.shared examples a lot of repetitive lines from your Rspec can be thrown out.

On one of my projects I managed to reduce the specs size by 20% and raise the visibility a lot :) I can't measure and therefore I can't express in numbers what "a lot" means but everybody was satisfied  and pleased with the result.

Professional DRY hacking ;)

Sunday, 28 December 2014

API Development in Ruby On Rails, Nested Entities

There are times when there is one-to-many association amongst the entities. And the mobile developer needs to show all of them in one screen. In these cases it is very easy to fell into the bad habit to execute n+1 queries from mobile side toward the server or from the server to the sql-server.

Here are the entities:

class DetailedGameEntity < Grape::Entity
  ...
  expose :reviews, using: ReviewEntity do |game, options|
    game.reviews
  end
end

class ReviewPresenter < Grape::Entity
  expose :user_id, as: :owner_id
  expose :comment
  expose :rating

  expose :image_urls do |review, options|
    review.images.map{|image| image.image}
  end
end

They are designed to return everything the mobile screen needs. The games contains their reviews. And the reviews contain their images. So in one query the whole mobile screen can be populated.

Let's see how a programmer in a hurry develops all this API:

games = Game.actual.limit(500)
present games, with: DetailedGameEntity

By developing the API this way the call will result in these queries:
...
  Review Load (0.5ms)  SELECT "reviews".* FROM "reviews"  WHERE "reviews"."reviewable_id" = $1 AND "reviews"."reviewable_type" = $2  [["reviewable_id", 2359], ["reviewable_type", "Game"]]
  Review Load (0.4ms)  SELECT "reviews".* FROM "reviews"  WHERE "reviews"."reviewable_id" = $1 AND "reviews"."reviewable_type" = $2  [["reviewable_id", 2358], ["reviewable_type", "Game"]]
...

As you can see for each returned game game another select is executed to fetch the belonging reviews. The same is done to fetch the belonging user and to fetch the images belonging to the review. This is a very ineffective way because as the review count grows more and more queries need to be executed. This problem is called n+1 query problem.

I do this in my code in these cases:

games = Game.actual.limit(500)
games = games.includes(reviews: [:images, :user] )
present games, with: DetailedGameEntity

The above code will do this sql query:

 Review Load (1.0ms)  SELECT "reviews".* FROM "reviews"  WHERE "reviews"."reviewable_type" = 'Schedule' AND "reviews"."reviewable_id" IN (2195, 2198, 2197, 9567, 9572, 9573, 9574, 9575, 9576, 9571, 9570, 9569, 9568, 2196, 2204, 2210, 2211, ...

So, the underlying Rails code will execute only one query per entity.

It is nice, isn't it?

Please use it ;)

Saturday, 27 December 2014

API Development in Ruby On Rails, Entities

In my last blog post I presented in general the features I like the most about the grape gem. Now I will present the grape entities which are related to the presentation of the returned data. I like these entities a lot because they are an OO way to present data and they help a lot to keep my presentation layer DRY.

The last statement of every grape call is returned to the caller as JSON.
Despite this simple efficiency I like to have more control over the returned data and I use the grape entities to format the returned data wherever I can. I can also rspec them and I am doing it extensively because I am a Test Driven Guy ;)

Let me present some examples of entities their usage and how do I spec them:

Here is the spec for the entity:

describe GameEntity do
  describe 'fields' do
    subject(:subject) { GameEntity }
    it { is_expected.to represent(:id) }
    it { is_expected.to represent(:user_id).as(:owner_id) }
...


Here is the entity itself:

class GameEntity < Grape::Entity
  expose :id
  expose :user_id, as: :owner_id
...

The part of the grape API:

  namespace :games do
    desc "Retrieve all the games"
    params do
      optional :include_reviews, type: Boolean, default: true, desc: 'Accepts: true/false.'
    end
    get do
      games = Game.actual.limit(500)
      present games, with: GameEntity

This was simple so far. But I can use a more sophisticated GameEntity in case I want to present the included reviews. In all other cases I will use my old simple GameEntity to present the Game.

Let's see that case too:

I modify the grape API to this:

    ...
    if params[:include_reviews]
      games = games.includes(:reviews)
      present games, with: DetailedGameEntity
    else
      present games, with: GameEntity
    end

I also spec the new entity:

describe DetailedGameEntity do
  describe 'special fields' do
      subject(:detailed_game) { DetailedGameEntity(game).as_json }
      specify { expect(detailed_game[:reviews]).to be_an(Array) }
  end

Of course the above was a very simple example.

In real world scenarios we can find ourselves that we are reusing the entities like this:

class DetailedGameEntity < Grape::Entity
  expose :game, using: GameEntity do |game, options|
    game
  end

  expose :venue, using: VenueEntity do |game, options|
    game.the_venue
  end

  expose :visitor_team, using: TeamEntity do |game, options|
    game.visitor_team
  end

  expose :host_team, using: TeamEntity do |game, options|
    game.host_team
  end
end

As you can see we are keeping our entity codebase DRY by calling the TeamEntity twice in the DetailedGameEntity. This is a typical use of delegation of the Entities.

Let me show a more complex example of our entities:

The usage of the entities:

  desc "Retrieve all the reviews belonging to a team"
    get do
      ...
      present reviews, :with => ReviewPresenter, user: current_user
    end

The entity itself:
  class ReviewPresenter < Grape::Entity
    expose :current_user_likes do |review, options|
      review.likes.pluck(:user_id).include? options[:user].id if options[:user].present?
    end
  end

And finally the spec for the entity:
describe ReviewPresenter do
  describe 'special fields' do
    context 'without passing a user' do
      subject(:presented_review) { ReviewPresenter.new(image.review).as_json }
      specify { expect(presented_review[:current_user_likes]).to be_nil }
    end

    subject(:presented_review) { ReviewPresenter.new(image.review, user: current_user).as_json }
    ...
      context 'user is the current_user' do
        specify { expect(presented_review[:current_user_likes]).to eq(true) }
      end
    end
    ...
  end
end

What I was doing here is to populate the field 'current_user_likes'
 to true or false depending if the current_user likes or not the presented review.

As you can see the presentation layer of any grape API could became more DRY and versatile by using the grape entities.

Enjoy using them ;)

Friday, 26 December 2014

API Development in Ruby On Rails

I met a technology and a set of gems which were designed to develop any API in Ruby On Rails about half years ago.

The set of gems were:


At first glance it seemed that it adds complexity to the Rails application compared to standard Rails JSON returns. After I played a bit with it I suddenly realized that its API formulating DSL is much more superior. So I started to like it :)

Let me list the elements of DSL I like most:

  • ability to mount API endpoints. This can be very handy if I want to unmount certain unused parts of the API.

      mount Caesars::Ping
      mount Caesars::PingProtected

  • ability to describe the API endpoint. This description will appear in the Swagger documentation. It is a help for the API user.
  • ability to specify the parameters (type, mandatory or required).
      params do
        optional :user_ids, desc: "JSON array of the user ids to be deleted."
        requires :group_id, type: Integer, desc: "The id of the group"
      end

By specifying the type our parameters will be converted automagically to the specified type. It is easier this way for us developers to write our core logic.
  • a complex DSL to organize the API hierarchy and use route params. I consider this the most powerful feature. I can easily organize the API to look more like Facebook graph API or Philips Hue API than standard Rails restfull API.
      namespace :teams do
        route_param :id do
          desc 'Retrieve all games belonging to teams'
          get 'scores' do
            #DO logic here
          end
        end
      end

The grape features are many more including versioning. Take a look for a complete description at grape site ;)

Monday, 1 April 2013

Editing the code with the speed of thought

Hi

I changed my code editor from Textmate2 to Sublime. A friend of mine told me about it and I was so curious that I needed to try it. I started to love it right in the first moment mainly because of the existence of the command palette and its modular feature approach and the multiple editing feature.

The command palette was for me like some sort of help combined with a quicksearch-autosuggest menu. When I thought to something I would do I typed in and if I had luck then the feature was already there. If not then I needed to search for a suitable package which did the desired functionality.
I saw a huge potential in this approach because I always wanted to edit my code like the code editor would be a extension of my body. In other words I wanted to edit code with the speed of my thought.
By default the desired functions were not there. I needed to search for some packages and even edit those packages to achieve my desires. I wanted to toggle between quote types 'a', "a". I wanted to convert a string to a symbol and vice versa "a", :a. I wanted to toggle the do end blocks with the braces blocks. I wanted to convert amongst sneak_case, pascalCase, dash-case, dot.case, slash/separated

So by using it continouosly for more than half year I installed these packages:

          For converting each other from any of them:
    • "sneak_case"
    • "pascalCase"
    • "dash-case"
    • "dot.case"
    • "slash/separated"
  • Clipboard History - for using multiple history feature
  • CoffeeScript - syntax for coffee script
  • Cucumber -  syntax for cucumber
  • Eco - syntax for eco template
  • ERB Insert And Toggle Command
          For toggling amongs these
    • <%= %> 
    • <%-  -%>
    • <%  %>
    • <%=  -%>
    • <%#  %>
  • Gist - I like to store my snippets on gist and access them from sublime
  • Missing Palette Commands
  • Plain Tasks - For storing my tasks in plain text format but nicely formated, like the below list
    • ✔ task 1 @done (13-04-01 21:28)
    •  ☐ implement sorting
  • RSpec - syntax for my specs. I am a TDD advocate. It saves me a lot of time.
  • Ruby Block Converter - for toggling amongst {} block and do end blocks.
  • SFTP - for editing remote files.
  • Toggle Symbol To String for toggling between "symbol"and :symbol
  • Toggle Quotes for toggling between 'string' and "string"
  • Advanced New File for creating deeple nested directories when I place a file into them.
  • Nettuts + Fetch for fetching the latest jquery.js or jquery-ui.js or backbone.js libraries.

Often the commands were missing from some of these packages. So when I searched for 'toggle block' then no result was shown however I wanted to convert a block from "{}" style to "do end" style. This was a bottleneck in my speedy coding... In these cases I needed to open the package add the command to the Default.sublime-commands file and then make a pull request. This way if I eventually forgot the key combination of a feature I could search for it in a command palette.

May I ask any developer who makes a sublime package to add this file to the package and enlist the commands there.
There is the multiple editing feature I like the most... :)
And it is a very fast editor.
I also like the snippet feature of it.

Overall all I can say s that it is a very handy code editor and it worth the money for buying it and spending time to learn it ;)

Thursday, 7 June 2012

Full Text Search Feature Of Postgresql

Hi,

I was using mysql for my development for a long-long time. But I decided to try postgresql as I watched Ryan Bates screencast which showed the fullt text search capabilities of postgresql.

I was motivated to try postgresql since heroku added a new addon which enables any application to use the lastest(9.1.13) postgresql. And as a bonus the capacity of that DB is unlimited... However I can hardly believe that it will remain free forever...

So...
I created a new DB on my heroku account for my application and I followed the migration manual and voila. It took me about no more than 15 minutes to transition to a new DB.

It was a nice experience. :)

Thursday, 6 October 2011

Design your User Interface

Hello everybody,

I have just finished the book "The Design Of Everyday Things".
It's a great book written by Donald Norman. The guy who wrote "The invisible computer", who worked for the Apple Computer's Advanced Technology Group.

In this book he talks about many-many interesting things:
-The constraints and strength of our short term & long term memory.
-The slips and mistakes our brain makes, while we do things.
-How can we make use of our cultural knowledge and knowledge in the world while design.
-The importance of natural mapping between controls and our actions.
-Interpretation of a system state, and our mental model about the system.
-The (missing) evolution of product line. The evaluation of the already selled products and the importance of debriefing then evolving the product to a better one.

He also describes how primitive were the computers at time of writing.
And he gives us some hints how he imagines the mobile phones should easy our everyday tasks. I had the impression the he was envisioning the nowadays iPhones. He also describes how a graphical user interface would help us to be more effective with the computer interaction. I often had the feeling that he was describing a vision which is very similar about nowadays iPads, iPhones or Microsoft Surface.

I was also symphatizing with him because I remember times when I needed to setup my Mouse and CD driver in the config.sys in order to use those devices. Nowadays everything is much more easier. Especially that, I am a Macintosh user I can spend my time focusing on relevant tasks, without even realizing that I am working.

I finished the book yesterday. After I put the book back onto my shelf I started to browse the new Apple products... They are just amazing. I was especially amazed by the new Siri feature in the iPhone. It remembers me to the "Computer" from Startreak. It is a good example that if we can imagine something then we can sooner or later achieve it. Another feature I imagined even myself and waited a lot is the iCloud. I tried the beta version by downloading Ethics from Aristotle on my Macbook and the book instantly appeared on my iphone too. :) Definetely it enhances our usability experience.

The next day, other news were expecting me. Sad news. The founder and main dreamer, envisioner and technology pioneer Steve Jobs died. So good journey Steve and rest in peace.


Saturday, 2 February 2008

Social Networking Softwares

I received an invitation to a "social network" software from India not long time ago.

I always subcribe to these software at least to check what are they capable of. I also like to observe the Darwinian evolution of software.

This software made me upset right in the first turn. It didn't harvest my relationships organically like FACEBOOK but it wanted to record my relationships using my Gmail contacts by borrowing my Gmail password. I didn't have any other choice if I wanted to see the web-app running, however borrowing my Gmail password was very suspicious to me. In the next step it enlisted all my Gmail contacts. There was a tiny checkbox to send the invitation to all of them or not. I didn't want to invite anybody to this social network, all I wanted to do is to check what it offers. Therefore I've unchecked the checkbox and went to the next page. Well... The service didn't offer me any excepcionally good quality. It was nothing to learn. So it didn't worth borrowing my precious Gmail password. For the sake of security I changed my Gmail password promptly.

I was surprized when many of my friends sent me a message in the next few days that they don't want to subscribe to this at all, nevertheless I unchecked the invitation checkbox. There were some of them who thought that this software is a VIRUS. Well I've unregistered myself from this "social networking" immediately

Well... This happens when a software harvests its users in an agressive manner rather than adapting the strategy to grow organically. The problem with it is that even sometimes even those people will be harvested who are not even my friends (I send them only one or two email messages in my whole life). So my "social network" will contain some foreign people to.

This precious information is now free and available for everyone, thanks to these softwares. But sometimes they also contain false relationships :)