In this post we will talk about building RESTful services. However before we do that I would recommend reading the previous part of the series

 

 

One of the biggest roadblock in developing REST base systems are influencing technical factors like Object Oriented design, abstractions or the technologies that the architects are familiar with. This situation could even become worse if the architect has good technical background in web technologies like MVC and thinks about implementation in these technologies while designing REST services.

 

Design Workflow for Agile Board

We need to think independently of any technology or other influential factors that might corrupt our design. We should not think about classes or methods while designing a RESTful system and I think the best way to do that is stay away from the computer and use pen and paper for the design purpose. To practice that let’s think of the process of managing the task tracking workflow with the example of an agile board.

Agile Board

Agile Board

 

Let’s think about the process of managing an agile task board. When we want to design the workflow, the first thing that we look at are the elements that enable the workflow. The elements that enable the workflow are:

  • Agile board – The agile board is divided into 5 sections to do, design, code, test, done. When a new task is created it would go in the “to do” list. When a team member will start of work on a task then it would be moved to the “design” list and so on to the different lists.

 

  • Sticky notes – These are the communication interface for communicating among the team members as well outside the team as well. The sticky note will have the story number, description, estimate and Assignee.
Sticky Note

Sticky Note

This would seem like a very simple workflow but we are making an assumption here that all the team members know how to move tasks across the agile board. We are also expecting the team to know that new tasks will be going in the “to do” as well. What if we change the process then we would have to share the same knowledge with complete team. We could eliminate all of this by a simple solution. We could define icon for each of the lists on the agile board.

Agile Board With Icons

Agile Board With Icons

We also add the corresponding items to the sticky node as well then we could easily track the current stage of the task. Now in the sticky we could see the icons of the various stages of the task starting from “to do” till “done”.

Sticky Note Icons

Sticky Note Icons

We can track it by simply crossing out the stage which the task has already passed through. In the image below it clearly shows that the task has passed though the “to do” and “design” list and currently in the code phase.

Sticky Note Stage 3

Sticky Note Stage 3

Whenever any changes happen in the sequence or number of stages we just need to update the board and the sticky and no other communication is needed to anyone. Suppose we remove the design phase from our systems. Now we could simply update the board and the sticky and everyone using it would be able to understand as what needs to be done. Whenever a task is being moved then the team member crosses off the icon of the current state and look at the next icon and move the task to that list.

Agile Board Without Design

Agile Board Without Design

Sticky Note Without Design

Sticky Note Without Design

Now if you image this kind of change to be done in an RPC system then it would require versioning and redeployment of the clients so that they are aware that they have to call a different method.  And this might be a reason for the failure of the RPC system. However in this model our client workflow hasn’t changed at all whereas the system workflow has changed. And hence it adheres to the RESTful principle where I could change the system workflow without changing the client workflow. The client workflow is still the same where the team member has to cross off the current matching icon and look for the next matching column on the board and place the sticky note in that column.

 

Software design for Agile Board

Now when we are comfortable with the idea and workflow of a physical agile board, we can go ahead and design for a software based agile board. Like any software system before we start working on the software we will list down a few high level requirements. Since this is an example we will keep the requirements pretty simple:

  • Add new tasks to the backlog
  • Move task to design
  • Move task to code
  • Move task to test
  • Move task to done

State Transitions

We have a simple domain hence we have a simple state transition model. From the starting state we could reach to different states each of which is a collection tasks. You could see in the diagram below that we have captured two types of information:

  • Application states such as entry and task collection. This will be useful when we will design the media type later in the module.
  • States that a task can go through like to do, design, etc. This information will be used in the next section of the module to identify resources.
State Transitions

State Transitions

Using this state transition our client would have to know about only one url that is the entry url and then the client should be able to discover all the other services from there.

 

Identifying the Resources

Now let us create the resources corresponding to the states for the application. The resources with their urls are shown below in the image.

Resources

Resources

  • /tasks – The entry point of the application. It supports the following operation

o   GET – fetches the hypermedia elements for navigation and addition of a new bug to backlog.

  • /tasks/todo – The list of the tasks that have not been started yet.

o   GET – fetches the list of tasks in the todo list

o   POST – adds a new task to the todo list

  • /tasks/design – The tasks that are currently in the design phase

o   GET – fetches the list of tasks in the design list

o   POST – adds a new task to the design list

  • /tasks/code – The tasks that are currently in code phase

o   GET – fetches the list of tasks in the code list

o   POST – adds a new task to the code list

  • /tasks/test – The tasks that are currently in test phase

o   GET – fetches the list of tasks in the test list

o   POST – adds a new task to the test list

  • /tasks/done – The tasks that are done.

o   GET – fetches the list of tasks in the done list

o   POST – adds a new task to the done list

 

Designing the representation of Resources

Once we have identified the application states, the transitions between them and mapped them onto the Uniform Interface element of resources and self-describing messages then we need to figure out how we want to represent our resources.

We have multiple options to decide from and Mike Amundsen in his book on Hypermedia API design, has broken these options into 4 questions.

 

  1. What format do we want to base our representation on?

This can be anything however the most common formats being used by the API today are JSON, XML, etc. We could choose to support multiple base formats as well.

  1. What type of state transfer will the API support?

The various types of state transfers could be

  • Read Only – In this type of state transfer, the client does not transfer any data to the servers.
  • Predefined – In this type of state transfer, the client is aware of the valid transfer elements beforehand via the media types shared by the server.
  • Ad-hoc – In this type of state transfer, the details of the valid transfer elements id shared with the clients in a representation.

 

  1. How will the domain elements map onto the base format?

There are 3 ways in which we could map the domain elements onto the base format.

  • Specific – In this style we create domain specific elements and attributes. For example in case of an XML based format we could create a task element having a title attribute.
  • General – This style as you might have guessed will be bound to a generic domain. It will not be specific to our business domain. One of the majorly used domain media type is Atom.
  • Agnostic – This style we would have a media type is completely unrelated to any specific business domain like HTML.
  1. What will be the application flow?

Application flow are the elements that allows a client to recognize and respond to possible state transitions. It could also be thought of as how the representation format will tell a client that a links exists and what the client can do with it. The application flow identifiers could be:

  • None – The client does not have any flow identifiers.
  • Intrinsic – The media type have built in elements for representing links and expressing the semantics.
  • Applied – The media type does not have the full capabilities to express the media type identifiers. This opens up the opportunity for the designer to apply attributes to the elements to express the identifiers. The example for this application flow could be seen in rel and class attributes of HTML.

 

Task Tracking Service Representation

Let’s make the choices for designing the task tracking service:

Base Format = HTML (since it’s easier to test)

State Transfer = Ad-hoc (since we will be using HTML forms to provide state transfer information to the clients. This way the client will just have to fill in the values and submit the form.)

Domain Style = Agnostic (Since HTML is domain agnostic)

Application Flow = Applied (Since HTML is a domain agnostic language so we will have to apply the custom application flow identifiers using an element of HTML)

 

Based on the above information and decisions we would need the elements for:

  • List of tasks
  • Link template to add a task
  • Link template to move task to to do
  • Link template to move task to design
  • Link template to move task to code
  • Link template to move task to test
  • Link template to move task to done
  • Navigation links

The advantage of using the Hypermedia design is that not all these elements will exits for each of the states. When a task is in to do state then only the design link will be visible and so on. This is how a hypermedia enabled design will help the client in deciding the right thing to do without any prior knowledge.

 

Mapping domain style to base format

Below is a table showing the mapping of domain style onto the base format which is HTML here. This is really useful way to document the representation. Here we are capturing all the details that client will need to recognize. Since in rest we do not have service description hence the client will look for a documentation top understand the representation.

 

Attribute Value AppliedTo Description
id task DIV Container for task state elements
name id INPUT[hidden] The task identity. Found as a child of FORM.move
  title Input[text] The task title state transfer element. Found as a child of FORM.new
class all UL List of tasks
  title SPAN Task title. Found as child of LI
  description SPAN Task description. Found as child of LI
  new to do FORM Application flow identifier for adding a new task to to do
  next FORM Hints to the client for ideal next state transition. Should be used with FORM.move class
rel index A Navigate to index resource; should not be more than 1
  to do A Navigate to to do resource; should not be more than 1

 

Id – is used to represent a unique data element or block in a representation. We are using it here to identify the various state formats in the task tracking representation.

Name – is used to identify a state transition element within a representation. For example input elements.

Class – is used to identify any non-unique data element or block in a representation. Here we have used title to identify a task in the task list. It can have multiple value separated by space. This is used by the client to show available transitions instead of showing the same one dimensional list.

Rel – is used to identify a non-unique process flow element within the representation. Here we have used rel to identify the purpose for the link. It can have multiple value separated by space. This is used by the client to show available transitions instead of showing the same one dimensional list.

 

Sample Markup

The table provides a good visualization on how the representation looks like however nothing beats a dummy markup as shown below.

Sample Markup

Sample Markup

I have not provided any href and action attributes since the actual link value do not matter to the client. In REST url is not the most important part.

Also you would notice that I have specified move active next in the class value for the form. This would tell the client that it could show the optimal path as per the workflow.

 

Modifying the workflow dynamically

We will now go ahead and modify the workflow and remove the design phase. This will eliminate the transitions from todo to design, design to todo, design to code and code to design. It will add new transitions which are from todo to code and code to todo.

Modified Workflow

Modified Workflow

This will not have any impact on the clients since the clients do not need to know about the urls. The only thing that a client would have to know is about the new relationship values of both forms class and the rel attribute anchors since we removed a relationship value called design. Even if the client do not understand these, still the client doesn’t necessarily need to crash.

 

As per the new design a task in the todo list will never see a link to move it to design anymore. This design lets the client see only the valid operations that could be performed.

 

Versioning

Generally rolling out the version 1 of a service is easy and difficulties come in rolling out new versions and fixing the bugs in the previous versions as there may be many clients using it. In REST the contract is the uniform interface.

Versioning within representation

In this form of versioning we do not need to do anything in form of formal versioning as we would be doing the versioning inside the representation. The core concept lies in the fact that the service could add whatever they want and clients would ignore whatever they do not understand.

Versioning the representation

When the semantics of a representation change as a result of the rename or deletion of an element then we might need to version the representation. Here we could make use of content negotiations to make sure the clients are able to get the representations they need in order function properly. We could transfer this representation metadata from the client to server by:

  • Embedding this information as part of custom media type name.
  • Using a parameter on http accept header like version or profile
  • Using custom http header

Versioning the resource

When the service is no longer able to keep the mapping between the resource and its associated profile then we would have version the resource itself. This would more creating a new resource rather than versioning the old one. But it will manifest in the resource identifier which is url in http terms.

In this post we will derive REST from constraints. However before we do that I would recommend reading the 1st part of the series here.

Deriving REST

Dr. Fielding in his dissertation talks about the method that he would use to define REST. This method is more constraints driven rather than requirements driven. A constraints driven approach identifies the factors that influence system behavior and then we apply the design so that the constraints works with those factors rather than working against them.

 

Requiements Vs Constratints

Requiements Vs Constratints

 

Many software architectures are built and designed for small set of requirements and as we get new requirement we grow our design to incorporate those. The PC architectures follow this pattern because the domain of the architecture and domain of the business are closely coupled. These designs solve programmer problems like encapsulation. These designs are designed and tested in a limited environment and then deployed at production where we discover that there are limitations that keep these designs from being broadly usable apart from the environment for which it was designed.

 

REST was designed to solve this problem by determining these constraints in a distributed architecture that restrict the design to be usable broadly. Then REST applies these constraints on a working design and thus shaping it incrementally. Hence we end up mapping the business domain on the architecture domain.

So as we conclude that REST is defined as the identifying the forces that are barriers in distributed computing then knowing these barriers might be helpful in understanding the significance of the individual constraints.

 

Fallacies of Distributed Computing

These are the set of assumptions that L. Peter Deutsch at Sun Microsystems (now Oracle Corporation) originally declared and it states the assumptions that the programmers unaccustomed to distributed applications invariably make. These assumptions ultimately prove false, ensuing either the failure of the system, a considerable reduction in system scope, or in giant, unplanned expenses needed to revamp the system to satisfy its original goals.

The 8 Fallacies of Distributed Computing are as below:

  • The network is reliable.
  • Latency is zero.
  • Bandwidth is infinite.
  • The network is secure.
  • Topology doesn’t change.
  • There is one administrator.
  • Transport cost is zero.
  • The network is homogeneous.

So we should design our architecture to work with these forces of nature rather than against them.

Constraints

Let’s have a look at few architectural constraints that define the RESTful style.

Client – Server constraint

This is one of the fundamental constraint and enforces the constraint in for the client server architecture. The constraint defines all the communication between nodes in a distributed architecture as being between a client and a server. A server is continuously listening for message and when a client sends a message to the server then the server processes it and returns a response. This constraints allows separation the concerns of server and client mainly for User Interface and thus allows different types of client to work with the server and also the client can evolve independently of the server.

Client Server

Client Server

The guiding forces for the Client-Server constraint are as follows.

  • Network security is improved as by scoping the connections between clients and servers we can make the system more secure.
  • Administration is easier as by scoping the connections between the clients and servers we limit the responsibilities of client server and hence they are easy to manage.
  • Heterogeneous network is workable by connecting and disconnecting any number of clients on multiple platforms with no impact on the server

The properties of this constraint are:

  • Client portability is more because the client structure is independent of the server
  • Scalability is better because the server does not have to worry about the user interface details
  • Independent client evolution happens as the server and client are independent.

Stateless constraint

In distributed application the stateless constraint is quite prominent. Stateless constraint does not imply that we should maintain no state of the application but Stateless constraint applies to the communication between the client and server. So the client server interaction must be stateless so that the server is able to process the request with just the information provided by the client request without any context available on the server. The design with Stateless constraint will imply that the state is stored on the client. This design is quite suitable in designs where clients and servers are constantly being added, removed or their network identities are being modified.

Statelessness

Statelessness

The guiding forces for the Stateless constraint are as follows:

  • Network Reliability is improved by storing the state in the client and we allow the interaction between the client and server to be stateless and this give the application the capability to recover from network errors.
  • Network Topology will be simpler since the state of the client is on the server we can add, remove clients and servers from the network without any corruption of data.
  • Administration will be simple when we have stateless interactions.

The various properties of this constraint comes are:

  • Visibility is improved since the system does look for any further than the current request so the full nature of the request is known easily.
  • Reliability of the system is more reliable because the system could recover from partial failures.
  • Scalability is better because the server does not have to worry about the state maintenance across various requests and servers.

There are a couple of design trade-offs that we would have to do when following this architecture.

  • Network Performance might decrease as we might me sending more or repetitive data in each request for the server to have enough information to process the request independently.
  • Client consistency might be lost as the state management is done on the client and the implementation might be different on different platforms.

Cache constraint

According to REST the response from the server should be implicitly or explicitly labeled as cacheable or not cacheable. When the response is cacheable then the client is allowed to reuse the response in equivalent requests. This could allow our applications to reap the benefits of caching at multiple levels (server, intermediate or client). This will majorly improve the network efficiency.

Cache

Cache

The guiding forces for the Cache constraint are as follows:

  • Latency is reduced as some the requests might be served on the client itself and some of them from other caches.
  • Bandwidth consumption is less since some requests might not even reach the server and served beforehand by cache.
  • Transport cost is reduced as the number of requests might be reduced.

The properties of Cache constraint are:

  • Efficiency is improved since the application might have less latency and sucks less network.
  • Scalability is improved since the application is more efficient it could handle more clients.
  • User perceived performance could be improved when the response from the request is coming from the cache.

The design trade-offs that we might have live with, in this architecture is

Decreased reliability on data if the data is stale and differs significantly from the one which would have been provided from the server (if requested).

Uniform Interface constraint

This is the major differentiator between the REST architecture and other network-based architectures. This constraint emphasizes on having a Uniform Interface for all the components in the architecture and could be achieved by applying the generality principle to the component interface and hence simplifying the overall system architecture and improving the visible interactions. So each component talks to the other via standard mechanism. Implementation of decoupling from the service could lead to independent evolution.

Uniform Interface

Uniform Interface

To achieve the Uniform Interface constraint we need to include the following elements in our design:

  • Identification of resources
  • Manipulation of resources through representation
  • Self-descriptive messages
  • Hypermedia as an engine of application state (HATEOAS)

The guiding forces for the Cache constraint are as follows:

  • Network reliability is improved when all the components of the design understand the message sin the same way.
  • Network topology could be simpler and evolve as the clients and serve communicate with each other following the same interface
  • Administration could be easier since we could introduce generic tools for network optimization
  • Heterogeneous network could be supported better because the communication interface is the same between different components.

The properties of Uniform Interface constraint are as below:

  • Visibility is more when we are exchanges the same Interface between all the components of the architecture.
  • Evolvability for each component will be easier as all the component talk the same language

The design trade-offs that we might have live with, in this architecture is

Decreased efficiency since the data will be transferred in standard format rather than the specific format in which it is needed by the application.

Layered System Constraint

Layered system constraint states that a component in a system should only know about the components of the layer with which it is interacting.

Layered System

Layered System

The guiding forces for the Layered System constraint are as follows:

  • Network topology could be simpler as the communication is restricted to the layers and when we change the network components then the only the elements that interact with that layer will be impacted.
  • Security will be better since we layering will allow us to place trust boundaries in layers know the possible components interaction.

The properties of Layered System constraint are as below:

  • Scalability is enormous when we have layered system and modern web is a living example of this.
  • Manageability is also great since each layer could be managed by different admins and still be perfectly operational and scalable. Example my browser know to manage the connection proxy which is managed by my company which know how to connect to Internet which is managed the ISP and so on and so forth. Each layer is managed by different system with different policies.

The design trade-offs that we might have live with, in this architecture is

Increased latency since the data might travel more layers as each component will be communicating with the layer it’s supposed to as compared to a direct connection. We can mitigate this trade off by usage of shared caches and intermediate load balancers.

Code on Demand Constraint

This is listed as an optional constraint in Dr. Fielding paper and this might be one of the reasons why it’s not talked about as much. Code on Demand states that along with provides the clients with the data and metadata, the servers could also provide executable code. The idea is to provide the client with readymade features so that they do not need to write or rewrite them.

Code On Demend

Code On Demend

The properties of Layered System constraint are as below:

  • Simplicity is increased since the client have less number of pre written features and these features could be made available by the server.

The design trade-offs that we might have live with, in this architecture is

Reduced visibility since the clients are downloading the readymade code and features and these might affect caching, manageability and security. So the key rule to applying this constraint is that we should apply this constraint is such a way that the clients who support it should be benefitted by it and the client who do not support this should not break.

 

Any questions, comments and feedback are most welcome.

 

I have been learning and working on REST for a while now. But I have on many blogs that there are disconnects between what REST actually is and what is perceived. So I wanted to write an article based on my understanding of REST. I would talk here about what REST actually is and how to design systems that follow the principles of REST. I will talk about following things in the series.

  • Components of modern distributed architecture
  • Properties of RESTful design
  • What REST is and what it is not?
  • The journey to RESTfulness
  • REST and the rest
  • RESTful Architecture
  • Elements of RESTful Architecture
  • Designing for RESTful Services
  • REST and Cloud

Components of modern distributed architecture

Distributed application development is more challenging in the modern times as we are dealing with of everything users, services, hardware, etc. Few of the major problems that we face today are:

Interoperability between heterogeneous applications

In simple words we want to integrate different applications which have been developed with different frameworks and might even run on different platforms. You must have seen multiple ways to sign up or share various website. It’s a live example of integration of heterogeneous applications in one place. These are different service providers who could not make assumptions for the applications and services provided by any other platform and yet we would like to use all these providers at one place.

Share

Share

 

Signup

Signup

 

 

We want these different integration pieces to simple, consistent and reliable.

Heterogeneous

Heterogeneous

Diversity in Devices

REST is based on the idea of a network based API rather than a library based API and this goes hand in hand with integration of heterogeneous applications and services available. Today we want the integration to be device independent. When we say devices we not only mean smartphones and tablets, it includes most of the electronic devices including cameras, navigation devices, watches, car in dash and what not.

Device Integration

Device Integration

 

 

We want our services and applications to work seamlessly for all the devices. Since most of these devices run native apps and not web based applications maintenance and updates are big challenges. And you could imagine the issues we might come across for performance and efficiency when working with multiple devices because of the network availability and amount of data that can be transferred on the network (data being paid as per usage).

To be or not to be: Cloud

Apart from the interoperability between different services and different devices a major problem that we face is the number of users simultaneously accessing the service. We could most out of the scalable infrastructure if have a scalable architecture. The various organizations have already been taking the advantage of the elastic infrastructure provided by various companies.  The elastic infrastructure allows the business to automatically grow or shrink the computing power and storage capacity of the applications according to the number of users and pay only for the resources that are used.

 

Cloud

Cloud

However we need to understand that cloud just provides the hardware and capability to scale our applications and services and we need to develop our applications and services in a way so that could utilize the various capabilities offered by cloud. We need to build saleable architecture to take advantage of saleable infrastructure.

We could design scalable architectures by neither depending on the middleware in our infrastructure nor on the hardware does that have inherent limitations of its own. We should build applications with transparency in mind so that in case of any errors or failures we don’t have to dig through for days.

The gives us the capability to eliminate the situations shown on the left and provide us a clean (that’s all you need to think) and maintenance free hardware.

 

Server Rooms

Server Rooms

Properties of RESTful design

There are various properties of REST design align with the solutions of the challenges that we discussed above.

  • Heterogeny – The ability to seamlessly interoperate with other participants regardless of language or platform.
  • Scalability – The ability to limit complexity between components in a distributed system, efficiently handling requests and scaling out horizontally when needed.
  • Evolvability – The ability for client and services to evolve independently of one another.
  • Visibility – The ability for value added components such as intelligent gateways to operate correctly without needing access to any hidden or proprietary state such as session state.
  • Reliability – The ability for clients to recover more reliably from failures by developing rich compensation strategies.
  • Efficiency – The ability for multiple components such as proxy servers and caches to participate in the handling of requests taking load away from your server.
  • Performance – The ability to use caches, greatly improving the speed in which a response can be delivered, giving the impression of increased performance.
  • Managability – The ability for simpler management due to interactions between components happening in a highly consistent and visible way.

One or more of these properties align with solving each of the challenges that we discussed previously.

What REST is and what it is not?

REST is not RPC – In RPC the design target of a network interaction is a remote function and the goal RPC of RPC to abstract all the network details so that the developer writing the code should not care about the components interacting over the network.

Whereas in REST the design target of a network interaction is a network resource. Also the network schematics are part of the design.

REST is not just HTTP – Http is the underlying architecture on which REST is based but using just HTTP verbs correctly does not make our services completely RESTful. However most RESTful systems use HTTP as the underlying platform

REST in not just URI – URIs hold an important place in RESTful design but extreme focus on URIs could push us back to thinking designs more the RPC way.

REST is not just anything that is not SOAP – SOAP is more of an implementation detail and REST is more like an architectural style. SOAP aligns iteslf with RPC design style and anything which not SOAP does not imply that it is REST.

Representational State Transfer better known as REST is an architectural style defined in the dissertation of Dr. Roy Fielding at University of California, Irvine in 2000. He designed REST for larger architectural concepts on which web was designed. As per fielding the phrase “Representational state transfer” represents for how a well-designed application behaves as virtual state machine of web pages where progress is made via links.

The journey to RESTfulness – Richardson’s Maturity Model

This model gaining attention and importance in the community and has been referred by Martin Fowler and books like The RESTful CookBook. It is a model we could use to grade our API as per the constraints of REST. The more our APIs adhere to these constraints the closer they are towards RESTfulness.

The different steps in the image below represent the incremental steps towards REST. These are in no way the levels of REST.

 

RESTfull

RESTfull

L0 – represents that we are following the RPC style with the Plain Old XML (POX). This is the most elementary level of the service maturity.

L1 – represents the use differentiated resources.

L2 – represents the usage of HTTP verbs and HTTP status codes.

L3– represents the use of hypermedia controls.

More details on the Richardson’s Maturity Model could be found at Martin Fowler’s blog.

 

Any Questions, Comments and feedback are always welcome.