API Development Tips

Organizations who are paying attention already know they need to have an open web API, and many already have under development or in the wild. Make sure you haven’t been caught by the pitfalls of many early API releases.

Multiple points of failure

  • Back-end systems: db servers/caches, hardware failures, etc.
  • Interconnections: router failures, bad cables, etc.
  • External Dependencies: fail whales, random cloud latency, etc.

The 5 tips

  1. Test it all
    • Unit test are not enough, they are just the beginning.
    • Test what users experience. Perform end-to-end black box tests.
    • Replay your access logs. Very accurate.
    • Validate return payloads. A stack trace is not valid XML.
  2. Plan for future versions
    • Versions are not sexy/semantic (but do it anyway).
    • Announce versions often.
  3. Embrace standards
    • APIs are better when predictable.
    • Standard approaches mean tools.
    • Avoid uncomfortable migrations. No one wants an OAuthpocalypse.
  4. Monitor everything & be honest
    • Trends are your friend.
    • Users are not your early-warning ops team.
    • Be open and honest, or your users will tweet that your API sucks!
  5. Fail well
    • Well-formed errors win friends and makes users more tolerant to failure.
    • Make monitoring easy.
    • Don’t punish everyone. Determine who gets hurt most by failures.

Watch the video here: Understanding API Activity by Clay Loveless

JavaScript: Retrieve and paginate JSON-encoded data

I’ve created a jQuery plugin that allows you to retrieve a large data set in JSON format from a server script and load the data into a list or table with client side pagination enabled. To use this plugin you need to:

Include jquery.min.js and jquery.paginate.min.js in your document:

<script type="text/javascript" src="js/jquery.min.js"></script>
<script type="text/javascript" src="js/jquery.paginate.min.js"></script>

Include a small css to skin the navigation links:

<style type="text/css">
a.disabled {
    text-decoration: none;
    color: black;
    cursor: default;
}
</style>

Define an ID on the element you want to paginate, for example: “listitems”. If you have a more than 10 child elements and you want to avoid displaying them before the javascript is executed, you can set the element as hidden by default:

<ul id="listitems" style="display:none"></ul>

Place a div in the place you want to display the navigation links:

<div id="listitems-pagination" style="display:none">
    <a id="listitems-previous" href="#" class="disabled">&laquo; Previous</a>
    <a id="listitems-next" href="#">Next &raquo;</a>
</div>

Finally, include an initialization script at the bottom of your page like this:

<script type="text/javascript">
$(document).ready(function() {
    $.getJSON('data.json', function(data) {
        var items = [];
        $.each(data.items, function(i, item) {
            items.push('<li>' + item + '</li>');
        });
        $('#listitems').append(items.join(''));
        $('#listitems').paginate({itemsPerPage: 5});
    });
});
</script>

You can fork the code on GitHub or download it.

Building a RESTful Web API with PHP and Apify

Apify is a small and powerful open source library that delivers new levels of developer productivity by simplifying the creation of RESTful architectures. You can see it in action here. Web services are a great way to extend your web application, however, adding a web API to an existing web application can be a tedious and time-consuming task. Apify takes certain common patterns found in most web services and abstracts them so that you can quickly write web APIs without having to write too much code.

Apify exposes similar APIs as the Zend Framework, so if you are familiar with the Zend Framework, then you already know how to use Apify. Take a look at the UsersController class.

Building a RESTful Web API

In Apify, Controllers handle incoming HTTP requests, interact with the model to get data, and direct domain data to the response object for display. The full request object is injected via the action method and is primarily used to query for request parameters, whether they come from a GET or POST request, or from the URL.

Creating a RESTful Web API with Apify is easy. Each action results in a response, which holds the headers and document to be sent to the user’s browser. You are responsible for generating the response object inside the action method.

class UsersController extends Controller
{
    public function indexAction($request)
    {
        // 200 OK
        return new Response();
    }
}

The response object describes the status code and any headers that are sent. The default response is always 200 OK, however, it is possible to overwrite the default status code and add additional headers:

class UsersController extends Controller
{
    public function indexAction($request)
    {
        $response = new Response();

        // 401 Unauthorized
        $response->setCode(Response::UNAUTHORIZED);

        // Cache-Control header
        $response->setCacheHeader(3600);

        // ETag header
        $response->setEtagHeader(md5($request->getUrlPath()));

        // X-RateLimit header
        $limit = 300;
        $remaining = 280;
        $response->setRateLimitHeader($limit, $remaining);

        // Raw header
        $response->addHeader('Edge-control: no-store');

        return $response;
    }
}

Content Negotiation

Apify supports sending responses in HTML, XML, RSS and JSON. In addition, it supports JSONP, which is JSON wrapped in a custom JavaScript function call. There are 3 ways to specify the format you want:

  • Appending a format extension to the end of the URL path (.html, .json, .rss or .xml)
  • Specifying the response format in the query string. This means a format=xml or format=json parameter for XML or JSON, respectively, which will override the Accept header if there is one.
  • Sending a standard Accept header in your request (text/html, application/xml or application/json).

The acceptContentTypes method indicates that the request only accepts certain content types:

class UsersController extends Controller
{
    public function indexAction($request)
    {
    	// only accept JSON and XML
        $request->acceptContentTypes(array('json', 'xml'));

        return new Response();
    }
}

Apify will render the error message according to the format of the request.

class UsersController extends Controller
{
    public function indexAction($request)
    {
        $request->acceptContentTypes(array('json', 'xml'));

    	$response = new Response();
        if (! $request->hasParam('api_key')) {
            throw new Exception('Missing parameter: api_key', Response::FORBIDDEN);
        }
        $response->api_key = $request->getParam('api_key');

        return $response;
    }
}

Request

GET /users.json

Response

Status: 403 Forbidden
Content-Type: application/json
{
    "code": 403,
    "error": {
        "message": "Missing parameter: api_key",
        "type": "Exception"
    }
}

Resourceful Routes

Apify supports REST style URL mappings where you can map different HTTP methods, such as GET, POST, PUT and DELETE, to different actions in a controller. This basic REST design principle establishes a one-to-one mapping between create, read, update, and delete (CRUD) operations and HTTP methods:

HTTP Method URL Path Action Used for
GET /users index display a list of all users
GET /users/:id show display a specific user
POST /users create create a new user
PUT /users/:id update update a specific user
DELETE /users/:id destroy delete a specific user

 

If you wish to enable RESTful mappings, add the following line to the index.php file:

try {
    $request = new Request();
    $request->enableUrlRewriting();
    $request->enableRestfulMapping();
    $request->dispatch();
} catch (Exception $e) {
    $request->catchException($e);
}

The RESTful UsersController for the above mapping will contain 5 actions as follows:

class UsersController extends Controller
{
    public function indexAction($request) {}
    public function showAction($request) {}
    public function createAction($request) {}
    public function updateAction($request) {}
    public function destroyAction($request) {}
}

By convention, each action should map to a particular CRUD operation in the database.

Building a Web Application

Building a web application can be as simple as adding a few methods to your controller. The only difference is that each method returns a view object.

class PostsController extends Controller
{
    /**
     * route: /posts/:id
     *
     * @param $request Request
     * @return View|null
     */
    public function showAction($request)
    {
        $id = $request->getParam('id');
        $post = $this->getModel('Post')->find($id);
        if (! isset($post->id)) {
            return $request->redirect('/page-not-found');
        }

        $view = $this->initView();
        $view->post = $post;
        $view->user = $request->getSession()->user

        return $view;
    }

    /**
     * route: /posts/create
     *
     * @param $request Request
     * @return View|null
     */
    public function createAction($request)
    {
        $view = $this->initView();
        if ('POST' !== $request->getMethod()) {
            return $view;
        }

        try {
            $post = new Post(array(
                'title' => $request->getPost('title'),
                'text'  => $request->getPost('text')
            ));
        } catch (ValidationException $e) {
            $view->error = $e->getMessage();
            return $view;
        }

        $id = $this->getModel('Post')->save($post);
        return $request->redirect('/posts/' . $id);
    }
}

The validation is performed inside the Post entity class. An exception is thrown if any given value causes the validation to fail. This allows you to easily implement error handling for the code in your controller.

Entity Class

You can add validation to your entity class to ensure that the values sent by the user are correct before saving them to the database:

class Post extends Entity
{
    protected $id;
    protected $title;
    protected $text;

    // sanitize and validate title (optional)
    public function setTitle($value)
    {
        $value = htmlspecialchars(trim($value), ENT_QUOTES);
        if (empty($value) || strlen($value) < 3) {
            throw new ValidationException('Invalid title');
        }
        $this->title = $title;
    }

    // sanitize text (optional)
    public function setText($value)
    {
        $this->text = htmlspecialchars(strip_tags($value), ENT_QUOTES);
    }
}

Routes

Apify provides a slimmed down version of the Zend Framework router:

$routes[] = new Route('/posts/:id',
    array(
        'controller' => 'posts',
        'action'     => 'show'
    ),
    array(
        'id'         => '\d+'
    )
);
$routes[] = new Route('/posts/create',
    array(
        'controller' => 'posts',
        'action'     => 'create'
    )
);

HTTP Request

GET /posts/1

Incoming requests are dispatched to the controller “Posts” and action “show”.

Feedback

  • If you encounter any problems, please use the issue tracker.
  • For updates follow @fedecarg on Twitter.
  • If you like Apify and use it in the wild, let me know.

Towards Community Cloud Computing

Cloud Computing is rising fast, with its data centers growing at an unprecedented rate. However, this has come with concerns of privacy, efficiency at the expense of resilience, and environmental sustainability, because of the dependence on Cloud vendors such as Google, Amazon, and Microsoft. Community Cloud Computing makes use of the principles of Digital Ecosystems to provide a paradigm for Clouds in the community, offering an alternative architecture for the use cases of Cloud Computing. It is more technically challenging to deal with issues of distributed computing, such as latency, differential resource management, and additional security requirements. However, these are not insurmountable challenges, and with the need to retain control over our digital lives and the potential environmental consequences, it is a challenge we must pursue.

Towards Community Cloud Computing (Visit Site | Download PDF)

The Cost of Hosting on Amazon

Mather Corgan, president of HotPads, gave a great talk on how HotPads uses AWS to run their real estate search engine. HotPads abandoned their managed hosting in December and took the leap over to EC2 and its siblings. The presentation has a lot of detail on costs and other things to watch out for, so if you’re currently planning your “cloud” architecture, you’ll find some of this really helpful.

HotPads on AWS

The Little Manual of API Design

This manual gathers together the key insights into API design that were discovered through many years of software development on the Qt application development framework at Trolltech (now part of Nokia). When designing and implementing a library, you should also keep other factors in mind, such as efficiency and ease of implementation, in addition to pure API considerations. And although the focus is on public APIs, there is no harm in applying the principles described here when writing application code or internal library code.

The Little Manual of API Design (PDF)

A Lifestreaming Zend Framework Application

There are just too many social sites out there for everyone to be able to keep up, and that’s where PHPLifestream steps in. PHPLifestream is an application that aggregates feeds from different sources and combines them into one. Developed by Johan Nilsson and built on top of the Zend Framework, PHPLifestream is a powerful Web 2.0 lifestreaming application that you’ll want to keep an eye on.

Great open source PHP application!

Demo
http://johannilsson.me/streams/list

http://johannilsson.me/graphs

Source Code
http://github.com/johannilsson/phplifestream/tree/master

Server-side Marker Clustering with PHP and Google Maps

with_cluster2

As maps get busier, marker clustering is likely to be needed. Marker clustering is a technique by which several points of interest can be represented by a single icon when they’re close to one another.

Mika Tuupola wrote a PHP library that divides the map into a given number of cells and represents all the markers present in the same cell by a single icon. This icon shows the number of markers it symbolizes.

He also wrote an excellent post explaining how marker clustering works.

Related Posts

Building a Web Service Client using the Zend Framework

UPDATE: Interested in creating a web API? Read this post instead.

In this post I’ll demonstrate how to use the Zend Framework to quickly and easily develop a Web application that consumes a Web API as a Web service.

The Zend Framework puts heavy emphasis on Web services. This is a good thing, considering the amount of Web services out there that can help lower costs and increase the value of your site.

The various steps that are involved in creating a Web service client are as follows:

  1. Find a Web Service provider.
  2. Identify Web service endpoints.
  3. Identify query string parameters to endpoints.
  4. Identify response types.
  5. Create a proxy object for the Web service.

Find a Web Service provider

In this example, I’ll develop a Web services client to Digg, a real-world Web services provider. The Digg API has been created to let users interact programmatically with Digg. The API accepts REST requests and offers several response types. More info here: http://apidoc.digg.com/

Identify Web service endpoints

A Web services endpoint is a resource where Web services messages can be targeted. For example:

GET /stories
    All stories.
GET /stories/popular
    Popular stories.
GET /stories/topic//popular
    Popular stories from a given topic.

Let’s assume you only need to target a single endpoint, for example, fetch popular stories by topic:

http://services.digg.com/stories/topic/programming/popular/

Identify query string parameters to endpoints

You can pass additional arguments to a Web service by specifying additional parameters and then passing values for those parameters. In theory, a GET should retrieve a representation of a resource identified by a URI, but many APIs make it extremely easy to view the URI not as a resource identifier, but as a convenient means to encode parameters.

A good example of this is the Digg API that allows you to specify parameters in the query string. For example:

appkey
    The value of the application key.
sort
    How to sort returned stories.
count
    Number of stories to retrieve.
offset
    Offset in complete story list.
domain
    Partial domain of linked article.
link
    URL of linked article.

Identify Response Types

The Digg API provides several response types, each designed to be useful in a variety of programming contexts:

  • text/xml: XML response type
  • application/json: JSON response type
  • text/javascript: Javascript response type
  • application/php: Serialized PHP response type

Create a proxy object for the Web service

We are now ready to create a proxy object to extract data from the Web service. Because the Digg API accepts REST requests, we’ll create a Zend_Service_Digg class that extends Zend_Rest_Client, this allows us to take advantage of some predefined accessor and mutator methods, such as getUri() and setUri():

class Zend_Service_Digg extends Zend_Rest_Client
{
    protected $_uri = 'http://services.digg.com/';

    public function __construct()
    {
        $this->setUri($this->_uri);
        $client = self::getHttpClient();
        $client->setHeaders('Accept-Charset', 'ISO-8859-1,utf-8');
    }
...

Digg provides different response types, so we need to create a method to allow the user define the context (Codepad):

...
    protected $_responseType = 'xml';
    protected $_responseTypes = array('php', 'xml', 'json');

    public function setResponseType($responseType)
    {
        if (!in_array(strtolower($responseType), $this->_responseTypes)) {
            throw new Zend_Service_Digg_Exception('Invalid Response Type');
        }

        $this->_responseType = strtolower($responseType);
        return $this;
    }

    public function getResponseType()
    {
        return $this->_responseType;
    }
...

Now we’ll create a setter and getter method to pass and retrieve additional parameters (Codepad):

...
    protected $_params = array();

    public function setParams($params)
    {
    	// Validate mandatory parameters to endpoint
    	if (!array_key_exists('appkey', $params)) {
            throw new Zend_Service_Digg_Exception('Param appkey missing');
        }

        foreach ($params as $key => $value) {
            switch (strtolower($key)) {
                case 'type':
                    $this->setResponseType($value);
                    break;
                default:
                    $this->_params[$key] = $value;
                    break;
            }
        }

        return $this;
    }

    public function getParams()
    {
    	return $this->_params;
    }
...

All the values of the arguments must be URL encoded. Fortunately, Zend_Rest_Client takes care of this for us. Next, we’ll create a method that fetches the most popular stories by a given topic (Codepad):

...
    public function fetchPopularStoriesByTopic($topic, array $params = array())
    {
        $this->setParams($params);

        $path = sprintf('/stories/topic/%s/popular/', trim(strtolower($topic)));
        return $this->sendRequest('GET', $path);
    }
...

And finally, we create 2 methods, one that sends the request and another one that formats the response data (Codepad):

...
    public function sendRequest($requestType, $path)
    {
        $requestType = ucfirst(strtolower($requestType));
        if ($requestType !== 'Post' && $requestType !== 'Get') {
            throw new Zend_Service_Digg_Exception('Invalid request type: ' . $requestType);
        }

        try {
            $requestMethod = 'rest' . $requestType;
            $response = $this->{$requestMethod}($path, $this->getParams());
            return $this->formatResponse($response);
        } catch (Zend_Http_Client_Exception $e) {
            throw new Zend_Service_Digg_Exception($e->getMessage());
        }
    }

    public function formatResponse(Zend_Http_Response $response)
    {
        if ('json' === $this->getResponseType()) {
            return $response->getBody();
        } elseif ('php' === $this->getResponseType()) {
            return unserialize($response->getBody());
        } else {
            return new Zend_Rest_Client_Result($response->getBody());
        }
    }
}

Usage

$digg = new Zend_Service_Digg();

$params = array('type'=>'json', 'appkey'=>'http://api.test.com');
$stories = $digg->fetchPopularStoriesByTopic('programming', $params);

Response:

http://services.digg.com/stories/topic/programming/popular/?type=json&appkey=http%3A%2F%2Fapi.test.com

Zend_Service_Digg class

That’s it! Time to open your editor and start building some cool WS clients :)

Four Great InfoQ Presentations

Hope you like these recommendations and if you know of any other good tech-related video, then please let me know.

1. Developing Expertise: Herding Racehorses, Racing Sheep

One of my favourites. In this presentation Dave Thomas (The Pragmatic Programmer) talks about expanding people’s expertise in their domains of interest by not treating them uniformly as they had the same amount of knowledge and level of experience.

Developing Expertise

2. Real World Web Services

Another good presentation. In this one Scott Davis provides a pragmatic, down-to-earth introduction to Web services as used in the real world by public sites, including SOAP-based and REST examples.

Real World Web Services

3. CouchDB and me

This presentation is different, and that’s why I like it so much. Damien Katz shares his experiences and reminds people how difficult but at the same time gratifying is to be an open source developer. He talks about the history of CouchDB development from a very personal point of view. His inspirations for CouchDB and why he decided to move my wife and kids to a cheaper place and live off savings to build this thing.

CouchDB and me

4. Yahoo! Communities Architectures

In this presentation, Ian Flint tries to explain the infrastructure and architecture employed by Yahoo! to keep going a multitude of servers running of different platforms and offering different services. Very interesting!

Yahoo! Communities Architectures