Getting Started With Message Queues

When you’re building an infrastructure that is distributed all over the internet, you’ll come to a point where you can’t rely on synchronous remote calls that, for example, synchronize data on 2 servers:

  1. You don’t have any failover system that resends messages if something went wrong (network outages, software failures).
  2. Messages are processed over time and you have no control if something goes overloaded by too many requests.

Even if you don’t have to send messages all over the Internet there are enough points of failures where something can go wrong. You want a reliable and durable system that fails gracefully and ensure.

Solutions

Dropr

Dropr is a distributed message queue framework written in PHP. The main goals are:

  • Reliable and durable (failsafe)-messaging over networks.
  • Decentralized architecture without a single (point of failure) server instance.
  • Easy to setup and use.
  • Modularity for queue storage and message transports (currently filesystem storage and curl-upload are implemented).

More info

Beanstalkd

Beanstalkd is a fast, distributed, in-memory workqueue service. Its interface is generic, but was originally designed for reducing the latency of page views in high-volume web applications by running most time-consuming tasks asynchronously.

It was developed to improve the response time for the Causes on Facebook application (with over 9.5 million users). Beanstalkd drastically decreased the average response time for the most common pages to a tiny fraction of the original, significantly improving the user experience.

More info

Zend Platform Job Queues

Job Queues is an approach to streamline offline processing of PHP scripts. Job Queue Server provides the ability to reroute and delay the execution of PHP scripts that are not essential during user interaction with the Web Server. Job Queues improve the response time during interactive web sessions and utilizes unused resources.

More info

Memcached as simple message queue

In this post, Olly explains how to use memcached as a simple message queue:

Some months ago at work we were in the need of a message queue, a very simple one, basically just a message buffer. The idea is simple, the webservers send there messages to the queue, the queue always accepts all messages and waits until the ETL processes request messages for further processing. As the webservers are time critical and the ETL processes aren’t you need something in between.

More info

Links

About these ads

6 thoughts on “Getting Started With Message Queues

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s