Command-line memcached stat reporter

Nicholas Tang wrote a nice little perl script that shows a basic memcached top display for a list of servers. You can specify thresholds, for instance, and it’ll change color to red if you exceed the thresholds. You can also choose the refresh/sleep time, and whether to show immediate (per second) stats, or lifetime stats.

To install it you only need to download the script and make it executable:

$ curl http://memcache-top.googlecode.com/files/memcache-top-v0.6 > ~/bin/memcache-top
$ chmod +x ~/bin/memcache-top
$ memcache-top --sleep 3 --instances 10.50.11.3,10.50.11.4,10.50.11.5

Here’s some sample output:

memcache-top v0.6       (default port: 11211, color: on, refresh: 3 seconds)

INSTANCE                USAGE   HIT %   CONN    TIME    EVICT/s GETS/s  READ/s  WRITE/s
10.50.11.3:11211        88.9%   69.7%   1661    0.9ms   0.3     47      13.9K   9.8K
10.50.11.4:11211        88.8%   69.9%   2121    0.7ms   1.3     168     17.6K   68.9K
10.50.11.5:11211        88.9%   69.4%   1527    0.7ms   1.7     48      14.4K   13.6K
AVERAGE:                84.7%   72.9%   1704    1.0ms   1.3     69      13.5K   30.3K   

TOTAL:          19.9GB/ 23.4GB          20.0K   11.7ms  15.3    826     162.6K  363.6K
(ctrl-c to quit.)

Project Home
http://code.google.com/p/memcache-top/

Database Replication Adapter for Zend Framework Applications

Last updated: 21 Feb, 2010

Database replication is an option that allows the content of one database to be replicated to another database or databases, providing a mechanism to scale out the database. Scaling out the database allows more activities to be processed and more users to access the database by running multiple copies of the databases on different machines.

The problem with monolithic database designs is that they don’t establish an infrastructure that allows for rapid changes in business requirements. Here is where database replication comes into play. Replication can be used effectively for many different purposes, such as separating data entry and reporting, distributing load across servers, providing high availability, etc.

Zf_Orm_DataSource is a Zend Framework Replication Adapter class flexible enough to support the most commonly used replication scenarios:

Single-Master Replication

In the simplest replication scenario, the master copy of directory data is held in a single read-write replica on one server called the supplier server. The supplier server also maintains changelog for this replica. On another server, called the consumer server, there can be multiple read-only replicas.

Configuration array:

$config = array(
    'adapter'        => 'Pdo_Mysql',
    'driver_options' => array(PDO::ATTR_TIMEOUT=>5),
    'username'       => 'root',
    'password'       => 'root',
    'dbname'         => 'test',
    'master_servers' => 1,
    'servers'        => array(
        array('host' => 'db.master-1.com'),
        array('host' => 'db.slave-1.com'),
        array('host' => 'db.slave-2.com')
    )
);

// or ...

$config = array(
    'adapter'        => 'Pdo_Mysql',
    'driver_options' => array(PDO::ATTR_TIMEOUT=>5),
    'dbname'         => 'test',
    'master_servers' => 1,
    'servers'        => array(
        array('host' => 'db.master-1.com', 'username' => 'user1', 'password'=>'pass1'),
        array('host' => 'db.slave-1.com', 'username' => 'user2', 'password' => 'pass2'),
        array('host' => 'db.slave-2.com', 'username' => 'user3', 'password' => 'pass3')
    )
);

In the setup above, all writes will go to the master connection and all reads will be randomly distributed across the available slaves.

Multi-Master Replication

This type of configuration can work with any number of consumer servers. Each consumer server holds a read-only replica. The consumers can receive updates from all the suppliers. The consumers also have referrals defined for all the suppliers to forward any update requests that the consumers receive.

$config = array(
    'adapter'        => 'Pdo_Mysql',
    'driver_options' => array(PDO::ATTR_TIMEOUT=>5),
    'username'       => 'root',
    'password'       => 'root',
    'dbname'         => 'test',
    'master_servers' => 2,
    'master_read'    => true,
    'servers'        => array(
        array('host' => 'db.master-1.com'),
        array('host' => 'db.master-2.com')
    )
);

Using a distributed memory caching system

Database connections are expensive and it’s very inefficient for an application to try to connect to a server that is down or not responding. A distributed memory caching system can help alleviate this problem by keeping a list of all the failed connections in memory, sharing that information across multiple servers and allowing the application to access it before attempting to open a connection.

To enable this option, you have to pass an instance of the Memcached adapter class:

class Bootstrap extends Zend_Application_Bootstrap_Base
{
    protected function _initCache()
    {
        ...
    }

    protected function _initDatabase()
    {
        $config = include APPLICATION_PATH . '/config/database.php';
        $cache = $this->getResource('cache');
        $dataSource = new Zf_Orm_DataSource($config, $cache, 'cache_tag');
        Zend_Registry::set('dataSource', $dataSource);
    }
}

And here is a short example of how the Replication Adapter might be used in a ZF application:

class TestDao
{
    public function fetchAll()
    {
        $db = Zend_Registry::get('dataSource')->getConnection('slave');
        $query = $db->select()->from('test');
        return $db->fetchAll($query);
    }

    public function insert($data)
    {
        $db = Zend_Registry::get('dataSource')->getConnection('master');
        $db->insert('test', $data);
        return $db->lastInsertId();
    }
}

Source Code:
https://github.com/fedecarg/zf-replication-adapter

Increase speed and reduce bandwidth usage

Apache’s mod_deflate module provides the DEFLATE output filter that allows output from your server to be compressed before being sent to the client over the network.

There are two ways of enabling gzip compression:

  1. Using Apache’s mod_deflate
  2. Using output buffering

Encoding the output and setting the appropriate headers manually makes the code more portable. Keep in mind that there are hundreds of Linux distributions, each slightly different to significantly different. To allow portability the application should not make assumptions about the OS or config involved.

Using Apache

1. Enable mod_deflate

Debian/Ubuntu:

$ a2enmod deflate
$ /etc/init.d/apache2 force-reload

2. Configure mode_deflate

$ nano /etc/apache2/mods-enabled/deflate.conf

#
# mod_deflate configuration
#
<IfModule mod_deflate.c>
 AddOutputFilterByType DEFLATE text/plain
 AddOutputFilterByType DEFLATE text/html
 AddOutputFilterByType DEFLATE text/xml
 AddOutputFilterByType DEFLATE text/css
 AddOutputFilterByType DEFLATE application/xml
 AddOutputFilterByType DEFLATE application/xhtml+xml
 AddOutputFilterByType DEFLATE application/rss+xml
 AddOutputFilterByType DEFLATE application/javascript
 AddOutputFilterByType DEFLATE application/x-javascript

 DeflateCompressionLevel 9

 BrowserMatch ^Mozilla/4 gzip-only-text/html
 BrowserMatch ^Mozilla/4\.0[678] no-gzip
 BrowserMatch \bMSIE !no-gzip !gzip-only-text/html

 DeflateFilterNote Input instream
 DeflateFilterNote Output outstream
 DeflateFilterNote Ratio ratio
</IfModule>

Using output buffering

Create a gzip compressed string in your bootstrap file:

try {
    $frontController = Zend_Controller_Front::getInstance();
    if (@strpos($_SERVER['HTTP_ACCEPT_ENCODING'], 'gzip') !== false) {
        ob_start();
        $frontController->dispatch();
        $output = gzencode(ob_get_contents(), 9);
        ob_end_clean();
        header('Content-Encoding: gzip');
        echo $output;
    } else {
        $frontController->dispatch();
    }
} catch (Exeption $e) {
    if (Zend_Registry::isRegistered('Zend_Log')) {
        Zend_Registry::get('Zend_Log')->err($e->getMessage());
    }
    $message = $e->getMessage() . "\n\n" . $e->getTraceAsString();
    /* trigger event */
}

Reference

Use mod_deflate to compress Web content delivered by Apache

Format a time interval with the requested granularity

This class, a refactored version of Drupal’s format_interval function, makes it relatively easy to format an interval value. The format will automatically format as compactly as possible. For example: if the difference between the two dates is only a few hours and both dates occur on the same day, the year, month, and day parts of the date will be omitted.

class DateIntervalFormat
{
    /**
     * Format an interval value with the requested granularity.
     *
     * @param integer $timestamp The length of the interval in seconds.
     * @param integer $granularity How many different units to display in the string.
     * @return string A string representation of the interval.
     */
    public function getInterval($timestamp, $granularity = 2)
    {
        $seconds = time() - $timestamp;
        $units = array(
            '1 year|:count years' => 31536000,
            '1 week|:count weeks' => 604800,
            '1 day|:count days' => 86400,
            '1 hour|:count hours' => 3600,
            '1 min|:count min' => 60,
            '1 sec|:count sec' => 1);
        $output = '';
        foreach ($units as $key => $value) {
            $key = explode('|', $key);
            if ($seconds >= $value) {
                $count = floor($seconds / $value);
                $output .= ($output ? ' ' : '');
                if ($count == 1) {
                    $output .= $key[0];
                } else {
                    $output .= str_replace(':count', $count, $key[1]);
                }
                $seconds %= $value;
                $granularity--;
            }
            if ($granularity == 0) {
                break;
            }
        }

        return $output ? $output : '0 sec';
    }
}

Usage:

$dateFormat = new DateIntervalFormat();
$timestamp = strtotime('2009-06-21 20:46:11');
print sprintf('Submitted %s ago',  $dateFormat->getInterval($timestamp));

Outputs:

Submitted 3 days 4 hours ago

TypeFriendly: A Documentation And User Manual Builder

TypeFriendly is a documentation generation script written in PHP5. It was designed to be easy in use and it allows to achieve the first results immediately, a couple of minutes after you start the work. The script contains everything you need to write clear, multilingual documentation for your project, so that you do not have to code everything on your own.

The most important features of TypeFriendly:

  1. Modular documentation structure – it is generated from text files and the structure and navigation are generated from the file names.
  2. Simple syntax – the text is written in intuitive and clean Markdown syntax.
  3. Multilingual support and tools – TypeFriendly allows you to create your manuals in many language versions. It also contains a tool that shows whether the derived languages are up-to-date.
  4. Configurable output formats – currently, TypeFriendly is able to generate the documentation in XHTML (many pages) and XHTML (single page). There is also a third format – metadata – still under development. It will allow to import the docs to a database in order to make an on-line version with, for example, user comments.
  5. Various add-ons such as syntax highlighting, references, class description fields.
  6. Navigation generators.
  7. It is portable – works under Linux, FreeBSD and Windows. All you need is the PHP interpreter available.

TypeFriendly is distributed under the terms of GNU General Public License 3, which means that you can use, modify and share it for free.

Demo
http://static.invenzzia.org/docs/tf/0_1/book/en/index.html

Screenshots
http://www.invenzzia.org/en/projects/typefriendly/screenshots

Source Code
http://svn.invenzzia.org/browser/TypeFriendly/trunk/

Website
http://www.invenzzia.org/en/projects/typefriendly

Real Time Web-Based Service Monitoring Tool

phpWatch is a general purpose service monitor that is able to send notifications of outages via e-mail or text-message (SMS). The purpose of this system is to allow administrators to easily check the status of many different services running on any number of servers and also allow developers to interface with the query and notification APIs.

Installation

The basic installation is very simple: chmod config.php to 777 and simply navigate to the install directory in your browser. Fill in the database information and the setup will create the required tables and setup the configuration file as needed. The only required task beyond the automated install is to add cron.php as a cron job (Unix/Linux) or scheduled task (Windows).

SMS Alerts

phpWatch uses pre-existing SMS gateways provided by the cell-carriers themselves. For example, to send a message to a Verizon subscriber with the phone number 123-456-7890, an e-mail can be sent to 1234567890@vtext.com and it will then be forwarded to the individual’s phone.

Links

Omeka: A Web-based Publishing System

omeka_1238110767043

The Center for History and New Media has released an open source version of their Web-based publishing system built on top of the Zend Framework.

Omeka is a Web-based publishing platform for scholars, librarians, archivists, museum professionals, educators, and cultural enthusiasts. Its “five-minute setup” makes launching an online exhibition as easy as launching a blog. Omeka is designed with non-IT specialists in mind, allowing users to focus on content and interpretation rather than programming. It brings Web 2.0 technologies and approaches to academic and cultural websites to foster user interaction and participation. It makes top-shelf design easy with a simple and flexible templating system. Its robust open-source developer and user communities underwrite Omeka’s stability and sustainability.

Thank you guys for open sourcing this great application!

Links