Zend Framework Automatic Dependency Tracking

When you develop or deploy an application, dependency tracking is one of the problems you must solve. Keeping track of dependencies for every application you develop is not an easy task. To solve this problem I’ve created Zend_Debug_Include, a Zend Framework component that supports automatic dependency tracking.

We all agree that dependencies cannot be maintained by hand, it’s almost impossible, specially if you are using packages from Zend, Solar, Maintainable and/or Zym (if you don’t know any of these frameworks, check them out, they are pretty cool). Every time you add an include/require statement or create an instance of an object, you introduce a new dependency. And that’s why tracking internal project dependencies can become complex when using a framework.

The concept behind Zend_Debug_Include is that the dependencies for each source file are stored in a separate file. If the source file is modified, the file containing that source file’s dependencies is rebuilt. This concept enables you to determine run-time dependencies of files using arbitrary components. This solution is also useful if you are deploying your application using Linux packages. But, dependency tracking isn’t just useful for deploying applications, it can also be used to evaluate packages. Sometimes packages create unnecessary dependencies and this is something that we need to monitor.

Zend_Debug_Include comes with 3 built-in adapters: File, Package and Url.

Tracking File Dependencies

To track file dependencies you need to create an instance of Zend_Debug_Include_Manager and set the Zend_Debug_Include_Adapter_File adapter. This needs to happen inside your bootstrapper file, before the Front Controller dispatches the request.

$included = new Zend_Debug_Include_Manager();
$included->setAdapter(new Zend_Debug_Include_Adapter_File());
$included->setOutputDir('/var/www/my-app/dependencies');

/* Dispatch request */
$frontController->dispatch();

This creates a zf-files.txt in your output directory containing all the files included or required on that request. Every time the Front Controller dispatches a request, Zend_Debug_Include checks for new dependencies and adds them to the zf-files.txt file:

/var/www/my-app/public/index.php
/var/www/my-app/application/bootstrap.php
/var/www/my-app/library/Zend/Loader.php
/var/www/my-app/library/Zend/Controller/Front.php
/var/www/my-app/library/Zend/Controller/Action/HelperBroker.php
/var/www/my-app/library/Zend/Controller/Action/HelperBroker/PriorityStack.php
/var/www/my-app/library/Zend/Controller/Exception.php
/var/www/my-app/library/Zend/Exception.php
/var/www/my-app/library/Zend/Controller/Plugin/Broker.php
/var/www/my-app/library/Zend/Controller/Plugin/Abstract.php
/var/www/my-app/library/Zend/Controller/Dispatcher/Standard.php
/var/www/my-app/library/Zend/Controller/Dispatcher/Abstract.php
/var/www/my-app/library/Zend/Controller/Dispatcher/Interface.php
/var/www/my-app/library/Zend/Controller/Request/Abstract.php
/var/www/my-app/library/Zend/Controller/Response/Abstract.php
/var/www/my-app/library/Zend/Controller/Router/Route.php
/var/www/my-app/library/Zend/Controller/Router/Route/Abstract.php
/var/www/my-app/library/Zend/Controller/Router/Route/Interface.php
/var/www/my-app/library/Zend/Config.php
... (63 files in total) ...

To change the name of the file:

$included = new Zend_Debug_Include_Manager();
$included->setOutputDir('/var/www/my-app/dependencies');
$included->setFilename('files.dep');
...

Tracking Package Dependencies

Similar to Zend_Debug_Include_Adapter_File, but groups all the files into packages.

$included = new Zend_Debug_Include_Manager();
$included->setAdapter(new Zend_Debug_Include_Adapter_Package());
$included->setOutputDir('/var/www/my-app/dependencies');

The code above creates a zf-packages.txt file and adds the following data:

Zend/Loader.php
Zend/Controller
Zend/Exception.php
Zend/Config.php
Zend/Debug
Zend/View.php
Zend/View
Zend/Loader
Zend/Uri.php
Zend/Filter
Zend/Filter.php

Now, if you introduce a new dependency, for example, Zend_Mail:

class IndexController extends Zend_Controller_Action
{
    public function indexAction()
    {
        $mail = new Zend_Mail();
        $this->view->message  = 'test';
    }
}

The next time the Front Controller dispatches the request and calls the index action, Zend_Debug_Include will automatically add the Zend_Mail package and all its dependencies to the zf-packages.txt file:

Zend/Loader.php
Zend/Controller
Zend/Exception.php
Zend/Config.php
Zend/Debug
Zend/View.php
Zend/View
Zend/Loader
Zend/Uri.php
Zend/Filter
Zend/Filter.php
Zend/Mail.php
Zend/Mail
Zend/Mime.php
Zend/Mime

You can then use this information to keep track of dependencies and tell your build tool the name of the files and directories you need to copy and package.

External Dependencies

Here is where everything starts to make sense. Zend_Debug_Include allows you to search for external dependencies as well, you just need to tell the Adapter the libraries you are using. For example:

$libraries = array('Zend', 'Solar');
$adapter = new Zend_Debug_Include_Adapter_Package($libraries);

$included = new Zend_Debug_Include_Manager();
$included->setAdapter($adapter);
$included->setOutputDir('/var/www/my-app/dependencies');

The Solar packages will also be added to the zf-packages.txt file:

Zend/Loader.php
Zend/Controller
Zend/Exception.php
Zend/Config.php
Zend/Debug
Zend/View.php
Zend/View
Zend/Loader
Zend/Uri.php
Zend/Filter
Zend/Filter.php
Zend/Mail.php
Zend/Mail
Zend/Mime.php
Zend/Mime
Solar/Base.php
Solar/File.php
Solar/Factory.php
Solar/Sql.php
Solar/Sql
Solar/Cache.php
Solar/Cache

URL Adapter

If you want to create a different file for each request, use the URL adapter instead:

$included = new Zend_Debug_Include_Manager();
$included->setAdapter(new Zend_Debug_Include_Adapter_Url());
$included->setOutputDir('/var/www/my-app/dependencies');

The URL adapter maps the URL path to a filename. So, if you request the following URI:

http://my-app/blog/2009/02/01

It creates the file blog_2009_02_01.txt.

SVN

$ cd libraries/Zend
$ svn co http://svn.fedecarg.com/repo/Zend/Debug

Get groovy for better shell scripts

Eric Wendelin wrote:

I often use shell scripts to automate mundane, repeatable tasks on my computer. Since I’ve found Groovy, though, I have discovered a great way to make writing those scripts easier and more enjoyable. This is especially true if I have anything complex to do, and it saves me a LOT of time.

I couldn’t agree more. Also, I’m quite impressed how easy is to operate on an XML document with Groovy.

Get groovy for better shell scripts

Building desktop Linux applications with JavaScript

During his keynote presentation at OSCON last year, Ubuntu founder Mark Shuttleworth described application extensibility as an important enabler of innovation and user empowerment. Citing the Firefox web browser and its rich ecosystem of add-ons as an example, Shuttleworth suggested that the Linux community could deliver a lot of extra value by making scriptable automation and plugin capabilities available pervasively across the entire desktop stack.

Mark Shuttleworth also described his strategy for accelerating the adoption of Linux. He discussed the importance of extensibility in open platforms, contemplated the challenges of adapting conventional software methodologies so that they can be used for community-driven development, and contended that the open source software community has the potential to deliver a user experience which exceeds that of Apple’s Mac OS X platform.

Ryan Paul: Building desktop Linux apps with JavaScript

The Importance of Branching Models

Among the branching models used in software configuration management, the branch-by-purpose model offers better support for parallel development efforts and improved control of both planned and emergency software releases. If you want to improve software quality, you must first understand your software. What are its pieces? How are they organized and related to one another? If you do not understand your code base, your odds of updating it without breaking something are poor.

The Importance of Branching Models (PDF)

Links

High-level Best Practices in Software Configuration Management

Memcached consistent hashing mechanism

If you are using the Memcache functions through a PECL extension, you can set global runtime configuration options by specifying the values within your php.ini file. One of them is memcache.hash_strategy. This option sets the hashing mechanism used to select and specifies which hash strategy to use: Standard (default) or Consistent.

It’s recommended that you set to Consistent to allow servers to be added or removed from the pool without causing the keys to be remapped to other servers. When set to standard, an older strategy is used that potentially uses different servers for storage.

With PHP, the connections to the memcached instances are kept open as long as the PHP and associated Apache instance remain running. When adding a removing servers from the list in a running instance, the connections will be shared, but the script will only select among the instances explicitly configured within the script.

So, to ensure that changes to the server list within a script do not cause problems, make sure to use the consistent hashing mechanism.

Top 100 Blogs for Developers

PHPImpact makes it to the top 100!

What defines the popularity of a blog? The ranking by Google? Traffic statistics? Feed subscriptions? Feedback from readers? Links from other sites and blogs? The answer is of course… All of them!

Jurgen Appelo announced the new edition of the Top 100 Blogs for Developers. The list is based on a weighed average of each blog’s Google PageRank, Alexa Rank, Technorati Authority, RSSMicro FeedRank, Google hits, and blog comments.

Top 100 Blogs for Developers

Detect Replay Attacks in your Web Services

Many threats that are common to distributed systems are common to Web services as well. There are a few specific threats associated with the Web services processing model, such as:

  • Message replays: An attacker may re-play an entire message or a part of a SOAP message.
  • Man in the middle attack: An attacker may view and modify a SOAP message without the knowledge of either sender or the receiver.
  • Identity spoofing: An attempt to construct credentials that seems to be valid but not.
  • Denial of Service (DOS) attacks: An attempt to make a system expend its resources so that valid requests cannot access a service.
  • Message alteration: An attempt to alter a message compromising its integrity.
  • Confidentiality issues: Access to confidential information within a message by unauthorized parties.

Dimuthu wrote an interesting post about how to prevent replay attacks using WSF/PHP. He also shows how to detect them using WS-Addressing and WS-Username token headers.

How to kill an idea, or help it grow

It is far easier to kill an idea than to encourage it and turn it into a useful solution. Be on a constant watchout for putting down an idea too early without understanding the positive reasons for it being suggested. Hopefully you will see that there are many ways in which you can be constructive.

To kill an idea, say:

  • It’s not part of your job
  • That’s not what we do here
  • Costs too much
  • Against the company policy
  • It’s not budgeted, maybe next year
  • Let the other department handle that
  • It is not our problem
  • Why would you do something like that?
  • We have been doing it another way for a long time and it works fine
  • If it’s so good, why hasn’t someone suggested it already?
  • Has anyone else tried it successfully?
  • We have tried that before and it didn’t work
  • Is anyone crazy enough to try that?
  • We’re already doing that

To help an idea, say:

  • Yes, and…
  • Great, let’s try it
  • How can we make time to see if it will work?
  • What resources would we need to do it? Tell me more
  • How can we make it work?
  • What are the advantages?
  • How can we remove the dis-advantages?
  • What can I do to help this happen?
  • I like it
  • That sounds interesting, tell me more
  • How can we convince everyone else?

Getting Started With Message Queues

When you’re building an infrastructure that is distributed all over the internet, you’ll come to a point where you can’t rely on synchronous remote calls that, for example, synchronize data on 2 servers:

  1. You don’t have any failover system that resends messages if something went wrong (network outages, software failures).
  2. Messages are processed over time and you have no control if something goes overloaded by too many requests.

Even if you don’t have to send messages all over the Internet there are enough points of failures where something can go wrong. You want a reliable and durable system that fails gracefully and ensure.

Solutions

Dropr

Dropr is a distributed message queue framework written in PHP. The main goals are:

  • Reliable and durable (failsafe)-messaging over networks.
  • Decentralized architecture without a single (point of failure) server instance.
  • Easy to setup and use.
  • Modularity for queue storage and message transports (currently filesystem storage and curl-upload are implemented).

More info

Beanstalkd

Beanstalkd is a fast, distributed, in-memory workqueue service. Its interface is generic, but was originally designed for reducing the latency of page views in high-volume web applications by running most time-consuming tasks asynchronously.

It was developed to improve the response time for the Causes on Facebook application (with over 9.5 million users). Beanstalkd drastically decreased the average response time for the most common pages to a tiny fraction of the original, significantly improving the user experience.

More info

Zend Platform Job Queues

Job Queues is an approach to streamline offline processing of PHP scripts. Job Queue Server provides the ability to reroute and delay the execution of PHP scripts that are not essential during user interaction with the Web Server. Job Queues improve the response time during interactive web sessions and utilizes unused resources.

More info

Memcached as simple message queue

In this post, Olly explains how to use memcached as a simple message queue:

Some months ago at work we were in the need of a message queue, a very simple one, basically just a message buffer. The idea is simple, the webservers send there messages to the queue, the queue always accepts all messages and waits until the ETL processes request messages for further processing. As the webservers are time critical and the ETL processes aren’t you need something in between.

More info

Links