Archive for June 2008
Flickr Architecture
Flickr Platform
- PHP
- MySQL
- Shards
- Memcached for a caching layer.
- Squid in reverse-proxy for html and images.
- Linux (RedHat)
- Smarty for templating
- Perl
- PEAR for XML and Email parsing
- ImageMagick, for image processing
- Java, for the node service
- Apache
- SystemImager for deployment
- Ganglia for distributed system monitoring
- Subcon stores essential system configuration files in a subversion repository for easy deployment to machines in a cluster.
- Cvsup for distributing and updating collections of files across a network.
Flickr Architecture
A Modular Approach to Web Development
MVC is about loose-coupling, and Modular Programming takes that concept to the extreme. A modular application can dynamically load and unload modules at runtime, completely separate applications in their own right, which interact with the main application and other modules to perform some set of tasks.
In this article you will presented with a different approach to Web development, one that doesn’t focus on a design pattern, instead, it focuses on a programming paradigm known as Modular programming.
Modularity
Modularity is an important design principle, its goal is to design systems so that modules can be optimized independently of other modules, so that failure of one module does not cause other modules to fail, and in general to make it easier to understand, design and manage complex interdependent systems.
Principles
- Modular Decomposability: A software construction method satisfies Modular Decomposability if it helps in the task of decomposing a software problem into a small number of less complex sub-problems, connected by a simple structure, and independent enough to allow further work to proceed separately on each item.
- Modular Composability: A method satisfies Modular Composability if it favours the products of software elements which may then be freely combined with each other or produce new systems, possibly in an environment quite different from the one in which they were initially developed.
- Modular Understandability: A method favours Modular Understandability if it helps produce software which a human reader can understand each module without having to know the others, or, at worst, by having to examine only a few of the others.
- Modular Continuity: A method satisfies Modular Continuity if, in the software architectures that it yields, a small change in the problem specification will trigger a change of just one module, or a small number of modules.
- Modular Protection: A method satisfied Modular Protection if it yields architectures in which the effect of an abnormal condition occurring at run time in a module will remain confined to that module, or at worst will only propagate to a few neighbouring modules.
Advantages
One of the advantages of modularity, where modules are self contained, is that you can replace or add any module without affecting the rest of the system.
Some frameworks allow you to re-organize your directory structure, but because they were not created with a Modular architecture in mind, they fail to provide most of the advantages that this architecture has to offer.
Example: Traditional MVC implementation
admin/
controllers/
models/
views/
app/
controllers/
ContactUsController.php
NewsController.php
models/
ContactUsModel.php
NewsModel.php
views/
contactus/
index.html
news/
Using an MVC framework forces you to divide and organize your code according to the framework conventions. Presentation code goes to the view, data manipulation code goes to the model, and the request manipulation logic goes to the controller.
This is all theory, of course, and even though it’s not written anywhere that you should place 200 html files inside a folder called “views”, some systems encourage you to do it. And because everyone is doing it, you think is the right thing to do, right? Wrong.
Modular Programming
Modular programming is the breaking down of a website into parts that are of more manageable size and have a well-defined purpose. Modules should fit intuitively together into a system. In any development environment, modular programming is how larger, more complicated websites are constructed. The first step is to break the task into its basic parts, which leads to defining intermediate steps, and ultimately devising a comprehensive and efficient solution. Modules can be developed individually, validated, and then used throughout an organization, promoting teamwork, efficiency, and innovation. The advantages of modularity are obvious. Code only needs to be written once, which allows quick modifications. It provides a framework that dictates how subsequent programming should be incorporated. Modularity fits an environment where several programmers share work.
Modules
A module is a self-contained component of the system, which has a well-defined interface to the other components and can have its own MVC implementation. For example:
modules/
home/
controllers/
models/
views/
contact-us/
controllers/
ContactUsController.php
models/
views/
A module can also contain configuration data, user defined routes, dependencies information and language translations:
modules/
contact-us/
controllers/
i18n/
en/
models/
properties/
ContactUsConfig.php
ContactUsRoutes.php
ContactUsDependencies.php
views/
Composability
Composability refers to a collections of modules that can be coordinated and assembled to produce new systems. Pushing this concept a little bit further, we could create a ModuleCoordinator component that acts as a connector between modules. This will allow us to assemble and coordinate new modular subsystems.
Separation of Concerns
In an MVC framework that implements a modular architecture, separation of concerns (SoC) is the process of breaking a module into distinct features that overlap in functionality as little as possible. A concern is any piece of interest in a module. Based on this concept we are going to separate the functionality of a module into two groups, front-end and back-end:
modules/
contact-us/
controllers/
backend/
ContactUsBackendController.php
frontend/
ContactUsFrontendController.php
i18n/
en/
models/
properties/
views/
backend/
frontend/
The router of the framework will then determine which module, controller, and action of that controller should receive the request. For example:
Front-end Action Controller
The first segment of the URL path maps to the module’s front-end controller. The second segment is usually called “action” and maps to a method within that controller.
URI: http://www.example.com/contact-us/support/product/iphone
Module: contact-us
Controller: ContactUsFrontendController.php
Method: supportAction()
Params: array('product', 'iphone')
Back-end Action Controller
URI: http://cms.example.com/contact-us/view-departments/10
Module: contact-us
Controller: ContactUsBackendController.php
Method: viewDepartmentsAction()
Params: array('view', 10)
Content Management
Based on this architecture, you can create a Content Management System (CMS) that allows users to install, activate and assign modules to nodes (or pages). For example:
Page Module URL Action ----------------------------------------------------------------- - Home home home edit | delete - News news latest-news edit | delete - Products products products edit | delete + Sales products on-sales edit | delete + Shops map where-to-find-us edit | delete - Contact Us contact-us contact edit | delete
In the example above, we created a vertical module structure, where the module “maps” depends on the module “products” and inherits its properties. They have a parent/child relationship. However, if you want to minimize dependencies between modules, you can create a flat module structure.
Systems like Joomla and other Content Management Systems have adopted similar architectures, this confirms that a Modular Architecture is a good approach to Web development and RAD.
Conclusion
A system architecture should not only be based on a design pattern, such as MVC, it should also be based on different programming paradigms.
In a typical website, a design is implemented so that it meets a set of requirements at the time of development. Often, after a website is delivered, the user will want added functionality, or different users will require custom functionality based on specific needs. In order to accommodate these situations without a complete re-write, a framework that allows for future additions of modules without breaking the existing code base needs to be implemented.
In theory, putting the models in the models/ directory, the controllers in the controllers/ directory and the views in the views/ directory makes perfect sense, but in practice, it doesn’t. Remember, designing a flexible and scalable system architecture is way beyond your directory structure.
Foundations of Programming: Building Better Software
Karl Seguin has released the official, and completely free, Foundations of Programming eBook.
Although simplistic, every programming decision I make is largely based on maintainability. Maintainability is the cornerstone of enterprise development. Frequent readers are likely sick of hearing about it, but there’s a good reason we talk about maintainability so often – it’s the key to being a great software developer. I can think of a couple reasons why it’s such an important design factor. First, both studies and first hand experience tell us that systems spend a considerable amount of time (over 50%) in a maintenance state – be it changes, bug fixes or support. Second, the growing adoption of iterative development means that changes and features are continuously made to existing code (and even if you haven’t adopted iterative development such as Agile, your clients are likely still asking you to make all types of changes.) In short, a maintainable solution not only reduces your cost, but also increases the number and quality of features you’ll be able to deliver.
Create professional-looking charts quickly and easily with pChart

pChart is a PHP class oriented framework designed to create aliased charts. Data can be retrieved from SQL queries, CSV files, or manually provided. To have a complete overview of what pChart can do for you, we invite you to take a look on the on-line documentation which is trying to show all basic & advanced functionalities of this library.
From Swift Mailer to Zend_Mail
I’ve recently switched from Swift Mailer to Zend_Mail and to be honest, I’m loving it. Finally someone developed a lightweight, powerful and easy to use email component! Zend_Mail creates and sends e-mail messages with the format text or HTML and it allows e-mail to be sent in an easy form while preventing mail injection. Zend_Mime is a support set for handling multi-part MIME messages.
Simple email with Zend_Mail
$mail = new Zend_Mail();
$mail->setBodyText("This is the text of the mail");
$mail->setFrom("sender@example.com", "Some Sender");
$mail->addTo("recipient@example.com", "Some Recipient");
$mail->setSubject("Test Subject");
$mail->send();
For security reasons, Zend_Mail filters all header fields to prevent header injection with newline (\n) characters. You also can use most methods of the Zend_Mail object with a convenient fluent interface. A fluent interface means that each method returns a reference to the object on which it was called, so you can immediately call another method.
Zend_Mail fluent interface
$mail = new Zend_Mail();
$mail->setBodyText("This is the text of the mail.")
->setFrom("sender@example.com", "Some Sender")
->addTo("recipient@example.com", "Some Recipient")
->setSubject("TestSubject")
->send();
Zend_Mail messages can be sent via SMTP, so Zend_Mail_Transport_SMTP needs to be created and registered with Zend_Mail before the send() method is called.
Sending email via SMTP
$tr = new Zend_Mail_Transport_Smtp("mail.example.com")
Zend_Mail::setDefaultTransport($tr);
$mail = new Zend_Mail();
$mail->setBodyText("This is the text of the mail.");
$mail->setFrom("sender@example.com", "Some Sender")
$mail->addTo("recipient@example.com", "Some Recipient");
$mail->setSubject("Test Subject");
$mail->send();
Sending multiple emails per SMTP connection
$mail = new Zend_Mail();
$tr = new Zend_Mail_Transport_Smtp("mail.example.com");
Zend_Mail::setDefaultTransport($tr);
$tr->connect();
for ($i = 0; $i < 5; $i++) {
$mail->setBodyText("This is the text of the mail.");
$mail->setFrom("sender@example.com", "Some Sender");
$mail->addTo("recipient@example.com", "Some Recipient");
$mail->setSubject("Test Subject");
$mail->send();
}
$tr->disconnect();
To send an e-mail in HTML format, set the body using the method setBodyHTML() instead of setBodyText(). The MIME content type will automatically be set to text/html then. If you use both HTML and Text bodies, a multipart/alternative MIME message will automatically be generated:
Sending an HTML email
$mail = new Zend_Mail();
$mail->setBodyText("My Nice Test Text");
$mail->setBodyHtml("My Nice Test Text");
$mail->setFrom("sender@example.com", "Some Sender");
$mail->addTo("recipient@example.com", "Some Recipient");
$mail->setSubject("TestSubject");
$mail->send();
How easy was that!? If you haven’t used any of the Zend Framework components yet, then this is a great opportunity to do so.
Articles
Is this the future of Web application development?
Drag and drop widgets to build Web applications, in minutes, with minimal code.
WaveMaker Visual Ajax Studio is an easy-to-use visual builder that enables the drag & drop assembly of scalable, web-applications using Ajax widgets, web services and databases. WaveMaker Studio will look and feel especially familiar to client/server developers who are used to working with visual tools. WaveMaker’s Studio enables data-driven and web-services based applications to be quickly created without complex code, forms, patterns or portal frameworks.
Features
- Drag & Drop Assembly
- LiveLayout
- Push to Deploy: One-touch application deployment
- Visual Data Binding
- SOAP, REST and RSS web services
- Leverage existing CSS, HTML and Java
- Deploys a standard Java .war file
- It’s free!
See it in Action
Learn to Build Robust, Scalable and Maintainable Applications using MVC
MVC is about loose-coupling, and Modular Programming takes that concept to the extreme. A modular application can dynamically load and unload modules at runtime, completely separate applications in their own right, which interact with the main application and other modules to perform some set of tasks
This document (PDF) discusses the classes and interfaces of the PureMVC framework; illustrating their roles, responsibilities and collaborations with simple UML diagrams. The PureMVC framework has a very narrow goal, that is to help you separate your application’s coding concerns into three discrete tiers; Model, View and Controller. In this implementation of the classic MVC design meta-pattern, the application tiers are represented by three Singletons (a class where only one instance may be created). A fourth Singleton, the Façade, simplifies development by providing a single interface for communications throughout the application.
PureMVC Framework
Intrusion Detection For PHP Applications With PHPIDS
This tutorial explains how to set up PHPIDS on a web server with Apache2 and PHP5. PHPIDS (PHP-Intrusion Detection System) is a simple to use, well structured, fast and state-of-the-art security layer for your PHP based web application. The IDS neither strips, sanitizes nor filters any malicious input, it simply recognizes when an attacker tries to break your site and reacts in exactly the way you want it to. Based on a set of approved and heavily tested filter rules any attack is given a numerical impact rating which makes it easy to decide what kind of action should follow the hacking attempt. This could range from simple logging to sending out an emergency mail to the development team, displaying a warning message for the attacker or even ending the user’s session.
Agile Database Deployment Using Phing
Phing allows you to use SQL to define changes to your database schema, making it possible to use a version control system to keep things synchronized with the actual code.
A common way to automate development and deployment tasks is by writing shell scripts, however, Phing provides some advantages over shell scripts for task automation. Most of the shell scripts I created so far help me automate the most tedious tasks, from configuring, building, testing and documenting applications to synchronizing files and migrating database schemas. But, having a large collection of shell scripts can lead to a maintenance nightmare, reason why I decided to port some of them to PHP as Phing tasks. I chose Phing because it’s simple, powerful and very easy to extend.
DbDeployTask
Phing offers an optional task for making revisions to a database, it’s called DbDeployTask. Dave Marshall wrote a nice article about it. After adding this task to my build script and testing it, I came to the conclusion that DbDeployTask is not very effective if you want to roll out incremental changes to the databases. Also, it introduces new problems to the database migration process.
For example, if you use DbDeployTask, the SQL script names must begin with a number that indicates the order in which they should be run:
basedir/
database/
deltas/
1-create_foo_table.sql
2-alter_foo_table.sql
3-add_constraint_id_to_foo.sql
history/
up_20080622120000.sql
down_20080622120000.sql
public/
But, what happens when 2 or more developers are making changes to the same database design? How do they know which scripts have been executed? What if they don’t use a central development database, how do they merge individual changes? To be honest, I don’t know.
Agile Database Deployment
A more Agile way to manage database changes is for the developers to connect to a central development database. After making any changes to the database, the developers will have to:
- Update the local database/ directory before modifying any script.
- Copy the changes made to the database to the corresponding SQL file (up and down).
- Check the files back into the repository.
Finally, the changes can be propagated to other databases by executing a script or setting up a Cron job.
This is how the database directory can be structured:
basedir/
database/
down/
alter/
constraints/
create/
data/
up/
alter/
constraints/
create/
data/
public/
To put your database definition under version control, you need an off-line representation of that database. The directory up/ is used when migrating to a new version, down/ is used to roll back any changes if needed. These directories contain all the changes made to the database:
up/
alter/
foo.table.sql
constraints/
foo.table.sql
create/
foo.table.sql
data/
This directory structure is based on the one suggested by Mike Hostetler, and it clearly offers more advantages. After the team has made all necessary changes, the next step is to deploy the schema. If you are responsible for performing that task, you can get the most recent version of the database project from version control, build the deployment script, manually update that script as necessary, and then run the script to deploy the schema changes into development, staging or production.
Learn More
Database Schema Deployment (PDF) – By Lukas Smith
Symfony’s plugins tell us a lot about what developers need
If you want to use the power of the Rails framework without having to learn Ruby, then Symfony is the right framework for you. After spending more than 10 months playing around with Rails, I can say that Symfony is a great alternative to Rails for programmers who already know PHP.
What is a plugin?
A plugin is a packaged extension for a Symfony project. Plugins can contain classes, helpers, configuration, tasks, modules, schemas and model extensions of their own. The symfony autoloading mechanism takes plugins into account, and the features offered by a plugin are available in the project as if part of the framework. The PEAR packaged plugins have the advantage of managing dependencies, being easier to upgrade and automatically discovered.
Installing a Plugin
Since I last used Symfony, there have been great improvements, and the plugin system is one of them.
Stefan Koopmanschap wrote:
For symfony 1.1, the complete plugin system has been rewritten from scratch. This has been done to allow some serious improvements in the way plugins work, and to make it even simpler to work with plugins in your symfony project.
Installing a plugin has become much easier with symfony 1.1. The whole plugin system is now based on a full implementation of PEAR channels, which enables us to use all the advantages of this, such as the easy management (and even installation) of dependencies, or even the installation of plugins from different PEAR channels than the default symfony channel. You could even set up your own plugin channel to easily install the plugins you’ve written yourself!
Learn how to work with plugins in symfony 1.1
Popular Plugins
A quick look at the analytics of the Symfony website tells us a lot about what people need in addition to the Symfony core. Here is a list of the most popular plugins:
- sfSimpleCMSPlugin
- sfGuardPlugin
- sfDoctrinePlugin
- sfMediaLibraryPlugin
- sfSimpleForumPlugin
- sfFeed2Plugin
- sfThumbnailPlugin
- sfUJSPlugin
- sfControlPanelPlugin
- sfMogileFSPlugin
If you want to learn more about plugins, how they extend the framework and how you can package the features that you use across several projects into a plugin, read the plugin chapter of the Symfony book.
The plugin system ensures a great flexibility for distribution, upgrade and clean separation with the rest of the framework.
Once again, I can hear the Symfony play, and it’s wonderful.