Sankuru

The upstream developers

The Joomla team keeps creating new versions of the Joomla CMS. The Virtuemart team keeps creating new versions of the Virtuemart webshop. Other upstream Joomla extension developers keep improving their extensions, version after version.

We are downstream developers. Using the work delivered by the upstream developers, we add plugins, features, and generally customize the entire body of source code to suit the users' needs.

One single thing complicates our job massively: when we need to change the source code coming from upstream developers, and these developers bring out a new version of their code, we have to reapply the changes to the new version too. This activity is labour-intensive, costly, error-prone, and generally requires intense debugging sessions.

Therefore, downstream developers avoid changing the upstream source code. Unfortunately, changing upstream source code can all too often not be avoided.

Avoiding source code changes in Php with the runkit extension

Very often, we need to run some code before or after a particular function or method has been executed. For example, if the upstream source code contains a method aRecord.save(), we may want to do additional validation before this function, and do some logging after this function. Therefore, what we want is to replace the aRecord.save() method with the following sequence:

if( record.beforeSave() ) { record.save(); record.afterSave() );

Many source code extension issues can be solved in this manner. It requires two things:

• First, after loading the script that contains the aRecord.save() method, we need to load the script that will modify its definition. This requires modifying the builtin Php functions include(), include_once(), require(), and require_once() to make sure this mod is loaded everywhere the application uses it.
• Second, it requires methods to change existing methods at runtime. This practice is called monkey patching or duck punching.

So, the first thing we need to do, is to replace the builtin require() function by the following sequence:

if( beforeRequire($filePath) ) { require($filePath); afterRequire($filePath) };

The simplest way to achieve this in Php, is by using the Php PECL runkit extension:

function beforeRequire($filePath) { ... }
function afterRequire($filePath) { ... }

runkit_function_copy('require','oldRequire');

function newRequire($filepath)
{
 if( beforeRequire($filePath) )
 {
 oldRequire($filePath);
 afterRequire($filePath);
 }
}

runkit_function_remove('require');
runkit_function_copy('newRequire','require');

The code above, should allow us to override the builtin Php functions for including script files. In the afterRequire() function, it should be possible to make Php automatically load a patch file for the script file. In a next blog post, I will weigh in on the results for testing this strategy.

Minimizing the amount of patching needed by avoiding deeply nested anonymous structures

One could write an excerpt of code in the following way:

function f($parm)
{
 foreach($dd as $d)
 {
 $gg=$d->gg();
 foreach($gg as $g)
 {
 $hh=$g->hh();
 foreach($hh as $h)
 {
 do_something($h); // <-- we need to modify this
 }
 }
 }
}

The line, to modify above, is deeply nested anonymously inside a long function/method body.  In order to patch this line, we would need to rewrite and replace the entire function/method body. As you can imagine, replacing a large method implementation, increases the likelihood of a merge conflict with the next version of this method's implementation.

The only reference we could have to the line, is some kind of line number. But then again, this line number may not remain stable from version to version. Traditional diff tools will locate the line by using context lines. But then again, if the next version of the method implementation adds a line before or after the line we want to modify, the diff tool may fail to locate this line in the next version.

The functions/methods boundaries constitute the only reliable address space for locating fragments within the source code. The less granular the address space, the more difficult is becomes to target a particular line within the source code, and the more merge conflicts there will arise during future upgrades.

Upstream developers can address this problem by avoiding anonymously nested container structure, and by instead, choosing to name them. We could rewrite the function above as following:

function f($parm)
{
 foreach($dd as $d)
 {
 f_process_d($d);
 }
}

function f_process_d($d)
{
 $gg=$d->gg();
 foreach($gg as $g)
 {
 f_process_g($g);
 }
}

function f_process_g($g)
{
 $hh=$g->hh();
 foreach($hh as $h)
 {
 do_something($h); // <-- we need to modify this
 }
}

Explicitly naming and unnesting anonymous containers: a real-world example

The tcpdf library is quite a popular library to generate pdf documents in the Php world. Joomla uses it too. It is a fast and useful upstream piece of code. However, it is laced with deeply nested anonymous containers. As a result, it is:

• Hard to extend or patch
• Hard to read or actually understand

By unnesting its anonymous containers, the tcpdf library source code would become much easier to understand and much easier to extend. Take for example, the tcpdf.setFont() method:

function SetFont($family, $style='', $size=0) {
 // save previous values
 $this->prevFontFamily = $this->FontFamily;
 $this->prevFontStyle = $this->FontStyle;

 //Select a font; size given in points
 global $fpdf_charwidths;

 $family=strtolower($family);
 if($family=='') {
 $family=$this->FontFamily;
 }
 if((!$this->isunicode) AND ($family == 'arial')) {
 $family = 'helvetica';
 }
 elseif(($family=="symbol") OR ($family=="zapfdingbats")) {
 $style='';
 }
 $style=strtoupper($style);

 if(strpos($style,'U')!==false) {
 $this->underline=true;
 $style=str_replace('U','',$style);
 }
 else {
 $this->underline=false;
 }
 if($style=='IB') {
 $style='BI';
 }
 if($size==0) {
 $size=$this->FontSizePt;
 }

 // try to add font (if not already added)
 if($this->isunicode) {
 $this->AddFont($family, $style);
 }

 //Test if font is already selected
 if(($this->FontFamily == $family) AND ($this->FontStyle == $style) AND
 ($this->FontSizePt == $size)) {
    return;
 }

 $fontkey = $family.$style;

 //Test if used for the first time
 if(!isset($this->fonts[$fontkey])) {
 //Check if one of the standard fonts
 if(isset($this->CoreFonts[$fontkey])) {
 if(!isset($fpdf_charwidths[$fontkey])) {
 //Load metric file
 $file = $family;
 if(($family!='symbol') AND ($family!='zapfdingbats')) {
 $file .= strtolower($style);
 }
 if(!file_exists($this->_getfontpath().$file.'.php')) {
 // try to load the basic file without styles
 $file = $family;
 $fontkey = $family;
 }
 include($this->_getfontpath().$file.'.php');
 if (($this->isunicode AND !isset($ctg)) OR ((!$this->isunicode) AND 
      (!isset($fpdf_charwidths[$fontkey]))) ) {
 $this->Error("Could not include font metric file [".$fontkey."]: ".
               $this->_getfontpath().$file.".php");
 }
 }
 $i = count($this->fonts) + 1;

 if($this->isunicode) {
 $this->fonts[$fontkey] = array('i'=>$i, 'type'=>$type, 'name'=>$name,
   'desc'=>$desc, 'up'=>$up, 'ut'=>$ut, 'cw'=>$cw,
   'enc'=>$enc, 'file'=>$file, 'ctg'=>$ctg);
 $fpdf_charwidths[$fontkey] = $cw;
 } else {
 $this->fonts[$fontkey]=array('i'=>$i, 'type'=>'core', 
    'name'=>$this->CoreFonts[$fontkey], 'up'=>-100, 'ut'=>50,
    'cw'=>$fpdf_charwidths[$fontkey]);
 }
 }
 else {
 $this->Error('Undefined font: '.$family.' '.$style);
 }
 }
 //Select it
 $this->FontFamily = $family;
 $this->FontStyle = $style;
 $this->FontSizePt = $size;
 $this->FontSize = $size / $this->k;
 $this->CurrentFont = &$this->fonts[$fontkey];
 if($this->page>0) {
 $this->_out(sprintf('BT /F%d %.2f Tf ET', $this->CurrentFont['i'], $this->FontSizePt));
 }
 }

After unnesting it, it could become:

function SetFont($family, $style='', $size=0)
{
 $this->_savePreviousFont();

   $family=$this->_validateFontFamily($family);
 $style=$this->_validateFontStyle($style);

 // try to add font (if not already added)
 if($this->isunicode) {
 $this->AddFont($family, $style);
 }

 //Test if font is already selected
 if(($this->FontFamily == $family) AND 
 ($this->FontStyle == $style) AND 
 ($this->FontSizePt == $size)) {
 return;
 }

 $fontkey = $family.$style;
 $this->_loadCachedFontFile($fontkey);
 $this->_selectFont($family,$style,$size,$fontkey);
 }


The lower levels which do the actual work:

function _validateFontFamily($family) {
 $family=strtolower($family);
 if($family=='') {
 $family=$this->FontFamily;
 }
 if((!$this->isunicode) AND ($family == 'arial')) {
   $family = 'helvetica';
 }
 elseif(($family=="symbol") OR ($family=="zapfdingbats")) {
 $style='';
 }
   return $family;
}

function _validateFontStyle($style)
{
 $style=strtoupper($style);

 if(strpos($style,'U')!==false) {
 $this->underline=true;
 $style=str_replace('U','',$style);
 }
 else {
 $this->underline=false;
 }
 

if($style=='IB') {
 $style='BI';
 }
 

if($size==0) {
 $size=$this->FontSizePt;
 }

 return $style;
}

function _savePreviousFont()
{
 $this->prevFontFamily = $this->FontFamily;
 $this->prevFontStyle = $this->FontStyle;
}

function _validateFontMetricFilename($fontkey,$family,$style)
{
 $file = $family;
 if(($family!='symbol') AND ($family!='zapfdingbats')) {
 $file .= strtolower($style);
 }
 if(!file_exists($this->_getfontpath().$file.'.php')) {
 // try to load the basic file without styles
 $file = $family;
 $fontkey = $family;
 }
 return array($fontkey,$file);
}

function _loadFontMetricFile($fontkey,$family,$style)
{
 //Select a font; size given in points
 global $fpdf_charwidths;
 if(!isset($fpdf_charwidths[$fontkey])) {
 ($fontkey,$file)=$this_validateFontMetricFileName($fontkey,$family,$style);
 include($this->_getfontpath().$file.'.php');

if (($this->isunicode AND !isset($ctg)) OR ((!$this->isunicode) AND
(!isset($fpdf_charwidths[$fontkey]))) ) {
 $this->Error("Could not include font metric file [".$fontkey."]:    

 ".$this->_getfontpath().$file.".php");
 }

 return array($fontkey,$file);
}

function _registerFont($fontkey,$file)
{
 $i = count($this->fonts) + 1;
 if($this->isunicode) {
 $this->fonts[$fontkey] = array('i'=>$i, 'type'=>$type, 'name'=>$name, 
     'desc'=>$desc, 'up'=>$up, 'ut'=>$ut, 'cw'=>$cw, 
     'enc'=>$enc, 'file'=>$file, 'ctg'=>$ctg);
 $fpdf_charwidths[$fontkey] = $cw;
 } else {
 $this->fonts[$fontkey]=array('i'=>$i, 'type'=>'core',
      'name'=>$this->CoreFonts[$fontkey], 'up'=>-100, 'ut'=>50,
      'cw'=>$fpdf_charwidths[$fontkey]);
 }
}

function _loadFont($fontkey,$family,$style)
{
 list($fontkey,$file)=$this ->_loadFontMetricFile($fontkey,$family,$style);
 $this->_registerFont($fontkey,$file);
}

function _loadCacheFontFile($fontkey,$family,$style)
{
 //skip if the font is cached already
 if(isset($this->fonts[$fontkey]))  return;

 if(isset($this->CoreFonts[$fontkey]))
 {
 $this->_loadFont($fontkey,$family,$style);
 }
 else
 {
 $this->Error('Undefined font: '.$family.' '.$style);
 }
}

function _selectFont($family,$style,$size,$fontkey)
{
 $this->FontFamily = $family;
 $this->FontStyle = $style;
 $this->FontSizePt = $size;
 $this->FontSize = $size / $this->k;
 $this->CurrentFont = &$this->fonts[$fontkey];
 if($this->page>0) {
 $this->_out(sprintf('BT /F%d %.2f Tf ET', 
     $this->CurrentFont['i'], $this->FontSizePt));
 }
}

Unnesting this method makes it easier to extend it. In conjunction with Runkit, the strategy of explicitly naming and unnesting anonymous containers, allows us to patch the scripts granularly at runtime, and avoid touching the original source code.