Blog of a LAMP Developer based near Guildford, Surrey

I’m available for hire.

I have a solid understanding of Object Oriented PHP 5, SQL, MYSQL, PHP Frameworks, XHTML, CSS, Apache, Web 2.0 Ajax, Creative Design, Javascript, JQuery, Linux, Email Servers, Apache, Network Protocols and Systems Security.

Anyone who requires and freelance web developer. No project to big or to small. Give us a call.

I’m able to work from home, London or within 1 hour of Godalming, Surrey.

After the sad death of K T Ligesh I decided, like many people, to move away from HyperVM.

I searched and tried many control panels.

Openqrm - I found this over complicated
karesansui - This is very nice looking and polished but seemed quite slow.
VDS Manager - Still in beta for Xen

I’ve now decided to create the VM’s by hand. I then hope to make a PHP control panel of my own. Here are my steps for creating Xen VPS’s using the command line and Centos 5.

I’m going to create a vm called testvm

First I needed to find out what Volume groups exist on my system and what the VG Name was

lvdisplay

VG Name                VolGroup00

Create the disks for the VM and format

lvcreate -L8.5G -n testvm_rootimg VolGroup00
lvcreate -L 384M -n testvm_vmswap VolGroup00
mke2fs /dev/VolGroup00/testvm_rootimg
tune2fs -j /dev/VolGroup00/testvm_rootimg
mkswap /dev/VolGroup00/testvm_vmswap
mkdir /mnt/img
mkdir /mnt/vm

Now mount your disk image. I’ve created a custom hardened Centos distro, but you could use Jailtime Images

mount -o loop centos.5-3.64-smudge.img  /mnt/img
mount /dev/VolGroup00/looop_rootimg /mnt/vm

Copy the image onto your new partition

cp -a /mnt/img/* /mnt/vm/

Now all you need to do is configure the hostname, ip etc on the VM.

vi /mnt/vm/etc/sysconfig/network-scripts/ifcfg-eth0

DEVICE=eth0
BOOTPROTO=static
IPADDR=87.xxx.xxx.xxx
NETMASK=255.255.255.xxx
ONBOOT=yes

Change the hostname

vi /mnt/vm/etc/sysconfig/network

If your using Jailtime images people have advised to change this line to the following:

vi /mnt/vm/etc/inittab
1:2345:respawn:/sbin/mingetty console

vi /mnt/vm/etc/ssh/sshd_config
PermitRootLogin yes

vi /mnt/vm/etc/fstab

Add the line:

/dev/sda2     swap   swap     defaults     0     0

umount /mnt/img /mnt/vm

You will need a ram disk to boot from. This can be created by running:

mkinitrd /boot/xen-guest-initrd `uname -r` –with xenblk –with xennet –preload xenblk –preload xennet.

Now all you need to do is create the config file and give it an IP/ Mac address.Please check which kernel you are using first and change - kernel = “/boot/vmlinuz-2.6.18-128.1.14.el5xen”…

kernel = “/boot/vmlinuz-2.6.18-128.1.14.el5xen”
ramdisk    = ‘/boot/xen-guest-initrd’
maxmem = 192
memory = 192
vcpus = 1
name = “testvm”
vif = [ 'ip=87.xxx.xxx.xxx,bridge=xenbr0, rate = 1000KB/s, mac=aa:00:23:b0:de:34' ]
serial     = ‘pty’
disk       = ['phy:/dev/VolGroup00/testvm_rootimg,sda1,w', 'phy:/dev/VolGroup00/testvm_vmswap,sda2,w']
root = ‘/dev/sda1 ro’
on_poweroff = ‘destroy’
on_reboot   = ‘restart’
on_crash    = ‘restart’

All is ready to go.

Just run:

xm create testvm

And you VPS should be now running:

check with
xm console testvm

Hope this helps.

Smudge IT will be lauching  a new site and Xen Vps systems soon. Watch this space..

I used to go down the road of doing a for each on my source table, which I always hated and thought was bad.

This way I can use table locking for MyISAM or transactions, for InnoDB, to make sure the data gets copied across.

I mainly use this method for archiving tables and keep the primary keys the same.

I have 2 tables created called ads and adsarchives.

What you do is make a copy of your database structure called adsarchives, and change the primary key so that it doesn’t auto increment. This mean that you can’t copy the same data twice. Also the second query won’t delete your data from the source on a fail.

( Oh by the way, my first example is using Cakephp on a MyISAM table) It would be much better to do transactions for this. But I wasn’t updating an InnoDB table.

function archiveads($monthsold=false){

if(is_numeric($monthsold) & $monthsold > 1){

$range = date('Y-m-d H:i:s', strtotime("-{$monthsold} months"));

$db =& ConnectionManager::getDataSource($this->useDbConfig);

$this->query('LOCK TABLES ads READ, adsarchives WRITE');
$this->query("INSERT INTO adsarchives SELECT ads.* FROM ads WHERE ads.enddate < '{$range}'");

$error = $db->lastError();

$this->query('UNLOCK TABLES');

if(!$error) { $this->query("DELETE FROM ads WHERE  ads.enddate < '{$range}'") ; return true;}else{

$this->log('FATAL Error archiving the Ads |'.$error);

}

}else{

return false;
}

}

If you want the raw SQl way of doing it try this. I’m archiving 3 month old data.

LOCK TABLES ads READ, adsarchives WRITE;
SET @age=(CURRENT_DATE()- INTERVAL 3 MONTH), @today=CURRENT_DATE();

INSERT INTO adsarchives
SELECT ads.*
FROM ads WHERE ads.enddate < @age;

UNLOCK TABLES;
DELETE FROM ads WHERE  ads.enddate < @age;

After plenty of research and trials, it is possible to use mysql transactions for custom queries.

You still have to create the table with InnoDB for row based locking to work.

ALTER TABLE table_name type=InnoDB;

And set the transactions variable in your model

var $transactional = true;

Now in your method use the following:

class YourModel extends AppModel {

var $transactional = true;

function mymethod()
{

  $db =& ConnectionManager::getDataSource($this->useDbConfig);

 #start the transaction
  $db->begin($this);

 $success = $db->query('my sql');
  if ($success  === true) {
       return $db->commit($this);
  } else {
       $db->rollback($this);
       return false;
  }
}

}

Converting to flv

ffmpeg -i Head Office Version.ts -ar 22050 -ab 128 -f flv -s 448×252 - | flvtool2 -U stdin video.flv

or with better bitrate

ffmpeg -i happy_feet_h.i480.mov -ar 22050 -ab 128 -b 800 -f flv -s  480×204 - | flvtool2 -U stdin happyfeet.flv

Generate thumbnail

This creates a thumb of 12 seconds into the video

ffmpeg -y -i head_office.flv -f image2 -ss 12 -vframes 1 -an head_office.jpg

This is just a one liner as my class uses lots of method etc that you probably don’t need.

The money shot is this:

if (! file_exists ( $thumb_dir . $thumbname )) {
$draw_thumb = shell_exec ( $this->ffmpeg . ' -y -i ' . $video_file . ' -f image2 -ss 10 -vframes 1 -an ' . $thumb_dir . $thumbname );
}

($this->ffmpeg = the location of the ffmpeg binary)

This will generate a thumbnail of the video at 10 seconds.

Here is a method, that I pulled out of one of my video conversion classes.

All you need to do is set the location of ffmpeg ($this->ffmpeg)

It will then parse and return the video dimensions. More video methods to follow

/**
* Get the dimensions of a video file
*
* @param unknown_type $video
* @return array(width,height)
* @author Jamie Scott
*/
function __get_video_dimensions($video = false) {

if (file_exists ( $video )) {
$command = $this->ffmpeg . ' -i ' . $video . ' -vstats 2>&1';
$output = shell_exec ( $command );

$result = ereg ( '[0-9]?[0-9][0-9][0-9]x[0-9][0-9][0-9][0-9]?', $output, $regs );

if (isset ( $regs [0] )) {
$vals = (explode ( 'x', $regs [0] ));
$width = $vals [0] ? $vals [0] : null;
$height = $vals [1] ? $vals [1] : null;
return array ('width' => $width, 'height' => $height );
} else {
return false;
}
} else {

return false;
}

}

I found a simple solution for submitting a form using jquery + ajax. One that was compatible with CakePHP.

You need to create a from with an ID.

<?php echo $form->create('Test',array('url' => 'edit/', 'type' => 'get','id' => 'TestID'));?>

Then bind a listener to the form submit

// do an ajax from instead
$('#TestID').submit(function() {

var inputs = [];
$(':input', this).each(function() {
inputs.push(this.name + '=' + escape(this.value));
})

jQuery.ajax({
data: inputs.join('&'),
url: this.action,
timeout: 2000,
error: function() {
console.log("Failed to submit");
},
success: function(r) {
console.log(r);
}
})

// return false
return false;
});

This will work with checkboxes, input fields, selects but not upload file types.

It will create a get string and send the request to the form action using ajax.

CakePHP usually uses $this->data in its baked pages. As it is now a get request, I will now be using $this->params.

If you want to use Cake’s baked pages, just add this ternary operator to the beginning of the corresponding methods.

$this->data = isset($this->params['url']['data'])?$this->params['url']['data']:$this->data;

This is another problem that I came across, as unoconv doesn’t let you convert a PDF to anything.

I found some more cool tools for handling PDF files.

The package is called poppler and can be installed on linux or if you are using OSX there is a macport.

Poppler is a PDF rendering library based xpdf

To install on linux run the command

yum install poppler-utils

On the mac install macports then run

sudo port install poppler

Now you have the following binaries at your disposal.

pdftotext, pdftohtml

You can generate a text version of a pdf by running

pdftotext -enc UTF-8 -eol unix input.pdf output.txt

I’m writing a PHP script that will talk to these binaries. I will upload soon.

Here is the man page for the binary:

NAME
pdftotext - Portable Document Format (PDF) to text converter (version 3.00)

SYNOPSIS
pdftotext [options] [PDF-file [text-file]]

DESCRIPTION
Pdftotext converts Portable Document Format (PDF) files to plain text.

Pdftotext reads the PDF file, PDF-file, and writes a text file, text-file. If text-file is not specified, pdftotext con-
verts file.pdf to file.txt. If text-file is ´-’, the text is sent to stdout.

CONFIGURATION FILE
Pdftotext reads a configuration file at startup. It first tries to find the user’s private config file, ~/.xpdfrc. If
that doesn’t exist, it looks for a system-wide config file, /etc/xpdf/xpdfrc. See the xpdfrc(5) man page for details.

OPTIONS
Many of the following options can be set with configuration file commands. These are listed in square brackets with the
description of the corresponding command line option.

-f number
Specifies the first page to convert.

-l number
Specifies the last page to convert.

-layout
Maintain (as best as possible) the original physical layout of the text. The default is to ´undo’ physical layout
(columns, hyphenation, etc.) and output the text in reading order.

-raw Keep the text in content stream order. This is a hack which often “undoes” column formatting, etc. Use of raw
mode is no longer recommended.

-htmlmeta
Generate a simple HTML file, including the meta information. This simply wraps the text in

 and

and
prepends the meta headers.

-enc encoding-name
Sets the encoding to use for text output. The encoding-name must be defined with the unicodeMap command (see
xpdfrc(5)). The encoding name is case-sensitive. This defaults to “Latin1″ (which is a built-in encoding).
[config file: textEncoding]

-eol unix | dos | mac
Sets the end-of-line convention to use for text output. [config file: textEOL]

-nopgbrk
Don’t insert page breaks (form feed characters) between pages. [config file: textPageBreaks]

-opw password
Specify the owner password for the PDF file. Providing this will bypass all security restrictions.

-upw password
Specify the user password for the PDF file.

-q Don’t print any messages or errors. [config file: errQuiet]

-cfg config-file
Read config-file in place of ~/.xpdfrc or the system-wide config file.

-v Print copyright and version information.

-h Print usage information. (-help and –help are equivalent.)

BUGS
Some PDF files contain fonts whose encodings have been mangled beyond recognition. There is no way (short of OCR) to
extract text from these files.

EXIT CODES
The Xpdf tools use the following exit codes:

0 No error.

1 Error opening a PDF file.

2 Error opening an output file.

3 Error related to PDF permissions.

99 Other error.

AUTHOR
The pdftotext software and documentation are copyright 1996-2004 Glyph & Cog, LLC.

SEE ALSO
xpdf(1), pdftops(1), pdfinfo(1), pdffonts(1), pdftoppm(1), pdfimages(1), xpdfrc(5)
http://www.foolabs.com/xpdf/

Taken from hostingspeeds.com

‘February 1, 2009 - “Thanks for your email. That’s great news.” winning coder Jamie Scott and owner of smudge-it.co.uk replied today when he found out he won. Jamie will code hostingspeeds.com into a full featured web 2.0 hosting speed measuring service for webmasters and web hosts with plugins for Wordpress, Joomla and other great blog and portal scripts ppl use these days.

Jamie Scott has won because of his programming skills, his experience with web hosting business and his understanding of what web 2.0 is all about. I thank all of you that offered to code v3.’

I have already started rewriting the graphing code, so that the data is stored in RRD databases and there will be pretty graphs using RRDtool.

The system will be fully scalable, by using a worker node system to poll the information and will encorporate a complex advertising and server monitoring system.

This has been an issue that has bothered me for a while. I finally found a solution that worked and doesn’t kill your server in the process.

I give to two words. OpenOffice, or is that one ?

This is what I’m running for this test:

OS: CentOS release 5.2 (Final)
PHP: PHP 5.2.8
Openoffice 1.2.3

Firstly I installed several programs using yum. You will need to use DAG’s repo:

rpm -Uhv http://apt.sw.be/redhat/el5/en/x86_64/rpmforge/RPMS//rpmforge-release-0.3.6-1.el5.rf.x86_64.rpm

yum install unoconv openoffice.org-headless openoffice.org-writer

unoconv is a handy tool that can be run as a demon and talk to the open office binary, via the command line.

In order to run the commands via apache you need to change the apache home directory and make it writable.

mkdir /home/apache
chown apache:apache /home/apache
usermod -d /home/apache apache
chmd 755 /home/apache

Now the apache user can create the hidden .openoffice.org2.0 directory.

With the setup out of the wa,y we need to start the open office deamon.

I did this as root but you could start this as apache.

unoconv –listener &

This basically creates the following deamon

soffice.bin -nologo -nodefault -accept=socket,host=localhost,port=2002;urp;StarOffice.ComponentContext

You can now send requests to port 2002 using unoconv

/usr/bin/unoconv --server localhost --port 2002 --stdout -f pdf input.doc

This will output the PDF file to the stdout.

Here is a cakephp component that I wrote to talk to unoconv. Please note this is very alpha and has only had a small amount of testing but works If you want to use it you must create these directories in your cake install.

‘TMP_FOLDER’, TMP . ‘filegenerator/’
ROOT . ‘/uploads/generatedpdfs/’
ROOT . ‘/uploads/docfiles/’

It can be used via a form upload

$this->Filegenerator = new FilegeneratorComponent ($this->params["form"]['uploaddocfile']);
// if the filegenerator did all it's magic ok then process
if($this->Filegenerator){

// returns the text version of the PDF
$text = $this->Filegenerator->convertDocToTxt();
// returns the html of the PDF
$html = $this->Filegenerator->convertDocToHtml();
// returns the generated pdf file
$pdf = $this->Filegenerator->convertDocToPdf($doc_id);

}

The component called filegenerator.php

<?php
/**
* Class Used to convert files.
*@author jamiescott.net
*/
class FilegeneratorComponent extends Object {

// input folder types
private $allowable_files = array ('application/msword' => 'doc' );
// variable set if the constuctor loaded correctly.
private $pass = false;
// store the file info from constuctor reference
private $fileinfo;

/**
* Enter description here...
*
* @param array $fileinfo
* Expected :
* (
[name] => test.doc
[type] => application/msword
[tmp_name] => /Applications/MAMP/tmp/php/php09PYNO
[error] => 0
[size] => 79360
)
*
*
* @return unknown
*/
function __construct($fileinfo) {

// folder to process all the files etc
define ( 'TMP_FOLDER', TMP . 'filegenerator/' . $this->generatefoldername () . '/' );

// where unoconv is installed
define ( 'UNOCONV_PATH', '/usr/bin/unoconv' );
// where to store pdf files
define ( 'PDFSTORE', ROOT . '/uploads/generatedpdfs/' );
// where to store doc files
define ( 'DOCSTORE', ROOT . '/uploads/docfiles/' );
// apache home dir
define ( 'APACHEHOME', '/home/apache' );
// set some shell enviroment vars
putenv ( "HOME=".APACHEHOME );
putenv ( "PWD=".APACHEHOME );

// check the file info is passed the tmp file is there and the correct file type is set
// and the tmp folder could be created
if (is_array ( $fileinfo ) &amp;amp;&amp;amp; file_exists ( $fileinfo ['tmp_name'] ) &amp;amp;&amp;amp; in_array ( $fileinfo ['type'], array_keys ( $this->allowable_files ) ) &amp;amp;&amp;amp; $this->createtmp ()) {

// bass by reference
$this->fileinfo = &amp;amp;$fileinfo;
// the constuctor ran ok
$this->pass = true;
// return true to the instantiation
return true;

} else {
// faild to instantiate
return false;

}

}

/**
*      * takes the file set in the constuctor and turns it into a pdf
* stores it in /uploads/docfiles and returns the filename
*
* @return filename if pdf was generated
*/
function convertDocToPdf($foldername=false) {

if ($this->pass) {

// generate a random name
$output_pdf_name = $this->generatefoldername () . '.pdf';

// move it to the tmp folder for processing
if (! copy ( $this->fileinfo ['tmp_name'], TMP_FOLDER . 'input.doc' ))
die ( 'Error copying the doc file' );

$command = UNOCONV_PATH;
$args = ' --server localhost --port 2002 --stdout -f pdf ' . TMP_FOLDER . 'input.doc';

$run = $command . $args;

//echo $run; die;
$pdf = shell_exec ( $run );
$end_of_line = strpos ( $pdf, "\n" );
$start_of_file = substr ( $pdf, 0, $end_of_line );

if (! eregi ( '%PDF', $start_of_file ))
die ( 'Error Generating the PDF file' );

if(!file_exists(PDFSTORE.$foldername)){
mkdir(PDFSTORE.$foldername);
}

// file saved
if(!$this->_createandsave($pdf, PDFSTORE.'/'.$foldername.'/', $output_pdf_name)){
die('Error Saving The PDF');
}

return $output_pdf_name;

}

}

/**
* Return a text version of the Doc
*
* @return unknown
*/
function convertDocToTxt() {

if ($this->pass) {

// move it to the tmp folder for processing
if (! copy ( $this->fileinfo ['tmp_name'], TMP_FOLDER . 'input.doc' ))
die ( 'Error copying the doc file' );

$command = UNOCONV_PATH;
$args = ' --server localhost --port 2002 --stdout -f txt ' . TMP_FOLDER . 'input.doc';

$run = $command . $args;

//echo $run; die;
$txt = shell_exec ( $run );

// guess that if there is less than this characters probably an error
if (strlen($txt) < 10)
die ( 'Error Generating the TXT' );

// return the txt from the PDF
return $txt;

}

}

/**
* Convert the do to heml and return the html
*
* @return unknown
*/
function convertDocToHtml() {

if ($this->pass) {

// move it to the tmp folder for processing
if (! copy ( $this->fileinfo ['tmp_name'], TMP_FOLDER . 'input.doc' ))
die ( 'Error copying the doc file' );

$command = UNOCONV_PATH;
$args = ' --server localhost --port 2002 --stdout -f html ' . TMP_FOLDER . 'input.doc';

$run = $command . $args;

//echo $run; die;
$html= shell_exec ( $run );
$end_of_line = strpos ( $html, "\n" );
$start_of_file = substr ( $html, 0, $end_of_line );

if (! eregi ( 'HTML', $start_of_file ))
die ( 'Error Generating the HTML' );

// return the txt from the PDF
return $html;

}

}
/**
* Create file and store data
*
* @param unknown_type $data
* @param unknown_type $location
* @return unknown
*/
function _createandsave($data, $location, $file) {

if (is_writable ( $location )) {

// In our example we're opening $filename in append mode.
// The file pointer is at the bottom of the file hence
// that's where $somecontent will go when we fwrite() it.
if (! $handle = fopen ( $location.$file, 'w' )) {
trigger_error("Cannot open file ($location$file)");
return false;
}

// Write $somecontent to our opened file.
if (fwrite ( $handle, $data ) === FALSE) {
trigger_error("Cannot write to file ($location$file)");
return false;
}

fclose ( $handle );
return true;

} else {
trigger_error("The file $location.$file is not writable");
return false;
}

}

function __destruct() {

// remove the tmp folder

if (file_exists ( TMP_FOLDER ) &amp;amp;&amp;amp; strlen ( TMP_FOLDER ) > 4)
$this->removetmp ();

}

/**
* Create the tmp directory to hold and process the files
*
* @return unknown
*/
function createtmp() {

if (is_writable ( TMP )) {

if (mkdir ( TMP_FOLDER ))
return true;

} else {

return false;
}

return false;

}

/**
* Delete the tmp dir
*
* @return unknown
*/
function removetmp() {

if (strlen ( TMP_FOLDER ) > 3 &amp;amp;&amp;amp; file_exists ( TMP_FOLDER )) {

if ($this->recursive_remove_directory ( TMP_FOLDER ))
return true;

}

return false;
}

/**
* Return a rendom string for the folder name
*
* @return unknown
*/
function generatefoldername() {

return md5 ( microtime () );

}

/**
* Recursivly delete directroy or empty it
*
* @param unknown_type $directory
* @param unknown_type $empty
* @return unknown
*/
function recursive_remove_directory($directory, $empty = FALSE) {
// if the path has a slash at the end we remove it here
if (substr ( $directory, - 1 ) == '/') {
$directory = substr ( $directory, 0, - 1 );
}

// if the path is not valid or is not a directory ...
if (! file_exists ( $directory ) || ! is_dir ( $directory )) {
// ... we return false and exit the function
return FALSE;

// ... if the path is not readable
} elseif (! is_readable ( $directory )) {
// ... we return false and exit the function
return FALSE;

// ... else if the path is readable
} else {

// we open the directory
$handle = opendir ( $directory );

// and scan through the items inside
while ( FALSE !== ($item = readdir ( $handle )) ) {
// if the filepointer is not the current directory
// or the parent directory
if ($item != '.' &amp;amp;&amp;amp; $item != '..') {
// we build the new path to delete
$path = $directory . '/' . $item;

// if the new path is a directory
if (is_dir ( $path )) {
// we call this function with the new path
recursive_remove_directory ( $path );

// if the new path is a file
} else {
// we remove the file
unlink ( $path );
}
}
}
// close the directory
closedir ( $handle );

// if the option to empty is not set to true
if ($empty == FALSE) {
// try to delete the now empty directory
if (! rmdir ( $directory )) {
// return false if not possible
return FALSE;
}
}
// return success
return TRUE;
}
}
}
?>