Transferring large files on FAT32

Just encountered a problem today. Trying to copy a 4.2gb file onto a USB portable drive.

In Linux I got the error message "error splicing file: file too large"

Turns out that most USB portable drives are formatted as FAT32 which has a limit of 4gb per file. Never knew that...

All is not lost though, there is a terminal command called "split". I simply opened a terminal and entered

split -b 2000m mybigfile.ext myprefix

This created 3 files called myprefixaa, myprefixab, myprefixac - the first 2 were 2gb each, the last one was about 200mb. The -b 2000m switch simply says split into 2000m files, which is 2gb.

Once copied, you can reverse the split by using "cat" - This will join or concatenate the files together.

cat myprefixaa myprefixab myprefixac > mybigfile.ext

Thats it...

Excellent!!!

Let me google that for you...

http://lmgtfy.com/

What memory is this?

Restoring a single file from git

Okay, so you've done a few commits, pushes and pulls - but you've made a mess of a file? How do you get it back?

Simplest way is to first get the commit number from the log

git log

Then use git checkout with the commit number

git checkout [commit-ref] [filename]

Simples :)

Web apps for mobile

Very simple, use jquery for mobile :)

http://jquerymobile.com/

Tracking slow MySql queries

Useful tool to track slow queries, no need to do any coding, just need to change the MySql config :

gksudo gedit /etc/mysql/my.cnf

Search for "slow" and change the lines to

log_slow_queries = /var/lib/mysql/mysql-slow.log

long_query_time = 1

log-queries-not-using-indexes

Restart mysql 

sudo service mysql restart

Run your applications, then any queries that take longer than 1 second will be stored in the log. This can be viewed with

sudo mysqldumpslow | more

This will display the slowest queries first. Useful to keep running while in production to optimise your system.

To see the queries with the actual data

sudo more /var/lib/mysql/mysql-slow.log

Credit and thanks to Laurent David @ TDM

Really really simple method to export data from php to excel or another other spreadsheet.


<?php
// Excel export using The KISS method
$filename = $_REQUEST['name'] ? $_REQUEST['name'] : 'excelreport';
$contents = $_REQUEST['data'] ? $_REQUEST['data'] : "testdata1\ttestdata2\ttestdata3\ntestdata4\ttestdata5\ttestdata6\n";


if (substr(strtolower($filename)-4,4)!='.tsv') {
$filename .= '.tsv';
}


// header('Content-type: application/ms-excel');
header("Content-Type: text/tab-delimited-values");
header('Content-Disposition: attachment; filename='.$filename);


echo $contents;
?>

Kineo

Had a fantastic week last week working at Kineo on behalf of TDM. Really great working environment - relaxed but fast paced and very friendly people. Things would just turn up like boxes of fruit and packets of doughnuts that we could all tuck into as well as constant supply of proper coffee.

Mainly bug fixing on a Totara project using PHP but also picked up a bit about Postgresql (database), Trac (ticketing system), Passpack (shared passwords) and single sign on.

Aside from that... Brighton is a fab place!! Highly recommend it

Want to be a web developer? (part 6)

Version control / backup

Git is great for this - you will need a repository to store your code. This can either be a public or private repository. git hub has both public and private, codebase is private only. You can also setup your own repository on your host but it will probably need to be on a dedicated server rather than a shared server. Using git is great for collaborative development and merging code.

The basic procedure is

• Go to your development directory then enter
• git add * - to add files to the git repository
• git commit - commit changes to your local repository
• git pull - pull down any changes so you can merge
• git commit - another commit with merged changes
• git push - finally push them onto your remote repository

You should also install Meld - this is a fantastic tool for showing differences in 2 files and/or folders - but also allowing you to make changes to the files. To install

sudo apt-get install meld

Documentation

Always a good idea to have a bit of documentation! You'll notice a lot of code will have tags in the comments in front of a function or class. These are tags that are recognised by documenting systems. PHPDocumentor used to be the one to go for, but it hasn't been updated since 2008. Doxygen is a great alternative and has been kept up to date. To install

sudo apt-get install graphviz
sudo apt-get install doxygen
sudo apt-get install doxygen-gui

Then to run the program use

doxywizard

Certification

From a career point, you might want to get an industry certification - for PHP this is a Zend certified engineer. You might also want to consider getting an LPIC for Linux and Apache

And finally...

Thats pretty much it! I don't think I've left anything out - let me know if I have. Hope you have have found this series helpful

Cheers, Russ

(Part 5)

Want to be a web developer? (part 5)

Improving your code

Now you have your lovely application, can you improve it? Here are some tools that can check your code for fine tuning. Note: Some of the following tools require pear, to install pear use

sudo apt-get install php-pear

phpcpd - duplicate code

phpcpd - this is a command line tool that scans your code for duplicate code. Install using pear

pear channel-discover pear.phpunit.de

pear channel-discover components.ez.no

pear install phpunit/phpcpd

Usage :

phpcpd --log-pmd phpcpd.xml yourprojectfolder

This will generate a file called phpcpd.xml with all the duplicate code. There are other output options too, plain text for example.

phpcs - coding standards

phpcs - command line too that will check for coding standards

pear install PHP_CodeSniffer

Usage :

phpcs --report=xml --report-file=phpcs.xml yourprojectfolder

phpmd - mess detector

phpmd - This will look for "messy code", one of the best command line tools

Install using pear

pear channel-discover pear.phpmd.org

pear channel-discover pear.pdepend.org

pear install --alldeps phpmd/PHP_PMD

Usage

phpmd yourprojectfolder xml codesize,unusedcode,naming,design --reportfile phpmd.xml

profiling

Kcachegrind is a very useful tool for analysing profile results. This will highlight processes that are taking too long or are using too many resources. Install kcachegrind and xdebug if you haven't done so already

sudo apt-get install php5-xdebug
sudo apt-get install kcachegrind

Modify your php.ini file to include the following.

; Profiler Settings for kcachegrind copied from http://www.odino.org/5/profiling-with-xdebug-and-kcachegrind

; and http://xdebug.org/docs/profiler

xdebug.profiler_enable=1

xdebug.profiler_output_dir = "/var/benchmark/"

Then after running your application, you'll find cachegrind files in /var/benchmark. Open these by going to Applications -> Programming -> Kcachegrind

phpunit - unit tests

PHPUnit tests provide a great way to ensure you are getting the results you should be getting. This can be checked constantly so any code changes can be checked to see if they match the tests. For setting up this, refer to the Netbeans unit test documentation.

(Part 4) (Part 6)

Want to be a web developer? (part 4)

Open Source

Okay, so now you want to do more coding! Most of the time you are probably going to be amending an existing application rather than starting from scratch. Whenever anyone asks me for an application, I always look for an open source php solution first - mainly so I don't have to reinvent the wheel.

Of these Content Managment Systems are by far the most popular applications for websites. They are perfect for end users to manage their website without anyone techy having to do it for them. The techy bit comes from installing and making modifications. The end user can modify the content - this is very important for marketing, nobody knows their business like the business owner - so let them manage the content. Also SEO likes lots of content, especially lots of new content.

I would suggest downloading Joomla, Drupal and Wordpress to your /var/www folder. eg /var/www/joomla, /var/www/drupal, /var/www/wordpress. There are installation instructions for each. Usually you create a database, extract the zip and browse to the folder.

  

Once installed, create a project for them in Netbeans. This will then give you an idea of coding of a large project. There are plenty of other open source systems other than CMS (I'll add links later - I've plagerised this list, I think most of them are php and there are others missing from the list)

Blogs: b2evolution, Nucleus, pMachine Free, WordPress, Pixelpost, Life Type

Content Management Software (CMS): Drupal, Drupal7, Concrete5, Geeklog, Joomla, Mambo Open Source, PHP-Nuke, phpWCMS, phpWebSite, Post-Nuke, Siteframe, Typo3, Xoops, ocPortal, MODx, Moodle, e107, Zikula, Geeklog, Mahara, phpwcms, Hotaru CMS, Textpattern, Surus CMS

Customer Support/Chat Software: Crafty Syntax Live Help, Help Center Live, osTicket, PerlDesk, PHP Support Tickets, Support Logic Helpdesk, Support Services Manager, phpFreeChat, Vtiger CRM,   SugarCRM, 

Discussion Boards: phpBB2, Simple Machines Forum (SMF), bbPress, MyBB, Advanced Electron Forum, PunBB

E-Commerce: CubeCart, OS Commerce, Zen Cart, Open Cart, Magento, PrestaShop, TomatoCart, PhpCOIN

Social Networking Software: Pligg, Jcow

FAQ: FAQMasterFlex, phpMyFAQ

Hosting Billing: phpCOIN

Image Galleries: 4images Gallery, Coppermine Photo Gallery, Gallery, Zenphoto, Piwigo,

Mailing List: PHPlist, poMMo, Dada Mail

Polls and Surveys: Advanced Poll, phpESP, PHPSurveyor, LimeSurvey

Project Management: PHProjekt, dotProject, Mantis Bug Tracker, PHP Guest Book

Site Builders: Templates Express

Wiki Software: PhpWiki, TikiWiki CMS/Groupware, WikkaWiki, PmWiki

Other Scripts: Dew-NewPHPLinks, Moodle, Noahs Classifieds, Open-Realty, phpAdsNew, PHPauction, phpFormGenerator, WebCalendar

Other Software: Z-Push, Elgg, OpenDocMan, Sitebar, Churchinfo, Lazarus

There are others missing from the list... If you are looking for something particular then just google "open source" and the name of the application you are looking for.

PHP Frameworks

Here's where it gets interesting... A framework allows you to concentrate on the actual development, letting the framework to do most of the repetitive work for you. This is rapid application development. Frameworks tend to be used for web applications rather than web sites - the distinction is that websites are like portals for the world to see, a web application is usually created for specific purpose and requires a login. For example, an accounts package. A framework also has the advantage that other developers can take it over if they are familiar with the framework.

There are plenty of PHP frameworks to choose from, but the main ones are Zend, Symfony, Cakephp, Codeigniter, Kohana and Yii.

I've tried them all - Zend is by far the most popular but I like Yii the best. Its simpler than Zend and Symfony, very quick to run and install, really solid code. It has a small but loyal following.

Netbeans can be setup for code completion most of the frameworks including Yii

(Part 3) (Part 5)

Want to be a web developer? (part 3)

Writing code

While you are practicing, use a simple text editor such as Geany.

sudo apt-get install geany

This has syntax highlighting for several languages including php and sql. I often use it to make a quick change to code because it loads quickly.

Once you become more advanced, it is better to use an IDE (Integrated Development Environment) for writing code. IDE's have loads of features - not only syntax highlighting but it will also check for errors as you are typing, completes code for you and most important of all - will allow you to debug your project.

The choice here is either Eclipse or Netbeans. Both are pretty equal in features, I've tried both and personally prefer Netbeans. To install Netbeans

EDIT 17-Jan-2011
Ubuntu 11.10 doesn't include Netbeans in the repository. So to install, download Netbeans with the PHP pack. Then open a terminal, change to the download directory cd ~/Downloads and run sudo bash [name of the netbeans download]

sudo apt-get install netbeans

Ubuntu 11.04 uses Netbeans 6.9, to install the latest version download the PHP bundle from the Netbeans website. The default skin looks a bit naff on Ubuntu, so I've added a launcher to the top panel (right click the top bar/panel, select "add to panel", "Application Launcher", Programming, Netbeans) then changed the properties -> command to

/usr/bin/netbeans --laf Nimbus

This will use the Nimbus skin which is much nicer.

Launch Netbeans and assuming your project has an index.html file - go to file -> new project -> php -> php project from existing sources -> then choose your project folder, eg: /var/www/projectname. Or if you are starting from scratch, then simply create a new project.

There are tutorials for Netbeans at http://netbeans.org/kb/trails/php.html and in particular http://netbeans.org/kb/docs/php/editorguide.html.

The commands I use often are Ctrl+F (Find), Ctrl+H (Find and Replace), Ctrl+click name of function will take you to the source of the function, Ctrl+hover will show the parameters, code complete press Enter after typing a function, right click a folder in the folders pane for finding keywords and of course the debugger play button.

Debugging

For debugging you will need to install xdebug and modify your php.ini file - http://wiki.netbeans.org/HowToConfigureXDebug

Edit : 19-Jan-2012
To install xdebug use
sudo pecl install xdebug

These are the xdebug settings from my php.ini

; This setting is on by default - turned off here because it can interfere with xdebug

report_zend_debug = 0

; Settings from http://wiki.netbeans.org/HowToConfigureXDebug

zend_extension = /usr/lib/php5/20090626+lfs/xdebug.so

xdebug.remote_enable=1

xdebug.remote_handler="dbgp"

xdebug.remote_mode="req"

xdebug.remote_host="localhost"

xdebug.remote_port=9000

; also http://xdebug.org/docs/remote

; Which IDE to use

xdebug.idekey="netbeans-xdebug"

; Automatically debug

xdebug.remote_autostart=1;

Now in Netbeans you can click the line number on the left of your code, this will create a debugging bookmark called a breakpoint. Then when you click the run with debug icon in the toolbar - it will open a new browser window with your application. When the breakpoint is reached, Netbeans will open and display the line of code. You can then use step over, step into, step out etc. and in the panel below the code you can have a look at the variables and their contents. For more details go to http://netbeans.org/kb/docs/php/debugging.html.

MySql

You should have already installed phpmyadmin, this can be accessed via http://localhost/phpmyadmin PhpMyAdmin is a great tool for managing your databases. You should become familiar with it. There are other tools, but phpmyadmin is so widely used, including on web hosts. There is a basic tutorial here http://www.devshed.com/c/a/PHP/Doing-More-With-phpMyAdmin-Part-1/

If you are already familiar with SQL, phpmyadmin also allows you to run SQL commands from a window.

One the most useful features of phpmyadmin is being able to do a data dump - this can act as a backup but also allows you to transfer the database to another server. Just choose the export or import option from the toolbar - select which tables you want to export, then the format - SQL if you want a full dump.

Another tool you might to install is the MySql Workbench. This is similar to phpmyadmin, but is a graphical desktop application. It can be useful for designing your database and testing queries.

(part 2) (part 4)

Want to be a web developer? (part 2)

Learning PHP and MySQL

Okay, I'm not going to teach you PHP this is up to you. To start learning PHP, go through the w3schools tutorial. For reference use the online manual at php.net. Each page has programming examples, for example the str_len() function. If you get stuck, use a forum such as http://www.devnetwork.net/. Also download the free php 5.3 study guide.

You will also need to know SQL and learn MySql - PHP can communicate with several databases but MySql is the most popular with PHP. For reference the online manual is at http://dev.mysql.com/doc/. which also has examples, for example length(). And if you get stuck, use the forums http://forums.mysql.com/.

Getting started

To run php scripts you'll basically need a web server. If you already have a web host (for example hostgator) then you can already run php scripts. Just create a text file on your computer, give it a .php extension and copy it to your website (either through the websites control panel or by using an ftp program such as filezilla)

For example the classic hello world program - copy this and save as helloworld.php, upload it to your web host and then call http://www.yourwebsite.com/helloworld.php

<html>
 <head>
  <title>PHP Test</title>
 </head>
 <body>
 <?php echo '<p>Hello World</p>'; ?>
 </body>
</html>

Installing a local web server

You can continue to use a remote web server, nothing wrong with that, but it does depend on having an internet connection and it is a live site so you need to be careful. For development, it would be better to install a local web server. Then its not so critical if you make a mistake, then transfer to your webhost when you are ready.

The main web server used for PHP development is Apache. This can be installed as part of a bundle called LAMP - which contains Linux, Apache, Mysql and PHP. There are bundles for Windows and Mac too, the most popular being XAMPP.

I got pretty fed up of developing in Windows for lots of reasons so I moved to Ubuntu (Linux) in 2008.   I would strongly recommend switching to Ubuntu for PHP development. The main reasons are its quicker, its really easy to install stuff and best of all, its all free. Rather than make a complete jump, I installed Ubuntu on my laptop as a dual boot alongside Windows. But I very rarely use Windows now so its just eating disk space now. So for the rest of the blog, I will be assuming you are using Ubuntu or any other flavour of Linux.

To install LAMP on Ubuntu, just follow these instructions. Basically go to the terminal Applications -> Accessories -> Terminal and enter

sudo apt-get install apache2 php5-mysql libapache2-mod-php5 mysql-server

Make sure you make a note of the root password for mysql.

Once installed you can go to http://localhost to test your webserver is working - if it says "It Works!" then erm... it works :)

The default folder for Apache web files is /var/www - so the "It works!" display is in the file /var/www/index.html. You can create another html file and drop it into /var/www

To test that php is working, try copying the helloworld.php file into /var/www then calling http://localhost/helloworld.php.

You will also need phpmyadmin which is a great application for managing MySql. To install it, again from the terminal enter

sudo apt-get install phpmyadmin

You will then be able to access it from http://localhost/phpmyadmin.

Files, Folders and Permissions

I like to keep all my code in /var/www/projectname type folders. But the problem is a normal user doesn't have access to the directory by default. So I run the following commands from the /var/www directory. Linux purists will say its a terrible thing to do, but it works for me.

sudo chgrp -R www-data projectnamefolder && sudo chown -R myusername projectnamefolder && sudo chmod -R 775 projectnamefolder

Replace projectnamefolder with your website folder name and myusername with your username.

You can also have a local directory eg /home/username/public_html and do lots of wizardry things with Apache virtual host, but I'd rather be developing than messing with Apache.

You might need to make changes to the php.ini configuration file. This is in /etc/php5/apache2/php.ini. You'll need to edit the as root - for example

gksudo gedit /etc/php5/apache2/php.ini

(part 1) (part 3)