May 4, 2015

Viewing the progress of a MySQL command line Import

khallmark @ 4:20 pm —

Instead of importing from a dump file like this:

mysql -uxxx -pxxx dbname < /sqlfile.sql

You can use a utility called “pv” (pipe viewer) to view the progress of an import. This will show you a nifty progress bar that will let you see the estimated progress through the file. This program works by reading each line of the file, updating the progress bar, and passing the line of SQL code to the MySQL console.

pv sqlfile.sql | mysql -uxxx -pxxxx dbname

You can install the command in an Ubuntu environment using the command:

sudo apt-get install pv

or you can install it on a Mac using MacPorts with the command:

sudo port install pv

March 30, 2011

Software design is about iteration, not perfection

Kevin Hallmark @ 1:07 pm —

Over the years, I’ve learned some fundamental things about software development. Writing perfect code with a perfect design is a fools errand. Perfection used to paralyze me. These are some principles I use to help me make coding decisions.

1. Don’t rewrite your application. Ever.

This explains it better than I ever could.

2. DRY – Don’t repeat yourself

“Every piece of knowledge must have a single, unambiguous, authoritative representation within a system.”

While it can apply to data sources as well, I am applying this to  code itself. Every single line of code you write is a potential bug. You should strive to write the least amount of code possible so you can depend on “known working” code.

If I need to get a record from the database, I can either write the select/execute call every single place I need it. This works for one off calls, but if I need to get that more than once or twice, a function is a better choice. I won’t have to worry about making a mistake each time I write the code. This is a contrived, but relevant, example.

3. YAGNI – You Ain’t Gonna Need It

“Always implement things when you actually need them, never when you just foresee that you need them.”

Simply put, if you don’t need it today or tomorrow (soon), don’t write for it. Maybe put a little underlying support code in, but don’t implement it until you have to. It takes time to add code, debug it and document it, time that might be better spent on more pressing features. If you add the feature too early, you need to support that bloat. If you need to refactor, you are refactoring more code. You have to make sure it doesn’t break.

After all that effort, you might even find that you don’t need the feature (ever), or you need something slightly different. The way you implemented it might be mutually exclusive with what you actually need. Adding it before it is needed might actually make you unable to implement a feature you need, or may lead to an incomplete implementation.

A good example is scaling. Why would you write software to scale your servers when you aren’t even using the power of one? Why write some awesome abstraction when it’s not going to be used by anything?

4. Three simple steps:

  1. Make it work
  2. Make it fast
  3. Make it pretty

I used to worry about the perfect application. I wanted everything to be the creme de la creme design. Unfortunately, I found that I never actually wrote the application. I got stuck in the design and never built out all the features I needed.

In a time crunch, you need the code to work. It doesn’t matter if it’s fast or pretty, if it doesn’t work it’s worthless. Usually, I can make it fast enough with iteration one, but as a system grows you’ll need to make things faster. As each pain point in speed becomes apparent, whack it out. Eventually, you’ll discover where you need to optimize. Finally, making it pretty by identifying code duplication and eliminating it where needed.

These principles have helped me write better software. I hope they help you too.

April 3, 2009

Reverse Engineering MySQL Queries in web apps

Kevin Hallmark @ 10:26 am —

So here I am, sitting at work. I’ve been tasked with reverse-engineering the login system for this website. Ugh. The code is spaghetti. Queries are a mish-mash. How do I figure this out?

What I discovered is that you can log literally every single mySQL query made to the server. This is really useful if you need to figure out crazy things like what queries run when new user is inserted.

Add the following line to your my.cnf file:

log = /var/log/mysql/mysql.log

This gets rid of the hassle involved with finding and tracing mysql queries directly.

February 12, 2009

Getting xDebug Working on Mac OS X with MacPorts

Kevin Hallmark @ 3:25 pm —

xdebug is a powerful PHP debugging tool. With xdebug you can get all sorts of debugging out, stack traces, and memory usage (as well as delta). It also gives you classic code-stepping tools and breakpoint. Finally, you can debug your PHP applications! Getting it working on Mac OS X has been a bit confusing, but this how to guide will give you all the tools you need!

Before we proceed, you have MacPorts installed right?

So how do we go about getting xdebug? First, install xdebug:

sudo port install php5-xdebug

Next enable xdebug. All this goes in your php.ini

; NOTE: This line adds the xdebug extension. The macports install will give you the path, 
; or may even add this automatically. Be smart, look for a similar line in your config first.
; (Edit: 06/22/2012)

; General config
;Dumps local variables on exception
;Dump server variables
;Dump global variables

; Tracing
xdebug.trace_output_dir= /opt/local/php_traces/

; Profiler

; Debugging. You might need to specify your host with some additional options

For viewing profile output:

You will use this program to open the cachegrind.out files in /opt/local/php_trace

For using the debugger:

Just put “XDEBUG_SESSION_START=session_name” in your query string. Session_name can be any alphanumeric string.

You can set a breakpoint with: xdebug_break()

May 20, 2008

HTTP Error Codes

Kevin Hallmark @ 2:08 pm —

My co-worker posted a great summary of common HTTP error codes that require a custom page handler. Many times, you’ll find sites that don’t actually deliver the correct error codes. This list helps you make sure to handle the different conditions properly.

Check it out here

May 15, 2008

Zend Framework – An Example Model

Kevin Hallmark @ 3:48 pm —

Here is a little advice for people who want some guidance on creating their a model layer:

NOTE: This is merely a simple example, for guidance only. YMMV. There are a million different ways to do this kind of thing, this is merely one I have found effective.

In my apps, I generally create a BaseModel class, from which all others inherit. In this class, I’ll define a basic API that I want all models to subscribe to. This is usually something like __get, __set, setDataFromArray, validate (verify data is correct), filter (filter input data), save and other common methods.

This setup lets you change the backend storage mechanism easily. If all your model objects use the same API, you can change the underlying storage code of all your models simultaneously and your app is none-the-wiser.

An example:

Say I begin by using an array to store my data. Suddenly, I realize I want to store my data inside a database. I can change all the methods in my model to set data to a database object. Later in the project, I decide that XML would be a better choice. I can change my methods to write to an XML data file, and once again I don’t have to change my application code.

You can implement custom behavior by overriding methods and using the ‘parent’ keyword to call the BaseModel method implementation.

In most apps these days, my BaseModel class directly extends Zend_Db_Table_Row. Extending Zend_Db_Table_Row allows my model objects, with all their custom functionality, to be returned directly by Zend_Db_Table function calls. Creating lists is much faster when you don’t have to iterate the result set a second time to create model objects from them.

I usually won’t override the methods I want to use from Zend_Db_Table_Row (__get, __set, setDataFromArray). If I needed to change my storage engine I would implement these methods in my BaseModel to access the new data storage mechanism.

In each of my BaseModel subclasses, I’ll usually add some static getter methods.

For example:

Class User extends BaseModel {
Public static function getUser($user_id = null) {
//return a setup model object
$table = new users();//where users is your table class
If($user_id === null) {
$return = $table->createRow();
} else {
//get row from db

Return $return;

This function would return the model object for ‘$user_id’, or an empty object if null. You can call this method with

Each model then has a single point of origin. If I decide to change my implementation, I can be assured that “User::getUser” always will give me a fully qualified User object.

You can also create similar functions the will return a list of rows.

Hope this small bit of advice helps get your started.