The other side of the moon

I've seen a few questions about doing pagination with AJAX, and I don't like the thought of it. It smells of accessibility problems. I've addressed this issue before in my own toy pages (since I don't actually write production web code), so thought I'd share it.

This is a sort of continuation of my post on progressive enhancement.

Problem definition

Given a long list of data, display it to the user in pages to avoid scrolling. Typically you'd have a bunch of navigation links at the end with First, Last, Next, Previous links, or links to specific pages.

Six steps from vanilla HTML pages to AJAX pages

It's important to note that the problem definition does not mention AJAX, but people always like to make their apps buzzword compliant. So, forget about AJAX for the moment and concentrate on solving the problem — retrieve a database resultset in limited sized pages. Once we've done that, it's five more steps to accessible AJAXified pages:

1. Build the page as you would for html only pagination
2. When your pagination links load, attach onclick handlers to them.
3. The onclick handler makes an asyncRequest to this.href + '&js=1' (or something similar)
4. Modify your backend code to check for js=1 in the query string.
   
   If not found, then send the entire page with header and footer as before
   If found, then send one of the following:
   • The html for the paged data
   • XML for the paged data
   • A JSON object representing the paged data
5. In your callback for the asyncRequest, do one of the following:
   • Put the html into the innerHTML of your page's container object
   • Parse the XML and translate it to the DOM objects for your paged data
   • eval() the JSON and redraw the DOM for the paged data
6. Rewrite the hrefs in your paging links to point to new pages.

You now have pagination that works with and without javascript enabled.

The Code

Let's look at some of the code. I'll use the yui utilities for connection and event management since I've been playing with that.

For simplicity, I'll assume that we're representing our data as a <LI>st. A table is similar, except that you need to redraw the entire table since it's read-only in IE.

Step 1: Build HTML (PHP with MySQL)

<div id="page">
<ul>
<?php

// Fetch all results and print them
while($o = mysql_fetch_array($result, MYSQL_ASSOC))
{
?>
<li><?php print $o['name'] ?></li>
<?php
}
?>
</ul>
<?php

// Verify next/last page links
$prev_page = ($pg<=0?0:$pg-1);
$next_page = ($pg>=$num_pages?$num_pages:$pg+1);

// Display navigation links, disable (via css) links that cannot be selected
?>
<p class="navbar">
<a id="first-link" href="foo.php?pg=0"
class="<?php if($pg == 0) echo 'disabled' ?>">First</a>
<a id="prev-link" href="foo.php?pg=<?php print $prev_page ?>"
class="<?php if($pg == 0) echo 'disabled' ?>">Prev</a>
<a id="last-link" href="foo.php?pg=<?php print $num_pages ?>"
class="<?php if($pg == $num_pages) echo 'disabled' ?>">Last</a>
<a id="next-link" href="foo.php?pg=<?php print $next_page ?>"
class="<?php if($pg == $num_pages) echo 'disabled' ?>">Next</a>
</p>
</div>

Step 2: Attach onclick handlers

var nav_links = ['first-link', 'prev-link', 'next-link', 'last-link'];

YAHOO.util.Event.addListener(nav_links, 'click', navigationHandler);

Step 3: Make async call:

var callback =
{
success: gotResponse,
failure: failedResponse
}

var navigationHandler = function(e)
{
var url = this.href + '&js=1';

YAHOO.util.Connect.asyncRequest('GET', url, callback, null);

YAHOO.util.Event.preventDefault(e);
return false;
}

Step 4: Modify back end to check for js=1:

<?php
$js = $_GET['js'];

if($js)
{
header('Content-type: text/json');
}
else
{
?>
<div id="page">
<ul>
<?php
}

$json = array('n'=>$num_pages, 'p'=>$pg, 'l' => array());
// Fetch all results and print them
while($o = mysql_fetch_array($result, MYSQL_ASSOC))
{
if($js)
{
$json['l'][] = $o['name'];
}
else
{
?>
<li><?php print $o['name'] ?></li>
<?php
}
}

if($js)
{
print json_encode($json);

// nothing more to output, so quit
exit();
}
?>
</ul>

I've hilighted the code that changed, it's just a bunch of if conditions. Yeah, it's ugly, but cleaning it up is not the purpose of this article.

Step 5: Make your asyncRequest handler:

var gotResponse = function(o)
{
var json = eval("(" + o.responseText + ")") ;

var list = json['l'];
var num_pages = json['n'];
var page = json['p'];

var prev_page = (page<=0?0:page-1);
var next_page = (page>=num_pages?num_pages:page+1);

var lis="";
for(var i=0; i<list.length; i++)
{
lis += "<li>" + list[i] + "</li>\n";
}

var ul = document.getElementById('page').getElementsByTagName('ul')[0];
ul.innerHTML = lis;

Step 6: Rewrite paging urls:

var fl = document.getElementById('first-link');
var pl = document.getElementById('prev-link');
var nl = document.getElementById('next-link');
var ll = document.getElementById('last-link');

var url = fl.href.substr(0, fl.href.indexOf('pg=')+3);

pl.href = url + prev_page;
nl.href = url + next_page;
ll.href = url + num_pages;

fl.className = pl.className = (page<=0?'disabled':'');
nl.className = ll.className = (page>=num_pages?'disabled':'');

}

Steps 5 and 6 are the same function of course, so don't split them up.

A brief explanation

Well, there you have it. If javascript is disabled, the default <A> behaviour is to make a GET request to foo.php with default values for pg. On every page call, the back end changes the value of pg in the Next and Previous links, and possibly in the Last link if records in the database have changed.

If javascript is enabled, we prevent the default href from being called with our return false;, and instead make the same call using asyncRequest, but with an additional query parameter saying that we want a javascript (json) object back.

The back end php script still hits the database as usual, and gets back a result set, which it now builds into a PHP hash, and then converts to a JSON object. The JSON object is sent back to the client where it is converted into HTML to push into the <UL>.

The page and num_pages variables allow us to rewrite the hrefs so that they point to up to date paging links, that you can, in fact, bookmark.

Improvements

To make the code cleaner, you may want to build your PHP hash at the start, and then based on the value of $js, either convert it to HTML or to JSON. This of course has the disadvantage of having to iterate through the array twice. If you're just looking at 20 records, I'd say it was worth it, and a better approach if you start off that way.

This is quite a simple implementation. You could get really creative with javascript, showing funky page transitions that keep the user busy while your asyncRequest returns.


Update: json_encode is available from the PHP-JSON extension available under the LGPL.

Update 2: The cleaner way to write the PHP code that I mentioned in Improvements above is something like this:

<?php
$js = $_GET['js'];

if($js)
header('Content-type: text/json');

$list = array();
while($o = mysql_fetch_array($result, MYSQL_ASSOC))
$list[] = $o['name'];

if($js)
{
$json = array('n'=>$num_pages, 'p'=>$pg, 'l' => $list);
print json_encode($json);

// nothing more to output, so quit
exit();
}
else
{
?>
<div id="page">
<ul>
<?php
foreach($list as $name)
{
?>
<li><?php print $name ?></li>
<?php
}
?>
</ul>
<?php

// Verify next/last page links
$prev_page = ($pg<=0?0:$pg-1);
$next_page = ($pg>=$num_pages?$num_pages:$pg+1);

// Display navigation links, disable (via css) links that cannot be selected
?>
<p class="navbar">
<a id="first-link" href="foo.php?pg=0"
   class="<?php if($pg == 0) echo 'disabled' ?>">First</a>
<a id="prev-link" href="foo.php?pg=<?php print $prev_page ?>"
   class="<?php if($pg == 0) echo 'disabled' ?>">Prev</a>
<a id="last-link" href="foo.php?pg=<?php print $num_pages ?>"
   class="<?php if($pg == $num_pages) echo 'disabled' ?>">Last</a>
<a id="next-link" href="foo.php?pg=<?php print $next_page ?>"
   class="<?php if($pg == $num_pages) echo 'disabled' ?>">Next</a>
</p>
</div>
<?php
}
?>

Update: I've put up a working example on sourceforge.

Have you ever visited a website and wondered, "Man, I wish I could drag that stuff out of the way"? Well, it really shouldn't be that hard. I've been doing this for a while on my own, and thought I'd share it.

To make it easy to use, I'd suggest you make a bookmarklet out of it that can be clicked on when you get to a page.

To start, download Yahoo's yui utilities:
Zip file. Unzip it into a convenient directory on your computer.
For this particular hack, I'd suggest putting the following files into the same directory:

• YAHOO.js - from any of the subdirectories
• event/build/event.js
• dom/build/dom.js
• dragdrop/build/dragdrop.js

Next, create your own script file, that looks like this:

function add_drag_drop(element)
{
   if(!element)
      element="div";
   var divs = document.getElementsByTagName(element);
   for(var i=0; i<divs.length; i++)
   {
      var id = "div-" + i;
      if(divs[i].id)
         id=divs[i].id;
      else
         divs[i].id = id;

      var dd = new YAHOO.util.DD(id);
   }

   return "Added drag-drop to " + i + " " + element + "s";
}

I'll call it adddragdrop.js. Use a javascript prompt at the top instead if you'd like to be prompted for the element's tag name.

The only thing this function does is iterate through all elements of the specified type, adding drag drop to them. If elements don't have an id, an id is added. Notice that it takes more lines of code to add ids than it does to add drag and drop. That's why I love this library.

You now need to add these five files to the page you're viewing. You can do it via the browser URL bar like this:

javascript:void(_s=document.createElement("script"), _s.src="YAHOO.js", document.body.appendChild(_s);
javascript:void(_s=document.createElement("script"), _s.src="event.js", document.body.appendChild(_s);
javascript:void(_s=document.createElement("script"), _s.src="dom.js", document.body.appendChild(_s);
javascript:void(_s=document.createElement("script"), _s.src="dragdrop.js", document.body.appendChild(_s);
javascript:void(_s=document.createElement("script"), _s.src="adddragdrop.js", document.body.appendChild(_s);

Of course, use the correct path for the files in there.

Finally, call the function from your url bar like this:

javascript:alert(add_drag_drop('div'))

All <div>s on the page should now be draggable.

Ok, so it's a pain to do this over and over, so turn it into a bookmarklet or greasemonkey script. This is what
my bookmarklet looks like:

<a href='javascript:void(
_s=document.createElement("script"), _s.src="http://localhost/philip/yui/YAHOO.js", document.body.appendChild(_s),
_s=document.createElement("script"), _s.src="http://localhost/philip/yui/event.js", document.body.appendChild(_s),
_s=document.createElement("script"), _s.src="http://localhost/philip/yui/dom.js", document.body.appendChild(_s),
_s=document.createElement("script"), _s.src="http://localhost/philip/yui/dragdrop.js", document.body.appendChild(_s),
_s=document.createElement("script"), _s.src="http://localhost/philip/yui/adddragdrop.js",
_s.onload=_s.onreadystatechange = function(){alert(add_drag_drop("div"));},
document.body.appendChild(_s))'>Add drag drop to page</a>

(Whitespace added for readability)

Drag that link to your bookmarks toolbar to make it easily accessible, and of course, change the links to point to your own versions of the files.

Ok, we're ready to go now. Visit any page, click on the bookmarklet, and all divs become draggable. Is that cool or what?

If it doesn't work, let me know and I'll try and figure it out.

Anyway, now that you've got the code, play with it and show me what you can do.

After my post about converting the geo microformat to a google map, this one adds support for Yahoo! Maps. What I found cool was how alike the two APIs were.

I changed my map creation code to this:

             if(mapAPI == 'G')
             {
                point = new GPoint(lon,lat);
                map = new GMap(adr[i]);
                map.addControl(new GSmallMapControl());
                map.addControl(new GMapTypeControl());
                map.centerAndZoom(point, 0);
             }
             else
             {
                point = new YGeoPoint(lat,lon);
                map = new  YMap(adr[i]);
                map.addPanControl();
                map.addZoomShort();
                map.drawZoomAndCenter(point, 1);
             }
             addMarkerToMap(map, point, html);

And the marker addition code to this:

       if(mapAPI == 'G')
       {
          var marker = new GMarker(point);
          map.addOverlay(marker);
          if(html)
          {
             marker.openInfoWindowHtml(html)
             GEvent.addListener(marker, 'click', function() {marker.openInfoWindowHtml(html);});
          }
       }
       else
       {
          var marker = new YMarker(point);
          map.addOverlay(marker);
          if(html)
          {
             html = "<div class='map-marker'>" + html + "</div>";
             marker.openSmartWindow(html);
             YEvent.Capture(marker, EventsList.MouseClick, function() {marker.openSmartWindow(html);});
          }
       }

Which creates as close an experience as possible in both maps.

The map provider should be selected at random, but if you'd like to force a particular map, add #G or #Y to the url. They can be found on the california reviews page.

Let me know what you think.

I'd been toying with the idea of adding geo data to my restaurant reviews. I thought it would be nice to have a map pointer right below the instructions for getting there. I started looking around for well estabilished methods to markup geo data in a blog post.

I came upon an article in linux journal that spoke about geotagging and geocoding for websites. It talks about ICBM and meta tags, which are great except for one thing. It goes into the page header, so can't be different for different sections of the page. It also talks about embedding the information in comments - which means that users and javascript can't easily read it, and about RDF/RSS feeds - which would be useful for my blog feed, but not for my blog itself.

I decided to try my own minimalistic markup, and came up with this:

   <address class="gmap" lat="yy" lon="xx" zoom="z">Some text</address>

Of course, I went through a couple of iterations to settle on this, and it was based largely on what the google maps api accepts.

A side note before I go on. The reason I chose google maps was because they provided aerial photos of a few major Indian cities, which is where most of my reviews are based. This turned out to be of no use though, because the aerial photos are not provided via the API. One has to visit Google Local to see them.

Ok, so this gave me the ability to easily add geo information to a post - just a single line. If I wanted to be really cool, I'd need to translate that to a map, so I started studying the google maps API, and after several iterations, came up with this:

   //! Add a marker with a callout containing the specified html
   function addMarkerToMap(map, point, html)
   {
      var marker = new GMarker(point);
      map.addOverlay(marker);
      if(html)
      {
         marker.openInfoWindowHtml(html)
         GEvent.addListener(marker, 'click', function() {marker.openInfoWindowHtml(html);});
      }
   }

   window.onload=function()
   {
      var adr = document.getElementsByTagName("address");
      for(var i=0; i<adr.length; i++)
      {
         if(adr[i].className == 'gmap')
         {
            // Grab HTML to put into callout
            var html=adr[i].innerHTML;
            var lat, lon, zoom;
            lat=1*adr[i].getAttribute('lat');
            lon=1*adr[i].getAttribute('lon');
            zoom=parseInt(adr[i].getAttribute('zoom'));
            adr[i].innerHTML = "";

            // Build map and center on lat/lon
            var map = new GMap(adr[i]);
            map.addControl(new GSmallMapControl());
            map.addControl(new GMapTypeControl());
            var point = new GPoint(lon,lat);
            map.centerAndZoom(point, zoom);
            addMarkerToMap(map, point, html);
         }
      }
   }

What it does is quite simple. After page load, it iterates through all the divs in the page, looking for divs with class 'gmap'. When it finds such a div, it looks at the lat, lon and zoom attributes (which are not standard HTML btw) of the div and uses that to draw the map.

I soon realised that I was only using zoom level 0, so dropped that attribute and hard coded it in.

I showed it to Nate a little later, and he mentioned that there was a microformat for geocoding. Had a look at it, and while it was slightly more verbose than my format, it achieved a little more, and wouldn't be much harder to parse.

Changed the div to this:

<address class="geo">
Some text<br>
<abbr class="latitude" title="37.801324">N 37° 48.079</abbr> <br>
<abbr class="longitude" title="-122.424903">W 122° 25.494</abbr>
</address>

This shows up neatly, and I could change my javascript to accept both classes, 'gmap' as well as 'geo', and change the parsing to this:

             if(adr[i].className == 'geo')
             {
                var ab = adr[i].getElementsByTagName('abbr');
                for(var j=0; j<ab.length; j++)
                {
                   if(ab[j].className == 'latitude')
                      lat = 1*ab[j].title;
                   else if(ab[j].className == 'longitude')
                      lon = 1*ab[j].title;
                }
             }
             else
             {
                lat=1*adr[i].getAttribute('lat');
                lon=1*adr[i].getAttribute('lon');
             }

The rest of the code remains the same.

It isn't too hard to replace Google maps with Yahoo! maps in this implementation. The parsing of the microformat is the heart of it all, after that it's just a matter of using your API of choice.

You can see both formats in action on my reviews of US restaurants. Note that one of the restaurants uses the old format that I've described above, and the other two use the geo microformat. I'll add more in time, and a few from the UK as well.

Update: Calvin Yu has written a Yahoo! Maps implementation of geo.

Update: Switched to use the <address> tag instead of a <div>. Just seems more semantic.

Update: I have a Yahoo! Maps version as well.

I'm not a web developer, and I'm not always aware of the quirks that experienced web developers are aware of. Like this one for example.

Populating list boxes through javascript

The standard way to populate a list box using javascript (or at least the way that I learnt, is to create Option objects, and add them to the list, like this:

var o = new Option('text', 'value');
var s = document.getElementById("my-selectbox");
s.options[s.options.length] = o;

Pretty simple, and when you're doing AJAX, it's just easy to return a javascript array, or maybe even the code that needs to be executed, and just eval that in the calling script.

HTML entities in a javascript populated list box

I'd been doing this for a while, when I got a bug report from someone saying that their single quotes were showing up as &#039;. Checked up, and found out that while populating a select box using javascript, you've got to pass it the raw text — barring the primary html entities that is; <, > and & (and possibly ")). So, I set about unescaping all text before handing it to the array that was to populate the list box.

Not too much trouble. I was doing it in PHP, and this is the code I had to put in:

?>
var sel = document.getElementById("my-select-box");
sel.options.length = 0;   // delete all elements from sel
<?php
while($o = mysql_fetch_array($result, MYSQL_ASSOC))
{
$on = preg_replace("/(['\\\\])/", '\\\\$1',
html_entity_decode($o['name'], ENT_QUOTES));
?>
sel.options[sel.options.length] = new Option('<?php print $on ?>', '<?php print $o['id'] ?>'));
<?php
}
?>

Now don't get confused with the mixture of code in there. The stuff inside the <?php ... ?> blocks are php code, and the stuff outside is javascript.

What that code basically does is, fetch a bunch of records from the database. The names are stored htmlescaped in the database (because that's how we were dealing with them initially). After fetching, we need to unescape them because we're populating a select box using javascript, but because we're enclosing our javascript strings in single quotes, we need the funny looking extra preg_replace to escape single quotes in the string.

This code worked perfectly (although I should really be puttin unescaped data into the database anyway), until, that is, someone tried putting japanese characters into the name field.

What happened was a series of things. To start with, the japanese characters were showing up url encoded everywhere. Now url encoding is different from html encoding. Url encoding is where a character is converted into its hex code and prefixed by a % sign. Japanese characters in utf-8 are two bytes long, and hence take up 4 hex digits (hexits?). The url encoded representation of them was something like %u3c4A, and I had a whole bunch of strings that were filled with that.

URL encoding to HTML entities

A simple regex converted the url encoded data to an html entity:

$text = preg_replace("/%u([a-z0-9A-Z]+)/", "&#$1;", $text);

Note that there may be major flaws in this code. For starters, I pick the longest string of text after the %u that matches a hex digit. This may not always be correct. I prolly need to test that it's a valid utf8 character first, but I'll leave that for later.

Ok, so the url encoded text is converted to an html entity, and I can do this before it goes to the db, and everyone is happy. Not really. Remember that the list box populated via javascript doesn't like html entities either. It needs plain text.

My code above should have fixed that, except that it wasn't prepared for utf-8 strings. The default character set used by php 4's entity functions is iso-8859-1. Looked up the docs (I'm not a PHP programmer), and found that the third parameter to html_entity_decode was the charset, so, I changed my code to this:

html_entity_decode($o['name'], ENT_QUOTES, 'UTF-8')

But, instead of working, I got a whole bunch of errors telling me that PHP 4.3 didn't support UTF-8 conversions for MBCS (Multi byte character strings). This was a combined WTF and Gaaah moment. I'd promised I18N support, and now either PHP or javascript were playing spoilt sport.

The exact error message I received (for the search engines to index) is this:

Warning: cannot yet handle MBCS in html_entity_decode()!

Tried several things. Found out that javascript's unescape function could in fact handle the string, and I tried wrapping my text with a call to unescape(), but it didn't work.

The Eureka moment

The Eureka moment was when I realised that it was only list boxes populated via javascript that had this problem. If I had a list box completely populated with html, then there was no problem.

I'd been using javascript arrays all along because it's far cheaper to send them across a network and process on the client side than it is to send complete rendered html. Still, it seemed like I had little choice.

I decided to take advantage of the innerHTML property of all container elements, and add the options to the select element as an HTML block. This didn't work, and made the select box have one single long option.

My next thought was to tackle the select element's parent and dump the entire list box as its innerHTML. There were other problems with this approach though. Some select boxes were in a container element along with other elements, and I didn't want to have to draw all of them. One of the select boxes had javascript event handlers attached to it from the window's onload handler, and deleting and recreating the select box would effectively wipe out those handlers. Still, this was the best approach.

So, I decided to first surround all my selects by a div that did nothing else, and build the entire select box and options list as an html block, and stick that into the innerHTML property of the div. Finally, I had to reattach any event handlers that had existed before. This was simply a matter of saving the event handler before wiping out the select box, and adding it again later.

Here's my final code:

var sel = document.getElementById("my-select-box");
var fn = sel.ondblclick;
var sp = sel.parentNode;

// sel is useless after this point because we basically wipe it from memory

// get all html up to the start of the first option
// just in case we have more than just this select box in the parent
var shtml = sp.innerHTML.substr(0, sp.innerHTML.indexOf("<option"));

// get all html from the closing select onwards
// for the same reason as above
var ehtml = sp.innerHTML.substr(sp.innerHTML.indexOf("</select>"));
<?php
while($o = mysql_fetch_array($result, MYSQL_ASSOC))
{
$on = preg_replace('/%u([0-9A-F]+)/i', '&#x$1;', 
preg_replace("/(['\\\\])/", '\\\\$1', $o['name']));
?>
shtml += '\n<option value="<?php print $o['id'] ?>"><?php print $on ?></option>';
<?php
}
?>

sp.innerHTML = shtml + ehtml;

// need to reget the element because its location in memory has changed
// reset the ondblclick event handler
document.getElementById("my-select-box").ondblclick = fn;

Finally, code that works, and shows all characters correctly, and lets me go home to have lunch.

I'd also tried stuff with the DOM, like using createElement, createTextNode and appendChild, but it had the same problems as before - javascript want unescaped strings.

Now, maybe this is just a problem with firefox, I'm not sure as I hadn't tested in IE, but anyway, I found it and fixed it for me, maybe it will help someone else.

Ciao.