Friday, January 29, 2010

Ruby and Sinatra Just Works

To be filed under the pleasant surprise for a late Friday technology jaunt, I decided to see how difficult it would be to install and get the basic "Hello World" web application running using Ruby with the Sinatra framework. The pleasant surprise was that it took less than 5 minutes! I was shocked!

Here are the steps I took to get it running on a Ubuntu 9.04 Virtual Machine:

1. Execute the following commands from the command prompt:
$ sudo apt-get install ruby
$ sudo apt-get install ruby-gems
$ sudo gem install sinatra
$ vi myapp.rb (see instructions here)
$ ruby myapp.rb

2. Point your web browser at http://localhost:4567/ and it just works! Amazing. Of course that is probably not the best way to install the latest version of ruby or ruby gems, but it works. I am going to quit and go home now. Have a great weekend.

Finger Pointing Internet Style

This morning was spent diagnosing problems with a web application. It would be one thing if it were our web application, but It wasn't. It was a 3rd party web application that one of my clients links to. And it didn't work.


Background
This problem has actually been going on for a couple of months. The organization responsible for this web application would provide my client with a link to use to access a landing page specifically designed for the client. My client would then test the link, and send back a message telling them that it was broken. Days or weeks would go by, and the process would repeat itself.


Finally it was getting down to the wire. My client has their own schedule, and this 3rd party application was becoming a problem. So after a flurry of emails, the 3rd party informed my client that everything was ready to go. And initially my client thought it was. Unfortunately it was not.


Problem 1: The Obvious
The first time a person accessed the web application, there is a server error, along with all the cryptic crap that gets spewed when a web application decides to fail, and has no graceful error processing/logging system to avoid exposing its gory details to the user.


This is a fairly common problem. One cause of this type of problem is when a link to a landing page is provided, but it requires specific information to have been previously set up in the user session. Normally users follow fairly predictable paths when accessing a web application. Early in that process, often certain session attributes are initialized for the user, for example an empty shopping cart, and the rest of the site just assumes that this already exists. But when a custom landing page is developed, and that is the first page that the user hits when coming to the site, those session attributes may not exist.


Interestingly, this was not the problem in this particular case. Sessions are often tracked with cookies, and after deleting the cookies from the browser, we discovered that the error screen was not generated. It still may be session related, but seems less likely.


The point of all this is that people who provide web applications should TEST what happens when a user comes into the site fresh. This should not be left as an interesting discovery for the users. 


Problem 2: The Obscure
The second problem occurred when my client tried to access the web application, using the link provided, from different document types. The link was placed in a word document, a PDF file, a web based email system, an Outlook based email system, and in another web page. In all cases, except the PDF file and Outlook, the link did not take the user to the correct page. The interesting thing was that doing a copy and paste of the link into the browser directly worked!


After some debugging (thanks to Firefox/Firebug), we determined that the only difference between the sequence of 4 pages that were redirected in the initial landing page was in the http header "referrer" field. Our tentative conclusion is that the web application is somehow sensitive to the referrer field, and is broken if that is set. One user of the referrer field is to control third party access to server resources, i.e. verify that the referrer is authorized to access the resource. This is just the opposite, here the referrer must be non-exist ant for the application to work.


Lessons Learned
The lesson for web application developers here? Test access to published links using a variety of different clients. And not just browsers, also client applications such as Microsoft Word, PDF files, email clients etc.


I hate to leave this discussion without definitive answers to the problems, but the truth is that since it is not our web application, there is only so much we can do. We isolated the fault domain (the 3rd party application), and provided them with sufficient documentation and test scenarios that they should be able to reproduce the problem. Whether they do so, fix the problem, and test the result remains to be seen.

Thursday, January 28, 2010

Creating Master Calendar from Separate Google Apps Calendars Using Zend Framework/PHP

Introduction
The Google Apps/Calendar API enables programs to access calendar and event information. It seemed like a good way to have a program take the information from several calendars and merge them into a single master calendar. At first blush this seemed like an easy thing to do, but there are some nuances in the Google API that caused some problems.

The technology I used for this educational project was Zend Framework/PHP, in particular the ZF Google Data APIs. This is the basis for the PHP documentation on the Google API developers site, and seemed as good a place as any to get started.

I am writing this as a task that will be executed as a cron job automatically, not as a web application. The key difference between the two is how authentication is performed. For a cron job, authentication uses ClientLogin, instead of AuthSub.

The Setup
The goal of this example is to access all of the calendars available in a single user's Google Apps account. These can be calendars created by that user, calendars shared with that user by other users, either public or in the Google Apps domain, or calendars that are generated from information from other calendar sources, such as iCal.

I am not going to go into detail to describe how to set up the Zend Framework environment for use by normal command line processing applications, as that is dealt with in detail elsewhere. Having said that, the following is the basic structure required to set it up on my system:


echo "\n\nStarting calendar program...\n";


// Add the Zend Framework library to the include path 
// so that we can access the ZF classes
set_include_path('/path/to/local/zend/framwork/library/' .
    PATH_SEPARATOR .
    get_include_path()
);


// Start Zend Framework autoloader
require_once 'Zend/Loader/Autoloader.php';
Zend_Loader_Autoloader::getInstance();

Authentication
The next step is to authenticate the user in the Google Apps domain:

// Substitute your own user account here
$user = "user@domain.com";
// Substitute your own password here
$pass = "mypassword";
$serviceName = Zend_Gdata_Calendar::AUTH_SERVICE_NAME;

// Create an authenticated HTTP client
try {
    $client = Zend_Gdata_ClientLogin::getHttpClient($user, $pass, $serviceName);
} catch (Zend_Gdata_App_AuthException $e) {
    echo 'Error: ' . $e->getMessage();
    if ($e->getResponse() != null) {
        echo 'Body: ' . $e->getResponse()->getBody();
    }
}


In one example I saw, there was an additional catch section before the one here in case Google required a captcha response. Presumably this would occur if Google detected suspicious behavior during login. Unfortunately if that happens, I don't know of any way to automatically respond. After all the whole point of captcha is to require require human intervention. So I left it out, and it seemed to work fine. Maybe I am just lucky.

Load Calendar List
The next step is to load the calendar list:


// Create an instance of the Calendar service
$gdataCal = new Zend_Gdata_Calendar($client);


try {
    $calFeed = $gdataCal->getCalendarListFeed();
} catch (Zend_Gdata_App_Exception $e) {
    echo "Error: " . $e->getMessage();
}


Load Events for Each Calendar
The final step is to load each calendar's events:


// Iterate through calendars in calendar feed
foreach ($calFeed as $calendar) {
    // Print calendar title
    echo $calendar->title->text . "\n";
    $links = $calendar->getLink();
    foreach ($links as $link) {
        // This is the mystery part
        if ($link->getRel() == 'alternate') {
            try {
                $eventFeed = $gdataCal->getCalendarEventFeed($link->getHref());
            } catch (Zend_Gdata_App_Exception $e) {
                echo "Error: " . $e->getMessage();
            }
            foreach ($eventFeed as $event) {
                echo "\t".$event->title . "\n";
            }
        }
    }
    echo "\n";
}


The main stumbling block here came in trying to specify which calendar I wanted to load the events for. The default is to load the default user calendar. I was able to determine that the calendar entry includes an array of links, one of which is tagged as "alternate". This link happens to coincide with the link that you can view in Google Apps under calendar settings. I am not sure this is the recommended way, but it seems to work.

The other issue I ran into is that there are two different ways to generate an event feed. In one you specify the URL, as is done here. I the other you specify a query, based on a number of query attributes. It would have been nice to use the query approach, because ideally the program would be able to select events based on some criteria, such as start time. Unfortunately because of the way that calendar links are generated, I was not able to use the information from the calendar feed to generate a query that would work. Maybe some helpful reader can shed some light on this.

At this point the program has everything it needs to put together a master calendar. I leave the rest as an exercise for the user!

Wednesday, January 27, 2010

Upload Any File Type to Google Docs

Until last week, my recommendation for organizations using Google Apps for collaborative document sharing was to use Google Docs for online editing, augmented by DropBox for general file storage and offline editing. With the new capability introduced by Google last week to upload and store any file type in the Google Docs storage space, (learn more) that may change.

While the Google Docs user interface continues to improve, for those used to Microsoft Office and other desktop office apps, it still leaves something to be desired. Network latency and performance issues aggravate the problem. So while documents in the cloud have a certain appeal, the reality is that for day to day work, documents have remained in local storage. For that reason, services such as DropBox have been used to provide local document editing while sharing those documents between different users and systems.

As of last week, Google now allows the Docs workspace to be used for general purpose storage of files regardless of their types. And for files of the standard Google Docs types (eg. documents, presentations and spreadsheets), the user can choose whether to upload in the desktop native format, or convert to the Google Docs online format.

The remaining difference between this capability and a service such as DropBox is that Google Docs are not automatically synchronized with  a users desktop. If a person wants to edit a document, they first must download it, edit it, and then upload it back to the Google Docs system. Here is where other 3rd party services come in. One company in particular, Memeo Inc. has introduced Memeo Connect for Google Apps, which provides a desktop folder that is automatically synchronized with Google Docs. Priced at $9/user per year, this is an affordable solution for organizations that want to have the ability to store online and offline editable documents in the Google Docs service.