TL;DR: if you have a web (or mobile) app and you want to make it easy for your users to import/sync files from their Dropbox, Box, Skydrive, Evernote, Facebook and more than half a dozen other sources, you can now use a single “proxy” service, InkFilePicker, and easily pick, import and then update/export files from and to these services into your web app with just a few lines of JavaScript. Unfortunately, the InkFilePicker model is well suited to importing/copying files and not as well suited to linking to them, which is essential if you want to make it easy for users to click through and see/edit files in Google Drive, but by combining InkFilePicker with Google’s Picker product you can get the best of both worlds. This post shows you how I did it in a PoC sense with AffinityLive.

Background

One of the features of AffinityLive that we’ve been meaning to upgrade for a while is our integration with cloud storage vendors. Dropbox, Box, Google Drive, Skydrive, Evernote.. the list of popular cloud storage platforms where our clients are storing their proposals, project files, contracts and more continue to be more and more popular.. and populous.

One of the challenges we faced when looking at the best way to integrate was the inverted mental model most of these platforms use. While in AffinityLive, a project is a shared collaborative space for internal (and in some circumstances, external) users to use, the model around almost all cloud storage services is personal and private. While folders in Dropbox can be shared (and the view of them is consistent), in Google Drive the way I arrange my files in folders is very different to the way someone else may arrange those exact same files in their own folder view. While our proof of concept work included creating a named/defined project folder in Google Drive, for example, the fact we had to create it in a single user’s account and then share it rather than it being a common, organizationally owned folder with access permissions struck us as a real square peg and round hole issue – and synchronization was always going to be painful and scary.

The other practical challenge is that there are so many cloud storage providers and the slight differences in use cases and focus areas means that users are unlikely to use just one of them. Even if a company decides they’re going to focus on Box, there’s a good chance some users are going to use Evernote’s awesome mobile note taking features to store meeting notes and ideas there… declaring a single, two way sync platform and forcing everyone to use it seemed like something from 1990’s enterprise tech and not realistic or desirable in this age of consumerization of the enterprise.

And finally, even if we could come up with a magical many to many sync which handled access controls and conflicts across half a dozen services, there’s also the commercial consideration – a solid two-way integration would realistically take an engineer 3-4 weeks to do properly, and if you multiple out that time by the (growing) number of popular services, if we started today we might not be finished for a (wo)man-year of engineering time.

But, we still wanted to give our users a convenient way to bring their files from their cloud storage providers into AffinityLive. What is to be done? The answer, it turns out, is InkFilePicker (with a Google Drive specific twist).

Our Model/Pattern

This post won’t go into the specifics of the InkFilePicker options/docs – you can see them yourself on their great developer site – but it is worth sharing our model/pattern of file storage and sharing for context.

AffinityLive uses a folder and file model to make it easy for our users to share files against clients, sales, projects, issues and retainers. Until now, files in AffinityLive have come in from four places:

• Files uploaded by users through our web attachments interface. This has been pretty cludgy and uses the traditional “browse” and upload model.
• Files attached by users through our web activities interface. Similar to composing an email, our inbox and activities screens allow users to attach files via an AJAX model when they’re making a note, writing an email or logging time.
• Email attachments captured in automatic email tracking. This way, if a client sends through a reply with a marked-up attachment, we’ll automatically store that attachment against the client, project or whatever their reply related to.
• Files uploade via our Forms API. In cases like the Angel Group (who have an job application page on their website) public users can upload files through a web form and have them go against a client, project, etc.

In all of these cases the files are stored in folders related to the project, issue, client etc, with the ability to create sub-folders to keep information organized.

When it came to planning our integrations, we had three (not always mutually exclusive) choices:

1. Delegate storage to a designated cloud provider on a per-object basis.
2. Link to objects that remained in the source cloud storage provider.
3. Import (and sync) objects from the cloud storage provider into AffinityLive’s own storage model.

The first two options have a lot of appeal, but they had shortcomings.

Delegation meant choosing a single cloud integration and limiting users to only using that. If you chose Box, for example, and uploaded a file to AffinityLive using one of the four interfaces above, we’d simply push that file across to Box. Want to connect to Evernote? Bad luck.

Linking to objects had the advantage that our users could link to anything and everything. But it meant that functionality – like attaching a file to an activity you were about to send to a client, which is super common for our users – would become confusing, frustrating, difficult or impossible. Your client probably wouldn’t have direct rights to see it in your Dropbox (and your colleagues might not either) so we’d need to be messing with complex ACLs… a problem that gets all the most complicated as you add in more services. So, the seemingly short-cut approach of “paste in the link to view in the web interface” model quickly becomes a nightmare.

Importing, while facing its own challenges, is actually the model we chose to go with. It means that a user chooses to add a file to their AffinityLive Project from Dropbox, and then it is against the AffinityLive project. Makes sense – nothing confusing around ACLs, consistent shared project file space. You can bring in files selectively from anywhere without worrying about sharing too much or having us try and swap/switch the model of the cloud storage provider from a user-centric private model with explicit sharing to an organization-centric collaborative model with ACLs. There are a couple of downsides with this approach, however, mainly around synchronizing the file changes that may occur in AffinityLive back into the cloud provider in question.

InkFilePicker Integration

The InkFilePicker integration is actually exceptionally easy to get started with. This is what we chose to do:

1. Use the Pick JavaScript API. This makes it easy for your users to pick a file from their cloud platform, which is then downloaded to the InkFilePicker servers and stored at a permalinked URL. Since we’re handling our own storage we didn’t use the S3 integrated Pick and Store, but if you are you might be able to save a step below.
2. Get the permalink & metadata. When the filepicker.pick command returns, we fetch the metadata (would be great if they made it an option to return this detail in the pick command to save the extra hit) and then POST via AJAX to our API.
3. Import/download/create the file to AffinityLive. On our API end, we fetch the file from InkFilePicker and store it in our AffinityLive storage model.

There’s the relevant code snippets. Note that this is rough, PoC code and not final code but ironically this code is likely easier to follow than our final optimized code will be.

JavaScript

function pickFile(collId, colDepth)
{
  // The collId tells AffinityLive the collection we want to use
  // The colDepth is a visual feature to indent the margin on the row in our PoC
  filepicker.setKey('YOURKEYHERE');
  filepicker.pick(
    function(InkBlob){
      var permalink = InkBlob.url.split('/').pop(); 
 
      // Build a new row for the resource. 
      // NB: during this process it is still being pulled back as a resource into AffinityLive
      var resourceTemplate = $('#clone_' + collId);
      var newResource = resourceTemplate.clone().show();
      newResource.children("td[title='Resource Title']").text(InkBlob.filename);
      var newRow = resourceTemplate.after(newResource);
 
      filepicker.stat(InkBlob, function(metadata){
        jQuery.ajax({
          type: "POST",
          beforeSend: set_ajax_api_key,
            data: {
            service : 'filepicker',
            action : 'import',
            url : InkBlob.url,
            collection_id: collId,
            title : metadata.filename,
            key : permalink
          },
          url: api_base+'/key/resource',
          success:function(data, code){
            newResource.children("td[title='Resource Title']").text(data.title);
            newResource.children("td[title='Size']").text(Math.round((data.filesize/1024)*100)/100 + ' KB');
            newResource.attr('id','resource_' + data.id);
          }
        });
      });
    },
    function(FPError){
      console.log(FPError.toString());
    }
 );
}

Note that the DOM we’re writing to has a simple three-col table with the folder/file name, the size of the file and a col for icons that edit/delete etc. The thing to note is that we create the row and update the title when we get the value back from InkFilePicker and then we update it with size and ID information once it has been saved to AffinityLive.

API Back-End

This back-end snippet (in Perl) shows how we’re doing the import from InkFilePicker using LWP and then storing the file in our resource area.

my $collection_id = $cgi->param('collection_id');
my $collection = IRX::Management::Collection->new($context, $collection_id);
return Apache2::Const::DECLINED unless $collection_id && $collection->get('id');
my $ua = LWP::UserAgent->new();
my $tempfile = $cgi->param('key');
$tempfile =~ s/\W/_/g;
$tempfile = '/tmp/filepicker-'. $tempfile;
my $req = HTTP::Request->new(GET => $cgi->param('url'));
my $res = $ua->request($req, $tempfile);
if($res->is_success)
{
 my $resource = $collection->build_resource($tempfile, $cgi->param('title'));
 $resource->set('service', 'inkfilepicker');
 $resource->set('service_id', $cgi->param('key'));
 $resource->save;
 
 $r->print($resource->to_json);
 return Apache2::Const::OK;
}
else
{
 $logger->debug(sprintf('Fetch from filepicker failed: %s', $res->status_line));
 return Apache2::Const::DECLINED;
}

Google Picker Integration

The problem with InkFilePicker, unfortunately, is that it exists solely to find and export files. Which means, if the user is using Google Drive for their file picking, it will export the file from Drive, rather than provide a link to view/edit the file. This means, when you Pick a Document in Google Drive, you’ll get a .docx export!

Thankfully, Google have their own Picker solution, and though it is harder to find now (they’re giving the Google Drive SDK all the top billing) it is still out there and perfect for this situation.

To find out more about Google Drive, check out https://developers.google.com/picker/.

The first thing to note about Picker is that it is designed to allow picking from a LOT of Google services. Image search, Picasa, you name it, it’s there. Unfortunately, if you’re just looking for Drive (which I was) the default mode isn’t ideal.

What worked best for us was:

1. Turn off the left hand pane. If you’re only using one service (Drive) it is a waste of real-estate.
2. Set the scope to be DOCS so you’re just showing the user files from Google Drive. While we wanted to browse on a folder view, you will want to use DOCS to get everything (otherwise you miss out on files in the root folder).
3. Get the user to OAuth (if you can) so you can show the *right* Google Drive. For users who might be logged into their browser via their own @gmail.com Google Account and their corporate/business Google Apps domain, this is important otherwise the Picker will just show the files in the first account they logged into.

JavaScript

The JS we used (again, this is ugly PoC shit but you get the point) is as follows:

var developerKey = 'YOURKEY';
jQuery(document).ready(function(){
 gapi.load('picker');
});
// Create and render a Picker object for searching images.
function pickDrive(collId, colDepth) {
 var picker = new google.picker.PickerBuilder().
   addView(google.picker.ViewId.DOCS).
   enableFeature(google.picker.Feature.NAV_HIDDEN).
   //setOAuthToken(AUTH_TOKEN). // This is where you force a specific Google User account to be used
   setDeveloperKey(developerKey).
   setCallback(function(data){
     var url = 'nothing';
     if (data[google.picker.Response.ACTION] == google.picker.Action.PICKED) {
       var doc = data[google.picker.Response.DOCUMENTS][0];
       driveUrl = doc[google.picker.Document.URL];
       driveId = doc[google.picker.Document.ID];
       driveName = doc[google.picker.Document.NAME];
       driveType = doc[google.picker.Document.MIME_TYPE];
       driveServiceID = doc[google.picker.Document.SERVICE_ID];
 
       var resourceTemplate = $('#clone_' + collId);
       var newResource = resourceTemplate.clone().show();
       newResource.children("td[title='Resource Title']").text(driveName);
       var newRow = resourceTemplate.after(newResource);
 
       jQuery.ajax({
         type: "POST",
         beforeSend: set_ajax_api_key,
         data: {
           action : 'import',
           service : 'googledrive',
           url : driveUrl,
           key : driveId,
           collection_id: collId,
           mime_type : driveType,
           title : driveName,
         },
         url: api_base+'/key/resource',
         success:function(data, code){
           newResource.children("td[title='Resource Title']").text(data.title);
           newResource.children("td[title='Size']").text('Google Drive');
           newResource.attr('id','resource_' + data.id);
         }
       });
     }
   }).
   build();
   picker.setVisible(true);
}

Back-end AffinityLive API

In our back-end we wanted to distinguish between a Google Drive element that was editable and that which was merely a stored file (like a PDF). The current approach isn’t final (is a hack) but one way is to interrogate the mime-type for a Google specific prefix. The other option is to use the other data that Google Picker returns – here’s the reference of what Google can send back: https://developers.google.com/picker/docs/results.

Another comment – we provide a desktop-mountable interface to the AffinityLive file system and we wanted to make it possible for people using Google Drive and who’ve installed the desktop sync application to be able to click on a link and open the file in Drive. That’s why we’re creating the .gdoc and .gsheet files (which contain a very simple plain-text JSON payload which tells your OS where to open the file on Google’s servers).

my $collection_id = $cgi->param('collection_id');
my $collection = IRX::Management::Collection->new($context, $collection_id);
return Apache2::Const::DECLINED unless $collection_id && $collection->get('id');
my $title = $cgi->param('title');
my $url = $cgi->param('url');
my $type = $cgi->param('mime_type');
my $key = $cgi->param('key');
my $tempfile = undef;
if($type =~ /^application\/vnd\.google-apps\.(.+)$/)
{
  my $gtype = $1;
  my $resource_content = sprintf('{"url": "%s", "resource_id": "%s:%s"}', $url, $type, $key); 
  $tempfile = $key;
  $tempfile =~ s/\W/_/g;
  $tempfile = '/tmp/gdrive-' . $context->get('system_domain') . '-' . $tempfile;
  my $gfile = open GFILE, "> $tempfile";
  print GFILE $resource_content;
  close GFILE;
  if($gtype eq 'document')
  {
    $title .= '.gdoc';
  }
  elsif($gtype eq 'spreadsheet')
  {
  $title .= '.gsheet';
  }
  elsif($gtype eq 'presentation')
  {
  $title .= '.gslides';
  }
  else
  {
  $logger->debug(sprintf('I don\'t know how to handle a google doc mime-type of %s', $type));
  }
}
if($tempfile && -e $tempfile)
{
 my $resource = $collection->build_resource($tempfile, $title);
 $resource->set('url', $url);
 $resource->set('service', $service);
 $resource->set('service_id', $cgi->param('key'));
 $resource->save;
 $r->print($resource->to_json);
 return Apache2::Const::OK;
}
elsif($title && $url)
{
 my $resource = IRX::Management::Resource->new($context);
 $resource->set('collection_id', $collection_id);
 $resource->set('title', $title);
 $resource->set('content', $url);
 $resource->set('url', $url);
 $resource->set('service', $service);
 $resource->set('service_id', $cgi->param('key'));
 $resource->set('owner_id', $context->get_current_user_id) if($context->get_current_user_id);
 $resource->save;
 $r->print($resource->to_json);
 return Apache2::Const::OK;
}
else
{
 $logger->debug(sprintf('Can not create a Google Drive link without a title (%s) and a url (%s)', $title, $url));
 return Apache2::Const::DECLINED;
}

End Result

The end result is a powerful set of import and connection features to over a dozen cloud storage providers with Google Drive getting special attention because of its in-line editing and cloud creation processes.

Future work around synchronization and integration with Office365 is also on the cards – we’re looking forward to shipping this new feature in a month or so with a brand new Angular built attachments tab in AffinityLive proper.