Client-side search providers

Hi,

  Fellow swhacker and www-archiver sbp and I are sick and tired of web
search and services being hard and are returning the address bar to the
user. As a first step we have designed a client side search provider for
use in the address bar, starting with the collection of services we use
a lot, would like to use more, or simply want to be easier to access.

Currently there are two components, one a list of service providers with
corresponding URL templates. For instance, searching with duckduckgo has
an entry like `duck: http://duckduckgo.com/?q=%s` as we know it from the
address bar search shorthands introduced with Internet Explorer 5 or so.

Those are hard to configure, maintain, and share though, and they have
to be specific, you can't say "geo placename" and get a list of services
for geo location specific services, you would have to configure each of
them individually, or rely on some meta search engine online, so we will
be placing the data on an easily editable public service and support the
selection among multiple services client-side.

Another problem solved thus is that some services require unusual input
forms, some require a full URL and don't allow plain domain names, other
services need a domain name and require you to strip off the URL scheme.
Yet others may be for, say, coordinates, and don't allow simple decimals
all of which requires simple conversions that can easily be implemented
by having proper types in the URL templates and implementing appropriate
conversions.

The second component is a user script. For testing I configured this in
the Opera web browser. After trying various things that did not work as
they should have, I've come to use a search provider that maps, for the
time being, `s` to data:text/html;charset=utf-8;secret=value,#%s and the
URL prefix has an associated user script that matches the query against
the service meta data to select appropriate services and then either re-
directs using location.replace() to the service, or offers a selection
if multiple services match a query.

With the earlier example, `s duck example` in the address bar would lead
to `http://duckduckgo.com/?q=example`. Eventually the idea is to make it
the default search provider, so you just have to type `duck example` and
you don't have to specifically configure the `duck` provider, you just
rely on the collaboratively maintained service description data.

The service description data will be online and the user script has to
fetch it every now and then, but it will be stored, at least for Opera
users, locally in the user script local storage, to avoid information
leaks as much as possible, and for performance reasons of course. There
is obviously an abuse problem, especially because we think it best to
allow anyone to edit the data without identifying themself or proving
that they have some reputation, the service where we intend to host it
has reasonable measures against abuse. If that assumption turns out to
be flawed, there are easy ways around this, from the perspective of the
user script all that is needed is some JSON-P service with the data (I
am assuming Opera user scripts can load cross origin data using JSON-P).

One usability issue is the use of secret=value in the address that ties
the search provider to the user script. Obviously you don't want sites
to interfere with this, so far having a user-unique and long secret in
the address is the best I could think of and get working.

The main design issue in this stage is the service selection based on
keywords in the query. Having quite a lot of services in the list makes
it difficult to use the (normal) default search provider when none of
the services match (should a query 'maps ancient mesopotamia' redirect
to Bing Maps, or would a duckduckgo query be more appropriate? Or do you
offer a selection because Open Street Maps also matches Maps?) I've yet
to think of suitable solutions here, so I will go with a special command
for the client-side search provider for a while and see how that goes.

Principially it is quite easy to get used to type "b" in front of a more
general web search query to go to Blekko, but on a desktop keyboard that
is much easier to type than on a phone with a virtual keyboard, so I'm
not sure how this will play out ultimately.

Also, if this works out well, it's quite possible that the client-side
mostly redirecting search could be turned into a meta search engine that
queries multiple services initially and offers to go to specific ones to
refine results. For 'maps ancient mesopotamia' it might load a map and
also textual search results and allow the user to go from there, just as
some existing search engines do. With the benefit of users being able to
control it as they see fit, such as having their custom layout applied
to the search results. No more "Instant" search results being inflicted
on you if you don't like it. If people bother to make the scripts for it
of course, which is likely a major problem there.

regards,
-- 
Björn Höhrmann · mailto:bjoern@hoehrmann.de · http://bjoern.hoehrmann.de
Am Badedeich 7 · Telefon: +49(0)160/4415681 · http://www.bjoernsworld.de
25899 Dagebüll · PGP Pub. KeyID: 0xA4357E78 · http://www.websitedev.de/

Received on Sunday, 4 September 2011 00:18:00 UTC