Why Search is Done by Geeks Not Authors

I love search…in all its forms. I love the fact that even as data explodes we have a way to navigate and in some meager way comprehend the complexity and scale of knowledge. So I was excited to try out a new search which I came across recently called smalldemons.com. This is our guide to the “storyverse” (get it?) it’s a search platform which purports to offer insights and connections on books, authors, places and characters along the lines of “drinks found in James Bond novels” or “places in War and Peace”. It also has a neat Pintrest like feature which allows you to assemble a story board linking items, people and places from and around books.

My first reaction was ‘how cool’…no idea how the ex Yahoo and Facebook guys behind the site will make money but I love the idea. Then (being the annoying search guy I am) I started some cursory testing.

Place search: “Krishnapur” the city at the center of the fabulous indeed award winning”Siege of Krishnapur” no results.

Character Search: “Samuel Vimes” hero of many of the enormously successful Discworld novels. Lots of Samuel but no “Samuel Vimes”…at least not on the first pages.
Item search:  “Victory Gin” the gut rot drink for George Orwell’s 1984.… lots of “Victory” no “Victory Gin”

Shakespeare Search: “Damned Spot” from Macbeth…actually this is a funny result. First result a recent novel called “Damned” second result “Where’s Spot” the kid’s book.
Great but somewhat older novel search: “Memoirs of a Fox Hunting Man“ nothing relevant

Movie Quote Search: “You played if for her you can play it for me” from Casablanca….nothing close to relevant.

I promise those were my first eight searches at random from the rat’s nest of literary pocket litter which passes for my brain. I was somewhat saddened, especially when Google and Wikipedia (except for Wiki which missed the last one) both easily nailed every search. I actually love the idea of a search which does the cool kind of conceptual thinking and linking this site sets out to do. However, if they don’t even have phrase match search figured out let alone conceptual stemming then they have a ways to go. I was thinking I might make my first story board one called “People, places, books, references and things” which you can’t find easily using smalldemons.com…..but that would be unkind no?

A Google Pre-Announcement?

I have just returned from the ILM BIA/Kelsey conference in LA where the great and the good of the surviving Yellow Book industry meet to discuss the demise of their industry and figure out how they can construct a digital life raft to get at least the women and children away from the Sinking Yellow Titanic before the boilers explode. In any event it was good to catch up with a bunch of great folks and look ahead. Most of what was discussed has already been heavily covered in these pages but I found the Google session especially interesting. It’s always interesting to attempt to read the runes of any Google spokesperson. The focus on telling you what you already know (but with great slides) they assure you of your importance and how great it would be to work together. This is almost entirely unlike the actual experience of working with the big “G” but let’s not let details get in the way. The Kelsey conference is very hip and allows the audience to tweet in questions rather than call for the microphone. I took advantage of that opportunity and tweeted the following question to the Google Guy.

“Search is growing rapidly away from desktop yet Google doesn’t syndicate beyond AdSense, is Google missing an opportunity?”

The background to my question is simple: The past decade or so has seen Google establish a hegemony over desktop search. People go to Google.com to search and they experience AdSense contextual ads on web pages as they navigate the web. From a publishers point of view AdSense is not an especially valuable product and it’s pretty much impossible to get hold of a full blown Google powered search/ad feed. Google understandably wants to retain all of the revenue they can, which is exactly what happens when people search on Google. If Google syndicates advertisers they have to share revenue and be cautious that the quality of search remains high. All that is changing as mobile rapidly catches up with desktop. Search is now happening in mobile Apps and on mobile browsers and the current Google offering just don’t fit well….hence my question.

The answer I got was fascinating. First the Google Guy thought long and hard…he stared into the middle distance and gained some time by saying how good the question was. I hesitate to suggest that he was squirming…but it was close. Then he made it clear that Google doesn’t pre-announce products so he couldn’t really say much (which in of itself is saying a lot). He then went on to comment on how important emerging search sources are and how important they are to Google for growth.

That sounds to me a lot like Google has a new search and ad syndication product probably optimized for mobile to allow them to get a larger piece of that opportunity and reward high quality publishers for their traffic. You heard it here first.

Stalking Made Simple

As I have mentioned many times in the past I’m not a huge fan of social media. Given my dislike of people and social situations in general that’s not entirely surprising. When I go to hell, I expect it to be an eternal ‘networking‘ session where I’m forced by little men with horns and forks to talk to complete strangers at trade shows. Having said that I do use LinkedIn and I have a Facebook page which I never post on. I welcome pretty much all invitations to friend or connect (because I don’t really value the platforms) and that has gotten me into trouble on occasion, where my maniacally social wife will grill me on who various women on LinkedIn or FB are….many of whom I actually barely know at all.

In any event I was looking at a new “people search” tool called ark.com which is currently lumbering up to the start. Their paradigm is to index the LinkedIn, Facebook and other social platforms so that users can search not for people they do know, rather it lets users specify the parameters of people they would like to connect with and approach them through the platform. If I were visiting Chicago and I wanted to play chess with a cat loving man of Vietnamese extraction who went to my old high school they could in theory hook me up….assuming that I and the aforesaid gentlemen happen to subscribe to Ark.com and the appropriate other interest or alumni programs.

This approach of using social media to hook up with people you don’t know but happen to be in physical proximity to is highly successful in certain areas such as the gay dating world where platforms like grinder.com facilitate exactly that kind of connection. Recent years have seen the launch of several other platforms which set out on the premise that it would be neat to be able to track down people you don’t actually know who may be nearby with similar interests but discovered that technology enabled stalking was problematic and have morphed into ways to track down people you do know instead.

The folks at ark.com are launching with colleges under the premise that if it worked for Facebook it will work for them, and maybe it will. In the event that it does start to take off I imagine Facebook will pull one of their famous programming all-nighters and will come up with an exactly similar solution which will allow existing FB users to discover people of like minds which they don’t currently know and that will be game over for our friends at Ark.com. Either way…I’m not playing.

Welcome to the Machine (It’s Sue, Eric and Bob)

As we all know search is fiendishly hard to do and is run by all seeing all knowing algorithms which are as close to deities as is reasonably possible given current limitations. It’s that dedication to anonymous selection which lets the search engines fend of constant assaults over censorship, monopoly and privacy such as they are experiencing in the EU and US currently. Yet as the recent Register.co.uk  story once again points out Google in facts has armies of part time out sourced workers whose task it is to rank results sets flagging the best as “vital” all the way down to “Porn” and “harmful.”  These QA folk certainly don’t build the algorithm and Google says they don’t drive relevance (in which case why bother at all if you don’t listen to the results?) but the smart money is on that they have some kind of impact.

Evidence of potential human intervention might be seen in the recent waves of algorithmic attack on thin or “over SEO’d” content frequently means that pages with next to no links at all do extremely well…the skeptical might suggest that at least to a point some level of human intervention is helping these high quality pages with almost no traditional search juice. We in search have known for a good while that a measurably large number of important search results are most likely either hand crafted or at least carefully vetted and guarded against assault by the over enthusiastic or unscrupulous.

So which is it?…or is it both?. If search is indeed heavily influenced by human editorial ‘opinion’ even in the form of rigorous QA some will argue that the search engines move from a legal safe harbor of hands off neutrality, to a place of suspicion and maybe corruption. As Google in particular acquires more sources of “answers” to compete with other content providers, and if there are many thousands of people focused on the  “best” end user experience isn’t there the possibility that “best” might even informally be conflated with the “in house” or “our” content. If that were true we’d have an interesting monopoly situation to pick through…exactly as the FTC is currently doing.

The Deal Which Wasn’t

I was watching the wires this morning catching up on the comings and goings of our space after a barely earned Thanksgiving break. The story which jumped out was the announced purchase by Google of ICOA which is a large Hot Spot company which specializes in supplying WiFi access in high traffic areas like airports. The story had a solid ring of authenticity about it as Google has sponsored access in the past (I remember they did a very welcome deal last Holiday season where travelers through most major airports got free WiFi for a month or so). There have also been many stories about Google buying up “dark fiber” and potentially looking to supply nationwide WiFi as a potential competitor to the cell phone carriers. WiFi as opposed to 3/4G is completely possible. With enough WiFi you won’t need a cell carrier. This has already happened in some smaller highly tech centric countries like South Korea and it would be much harder to do in a space as large as the US…but it’s feasible with enough coverage. So the story made a kind of gut level common sense.

Looking to clarify the brief item I then searched for ICOA and Tada! there is no acquisition, the whole thing seems to be a fake being denied by all sides. The news services are falling over themselves to deny and retract. The story was published through PR Web a press release service. In the time it took to read the denial it had spread all over the world being picked up and republished in at least six languages and dozens of online publications.

To use PR news services you have to register with contact and payment info…so in theory it should be easy to track down the smoking gun for this story. This is either a story which leaked early or a pretty naked attempt at stock manipulation. I checked the Google stock quote and during the period of the fake release the stock moved only a couple of bucks, but ICOA which is a penny stock on the PINK exchange was up 200% on a huge trading spike. So it rather looks like Google and ICOA were the victims of a stock sting…and the chances are someone is going to jail.

Too Large To Sue?

The clock is counting down on the FTCs two year investigation of Google. The government has spent a hefty chunk of our tax dollars to see whether Google is a monopoly which acts against the interests of the market. A horde of heavily biased search ‘also ran’ companies has lobbied to have Google either sued (probably the largest action since Microsoft in the 90′s) or reach a deal with Google which would in their minds level the playing field. Word is that most of the FTC favors moving ahead against Google….but I wander if that really would do any good.

It’s a tricky situation. Google is undoubtedly a monopoly, but if you didn’t have search life would go on…no babies in incubators would die. However they are the fulcrum of a huge amount of the new economy so to have them favor their own projects and harm the competition could be a problem. What I found rather distasteful was the spectacle of Silicon Valley Congress people writing to the FTC suggesting that they back their tanks off Google lawn lest their actions harm the economy….really? Google is now too big to sue?

Google is a monopoly in terms of overall search traffic and especially in terms of the crucial commercial search market. They are aggressive and unwavering in their pursuit of their interests….but this is America they are allowed to be. I long since gave up trying to keep up with the endless parade of “next Googles” in search. The short answer is the next Google might be Facebook…or Apple.   Remember when Microsoft was an evil Monopoly…before browsers took so much of the desktop over…before Apple surged back and Google cornered search, before Microsoft got fat and lazy? There are serious threats to Google in the search world. If Facebook cuts a deal with Yahoo, Siri continues to grow, Amazon continues to establish mastery of shopping search and we continue our dash to mobile there will be ample opportunity for Google to step up or others to step in.

I suggest the FTC strike a deal to the effect that Google will clearly label all advertising from any source and will undertake not to unfairly promote their own products above others….then let the market vote with its feet.

The Search Ponzi Scheme

It’s not every day I get to stare goggle eyed at my screen and just go “Wow”….then pause a little and say “Wow” again. As I write this piece I’d like you to add the word “allegedly” in front of every sentence or assertion I make. The details I’m quoting are from today’s Forbes piece but it’s also in WSJ and other leading publications so I think this counts as color commentary or my angle on this amazing story.

Join me in the way back machine and reset the clock to 2000. At the time I was doing biz dev for FAST Search and Transfer…an amazingly talented bunch of Norwegian and US search guys. We sold web search to websites (our competitors were companies like Alta Vista, Inktomi and an upstart with a goofy name you have probably never heard of called Google). We also sold “Enterprise Search” to big companies who were trying to make sense out of the ever growing amount of data they were generating. The problem then (and now) is that enterprise data lives in all kinds of systems and formats and it’s really hard to manage it at scale. For example; imagine you are a home improvement mega store with 1,000 outlets, 100,000 employees and 200 product depots. Keeping track of the people, product info, inventory sales and shipping across all the different systems involved is a nightmare. A simple question like “how many 1″ nails should I order” might require you interrogate five different sub systems. At the time FAST was developing and selling systems which do just that…they were subsequently bought by Microsoft to its Enterprise Search component, but that was long after my time.

Back in the day I was mostly involved in the web search side of the business but I did get involved in the corporate side too and was keenly aware of the competition. One of the competitors we constantly ran into was an outfit form the UK called Autonomy. They cultivated a fiercely academic style; they were always the smartest kids on the block…certainly smarter than us mere Vikings. They had what they called an Intelligent Data Layer which mobilized and wrapped data to make it intelligently searchable.  We went up against them many times and we won lots of the battles. The weird thing was that even though I did manage to sneak into a few actual demos I never saw the product work and I never spoke to a client who was using it successfully…or at all. I always figured that I just wasn’t close enough to the battle to see all the use cases and I was out of Enterprise Search completely in 2004 so I rather lost track of the plot. I did note that just as Microsoft had bought FAST so HP bought Autonomy a couple of years back, which I figured completed the narrative arc nicely….good for the clever Brits….Ka Ching!

Today I read that HP has just written off $8.8 Billion of the $10 Billion they paid for Autonomy alleging accounting irregularities. As I read the story I had to hold my jaw up lest I drool on my keyboard. Autonomy grew large and swallowed up other tech services such as Iron Mountain data storage and Interwoven content management. They would (allegedly) do big deals for these fairly mundane tech products and services in high volume and low margin and ascribe the revenue from the deal to their fancy search product…even where the company didn’t want the product and wasn’t using it…either because they didn’t need it or couldn’t get it working. This created the illusion of a massively successful and wide spread search company as opposed to a data storage company which gives away a phony baloney (allegedly) vaporware search product.

Apparently the financial coroners at HP have tracked down the bodies, dug them up and determined cause of death…and have written off the vast majority of the acquisition….they have left about a billion on the books as the intrinsic value of the actual search….my hunch is that may be a little high. In a year of “other shoes falling” this one is a Jimmy Choo….Wow, wow ….wow.

It’s Getting Very Complicated Out There

I have been going to search engine conferences on and off for the past decade or so. In 2000 I attended what later became known as “PubCon” mostly because they used to hold them in the upstairs meeting room above a London Pub. It was a couple of dozen hackers and spammers getting together to tell war stories and discuss strategies to get their clients to rank on search (by all and any means possible). How times have changed. Search is now a multi-billion dollar industry which is being driven up half crazy by the explosion of social and mobile. End users are abandoning the desktop for mobile devices and spend as much time interacting with apps as they do clicking on browsers.

A few years back I saw a demographic transition from the hacker-nerd mix of the early days to a more traditional mix of web guys and mainstream marketers. As the whole search marketing space evolved so the big search engines came up with tools and data to allow marketers to more efficiently spend their dollars with them and in parallel a small ecosystem of technology companies emerged to service the busy marketers…..then something bad happened….

The problem for the traditional marketing persons point of view is that social media doesn’t play nice because it’s much harder to control people than it is to buy media. Mobile makes life harder because although it delivers fabulous location information it does it on relatively tiny screens with much less real estate to target and apps have exploded and are taking users away from engaging with browsers and make content harder to get at from the search engines point of view. Along with the continued fragmentation of traditional media the explosion of online channels and the data they generate is making the typical marketers head explode.

For example at SES this week I took part in several “Meet the Experts Sessions” where the audience gets to ask “The Experts” anything they want. I was on the Local/Social table. A very nice and very well dressed agency lady who runs marketing for a major gaming platform wanted to know how to “impose their brand strategy on Social Media”….my short answer was…”you can’t.”

There is a whole new category emerging in our industry….broadly speaking it’s called “Big Data.”  To get at it you need to be able to build APIs and to understand it you pretty much need a double major in statistics and higher math. They don’t teach that at MBA school…..life is getting more complicated and in general terms harder and harder pretty much month by month. To make matters worth the kinds of people who do have the technical, stats and marketing mix needed to wrestle this mess to the ground are both few and far between and highly sought after …and retirement is still a long way off for most of the folk currently in it. Raising goats can seem an attractive option sometimes.

We Are a Small Industry

I have been as Search Engine Strategies in Chicago for the past couple of days and I’ll have some specific search items to report later but for now I have been struck by how small we are as an industry. The search business is worth about 50-60 Bn this year with the bulk of that revenue for what is effectively a media buy going to Google, Yahoo and Bing. In dollar terms that’s more than is spent on Magazines and Newspapers put together in the US. The search business has roughly 8 major trade shows per year scattered through the big cities. I’m guessing most folk hard core in the business attend at least one show per year….some like me do multiple shows. SES Chicago is one of the larger shows and at a reasonable guess there are about 1,000 attendees with another hundred or so folks working the dozen or so booths in a small exhibit. Google employs about 34,000 Googlers, Yahoo and Bing maybe another 10,000 between them. So at a rough guess the entire industry employs maybe 60,000 people total…..each of whom commands about a million dollars of revenue each. Obviously it’s a massive over simplification but how many people work for the 2,000 significant magazines and 1,200 daily and 5,000 weekly newspapers I wonder. It’s striking how much power and influence search has in all aspects of our lives…and relatively speaking it’s controlled by a tiny handful of people….the other word for that is a monopoly….right?

Global Search Wars Escalating..(Again)

Search is important, anyone still harboring any doubt need do nothing more than look at some of the bubbling and breaking search stories circling this week. Clearly several parts of the EU hate, hate, HATE Google and all its works…and unsurprisingly the French are leading the charge. I have commented on this battle a couple of times recently, for a while media owners have been railing against Google taking their content for free and monetizing it by serving ads on results sets. In a very French way they want the government to levy a news tax on Google and in return Google is threatening to stop indexing all French content. Google boss Eric Schmidt met with the French government and a bunch of EU publishers to discuss this issue this week. It strikes me that rather than escalate to an all out war the French publishers might do what the Brazilian ones did a few months back. They effectively blocked Google from indexing their content…if searchers want new content they are forced to go to the publisher’s sites and enjoy the ads displayed on the publisher instead of the ads displayed by Google. If the Brazilian publishers are to be believed this change had only a minimal impact in over all traffic…so maybe the French should try that out and see if it works for them too. The other weird war still on a slow burn is behind the Great Fire Wall of China.

China has been growing by leaps and bounds with Baidu as the market leader. Back in August an anti virus company Qihoo 360 launched a search on the back of the enormous exposure they already have through their antivirus products. This pretty much overnight captured 10% of market share (is anyone at McAfee taking notes?). A slanging match has ensued with companies getting more agitated and trying more and more edgy things to get back at each other.  In any event in a statement on Thursday the Internet Society of China supported by all the major search providers in China committed themselves to place nice and  “encourage innovation and create a fair competitive environment” and perhaps ominously to “stop online rumors” by beefing up real identification of people posting online. This is clearly an attempt to put the rumor genie back in the bottle. There have been multiple cases lately where some pretty damaging stories have been circulating anonymously about some major Chinese political figures.

The Iranians continue work on an Islamic web which returns results only from the Caliphate of the 7th century, the Chinese want to stop online rumors and the French would rather the whole Internet thing just go away…It would be more effective to try to clean up after Hurricane Sandy with a bucket and mop….have a great weekend.