Sentimental Software

Software generally isn't sentimental.

Ask a software program to find everything on the Web that talks about Barack Obama or Flat Panel TVs or Interocitors and it does pretty well. It doesn't matter how simple or obscure, how abundant or scarce, computers are very good at finding stuff.

However, ask a software program to find everything on the Web that says nice things about Obama, TVs or Interocitors and the program gets queasy and confused and apparently random in how it responds. Humans may understand nice, nasty and neutral in astonishing detail, and pick up on the slightest nuance in how things are expressed, but computers don't. We understand plain spoken feelings, we laugh at a joke, we grok sarcasm. Said another way, humans feel things, computers can't.

Lots of people have tried to make computers more sensitive creatures by writing elaborate programs that attempt to disentangle the tone of what humans write. They usually start by trying to get computers to understand written expression in the way we do, grammatically, syntactically and lexically. It's tricky. Stuff my three year-old understands in I Love You, Goodnight might be obvious to most software programs that try and detect tone, but my nine year-olds' Harry Potter would be a real stretch. And my New York Times would bamboozle most sentiment detection systems a lot of the time.

A lot of people are working on this problem, because there's a lot of commercial applications (Wikipedia has a nice summary of the technology, the business potential and some of the companies that are trying to solve this problem). It has great potential in automated trading applications, brand management, PR, politics... the list is long. Today, automated tone or sentiment detection is often built into media monitoring systems, but most companies selling these products acknowledge the results can be erratic and instead rely on human readers -- now there's a really, really interesting job . In my day job, I'd love an application that could accurately assess the nice-nasty-neutral tone of the stories my company gets. And I have to believe there's a great start-up opportunity in this space, especially serving some vertical market sectors.

If anyone knows of good sentiment detection software, let me know.

Obama in Hong Kong

Newspaper headline in Hong Kong daily: America Changes Color, next to a picture of Lincoln on the left and Obama on the right.

Living another man's dream

I'm in Hong Kong this week with work, the first time I've been here in over 20 years, and all eyes are on the US.

On Nathan Road and in Central they're selling Obama t-shirts, and the local papers are filled with the rich prospect of not if he'll become president, but what he'll do when he gets the office. No more White House is a joke I heard on the Star Ferry.

He's living another man's dream, voiced over 45 years ago. Just when you completely loose faith in American politics and values, this happens. If it wasn't for an exquisite confluence of events it would never have happened at all, but it is happening and I never thought it would. Amazing.

If this doesn't get Obama elected, nothing will

CNN Money: Biggest Government Bailout in History.

PR Measurement is giving me a headache...

I've spent most of my career in marketing and lately have been concentrating on PR, which I manage for a publicly-traded company.

To say that PR and the media business have changed in the last few years is a bit like saying Bill Gates is comfortably well-off, or Neil Armstrong is a seasoned traveler, or Sarah Palin is low key; it's really hard to overstate the turmoil in the media business, and as a consequence the upheavals in PR (for a great take on how this has impacted tech publishing, see Tom Steinert-Thelkeld's blog).

Nowhere is this more apparent than in the media measurement business. I'm old enough to remember when clips really were clips: pieces of newsprint cut-out from a magazine or newspaper by some exceedingly patient, far-off reader, then painstakingly collated, annotated and mailed to me in a big bulging brown envelope. Today, almost all the news is online and much of it doesn't come from a traditional news outlet, yet most media metrics and coverage monitoring still function as if in an ink-smudged era.

For sure there's a bunch of new companies that have addressed the new media reality, and focused on social networks and brand management: Biz360, Cymfony, BuzzLogic, Vocus and RatePoint are some examples. They all essentially follow the same formula of aggregating digital news using some kind of search and filter system (you can still get the pieces of paper if you need them, but each little clip will cost you more than the newsstand price of the whole publication). The algorithms at the heart of these systems are usually based on fixed keywords (company name, ticker symbol, product names, etc.) then some additional processing based on either rudimentary rules of grammar and syntax or or a series of logical operations, and is often called natural language processing (NLP), since it tries to emulate how humans read and understand text. The results, based on my limited experiences, range from the amazing to the bizarre, and most systems need human intervention to get at subtle things like tone.

In an attempt to add value and differentiate themselves from free services like Google News, these companies also have a vast array of reports and dashboards that slice and dice data to show share-of-voice, on target messaging, competitive coverage, salience and on and on. Again, result may vary from those advertised...

Pricing does not seem to vary much: all cater to a similar audience of complex multinationals, usually in the financial services, pharma, or legal businesses, and costs are high. Or at least they seem high to me.

This complexity has spawned a lot of blogs. K.D. Paines is excellent on PR measurement, although recent posts suggest a level of complexity in getting truly comprehensive metrics that is daunting and might account for the high costs. Ed Moed has a lot of good stuff to say, too (although Ed, I think “What's so funny...” was written by Nick Lowe), and intelligent measurement has a lot to say about social media.

But at the end of the day I get a headache. It should be easy. It should be straightforward. It should be inexpensive. And it isn't. If anyone has ideas on how to crack the PR measurement problem, let me know.

Novel Novels

In 2007 there were a staggering 50,071 new works of fiction published in the US, or about a book every 30 minutes. This reminded me of the famous Kurt Vonnegut quip that there’s really only two plotlines in existence – ‘man falls in hole’, and ‘boy meets girl’ – which if he’s right would make for an awful lot of repetition.

More than anything else this might account for the increasing, willful absurdity of a lot of modern fiction. Even in the last few years we’ve had books narrated by a murder victim (Lovely Bones) and an autistic teenager (The Curious case of the Dog at Night Time), both very successful, and both upending ideas of the ‘unreliable narrator.’

All of this came to mind as I read for the first time Nabakov’s excellent Pale Fire, published back in 1962. It’s a great example of a novel novel: a twisty tale masquerading as a definitive, annotated edition of the last work of a (fictional) famous poet. I guess Nabakov was teaching at the time at Cornell, and he has great fun skewing academics and the academic interpretation of literature. The poem of the title, which is at the heart of the book, alternates between gorgeous, haunting imagery and great jokes, and the daft annotations that follow are laugh-out-loud, like Lolita. I feel I’m missing 90 percent of the references, but this is still one of the best books I’ve read in a long time.

Not many writers can pull-off literary tricks like Pale Fire, but here are a few other novel novels that derive a lot of impact from pulling apart the conventional narrative structure.

Time’s Arrow is probably Martin Amis’ best book, and mostly overlooked. It starts from an end, and end with a beginning: the books is entirely in reverse, following the death to birth passage of a man’s life, a Nazi war criminal who from old age moves back through youth and to Germany, where in a concentration camp he brings back to life millions of Jews. The book works in a lot of different ways, not least as a comment on redemption and atonement.

Vikram Seth, like Amis, was the golden boy of literature for a while, and wrote Golden Gate just before the fame hit. The whole novel, set in contemporary San Francisco, is written in iambic pentameter. At one point midway through the book Seth breaks down the “fourth wall” and directly addresses the reader – not as some postmodern trick, but out of exasperation with having to invent more plot with the right meter.

Cloud Atlas by David Mitchell got raves in the UK but far less adulation in the US. Some of the criticism has merit, but this book is still wonderful. Here the multiple narrators begin their tales then suddenly switch mid-sentence, and the six stories are dovetailed together in an ascending and descending sequence, each written is a distinctive style and voice that is utterly convincing.

And lastly, I read the latest Bond opus, Devil May Care, written by Sebastien Faulks writing as Ian Fleming. This is a hoot, and Faulks really is writing as Fleming here – pitch perfect, sexist, racist, and with a subtly subversive plot perfectly preserved in some early 1960s world that never really existed at all.

Skin Deep Politics

American politics can often appear shallow, but who would have thought it was only skin deep.

Last week the New York Times, in conjunction with CBS News, published their latest poll on the US presidential race. And Oh, what a shock:

After years of growing political polarization, much of the divide in American politics is partisan. But Americans’ perceptions of the fall presidential election between Mr. Obama, Democrat of Illinois, and Senator John McCain, Republican of Arizona, also underlined the racial discord that the poll found. More than 80 percent of black voters said they had a favorable opinion of Mr. Obama; about 30 percent of white voters said they had a favorable opinion of him.

On the other side, about 35% of whites have a positive opinion of McCain, compared to only 5% of blacks. Layer on this more research showing that about 30% of Americans admit to feelings of racial bias.

Then this week John McCain, who has a history of melanoma, announced that he's had a spot removed from his face, which is undergoing further testing. In a speech yesterday, McCain urged all Americans to use sunscreen and stay out of the sun if possible.

Both presumptive candidates have skin in the election game, literally. For one, it is about race, and for the other, age.

National Insanity

In 1790 the population of what is now the USA was about 3.9 million, or 41/2 people per square mile of settled territories. About 2.3 million were immigrants, and about 20% of that population was African (but rising to 43% in slave states like South Carolina). Indigenous native Americans still represented a substantial part of the population, mostly in the west. The life expectancy at birth was less than 40 years. Possessions of colonists were few, but in a study of estate records from the time, as high as 70% of households had a firearm, compared to only 25% who owned a Bible. Most guns had a flintlock mechanism for firing, and required hand-loading with shot and powder.

In 2008 the population of the USA is about 301 million, or about 85 people per square mile. Almost all the population are descendants of immigrants, and about 16% are African American. The native American population has all but vanished. Life expectancy at birth is well over 70 years. Possessions are many, with about 35% of households reporting owning a gun and as high as 90% owning a Bible. There are something close to 200 million licensed firearms in the US (households with firearms typically own more than one), of which about 65 million are handguns capable of carrying up to 10 rounds that can each be fired in quick succession.

At the end of 1791 the Second Amendment to the US Constitution was ratified providing a right to bear arms. This last week the Supreme Court decided that the amendment provided individuals with these rights, and made it very difficult for states to pass legislation limiting the ability to own and carry guns.

I'm not expert on American political democracy, and as a Brit I'm more familiar with the parliamentary process and common law (we have no constitution providing fundamental rights and obligations). And the idea of enshrining 'unalienable rights' for the individual seems very American to me. All that said, has nobody noticed that things have changed around here??? At what point do you acknowledge that laws passed hundreds of years ago are irrelevant and need to be reexamined?

There are a lot of complex arguments about this issue, and there's a mountain of research that provides supporting evidence for all sides. But take this, from an NRA-sponsored report:


The United States has the highest male teen homicide rate in the industrialized world (23.0 per 100,00 among males aged 15 to 19 in 1996). A 1997 study that compared firearm death rates in 26 industrialized countries among children less than 15 years old found that the firearms homicide rate among U.S. children was nearly 16 times higher than the rate among children in the other 25 industrialized countries combined.32 In 1996 the rate of firearms homicide was highest among males aged 20 to 24 (30.3 per 100,0000)—more than five times the firearms homicide rate for all Americans (6.0 per 100,000).


What typically follows in any debate about the facts above is largely abstract, and centers on ideas of individual liberty. This is a national insanity. The focus of the debate on guns should not be on any sense of individual right, and we should instead be pouring all our resources into creating and enforcing strong federal laws that seriously limit access to all firearms. Because I'm sick of reading things like this. And this. And this.

The New Frugalist

Although England has a reputation for heritage and retrospection, judging by the national daily papers it's pretty enthralled by fad and fancy too. When I was in the UK last week I tried to read a couple of papers a day, and they were filled with the usual blink-and-you-miss-it celebrity froth, pop-culture impulse and political nonsense.


One thing did catch my eye – the new Frugalism. I say “new” Frugalism because the idea has been talked about a long time before now, in the context of Victorian industrial economics. But according to the Independent, today's Frugalist is motivated by more complex concerns. This is a blend of economic imperatives, green values, and a desire for a simpler, less consumer-centric life. It seems to me that it's the economics that is really new – with oil pushing $140 a barrel, with global food shortages, and with globalization accelerating, Frugalism may simply be a new way of expressing a modest, middle-class discomfort in the wallet, while for others it's a more brutal slide into poverty. I also think it's an expression of anti-globalism, and a falling back on regionalism: when you can't afford to travel, or to ship things long distances, it encourages a natural set of parochial values.

Around the same time in the same newspaper I read an excellent article by Rupert Cornwell about the implications of the oil crisis on the American way of life. One thing that strikes visitors to the US is how much the car is king, and how a sprawling suburban life is fuelled by cheap gas and endless miles of highway. Car culture is real and engrained, and is central to everything from urban planning to entertainment. All this could crash pretty soon.

Frugalism may just be another fad. But it could also be the first wave of post-consumerism.

London

I've just got back from a week in England, mostly for work but also a couple of days of visiting family in London and Newcastle.
I arrived in London on a gorgeous sunny day, and the city shone. I took a long walk from the Edgware Road through Hyde Park and over to the West End and Soho to see Ralph Fiennes in God of Carnage, then walked down Shaftsbury Avenue, through Trafalgar Square and down Whitehall to the river (Flickr updates to follow). The whole city was alive, vibrant. I loved it.

Tuned Out

There was a time that I'd go listen to live music any chance I got -- these days, I rarely have the time or, I confess, the inclination. Sadly, this is more a reflection of my age than of the quality of music being played, but might also reflect the changing digital dynamics of the music biz.

I went to see the Raconteurs this week in Boston, a band dubbed oxymoronically an 'alternative supergroup'. An uneasy spotlight fell on Jack White, with the rest of the band playing second fiddle to his manic guitar. The Raconteurs new release has got a lot of press for the music, but just as much because it's a pressing on vinyl. For White, it's the tangibility of the object, in the sleeve, and the experience of playing a two-part story in music. White's music sensibilities hark back to the blues, punk, and an era when singles and albums formatted songs acoustically and physically. An album could hold about 30 minutes of music a side, a single maybe 7 minutes, and this condensed things. And albums tracks, the artwork, gave a new release an indentity that a CD or MP3 download can't get.

Before the Raconteurs, the last music I went to hear was Elvis Costello at a small club in Boston, where he played a long retrospective set interspersed with covers ranging from Bacharach to The Beatles. He's just released a new album, Momofuku, that is also on vinyl, but even the CD has a "side one and side two" listing of the tracks, which he recorded in a low-fi studio in a couple of weeks.

Costello and White are looking back for a lot of reasons, and I suspect one of them is a hankering for a time when music mattered, especially live music. The digital economics of the music industry are a mess, but it's also turned music into an solitary ibudded experience. It's a shame, because seeing White belt-out guitar solos, hearing Costello scream The Beatles' "Hey Bulldog", it's what it's all about.

LSD RIP

You may not have heard of Albert Hoffmann, but without him Lennon & McCartney would never have written Lucy in the Sky with Diamonds, Timothy Leary would be unknown, and a lot of bad art and incomprehensible poetry would never have seen the light of day.

Hoffmann was the chemist who first synthesised LSD, in 1943, and died this week at age 102, suggesting acid may not have the life-threatening impact we'd imagined. He didn't just discover the drug, he was also the first to (accidentally) ingest it. The resulting trip led him to believe that acid could help with psychosis and an understanding of how the brain works, all of which echoed Freud's earlier hopes about cocaine. Both got it exactly wrong, but if you've ever read Aldous Huxley's Doors of Perception or anything about Synesthesia you wonder about how much the world you and I perceive is anything like the world others see, hear, or smell, and how fluctuating and subjective things are.

Snap on Schmap

My HDR photographs just get worse, but Schmap.com picked one of my pictures from a trip to the Boston Harbor National Park for its guide to Boston.

HDR Photography

I started using Flickr last year and while cruising photographs of my home town Newcastle Upon Tyne came across some startling pictures of the quayside. The detail in the pictures was amazing; rich blue sky with tendrilled vapor clouds, and yet the details in the shadows of buildings was just as rich, giving the impression you could see every individual brick.

It turned out they'd been shot using high dynamic range imaging, something that's been around for a few years but somehow I'd missed. There's a bunch of different groups on Flickr: the main HDR group has over 20,000 members and more than 120,000 pictures. There's many others, and this group has a collection of winning pictures voted on by Flickr members. Some landscapes can we startling, like this picture taken in Germany, others looks entirely unreal.

Pete Carr has a great beginners guide on HDR that uses Photomatix software to create the HDR image and do the tonemapping. Ryan McGinnis has a nice post too. There's other software you can use - Panotools has a nice wiki summary. I downloaded the free trial of Photomatix and have produced a couple of images - terrible, but you can see the potential. I've set my Nikon D50 to bracket at +/-2EV and used the three images to generate the HDR. I also had to lock the ISO rating at 200 (the lowest setting), and now realise I'll have to work in apeture priority mode to get around depth-of-field issues. Even at ISO 200 I can see noise artifacts, and Photomatix can quickly generate its own artifacts as you adjust settings. But even at default settings the resulting image has amazing tonal depth.

If anyone has advice on software, techniques or how to get the best results with HDR let me know.

PR for PR

Just got back from Puerto Rico - it was awesome. We stayed with friends at the Copamarina Resort near Guanica, in the south of the island. You can read my review here, and see pictures at Flickr.

Citizenship

Although I was born in England, I've lived most of my adult life in the US, but never as a citizen: I'm a Legal Permanent Resident with a 'green card', married to a US national. Last week, after a decade of internal debate, I made the decision to start the citizenship process.

Why? You can blame Obama, Clinton, McCain and the rest - the primaries have mostly infuriated me, but they've also made me realize that I can't be a passive critic anymore. And last weekend I found myself outside the local post office encouraging people to vote for a tax override to fund the school system - and I couldn't even vote myself. And my eldest daughter has dual citizenship, but my youngest is just a Yank. And so on...

So I visited the INS website and downloaded the mammoth forms I need to complete. It all reminded me of my original application for permanant resident status over 20 years ago: I was surprised they still asked "if you are, or have ever been, a member of a communist party." Seem a little anachronistic to you?

I also discovered I need to get a move on. I want to vote in the election this year, but I've been told the process can take 6 months to complete. And cost $600. And I have to pass a test.

I'll keep you posted on my progress - and let me know if you've experienced this firsthand.

Heathrow Sucks

I'm traveling in the UK this week and despite my best attempts I'm flying through Heathrow thrice.

I agree with many others - Heathrow pretty much sucks. Too big, too busy, and too disorganized. Even the bits that work well, like the train shuttle to Paddington in central London, doesn't really make traveling easier. Paddington connects with the London Underground, where you quickly grind to a dimly-lit halt somewhere short of Kings Cross.

I'm so glad I don't commute in London anymore.

Murder Plots

Last entry I ranted about the hapless New York Times' reports on the murder rates among Iraq veterans, and about the overall murder rate in the US.

All of which brought me to the NationMaster website, which collects national statistics on everything from population density to levels of happiness. On the murder front, they rank the US at #24, far and away the leader among western democracies. They also show what correlates with the national murder rates of all countries, implying that abortion, oil wealth, and early mortality are all associated with murder. Now pointing out that early mortality is linked to murder might be the same as pointing out that a lack of food is related to starvation, but the rest is worth review.

Correlation is not causation, of course. To say that murder and oil wealth are positively correlated, simply means that they vary together: as overall oil wealth rises, so often does the murder rate. There’s no way to know if either one causes the other to occur, or if there is actually no causal relationship. For example, it’s entirely possible that some underlying third variable is ‘causing’ both these variables to change. And there’s no guarantee that the correlation infers a linear relationship between the two variables – they could easily have a much more complex relationship. Isolating two things and seeing how they work together is interesting but usually artificial, and ignores the complexity of real life.

Then there’s the whole problem of measurement. How do you get a uniform reporting of “murder” across nations as diverse as India, Columbia and France? And Abortion? Looking at these correlations, you begin to wonder how much is masked, and how much measures of abortion, education, life expectancy, and female literacy are surrogates for a measure of oppression, poverty, and division, things far harder to quantify.

Dead Data

I usually find myself sympathetic to the editorial content of The New York Times (The paper's politics are only slight left of dead-center, but in the US this passes for radical), so when I read the recent series of articles about how Iraq war veterans are committing killings I was initially shocked.

The series began will detailed counts of recorded killings committed by Iraq veterans who'd returned to the US, and many of them were profoundly tragic. But as the articles wore on, I kept asking myself -- what evidence suggests that the rate of fatal crimes among returning military is any worse that other groups in the US? Our fair-minded friends at MAF ran the numbers, and found the murder rate by Vets is lower than the national average by a wide margin. The Times had narrowly discussed the Vet murder rate in terms of year-on-year growth, an alarming number for sure, but hardly noteworthy when compared to the frenzied killings committed year after year by the general public.

The Right has rightly raked the NYTs over the political coals. Too bad, because the bigger issue the NYT raised is the terrible treatment many vets receive on their return to civilian life. We've all seen the reports about poor medical treatment, but the care of mentally scared Vets is no better.

Even worse, no real analysis is offered of how absurdly high the US murder rate is. In the international league or murderers, the US ranks 24th (for the curious, Columbia, Jamaica and South Africa top the list). Mexico and Russia offer good competition, but the next nearest first-world, western democracy in #33 Portugal with a murder rate almost half that of the United States. All this is an old saw debate, but it's also a national insanity that reasonable gun control laws can't be passed. The aforelinked NationMaster website also has a bunch of stats on what correlates with murder rates - wonderful and potentially misleading stuff that I'll look at later.

HNY

My company, probably for tax reasons rather than anything altruistic, shuts down over Christmas week, so I enjoyed an enforced but welcome break. I took the opportunity to read, and got lucky with two books – Ian McEwan’s Atonement, which came out in 2001 and won all sorts of awards, and Cormac McCarthy’s The Road, which was similarly lauded in 2006.

Two very good and very different books – John Updike said that Atonement could not have been written by an American, and the reverse could be said of The Road – it felt absolutely American in style, tone, and narrative.

The Road, a post-apocalyptic story that defines ‘bleak’, may not be the kind of book to read at Christmas by the fire, but it has stayed with me. A lot's been said about McCarthy’s style, which is perfectly suited to the terminal, desolate world he creates. For no reason I can think of it reminded me of Steinbeck.

Atonement has all the trappings, at least initially, of a classic English novel: the old mansion house, an off-kilter family of wealth and connection, the extended cast of servants, cousins, village folk. McEwan keeps the romantic story moving, but the novel slowly departs from the E.M, Forster-ish origin, as stories become embedded in stories. The way he decomposes the whole novel-narrative felt a bit like the way Fowles worked in The French Lieutenant's Woman, but his redemptive intentions are very different.

Both great reads. Happy New Year.