Wednesday, July 15, 2009

Working with news publishers

(Cross-posted from the European Public Policy Blog)

Last week, a group of newspaper and magazine publishers signed a declaration stating that "Universal access to websites does not necessarily mean access at no cost," and that they "no longer wish to be forced to give away property without having granted permission."

We agree, and that's how things stand today. The truth is that news publishers, like all other content owners, are in complete control when it comes not only to what content they make available on the web, but also who can access it and at what price. This is the very backbone of the web -- there are many confidential company web sites, university databases, and private files of individuals that cannot be accessed through search engines. If they could, the web would be much less useful.

For more than a decade, search engines have routinely checked for permissions before fetching pages from a web site. Millions of webmasters around the world, including news publishers, use a technical standard known as the Robots Exclusion Protocol (REP) to tell search engines whether or not their sites, or even just a particular web page, can be crawled. Webmasters who do not wish their sites to be indexed can and do use the following two lines to deny permission:

User-agent: *
Disallow: /

If a webmaster wants to stop us from indexing a specific page, he or she can do so by adding '<meta name="googlebot" content="noindex">' to the page. In short, if you don't want to show up in Google search results, it doesn't require more than one or two lines of code. And REP isn't specific to Google; all major search engines honor its commands. We're continuing to talk with the news industry -- and other web publishers -- to develop even more granular ways for them to instruct us on how to use their content. For example, publishers whose material goes into a paid archive after a set period of time can add a simple unavailable_after specification on a page, telling search engines to remove that page from their indexes after a certain date.

Today, more than 25,000 news organizations across the globe make their content available in Google News and other web search engines. They do so because they want their work to be found and read -- Google delivers more than a billion consumer visits to newspaper web sites each month. These visits offer the publishers a business opportunity, the chance to hook a reader with compelling content, to make money with advertisements or to offer online subscriptions. If at any point a web publisher feels as though we're not delivering value to them and wants us to stop indexing their content, they're able to do so quickly and effectively.

Some proposals we've seen from news publishers are well-intentioned, but would fundamentally change -- for the worse -- the way the web works. Our guiding principle is that whatever technical standards we introduce must work for the whole web (big publishers and small), not just for one subset or field. There's a simple reason behind this. The Internet has opened up enormous possibilities for education, learning, and commerce so it's important that search engines makes it easy for those who want to share their content to do so -- while also providing robust controls for those who want to limit access.

Update on 7/20/2009: The word "crawling" in the fourth paragraph has been replaced with "indexing."


Dan Thies said...

Josh, you probably want it to say:

User-agent: *
Disallow: /

The post has the * on the same line as the Disallow directive... maybe Googlebot can sort that out but I wouldn't count on it.

David said...

This is a very nice way of saying, "You guys in the newspaper industry are dingbats. If you don't want us to send you traffic, we wont. Stop complaining."

Paul said...

Wasn'this just another example of Newspapers just seeking "Press" for their views? Obviously it wasn't an attempt to actually solve a problem, or they would have asked their own Web folks if they could do this....

Ironic, isn't it? Newspapers just trying to get a bit of Press?

ratecityhall said...

Publishers of news content want as many people as possible to come to their web sites and read their stories. Most of these publishers also offer advertising on their sites. They know how to remove content from search indexes, but also know the negative effect this would have on their page hits and ad rates, since less people will be reading their stories.
Publishers should be focusing on how to make their sites better, which would increase their popularity and ranking in Google, resulting in increased readership and ad rates. They know this, and I would bet any newspaper that takes their content off Google would quickly come crawling back after realizing their mistake.

David said...

@ratecityhall: You are making a blanket statement based on logic, that does not agree with the stand that the newspaper industry has been taking.

I think the real problem is that the newspapers are not making money directly anymore and are asking to get paid more. Since they don't understand the open nature of the internet, and how to create a working business model for it, they want to take their toys and go home. This article simply instructs them in how to do that.

Todd said...

Sounds like the the newpaper industry does not understand how to build revenue streams in the Web 2.0 era. Are they really so stuck in a subscription-based model that they cannot see the flexibility of the internet medium and how they can use it to drive up their readership and in-turn, revenues?

Not to blanket the entire industry, but they seem to be stuck in the 90s.

iammikecohen said...

If I can reference a recent South Park episode this is the News Papers telling everyone they want their share of "that internet money"

Mike said...

Good response.

I think what the newspapers really want is a check from Google. When they say, "Hey, you guys are making money off of our content," there is a silent "and we want you to pay us for it" at the end. They know that their prospects of monetizing from advertising and increasing subscriptions are not good and are looking to introduce a new revenue stream.

Unfortunately for the papers, there are lots more sources of content out there, and unless they offer unique analysis or breaking news, it is unlikely the content is worth much. Google is very politely telling them to take the traffic or leave it, but that is all they are going to get.

Durisseau said...

Sad how none of you have mentioned copyright, which is what all of these stories that Google is crawling, then doing something with at no cost. Additionally, Google is also controlling whether or not they show up in a search, which would hurt anyone that wants the exposure, not just newspapers. What if you were a column writer that wanted to get your name out there...or an artist?

Christopher said...

@durisseau: nobody is talking about copyright because there's nothing to say. Google is instructing a segment of the industry about how to preserve their rights, period. Links and excerpts of articles are not infringing activities... cite source if you find otherwise.

Copyright, in general, is an abused tool that typically cries out for misuse if only to balance the scales.

Bettawrekonize said...

These newspapers need to understand that you can't force google to use their services and then force them to pay. That's extortion.

William Yu said...

Okay, let's put it this way. problem = opportunity = money.

The Internet and search engines are fine the way it is with REP...etc. However, Newspapers want a share of THAT "internet money" without having to do any work or use their brain to come up with something new and novel.

What do they do? Create an artificial problem by pretending they know nothing about the current Internet technology. Create another standard that's inferior to what we have right now. Whine to create pressure to make people use them.

Then everyone will have to PAY THEM to NOT USE that sh*t standard.

Business model or extortion? You tell me.

Nick Van Dyke said...

@ratecityhall Newspapers are adored by the search engines and have incredible power and influence with them. They don't have problems ranking well. They do however have problems monetizing that traffic in the "Information is Free" internet world we live in.

@david and @todd You guys are right. The search engines are stuck in the old subscription business model way of doing things. At the heart of this issue is that no one has figured out how to monetize a news site.

@mike You're crazy if you think that newspapers aren't bringing anything to the table unless it's breaking news. Newspapers have essentially been paying for the journalism degrees of most reporters out there. If newspapers don't have the money to hire reporters, who bothers to get that degree? If they don't get that degree who knows how to do the investigative journalism, who knows how to write the fair and balanced piece, who has the contacts to call for a quote or some info when they're trying to write a piece? Newspapers bring a lot to the table and yeah they're not getting any money off of it. should they be? Probably. Who is going to pay for it? I don't know.

@Durisseau This article is essentially about copyright. If the newspapers want to protect their copyrighted material then follow the instructions. The problem the newspapers have is that if they do that they lose their rankings and traffic. But you're right, this absolutely is an issue for the small guy too. If I only make a couple bucks off my blog how can I afford to really spend the time tracking down the information I need to write a deep article? If I want to charge people to read my content, I think it's possible to still rank well (I might be wrong on that) but the bigger issue is that nobody would pay for it. It's not a viable business opportunity yet.

The argument needs to move away from how stupid the newspapers are for attempting to maintain the status quo towards how can the newspapers and writers justifiably receive fair wages for their ability/services.

Kevin Kennedy said...

Interesting article. I tackle some of the same issues as it impacts business newspapers in “Business newspapers sharpen focus on the future.”

I think the winning formula is free content to get potential readers to the site and premium content that the visitor can also read, but only if they purchase it. So you get the eyeballs, but also the opportunity upsell those eyeballs.

Aerel said...

@Nick Van Dyke: I don't know where you can get your journalism degree paid for by a newspaper because they certainly didn't pay for the four years I spent getting mine. After graduating I didn't even bother looking for work in the industry because the wages are ridiculous for the amount of work and crap you have to put up with - both my internships taught me that. Let's face it, the only people who become journalist do so because they love the work. They sure as heck don't do it for the money. Newspaper have a bottom line and it's always in dollar signs. Not in integrity, not in reader interest or public good - only what will sell and what will save them money.

Newspapers have been hit hard by the economy and they've survived by laying off reporters that you seem to think they value so highly. They don't, trust me. From someone who had been working in the industry for thirteen years it was very clear to me that money is the most important thing to a newspaper, not the reporters.

That's why they want their "cut of the Internet money" and lay the blame elsewhere rather than spending time and money on developing a business model that makes sense in the 21st century. It's easier to say Google did it.

geo said...

Here's a better idea.

If you don't want to be on the internet then don't post your articles on the internet.

Post them on a little server with proprietary communications means and advertise it in your newspaper. See how much traffic you get then.

Hey no one told newspapers they needed to be on the internet.

Stick to your paper and physical distribution and others will break the stories on the internet and get your traffic.

Or is what you are REALLY proposing that you'd like to copyright facts themselves?

Do we have to go through 10 years of misapplication of copyright law prohibiting linking on a voluntary network designed around linking that other people now want to take over like we did before with patent law before a legal decision of the preposterousness of the whole thing finally put it back right?


Harold Fowler said...

Well Id say its safe to say you hit that nail square on the head!


Larry said...

You only get paid a fair wage for your ability if that ability generates revenue. Nobody owes newspapers or journalists a living.

And it isn't about copyright. Google doesn't publish the story; it directs the user to the site where the story is published.

Newspapers generated revenue through classified ads, a market which they have now lost. They need to come up with a new model - not extortion - or die.

David Gerard said...

Don't forget that Noam Cohen, speaking for the New York Times, tried to pull this crap on Wikipedia too: "So, in essence, many Wikipedia articles are another way that the work of news publications is quickly condensed and reused without compensation." I commented asking where to send the NYT a bill for journalists' use of Wikipedia. Maurice Jarre was unavailable for comment.

planetjeffy said...

Almost all newspapers have no idea of how to transition to the web. These sites should be the local hub for everything - from shopping to news to connecting. These could be the ultimate social media sites - and nobody else does local like a local newspaper...theoretically. I have yet to see any newspaper site get it right. Signonsandiego, where I live is as good as any and I rarely use the site. I'm sure they want to turn down as much traffic as they can to their lousy sites. Traffic doesn't do anything and they wouldn't know what to do with it anyway. In 10 years, there won't be single newspaper in existence.

HealthyFoods said...

the main problem is that the newspapers, like the music and movie industries, are still unable to accept the fact that the 'golden age' has come to an end. they believe they have some kind of god-given right to continue receiving revenue commensurate to what they were able to obtain during the print era, or in the case of the music industry the CD era.

unfortunately there is not likely to be *any* web era business model that generates the same kinds of revenue streams because the web has dramatically, and irrevocably changed peoples value perceptions of these products.

i for one will never pay a print subscription for an online newspaper account - ever. and most other people in their 20s feel the same way. if the newspapers try to enforce the kind of idiotic content restrictions and tiered subscriptions they are talking about, they will just see a massive drop in traffic to their sites without likely any corresponding increase in subscription revenue. i and people like me will simply stop coming round.

AdvertisingMiami said...

Newspapers complain that Google is raking in the dough that they should be getting. I for one, have never seen an ad on or any of the links. So how is Google monetizing this news content???

Mario Grgic said...

These guys will continue working hard to make themselves even more irrelevant than they already are, and there is no stopping them.

By the way, in the Web world, if the search engine doesn't see you, you don't exist.

On the one hand you have people paying search engines so they can be seen first, and on the other you have technological dinosaurs like news publishers begging not to be found. I hope the next extinction age comes fast.

Raymond Turney said...

The problem is that newspapers are in competition with the Internet, not Google. Most newspapers, most of the time, print wire service reports for local readers. They combine this with local stories. Since their distribution model is obsolete, it's not surprising that they are unhappy with what is making it obsolete.

As an internet user, the problem I see is that we haven't evolved a way to pay for the wire service without using the newspaper company as an intermediary. One answer is specialized news related sites, like the ones maintained by Le Monde Diplomatique and Stratfor. If they either expand their reporting by forming an internet equivalent to UPI, or by subscribing to UPI on their own, they could perform the function of newspapers at a lower distribution cost.

M said...

Newspapers have the unique advantage to offer a hybrid solution to both readers and advertisers ... we have been trying to get the newspaper business into keyword linking for 2 years now ... it's a revenue generating system that would provide more traffic than all search engines combined, while enhancing readership enjoyment.

Matt said...

@David Not all of us. Just the ones running the show. ;)

Alex said...

problem = opportunity = money -- that's what I read up there somewhere in the comments. So why don't newspapers actually wholeheartedly endorse the web way -- that is why don't they actually turn around and make some serious web brands rather than letting upstarts like Twitter, Facebook and hell even Google eat their piece of the pie?

Imagine this -- New York Times and the major urban pubs unite and create a Facebook competitor.. Each regular newspaper subscriber gets a free account to access their content online. It's fully open and available to search engines and uses clever tech to allow Twitter/FB/etc signons without having to reregister. The papers usurp all sorts of existing technologies and existing customer bases to attract an actual online audience to their work.

Then they create experiences with content and news like never before. A flash timeline of Michael Jacksons death is NOT premium content. What is premium is SOMEHOW getting access to the news as it happens in unprecendented ways. Who else *BUT* the existing media like the NYT have such access to rock stars, news, product launches, presidential scandals, etc?

We all want to be superheroes and traditional media is in a position to give it to us -- they just have to wholeheartedly endorse the web and what it offers, and then step it up a notch -- a real notch -- in terms of the experience. Something better than dead trees, finally, and much better than FB/Twitter/Google put together.

I really think they can do this, iff, they band together and actually collaborate. I am sure there are many tech people that would want to see standard media succeed. Their success will interplay with new media's success and vice versa. It's a virtuous relationship, if done right.

Ryan said...

I agree that newspapers need a new revenue model other than subscriptions. Unlike services such as those from Gale Group where institutions such as universities or libraries will pay for access, newspapers are read by individuals who are unlikely to pay a subscription fee for one newspaper, when another newspaper offers the same content for free. Unless newspapers can come up with "premium" features that are highly desirable to people, they are not going to be able to make money effectively.

For a newspaper to make money off of content they provide for free, I believe two techniques are needed (assuming they already get lots of traffic).

The first is how many other websites make money: ads. They should have a few strategically placed classified ads, or targeted ads from a service such as Google AdSense. It could potentially give them the answer to "what now?" after reading an article. Now, that would only account for a small portion of the time the user spends reading an article. Most of the time is spent reading an article, so what if that time spent reading could be turned into money?

The second is embedding a lightweight applet, such as that offered by Plura Processing, on pages. The applet would have no effect on the user experience, and there is no reason it would need to be visible on the page. In the case of the above company, it would carry out some lightweight computations, then send the result back to some central servers. For each set of computations users complete, the newspaper receives some amount of money.

The obvious benefit is that the newspaper would make money during the time a person spends reading an article. However, it would also encourage newspapers to write quality articles that engage the reader: the more people who read the entire article, the more money they make. Combined with a list of suggested follow-up articles, a users could spend a lot of time on the newspaper's website, earning them quite a bit of money.

frvfilms said...

they should use new protocols that instead of sending packets send whole parcels.

diamond ed said...

what a bunch of crybabies, nothing happens until someone sells something...the newspaper industry's idea of a sales force is a 12 passenger van full of high-school kids canvassing, door-to-door, making unsustainable "deals" with unsuspecting consumers, in a movable feast of mobile boiler-room practices that are usually beneath despicable.

Few people in fact have the ability to close a deal, and those with the gift, don't work cheap. dig deep and hire some high yield sales people and you won't have to behave like record companies.

Chris said...

Fox News disappear? News Limited disappear? What a shame!

Rudd-O said...

You guys are navelgazing here. The issue is simple: newspaper owners are leveraging their influence and power over government with lies, in effect threatening Googlers with putting them in cages or bankrupting them unless they do their bidding and use their equipment that belongs to them to give newspapers special favors. If you were wondering why this whole story is so repulsive and disgusting now you know -- it is the application of the old but infallible "do as we say or our friends here will fuck you up", told in super slow motion.

If I were Google, I would be much less diplomatic: I'd delist these fuckers TODAY. Google is not obligated to provide service to anyone with their property, much less extortionists in suits and ties.

diamond ed said...

Rudd-O-I'm with you the old standard of "If I get mine, before you get yours, you don't get yours", is dead and stinkin'. Musicians are rapidly coming to the conclusion that they can do a way better job of producing and recording and most importantly, promoting their products with style and grace, and totally eliminate the "suede shoe" element. The brick and mortar end of news and information gathering reminds me of the record companies. Running around, hair on fire, heads exploding, acting like a bunch of spiteful children while their business model tanks, and rather than embrace the new technology ad behavior of music lovers the world over, the attack, insult, embarrass and alienate the very heart and soul of their future. But don't get me started. I agree with you, that Google continue to speak softly, but get a little busier with the stick.

amanfromMars said...

"The truth is that news publishers, like all other content owners, are in complete control when it comes not only to what content they make available on the web, but also who can access it and at what price."

How very true. But moderation which removes or fails to show a submission can tell its own, much deeper tale.

The following two tales in support of Google's Long March obviously ruffled a few feathers and cracked open a few golden eggs/exposed a few cores.

How Google responds will determine whether it can fend off years of intensive investigation and—maybe even more important to Brin, Page, and Schmidt—maintain its heroic reputation.

What would be simpler than transferring everything to a foreign and alien position ……. Creating a Novel Virtual Identity with an Executive Base in a Virgin Jurisdiction beyond such Petty Tampering as would seek to exercise a Covetous Remote Control over an Intellectual Superior Power Play.

And it is the most stupid of arrogant ignorant power plays, to poke with a sharp stick/blunt instrument, a contented hornet’s nest. Whatever is Ms Varney thinking? Although it is most probable that she’s only following orders and being well paid to do so. It is after all, the American Way, is it not, ….. Thirty Pieces of Silver for a Few Dollars More?

And as everyone loves an Underdog Triumphant Tale, the chances of Google failing to render the Attack as a Bay of Pigs Copy Action, are Better than just Good and that will render US weakened and susceptible to Punitive Revenge Action against which they will have no Effective Response and thus will they be forced to have No Live Option other than to Surrender and Follow a Foreign Lead.


But to antitrust watchdogs, customer convenience is often less important than preserving a competitive landscape. Google can use the surplus cash from its search business to subsidize the development of new, free products and services. That’s a frightening prospect for would-be competitors without such a robust revenue stream—potentially scary enough to discourage them from entering the market.

The Bottom Line …… Google [Search] Algorithm better uses Information to produce its Own Beta Intelligence Product, a Product which surpasses the Government’s requirements which are mired in Legacy Control Systems.

And their cash generation engine is directly in competition with, and superior to, the Federal Reserve Fractional Reserve and Print Money for the Illusion of Wealth and Power Operations.

But the Problem is not Google’s, it is in the Lack of Integrity, embedded with Perverse Deceit in lesser Corrupted Power Systems.

Nice One, Eric, stick it to the evil swine, for they richly deserve it?

Thoughts which are common enough but which are an inconvenient truth for those destroying systems with their selfish abuse of them but a self evident opportunity to those into Creating NeuReal Systems .....

Genghis said...

Brilliant response. I assume News Crap , oops a typo, News Corp, already know they can do this. They have another agenda.