Audience research is bullshit

A perennial topic on this blog is research. In my interview last week with former NBC President, Warren Littlefield he discussed it and the flaws in the system as he saw them. I received an interesting Friday Question though, right after the election, supporting the notion of research and seeking my response.  (I'm also trying to answer a few more of these questions and catch up a little.)   Here’s his Q and my A.

It’s from Hecky:

Given the success of sabremetrics and deft statistical analysis in both sports and now politics (Nate Silver corrected predicted everything but the senate race in North Dakota), how can you be so opposed to research testing in principle when it comes to entertainment? Certainly it's true that a lot of research is done poorly (e.g. bad methodology, unwarranted conclusions/inferences, sloppy handling of the data, etc.). The companies doing it for profit don't make their methods publicly available, so who knows if what they're doing is any good. But I don't think that justifies a wholesale rejection of the entire enterprise. Maybe we just haven't seen a Nate Silver of Nielsen yet.

All this stuff about "going with your gut" and just finding "great" material and having "vision" -- unquantifiable rules of thumb -- strikes me as complete hooey. It's the exact same sort of dogma that got so deliciously panned in "Moneyball" and in all the election post-mortems about FOX News predictions over the last two days. When done right, statistical research methods work, and it doesn't really matter what's being analyzed. It could be baseball, TV, the stock market, or politics. TV is about making money by generating ratings. And I don't see why we shouldn't expect proper research to aid in achieving that goal. It's just a matter of figuring out the right parameters by which to measure the performance of one's algorithm.

Thank you for your question, Hecky.  Let me first say this: in 1974 I worked in the NBC research department.  My educational background emphasized math.  I appreciate the value of statistics and have seen the process of audience testing first hand from both sides -- as the network and as a producer.   Okay -- that's my disclaimer.  Here's my answer: 

How do you measure art, Hecky? How do you assign a numeric value to creative endeavors? Yes, you can predict who will win an election. It’s simple. People tell you they’ll vote for candidate A or candidate B and you put a check in the appropriate column. If you’ve asked the right people, if you’ve asked a large enough sample of people, and they’re truthful then you can make a prediction with relative assurance (always taking into account a margin for error).

When you’re analyzing baseball players there are intangibles but their ultimate value can be determined by performance. How many hits in how many at bats? Strikes vs. balls? How many stolen bases and how many caught stealings? They're all numbers -- numbers that don't lie.  MONEYBALL found statistics that were overlooked. They discovered undervalued players. And in MONEYBALL, these statistics were used merely as one form of input. Scouting and intangibles were still taken into account, just not to the same extent.  And the advantage the Oakland A's had was that no one else knew these statistics, which gave them a competitive edge.  Today every team knows those same formulas.  So you better have someone with an eye for talent to go along with the computer readouts.

But turning to entertainment --

When a joke doesn’t get a laugh, is it because the writer isn’t good, the actor didn’t deliver the line well, the audience doesn’t like that actor, the audience doesn’t like the situation, the audience doesn’t understand the joke, the audience is tired because it’s late at night, the air conditioning isn’t working, they’ve heard a similar joke, they didn’t hear the joke correctly, they’re biased against jokes of that topic, they were distracted by something else going on on the set, a camera blocked their view, they were pre-occupied by problems at home, or any combination of the above? Plus, the audience you’re testing has little dials and is asked to twist them to the right or left depending on how much they liked said joke – what’s the standard? Two people may find the joke equally funny but one person gives it a +4 and the other gives it a +7.  Is one guy overly generous or is the other overly tough? 

So when a test audience is watching your show and that joke comes on the screen and a line on a graph determines how funny it supposedly is – how accurate do you think that is? And how helpful is that number in determining why the joke didn’t rate higher?

Okay, let’s say you ask each audience member why he didn’t laugh at the joke. Here’s the answer you’re going to get most of the time: it wasn’t funny. Yeah, we know it wasn’t funny. Why? You think they can tell you? I’ve watched focus groups where people didn’t like characters because of their shoes.

On the other hand, you poll a bunch of people on who they plan to vote for they can tell you. And if you ask why they can generally give you an answer. They like his tax plan. They think the other guy isn’t a friend of Israel’s. They always vote along party lines. Their reasons aren’t subconscious. When you laugh at a joke, when you hear a new band, when you see a certain painting how often can you accurately define and articulate what you like about it and to what extent? And then digitize it.

That’s what program research attempts to do. It takes your show and breaks it down into which characters the audience thought they liked, which jokes they thought they liked, and based on that – how popular the show might be.

There is one statistic I would love to see. It’s also the one statistic these audience research firms won’t show you. HOW MANY TIMES HAVE YOU BEEN WRONG?

Since the failure rate on television shows is over 90% and these were the shows that all tested well, my guess is that number they’re keeping from us is also well into the 90 percentile. So Hecky, I disagree with your theory that testing works. It doesn’t always.

Now do the math. If something doesn’t work 90+% of the time why keep doing it?

Nat Silver’s numbers worked. His information was accurate. Karl Rove’s was not. And neither was the research that said SEINFELD was a bomb and THE PLAYBOY CLUB was going to be a breakout hit.

So the answer here is not to put too much stock in audience research. It’s too flawed. As Mr. Littlefield said, any show with new ideas, hard-to-categorize premises or execution test poorly. But show Mother Teresa assisting orphans and it will test through the roof.What would you rather watch -- that or BREAKING BAD?   Guess which of those two shows the research company would recommend. 

And yet the networks make programming decisions based almost SOLEY on this flawed information. And that’s my big beef. So when a network president “goes with his guy” and discards research for what he believes is a good show, I say that’s just as valid or more valid an indicator of whether a show will succeed. And a whole lot cheaper.

I could see political strategists going to Romney and saying you need to appeal more to women and minorities. I can’t see advisors telling Picasso he needs more blue, or telling Shakespeare that 64.6% of playgoers don’t like Hamlet because he’s indecisive.

Read More

Why are some shows cancelled so soon?

It's the Fall Season.  Some new shows have already been given the quick hook (sorry MADE IN JERSEY).  I Got a Friday Question about that that has turned into its own post/rant. 

Joseph Scarbrough posed it:

In the old days, a network would wait until the end of the inaugural run of a new series to determine its ratings, and whether or not the series would be renewed (in other words, if a first season was successful, that would at least ensure a second season); nowadays, it's as if networks feel if seemingly nobody is watching after two episodes, they'll go ahead and cancel. Why? That seems like a waste of everybody's time (especially those who made the show), not to mention, how do they figure a few episodes of a brand new show is going to be enough to bring in huge ratings? That's not even enough time for an audience to invest in the series, let alone build up an audience for it.

First off, it’s always been like that. There was a show in the early ‘70s called TURN ON that was cancelled after one airing.  Some Heather Graham thing, EMILY'S REASONS WHY NOT received the same fate six years ago.  There was a show on the NBC schedule in the late ‘70s called SNIP that the network ultimately never even aired – despite my partner and I almost being hired to write one. (Or maybe that’s why it cancelled.)

With so many viewing options these days networks feel they don’t have the luxury of time anymore. Especially on shows they fall out of love with or never believed in in the first place. (Why would they put on a show they don’t believe in? Commitments, testing, it has a piece of desired talent, they have nothing better.) The second the network feels a show is dead they now yank it. Why throw good money after bad?

If, however, the network believes in the show they will tend to stick with it. We’re seeing that this year with THE MINDY PROJECT, BEN AND KATE, and THE NEW NORMAL.

Will they be rewarded?  Sometimes yes, you get CHEERS.  Sometimes no.  How many chances did CHUCK get?   (Now CHUCK fans may be up in arms, but the truth is it never found a large enough audience.)  

The other thing that makes it difficult to determine what is a hit is that today’s definition is so cloudy. The numbers shows like BEN AND KATE are getting are awful. But do they hold onto enough the desired demo? Do they tonally fit into the night? Are they critical darling so there’s some prestige value in staying with them despite bleak ratings? Look, the bottom line is if a network wants to keep something it can justify it fifteen different ways; if it doesn’t it can find fifteen equally compelling arguments to cancel it.

Remember too, we're now in November Sweeps.  Although they aren't as crucial as they once were, they still are important so networks tend to shuttle shows off the schedule they know will pull them down.

And then there are the unfortunate series that are cancelled simply because they're collateral damage.   

In 1996 our show ALMOST PERFECT was renewed but more because of business considerations. We were given a dreadful time slot – right behind a new Bochco comedy (that's right -- comedy) called PUBLIC MORALS. When another new show on the schedule wasn’t ready to premiere they put us in a different time slot to start the season and we won our time slot both weeks. Then they went to the normal schedule and we were behind the debuting PUBLIC MORALS.

The critics despised PUBLIC MORALS. The reviews were savage. (By the way, they weren’t wrong.) CBS had been taking shit all summer for this show.

The overnights for their premiere were abysmal. And they pulled our numbers way down too. Les Moonves cancelled us both. Just like that. Why did we get the axe? Because he decided to drop the whole notion of comedies in that hour. He threw on a drama instead and there was no room for us. Les told me later that “ALMOST PERFECT was the best show he ever cancelled.” On the one hand that's gratifying to hear, but on the other it's like the girl you were most in love with saying to you, “You’re the greatest guy I never slept with.”

But ALMOST PERFECT was an inherited show (developed and put on the schedule by the previous regime) and not owned by the network. Today, if CBS owned the show and there were already 34 episodes in the can they would find places to keep it on their schedule.

Still, at the end of the day – that’s television. As they said in GODFATHER II: “This is the business we chose.”

Read More

R.I.P. Lucille Bliss

Lucille Bliss, the first voice of Crusader Rabbit passed away this week.  She was 96.  She had a long career, did many cartoon voices including several for the Smurfs.  But for me as an impressionable dorky kid growing up, Crusader Rabbit was my favorite -- I suppose because of all the clever puns and wordplay.  So the voice was as important as the picture.  I never met Miss Bliss, but I understand she was a wonderful person.  She provided joy to many people.  There is no joy tonight in Galahad Glen. 

Read More

Sitcom Room 6

SITCOM ROOM 6 was held this weekend at the birthplace of comedy -- the Custom Hotel in Marina Del Rey (just a mile from LAX where Denzel Washington lands planes upside down). For two days twenty attendees experienced what it’s like to be on staff of a sitcom. As you read this they're still sleeping.  

Two came from Australia and two came from Alabama.  They all had the same accent.  

One of the reason we moved from the LAX Hilton to the chic and stylish Custom Hotel (walking distance to a Ralph's market and bowling alley) is because last year the Hilton had an Anime convention during our weekend and the place was a friggin' zoo.  So what do I encounter in the lobby of the more sedate and exclusive Custom on Saturday night?  

Four terrific comic actors -- Andy Goldberg, Wendy Cutler, Harry Murphy, and Annie Abrahams -- performed a scene that even Ed Wood would say needed work.   The scribes were broken into four teams and all four rewrote the scene Saturday night (and Sunday morning).  Later that day the actors returned and performed their scenes.  The difference was like Billy Carter and Jimmy Carter.  

Later we had a panel discussion that featured David Isaacs, Phoef Sutton (CHEERS, BOSTON LEGAL, TERRIERS), Robin Schiff (ROMY & MICHELE'S HIGH SCHOOL REUNION), and Dan O'Shannon (CHEERS, FRASIER, MODERN FAMILY). I played James Lipton asking pretentious questions and fawning all over them.  

Among the things the scribes learned over the weekend were:

You can't be possibly be funny if you eat carrots..

There’s always a way to solve story problems.  In my case, that's what David is for.  

How do you know if an idea is any good?  You don't really.  But Robin Schiff has an ingenious approach.  She says to herself,  "IF this were a good idea, what would I do with it?"  

How different is writing comedy and drama?

The pros and cons of gangbanging.

Servicing actors (not to be confused with gangbanging).

Jokes are easy. Stories are hard.

It takes two to tango... at least.

Norm stories on CHEERS were particularly hard.

The importance of formatting.

Single camera vs. multi-camera: what's better?  


Savers.



The difference between men and women (besides that).

When you’re finished eating the take-out Chinese food throw everything out. Immediately!

What shows to write for your spec. 

What shows not to write for your spec. (hint: I hope you’re not too far along on that EMILY OWENS M.D. script)

Runthrough etiquette.

Robin shared the secret for a long career in comedy writing.

The Volkswagen test.


How to handle network notes.

The FRIENDS lawsuit.


The smart way to write a dumb character.

Why a comedy writer should never wear a toupee or Hawaiian shirt (not that anyone should).

Ways to fix troubled scenes.

Working with a partner.

How to run a room.

How to get a laugh without a joke.

What's funny at 5:30 in the morning. 

You don’t have to be the funniest person in the writing room to be the most valuable.


The book that explains comedy.


Should you send a spec of a show to that show?

Nothing goes better with Oreo Double-stuffs than beer.

The rule of threes.

The best book for learning how to be funny is Improv Comedy by Andy Goldberg.

… and finally -- 5-Hour Energy works!!! (Oh wait, it’s me who learned that.)


Thanks to Dan O’Day, David Isaacs, Robin Schiff, Dan O'Shannon, Andy Goldberg, Wendy Cutler, Harry Murphy, Annie Abrams, Cliff Levine, Jonathan Emerson, the housekeeping staff who had to clean the four writing rooms (you might want to just seal them off for two months), and especially the attendees for making Sitcom Room 6 such a great experience.

Get on our mailing list for next time.

A few of those attendees may comment. I'm holding my breath. 

Read More

Social Network Rejection

Hello from the SITCOM ROOM where everyone is having way more fun than in Robert McKee's class.

Meanwhile, I've been subjected lately to the heartache and horror of “Social Network Rejection”. You might suffer this too. I’ll check my Twitter page and see two or three people have unsubscribed. And I wonder – hey, what did I do? Did saying something nice about Patty Heaton cause you to leave? Did you think to yourself, “Jesus, if I see one more goddamn post about CHEERS I’m going to scream”? After a couple of months did you just realize I'm not the Ken Levine who created BIOSHOCK? Did I misspell just too many words for you? What???

I accept the fact that readers come and go, but still – there’s a tiny part of me that wonders, “should I take this personally?”

Do you feel this way too?

Do you see that four friends have dropped you from Facebook and think, “Gee, sorry I’m not interesting enough for you.” Your next reaction is usually “screw you” but you still wonder… did they drop you but keep Heather who tweets every time she coughs up phlegm?

I mean, face it, we all drop other people. If someone wants me to join 72 causes I usually dump them by cause 20. I used to follow a certain baseball beat writer because I like his writing. Then I started getting a hundred updates a day on the Kansas City Royals. I couldn’t unsubscribe fast enough.

People drop former romantic partners; that I understand. But other people FIND OUT they’re being a dropped by reading Facebook. When your girlfriend’s status goes from “in a relationship” to “single”, well, there’s now 51 Ways to Leave Your Lover.

The point is, there’s usually a reason. Enough of a reason that you’re willing to seek out the unsubscribe icon and click on it. Not saying the people you purge from your life are not lovely but there’s something about them that clearly bugs the shit out of you.

The other, even greater personal rejection, is when you ask to befriend someone – someone you KNOW, someone who IS a friend in real life – and they ignore or reject you. You’re not asking them to donate a kidney, you’re asking to add your little picture icon to their friends page. This can be particularly painful when members of your immediate family dump your sorry ass. 

I'm in a weird Facebook situation in that they say I have too many friends and I can't add any more without dropping people.  Well, I don't want to just drop people.  Again, what did they do other than show interest in me?    I'm told I can start a "fan" page but come on.  I'm a writer, not an actor.  Isn't it a little egotistical to start a "fan" page for yourself? 

Maybe I’m just being too sensitive. I’d start a Facebook Group – “Coping with Social Network Rejection” but what if no one signs up?

And don't even get me started on "LIKE"...

Read More

How to be a good showrunner

Natalie Wood

Hello from the Custom Hotel where I am holding my annual SITCOM ROOM seminar this weekend.  Wish you could be here.  This is the type of stuff we discuss: 

The WGA recently held a seminar on “How to be a Good Showrunner”.  The answer: stay within budget, even if the budget is smaller than Napoleon Dynamite’s clothing allowance. You may think “shouldn’t the course be more about how best to realize your creative vision?” That would be NO. Because if you can’t keep the budget down you’re fired (even if you’re WEST WING'S Aaron Sorkin) and then there is no creative vision. Networks only give you creative freedom if you’re a runaway hit or they feel they can trust you to not give away the store. Abuse either of those and you’re back to getting memos questioning why the actors playing Klingons can’t bring their own wardrobe from home.

People ask me what’s it like to be a showrunner. I tell them “did you see the end of BONNIE & CLYDE?” It is a constant barrage of problems coming at you from all directions. It can be overwhelming…which is why I’d take the showrunner from any well run series and elect him or her the president of the United States.

Showrunners are responsible for the writing, casting, hiring, post production, promotion, dealing with network, censors, studio, non-writing pod producers, and the other 90% of time dealing with the actors.

You need to be a psychiatrist, an accountant, a CEO, a personnel manager, a Drill Sergeant, a Jewish mother, and work well under heavy medication. Once you’ve satisfied those requirements then you can add talent…but that’s optional.

To first time showrunners I would say this:

Hire the best possible support staff. Leave your ego at the door. Joe Torre is a great manager but it sure helped having Don Zimmer as his bench coach for years. Surround yourself with experienced people. You don’t have time to make rookie mistakes.

Be willing to delegate authority. You can’t do it all. You can’t be in four places at the same time. As Garry Marshall once said, “What good is all the money when you’re in Cedars?”

Be organized. You can’t plan far enough in advance. When you look at most TV show budgets, the overages come from indecision, midstream changes, scrambling last second to meet deadlines (double and triple overtime time), and confusion.  In other words: the Federal Government.

Be a cheerleader. With tight budgets you’re asking everybody to work harder, give more, eat less craft services. That only comes with good morale and that starts with you. Know everyone’s name and that includes the actors.

And this next point I can’t stress enough: DON’T TAKE CREDIT FOR EVERYTHING. Nothing will kill morale faster and nothing is more untrue than that. You’ll realize it bigtime when your staff has fled and you really DO have to do everything. Can you say “implode” boys and girls?

Finally, learn the fine art of bending over. You’re going to have to compromise. Pick your battles, prioritize what in the budget you really need and what is a luxury. On CHEERS one year we thought of a great gag that would require levitating Norm. The cost turned out more than the license fee of three episodes combined. We did a beer joke instead. And think in terms of the whole season not just one episode. If you’re using a lot of outside sets or special effects one week, plan on doing little or none the next.  One week HOMELAND was in the Middle East with exciting chase scenes and explosions.  A couple of weeks ago most of the show was set in an interrogation room.  Much cheaper.  And by the way, the interrogation episode was better. 

This is ultimately what I learned. At the beginning of the season the network and studio is wary of every showrunner. Be fiscally responsible right from the get-go. Because other shows won’t. And soon the suits and bean counters will leave you alone because who knows what the fuck Dan Harmon is up to? 

Just remember, creative freedom comes not at a high price but a low one.

Note:  In case you're wondering, whenever I can't find an appropriate photo I post a picture of Natalie Wood.  

Read More

More stuff you wanted to know

Here are some Friday Questions as I prepare for this year’s SITCOM ROOM seminar, which begins tomorrow. Looking forward to seeing you lucky attendees and causing you sleep deprivation for the next two days. But on to the business at hand. The answer to this one is the shortest I’ve ever given to any Friday Question.

It comes from Steve Murray:

When writing a script, how do you best express an incomplete sentence (interrupted)?

…

More Friday Questions next week. Leave yours in the comments section.










Huh? Yeah, okay.  Maybe I should answer a few more.

Stephen wonders:

Many shows experience a change in show-runners, COMMUNITY being a recent example. Does the network alone decide on the replacement? Do they consult with existing producers on the show? How do they lure the potential replacement without saying, "Hey, here's a show with so-so ratings that's already been on for a few seasons, with a fanbase that was fiercely loyal to the old guy ... wanna take over?"

Most of the time when a showrunner steps down it’s his own choice. He’s burned out after a number of years of 100 hour weeks. Usually there’s a transition period where he’ll train his successor and start doling out more responsibilities. And when the new showrunner takes over the original will still be available to oversee and consult.

My post tomorrow will be all about what it takes to be a good showrunner.

There have been instances when a showrunner is fired. Actually, it happens more than you think. There are the high visibility firings (Dan Harmon off COMMUNITY, Aaron Sorkin off WEST WING), but also creators who get booted off their shows even before the shows air.  This now occurs once or twice a season. 

When you ask whether it’s the network or studio’s decision, in most cases these days the network and the studio are the same.

So why do these showrunners get canned? I go into this more tomorrow but a big reason is the inability to stay within budget. Creative differences are another – and by creative differences I mean the showrunner won’t just bow to the network and do every note. If the showrunner becomes too difficult to work with and the show is not in the top three he’ll often be asked to step down and spend more time with his family.

And finally, there are occasions when the network looks at the first few episodes, decides they’re shit, and puts the blame on the showrunner.

How hard is it to get a new showrunner? Not hard at all. There are plenty of really good writers who are out of work or just on staff of someone else’s show and would love the opportunity to get back in the driver's seat. Think of baseball managers.

To me the problem comes when networks don’t get the writer/producer who is the best fit for the particular project. Instead they get whatever writer/producer they happen to have under contract.

Carl Tyler is up next.

I am not in the entertainment industry so I have a question for you. Have you ever written a script so that you could screw people up who were playing drinking games? Say for example, you had to do a shot every time someone said Sam on cheers, or every time someone suggested a a disease ended in -osis on House. Have you ever worked in a word knowing that people do these silly games?

Okay, second shortest answer to any Friday Question: No.

From MacGilroy:

Whenever I see a list of "classic sitcoms" that include somewhat recent shows such as "Seinfeld," I wonder what you thought of "Mad About You." Maybe I'm in the minority, but that's always on my list and I rarely see it mentioned. Was that not critically respected? Did it have an attitude or style that didn't click for you? Behind the scenes issues? Seriously curious.

I very much liked MAD ABOUT YOU at first. I thought the writing was smart and Helen Hunt was endearing.  My problem, and I’ll be 1000% honest, is that Paul Reiser ruined the show for me. He never shut the fuck up. He became so annoying that I just couldn’t take it anymore. I have no way of knowing whether the show got better or worse the last few years because I couldn’t watch a second of it.

I don’t think NBC ever really loved the show either. Their perception wasn’t that it was a signature prestige franchise like CHEERS or SEINFELD but that it was something reliable they could plug into the schedule to fill holes. It’s time slot changed almost every week. There’s no way to build a loyal following when you never know when a show is on. And the network knows that.

Kristen wants to know…

At the beginning of a season, was the whole story plot more or less laid out or did it tend to evolve depending on how the audience reacted to each episode. For example, before starting season one of Cheers, was it already decide that Sam and Diane would get together at the end or not?

The first year of CHEERS we really experimented and we let the audience tell us what they liked and didn’t. The Sam & Diane relationship was popular right from the moment Diane first walked into the bar in the pilot. But it was director Jimmy Burrows who came up to the office several weeks in and said, “Sam & Diane -- that's your show.”

We tried to get as much sexual tension mileage as possible that first season, but at a certain point if they didn’t get together it was a little juvenile. So the Charles Brothers decided to make that happen as the season finale.

After that some thought was given each season to a general overall arc. And to my knowledge CHEERS was the first sitcom to do that.

And finally, from Barbara C.:

I always find the dubbing out of curse words interesting. It's obvious that in the original scene different dialogue was used. Do the actors have to re-dub certain scenes during the movie making process for when the show eventually airs on a network television?

In many cases separate takes are filmed in movies to substitute more acceptable words for broadcast television. God, wouldn’t you love to see an episode of DEADWOOD that was cleaned up for general audiences?

Read More