So, similar to yesterday we’re going to have our three morning speakers make their presentations, and then Dr. Michael Wolfson, who is the Assistant Chief Statistician, Analysis and Development at Statistics Canada, will moderate the question and discussion session, which will run from 11:30 to noon. And our first speaker today is Dr. Scott de Marchi, and he is going to provide the view from political science. Scott is an Associate Professor of Political Science at Duke University, and Scott’s research focuses on the fields of computational political economy and other mathematical methods, individual decision-making, the presidency, and public policy. At Duke, he teaches a class on the nature of freedom, and he’s written a book on the foundations of mathematical methods in the social sciences entitled Computation and Mathematical Modeling in the Social Sciences. So, I’d like to welcome Scott. Thanks.
Scott de Marchi:
Hi. I -- my training, like a lot of people, is a little bit idiosyncratic, so I’m not going to be able, like some of the speakers yesterday, to say a lot about political science directly. Most of my life was spent in a computer science department and I left, by and large, because computer science, and I think this is one of the signs of the decay of a discipline, turned to formalization, increasingly. There was a battle, and that’s not really overblown, between the people that wanted to solve problems and the people that wanted to work on math. And a lot of times those are intention. And yesterday, if you saw the talks, I think the same claims are more or less made by a lot of the speakers about current microeconomics in economics departments.
I switched, believe it or not, to history for a couple of years, and the same battle’s being fought but had already been lost. Most of you will notice that history departments are a little bit different than they were 20 or 30 years ago. And the people that wanted to study sort of, you know, wars and diplomacy were losing out to people that wanted to do something very different. And so I’ve switched now, again, back to more mathematical stuff. And I’m thankful that you folks invited me to talk because I think the computational crowd generally, in some ways, needs you more than vice versa. If we don’t do applied work, and it’s going to be one of the threads of my talk, I think bad things will happen. And bad things have happened to other fields and other fads in methodology.
So, a couple of things I would like to ask your tolerance for. I’m going to run over a brief, you know, story from history. And then I’m going to switch to games, toy models of the sort that Scott Page was talking about yesterday. And like a lot of you, I’m going to base this on statistics. I would guess, given your training, that most of your time is spent studying statistical models, and it turns out that even though I do computational work, about 80 percent of my time is spent either teaching econometrics, which is what you folks call biostatistics, or doing statistical work myself. So I’m going to make the assumption that stats is the lingua franca of everyone in the room. If that’s wrong, it’s too late.
So how did we get here? History, believe it or not, used to be the sort of parent of all of the social science disciplines: economics, politics, sociology and the rest. And they made claims that are not so different from some of the claims that were made yesterday by the epidemiologists in the room. It started off in a bad way. In the 19th century, you probably know there weren’t very many colleges or universities. They didn’t do research. They were finishing schools. Some might accuse schools of Duke like being finishing schools, given the tuition costs and the people that go there, but we’re not.
But research was not the first goal; religion was the first goal. Inculcating gentlemen with the right values was the second goal. And a lot of disciplines wanted to professionalize. And professionalization, ultimately, you know, meant math, believe it or not. But if you look at the people at the turn of the century when they started to do sort of theory work, they do not sound so different from you. Frederick Jackson Turner is one of the most famous historians ever, and if you read this quote, he’s talking about systems theory: “Up until our own great day, American history’s been in large degree the history of the colonization of the Great West. The existence of an area, free land, its continuous recession, and the advance of American settlement westward… Behind institutions, behind constitutional forms and modifications, lie the vital forces that call these organs into life and shape them. The civilization in America has followed the arteries made by geology, blah, blah, blah, blah, blah.”
He had a theory, though, and the theory was kind of neat. The theory was as long as we had a frontier, you know, westward territory, we were going to avoid the perils that had beset Europe. So we wouldn’t have urban strife, we wouldn’t have a lot of the problems that people had in Europe. And he had a frontier theory, and this actually guided policy for the better part of 70 or 80 years in American foreign policy. A lot of people wanted to push us on a natural science model, us being historians. So Henry Adams, Herbert Baxter Adams, they could read Darwin.
And one of the things that’s hard to remember about this time period is social scientists could read natural scientists, which is not so much true anymore, by and large. And they based a lot of their models on evolutionary models. Herbert Baxter Adams, for example, advanced the Teutonic seeds theory. I imagine no one’s heard of it, but it’s the idea that everything good in German -- in civilization comes from sort of German Teutonic seeds that sort of percolated through England and then ended up on our shores. And they were, you know, claiming they were doing evolutionary work.
Things went south. And the moral of the story is things can go south surprisingly quickly. By 1931, the interwar period, Becker, who was one of the presidents of the American Historical Association, wrote this: “It must be obvious that living history, the ideal series of events that we affirm and hold in memory, since it is so intimately associated with what we are doing and what we hope to do, cannot be precisely the same for all at any given time, or the same for one generation as for another. History in this sense cannot be reduced to a verifiable set of statistics or formulated in terms of universally valid mathematical formulas. It is rather an imaginative creation, a personal possession with each and every one of us, Mr. Everyman, fashions out of his individual experience, adapts to his practical or emotional needs, and adorns as well as may be to suit his aesthetic tastes.”
This is a cliff, right? In terms of a discipline that thought it was doing a natural science model, in about 40 years, you end up with this, and it keeps going. The next -- you know, skip a president, Beard writes more or less the same thing. And if you think about it, the question is what happened and could it happen to us in the social sciences? There was a brief consensus after World War II, driven by and large by the government wanting historians to be useful for a while. And the interesting thing, for my story at least, is they didn’t agree on what the end of history was, that they were doing causal work. There was significant disagreement about whether or not you could do causal work.
They did agree, though, on method, okay? And historical method, or historiography, as it’s called, is a little bit wacky. It’s mostly getting the facts right, and it’s hard to overstate how much they cared about facts, but the training of a historian is you get two years of coursework, they send you off to an archive, and they give you some sort of nice advice, like bring Kentucky bourbon, since they usually can’t get that for a cheap price, to bribe the people in the archives to let you see the good stuff. And make sure you have cards that are all the same size so you can put them in a nice little box to bring home with you and have a record of what you actually looked at in the archives. It’s not very sophisticated in terms of methodology. But they did care about it, which may seem odd to you.
And I want you to tell the -- a story about David Abraham and the end of history as we know it, and how at most universities, history went from a social science to a humanities department. David Abraham was a fellow who, not so long ago in the ‘70s, got trained in German history. And in German history, there’s a debate, and it’s an interesting debate about whether the Germans are evil, you know, innately, or whether the Nazis were just bad and hoodwinked them. And this matters at the margin -- I don’t mean to make light of it. And the reason it matters is if it could happen anywhere, that indicates we should all be worried. Or it could be unique to the Germans, in which case we don’t so much need to worry. Okay? So it’s an interesting historical debate as far as historical debates go.
Abraham was a socialist and he thought that German industrialists were complicitous in the rise of Nazi party to get war profits. So it’s, you know, a military industrial complex story, with sort of a Marxist crunchy exterior. The bad news is that there were a bunch of famous historians who didn’t agree with this story. And Abraham, like a lot of young people going to the archives, made a bunch of errors in his book. It was accepted at Princeton, it was up for an award. He got a job at Princeton, a tenure-track job, and the older historians who didn’t much care for him sent their graduate students to the archives to replicate his work.
Now, have any of you ever looked at a history book, academic history? About each page, half of the page is footnotes. There’s very little text in these monographs, mostly they’re footnotes. And this is, you know, privileging the fact above all else. And they found two or three hundred mistakes in a four or five hundred page book. So for them, the antagonists, that was the end of it. He’d basically been falsified and he was malicious in their view. So they began a phone call, you know, campaign to drive him from the discipline. They got him -- his book taken off the press. He was taken off of the list for the award, and he was ultimately driven from the discipline. He’s an investment banker now, or is he a lawyer? He’s one of the two. Something that makes the story, you know, make sense in a cosmic way.
But the -- again, the interesting thing for our purposes is that one of the journals actually took the time to print a list, like Excel spreadsheet format, of all of his errors and then a committee evaluated whether the error was neutral to his argument, helped his argument or hurt his argument once it was corrected. And the crisis for history, and again, I can’t overstate this, is that people realized that this was not of itself sufficient to prove that something was false. You could make errors but your theory might still be right. And the converse is that you might make no errors and your theory could ultimately still be wrong.
So they actually had a crisis of method, and history’s changed a little bit. Natalie Zemon Davis is an historian at Princeton and is famous by their lights, and method has changed. If you look at the brief quote, you probably -- some of you are not old enough to have watched this movie, it was “The Return of Martin Guerre,” it had Gerard Depardieu in it. I don’t even think anyone knows who he is anymore, but her claim about her historiography and method is distinct from the natural science model was, “Watching him, the actor, feel his way into the role of the false Martin Guerre gave me new ways to think about the accomplishments of the real impostor. I felt that I had my own historical laboratory, generating not proofs, but historical possibilities.”
So the brief story is this is about a man who in the 15th century disappears for a war in France, and someone comes back and claims that he is this man and is the husband of the wife that was left behind. They hook up, and it’s discovered that, after the fact, that he’s an imposter, and in that time period, that’s a hanging offense. They actually did kill the guy. That’s not so good in that time period. But instead of going through the archives and the documents and all the rest, of which there are not many from the 15th century for peasants, they did not write a whole lot, it turns out. She looked at a movie production that she was a consultant on and used that as her historical laboratory. That’s a different kind of history, obviously. It’s clearly not the natural science model, and most historians are self-identified as members of the humanities these days. They’re not social scientists anymore.
So, as with all fads and fashion in methods, the question is can computational science help you? And unlike -- like Scott Page, I’m going to present you a couple of toy models today so you can think about these issues. I think that’s a good way to go. Unlike Scott, my parents were not as nice to me when I was a child so I’m a little more pessimistic than he is. I think there’s enormous opportunity for computational modeling to help with big questions. I also think that, like a lot of fads and methods, that there is some chance that it might disappear, so we’ll see. But the bottom line is, we have to do applied work.
So, political science -- this will be about all I say about it. The good news about us as a group is that we are very ecumenical and we study behavior, which is probably of interest to you. We have, you know, people doing fMRIs, we have social psych people, we have a lot of people who increasingly are good at pure statistics or applied statistics and do bargaining models and computer science. So my training is not really all that idiosyncratic in one sense. And we do study most everything, from ethnic violence to interstate conflict to individual decision making, from elections to all sorts of public health concerns and public policy.
And like you, we study statistics, by and large. It’s the language that everyone uses to convey results, and we do a lot of hand wringing about it. I don’t know how much hand wringing you do, but the causation question and work like that of Judea Pearl and computer science, we actually read that stuff and take it seriously and worry about correlation versus causation, and you know, again, lots of hand wringing. And unlike history, we would not study something like the causes of World War I. We don’t care. There’s no such thing as World War I. We’d study war. If you can’t make it a general class, you can’t do statistics. If you can’t do statistics, we don’t so much care about it. So that is where political science is currently.
My first example is taken from Scott Page. He didn’t use it, so I’m going to. And I want to briefly, even though it was done yesterday -- I’ve cut a lot of this last night to try and not be redundant -- talk a little bit about how we do things and how our methods are different. We have three main schools. We have the applied stats group, which you will understand. The deductive group, which was talked about yesterday, mostly in negative terms, and it’s fortunate we don’t have members of the econ department that would rush the podium today. And the computational perspective. So here’s the problem. Has anyone ever been in a crowd and you wonder whether or not you should stand and ovate at the end of it? Has this caused any of you any angst? It might. It’s something you might want to think about. The thing you want to be clear on, you do not want to be the only fool standing, clapping wildly. Okay? For most people that would be humiliating.
On the other side, you do not want to be the only person sitting in your seat like the Grinch, okay? So I’m going to actually sound game theoretic for a moment here. You have a utility function. And your utility has terms for things like not being humiliated, okay, we can all agree on that. The question is how do you study this problem? The one approach that you’ll be most familiar with is the applied stats approach. You would come up with the dependent variable, okay? It could be the length of the ovation, it could be whether or not there is one, and then you do a logit to probit model. It could be how many people are standing so that you get a continuous variable with more variants, woo hoo.
You’d ask the usual questions of all the people that are in your observation: demographics, performance type, audience size. You’d run a model, and you’d have the usual problems. Your data would probably not be IID, right? Performances might have inter-reactions between them, you know? There’s buzz, positive buzz or negative buzz. There might be interactions between people in the audience, so if your unit of observation is an individual, they sit together and they don’t come in individually, they’re not independent. People don’t talk about non-IID, it lurks behind a lot of models and it’s kind of an ugly problem. But we do like sample bias and multicollinearity, and you know, all sorts of other stuff. And we would talk about that in our model. And the information we would get would not be valueless. We’d actually come up with a model that could more or less predict, possibly, if we’re interested in out of sample work, the incidence of whether or not there’s going to be a standing ovation or how long it’s going to last, and the like. Okay? So the information is not useless, and this is the kind of work that I would say about 80, 85 percent of the social scientists spend their time doing, okay?
Game theory’s a little different. Game theory, we would actually write down a utility function, and we’d write down the sequence of moves that are available to you. And in this case, it’s pretty easy. Your utility function is probably to avoid humiliation and to reward a good performance. And the sequence or strategies available are you can stand up or you can sit down. It’s not rocket science. What might be rocket science is how you would come up with an equilibrium play, and backwards induction, as was noted yesterday, is the sort of tool of choice. I don’t think this should be as privileged as it is. There are claims made that backwards induction is rationality by a lot of social scientists. That’s obviously goofy and wrong. But backwards induction is an algorithm, it does solve some problems, and most problems are put in the form of a tree, so backwards induction works.
But as noted yesterday, there’s no dynamics, there’s no knowledge of what happens in this equilibrium. The games are brittle, there’s no equivalence classes for games. If I have a game for standing ovation and someone else does, there’s no correspondence between them, they’re just different. There’s no way to say that they’re kind of sort of similar. And usually, questions like how does everyone start standing up? If we all have this in the back of our heads, not wanting to be humiliated, would anyone ever stand first in a deductive model? It’s actually tough, so the question is, how do you get it to happen? You probably have some exogenous parameter for, you know, a move by nature. How good the performance is. And then some people would have a function that they respond to that and stand up by force, more or less, and all the interesting work in the model, the standing up and then the sitting down again -- which is equally problematic. When do you stop ovating? You don’t want to be the first person, right? All of that, you know, problematic stuff is swept under the rug of exogenous parameters. A move by history. So I find this deeply unsatisfying even though, again, I spend a lot of my time doing deductive work of this kind. Okay, so that’s game theory.
And one last note on that. Deductive work does, you know, work in the sense that we do have some really, really strong results that are really general. For example, all of you who believe in democracy, you should Google when you get home the Arrow result, A-R-R-O-W, Kenneth Arrow, won a Nobel Prize for it. But democracy doesn’t work. It’s not a good aggregation mechanism for preferences. It’s just wrong and evil.
And can be manipulated. So when you are in a room or in a meeting with a social scientist, we are good at one thing, and that is manipulating other people to come out with outcomes that we prefer and you might not. So deductive work has its role and I don’t want to understate that.
Computational models. It’s not so different from game theory in the sense that you’re going to write down a utility function ultimately for your agents. What is different, as I think was abundantly clear yesterday, is you can include a lot more, and it’s way, way, way easy to add in things that are interesting and you think essential to the problem. Now, the converse again is the kitchen sink approach, which is you just add in everything that might be plausibly be described or be interesting to you, does not work. And the problem is, and I think you can understand this from a stats perspective, is it’s the same reason you don’t add everything to a statistical model, okay? A lot of times, if you open up a paper and you look at it and they have eight or 50 independent variables, and then they have every possible polynomial up to some order of all those variables, and then they have all the interactions of those variables, well, what would you do with that paper? You might put it down if you’re sensible, right?
And the thing that people learn that’s pathological is degrees of freedom, you know, more observations than you need, that’s actually wrong and false and we’re going to talk about that. You actually need to span your parameter space. So one of the problems with computational models, and this is true not just of them but also of a lot of statistics, like non-parametric stats, is that the dynamics are opaque because there’s too many moving parts and the parameter spaces are too big. So the question I’d like to ask is how can we actually make everything better?
So the limiting questions. Why are things so hard is the first question. Second question is how can these different approaches co-exist? I think there’s a case to be made that they should co-exist and the only way to make real progress is if we come up with a system to make them co-exist. Third question is how to make interdisciplinarity work, which I don’t think is easy. And last, how does one build more complicated models? So I’m going to get to the crunchy toy games part of the talk.
But first, everybody lies. We have a shared methodology in political science -- that is Gregory House, a rotten television show. My friends bought it for me because they claim that I was as horrible as he was. My wife says that I am not because I don’t have writers, so I can’t be.
So that’s a plus and a minus. I’ll take it as a compliment. But one of our better statistical people, Chris Achen, in 2002, wrote this about our empirical work: “Empirical work, the way too many political scientists do it, is relatively easy. Gathering the data, run the regression or MLE with the usual list of control variables, report the significance, and announce that one’s pet variable has passed. This dreary hypothesis-testing framework is sometimes even insisted upon by journal editors. Being purely mechanical, it saves a great deal of thinking and anxiety, and cannot help but being popular. Propose the following simple rule: any statistical specification with more than three independent variables should be disregarded as meaningless.”
How many of you would still have publications if we adhered to this rule? I would not, I will confess. I would have no publications if I only had three independent variables. And the question you should ask is does this apply to computational work? Yesterday, and I think I agree with this, is the allure is that you can add all these things you think are important. But the question is, is computational somehow exempt from rules like this that you might want to apply to your statistical models? And some ugliness, I can’t help myself, this is from political science. If you look over here, these are, you know, more or less whether or not something is significant for coefficients that people thought were interesting in their models, and this should cause you a little bit of a belly laugh. What does this slide tell us? It tells us that the .05 mark is magically wonderful, and that should not be, right? If we run model after model, we should get a nice monotonic decreasing slope where we get more .10s than .09s, more .09s than .08s. And instead what we find is we get this little peak at .05, which if you think about it, and that’s an awfully long tail that doesn’t really correspond to what you’d expect there either. That’s reasonably humiliating.