Are there actual psychics walking among us?
University of Pennsylvania professor Phil Tetlock probably wouldn’t go that far. However, the author of the book “Superforecasting: The Art and Science of Prediction” argues that certain people do possess astute predicting prowess. They don’t use ESP, and they don’t have to be experts in a particular field. Instead, they approach forecasting in a distinct way.
Tetlock has made a career of studying predictions and forecasting. In 2011, he co-created the Good Judgment Project, a program designed to harness the collective wisdom of people with especially strong forecasting capabilities. Later he founded Good Judgment Inc., a consulting firm specializing in forecasting.
Tetlock spoke with REIT magazine about the attributes of these so-called superforecasters and applications of superforecasting in the modern business world.
REIT: Is prediction more art or science?
Phil Tetlock: Depends on the domain.
In poker—or, more generally, casinos—it is closer to a science, where classic frequentist statistics apply to repeatable games with well-defined sampling universes: There are only so many cards in a deck.
But in the messy real world, it is much more of a mix of art and science. Carefully crafted questions and good feedback help move the dial toward science, but subjective probability estimation necessarily requires an artist’s touch.
REIT: My impression from “Superforecasting” is that good forecasters follow processes, but not rules. Would you say that is accurate?
Tetlock: Nicely put—fully agree.
REIT: What are some of the hallmarks of people who excel at forecasting?
Tetlock: Superforecasters embrace uncertainty and the idea that reality is infinitely complex, as described in chapter eight of the book. They test their beliefs, are intellectually curious and leave personal agendas at the forecasting door. They value a diversity of views and will seek to improve their forecasting skills in a spirit of “perpetual beta.”
REIT: Obviously, this leads to questions about superforecasters playing the market. Have you come across any superforecasters who are particularly skilled investors? What about the application of superforecasters’ techniques in investment decisions?
Tetlock: Superforecasters come from all over the world with a variety of backgrounds, including several financial professionals, in the U.S., Europe and farther afield.
Successful investors are already good forecasters almost by definition, and the superforecasters’ techniques can help them gain an added edge on their competitors. To take just one concrete example, investors who forecast a price target can also track the level of conviction they have in that price target; Good Judgment can help show them how to do so using probabilities.
The superforeforecasters are professionals at making probability estimates in an uncertain world. As a firm, that’s Good Judgment’s specialty. Running a successful portfolio requires many other specialized skills, so at this point we are partnering with firms who have those skills and recognize the value that Good Judgment offers to their investment processes.
REIT: Let’s say you’re an executive at a real estate company who’s trying to put together a forecast for a specific aspect of the market—demand for warehouse space, et cetera. What should your process or guidelines for that project be?
Tetlock: Before getting into the specifics of that market, Superforecasters will start with an “outside view” and collect broader information. For instance, without necessarily having any particular expertise, they’ll ask about things like where the economy is in the business cycle, then compare that to demand for warehouse space to arrive at an initial base rate estimate of what to expect.
Then they’ll dig deeper into the current trends to take an “inside view” of warehouse demand by looking, for example, for indicators that are in line or deviate from earlier cycles. Does the surge in online commerce add or subtract from warehouse demand? How geared is warehouse demand to particular areas of the economy that are on the upswing, such as housing, as opposed to a downturn, such as the manufacturing and energy sectors?
REIT: Can you tell us a little bit about your forecasting tournaments?
Tetlock: At Good Judgment, we are big believers in forecasting tournaments as a way to improve forecasting accuracy. For those hearty souls who would like to try their hand, the Good Judgment Open includes a variety of questions on geopolitics, sports, economics and so forth, where anyone can hone their skills and get the same kind of feedback that is provided to the Superforecasters.
Good Judgment also hosts in-house tournaments for organizations so they can identify and cultivate their own Superforecasters. And Good Judgment’s professional cadre of veteran Superforecasters are actively engaged in proprietary questions posed by clients in government, non-profits and the private sector.
REIT: What are your greatest takeaways from the tournaments?
Tetlock: Subjective probability estimation of messy real world events is a skill that can be cultivated and is worth cultivating—if you value accurate odds estimation.
REIT: How hard is it to incorporate what you consider to be principles of good judgment into the culture and everyday life at an organization or company?
Tetlock: Very hard—stale status hierarchies resist change—as described in the book.
But it’s not impossible. If the leaders and key individuals seek to change an organization’s culture, the principles uncovered by Good Judgment can help them do so.
Phil Tetlock is a professor at the University of Pennsylvania, where he teaches courses in the fields of psychology, politics and business. His research has focused on “the challenges of assessing ‘good judgment’ in both laboratory and real-world settings” and “the criteria that social scientists use in judging judgment and drawing normative conclusions about bias and error.” Tetlock is the author of multiple books, including his latest, “Superforecasting: The Art and Science of Prediction,” published in 2015.