Applied Rationality Workshop
November 2013
Table of Contents
Rationality Checklist...3
Annotated Schedule...9
What is Applied Rationality?...13
Getting the Most out of the Workshop...16
Prediction Markets...19
Opening Session – Further Resources...22
Friday – How to Use Your Brain...26
Building Bayesian Habits...26
Your Inner Simulator...52
Emotional Re-Association...60
Goal Factoring...76
Attention...80
Saturday – Try Things...82
Aversion Factoring and Calibration...82
Againstness...93
Implementation Intentions...97
Curating Your Emotional Library...100
Comfort Zone Expansion (CoZE)...104
Sunday – Compound Returns...113
Delegating to Yourself...113
Propagating Urges...118
Offline Habit Training...135
Turbocharging Training...148
Value of Information...152
Names and Faces: Staff...161
Names and Faces: Volunteers...165
Names and Faces: Participants...168
Contact Info...176
Rationality Checklist
This checklist is meant for your personal use so you can have a wish-list of rationality habits, and so that you can see if you're acquiring good habits over the next year—we're not using it to decide how rational you are at the start of the program.
1. Reacting to evidence / surprises / arguments you haven't heard before; flagging beliefs for examination.
a) When I see something odd - something that doesn't fit with what I'd ordinarily expect, given my other beliefs - I successfully notice, promote it to conscious attention and think "I notice that I am confused" or some equivalent thereof. (Example: You think that your flight is scheduled to depart on Thursday. On
Tuesday, you get an email from Travelocity advising you to prepare for your flight “tomorrow”, which seems wrong. Do you successfully raise this anomaly to the level of conscious attention? (Based on the experience of an actual LWer who failed to notice confusion at this point and missed their plane flight.))
Date of last example: □ Never
□ Today/yesterday □ Last week □ Last month □ Last year
□ Before the last year
b) When somebody says something that isn't quite clear enough for me to
visualize, I notice this and ask for examples. (Recent example from Eliezer: A
mathematics student said they were studying "stacks". I asked for an example of a stack. They said that the integers could form a stack. I asked for an example of something that was not a stack.) (Recent example from Anna: Cat said that her boyfriend was very competitive. I asked her for an example of "very competitive." She said that when he’s driving and the person next to him revs their engine, he must be the one to leave the intersection first—and when he’s the passenger he gets mad at the driver when they don’t react similarly.)
Date of last example: □ Never
□ Today/yesterday □ Last week □ Last month □ Last year
□ Before the last year
c) I notice when my mind is arguing for a side (instead of evaluating which side to choose), and flag this as an error mode. (Recent example from Anna: Noticed
myself explaining to myself why outsourcing my clothes shopping does make sense, rather than evaluating whether to do it.)
Date of last example: □ Never
□ Today/yesterday □ Last week □ Last month □ Last year
□ Before the last year
d) I notice my mind flinching away from a thought; and when I notice, I flag that area as requiring more deliberate exploration. (Recent example from Anna: I
have a failure mode where, when I feel socially uncomfortable, I try to make others feel mistaken so that I will feel less vulnerable. Putting this thought into words required repeated conscious effort, as my mind kept wanting to just drop the subject.)
Date of last example: □ Never
□ Today/yesterday □ Last week □ Last month □ Last year
e) I consciously attempt to welcome bad news, or at least not push it away. (Recent example from Eliezer: At a brainstorming session for future
Singularity Summits, one issue raised was that we hadn't really been asking for money at previous ones. My brain was offering resistance, so I applied the "bad news is good news" pattern to rephrase this as, "This point doesn't change the fixed amount of money we raised in past years, so it is good news because it implies that we can fix the strategy and do better next year.")
Date of last example: □ Never
□ Today/yesterday □ Last week □ Last month □ Last year
□ Before the last year
2. Questioning and analyzing beliefs (after they come to your attention).
a) I notice when I'm not being curious. (Recent example from Anna: Whenever
someone criticizes me, I usually find myself thinking defensively at first, and have to visualize the world in which the criticism is true, and the world in which it's false, to convince myself that I actually want to know. For example, someone criticized us for providing inadequate prior info on what statistics we'd gather for the Rationality Minicamp; and I had to visualize the consequences of [explaining to myself, internally, why I couldn’t have done any better given everything else I had to do], vs. the possible consequences of [visualizing how it might've been done better, so as to update my action-patterns for next time], to snap my brain out of defensive-mode and into should-we-do-that-differently mode.)
Date of last example: □ Never
□ Today/yesterday □ Last week □ Last month □ Last year
□ Before the last year
b) I look for the actual, historical causes of my beliefs, emotions, and habits; and when doing so, I can suppress my mind's search for justifications, or set aside justifications that weren't the actual, historical causes of my thoughts. (Recent
example from Anna: When it turned out that we couldn't rent the Minicamp location I thought I was going to get, I found lots and lots of reasons to blame the person who was supposed to get it; but realized that most of my emotion came from the fear of being blamed myself for a cost overrun.)
Date of last example: □ Never
□ Today/yesterday □ Last week □ Last month □ Last year
□ Before the last year
c) I try to think of a concrete example that I can use to follow abstract arguments or proof steps. (Classic example: Richard Feynman being disturbed that
Brazilian physics students didn't know that a "material with an index" meant a material such as water. If someone talks about a proof over all integers, do you try it with the number 17? If your thoughts are circling around your roommate being messy, do you try checking your reasoning against the specifics of a particular occasion when they were messy?)
Date of last example: □ Never
□ Today/yesterday □ Last week □ Last month □ Last year
□ Before the last year
d) When I'm trying to distinguish between two (or more) hypotheses using a piece of evidence, I visualize the world where hypothesis #1 holds, and try to
consider the prior probability I'd have assigned to the evidence in that world, then visualize the world where hypothesis #2 holds; and see if the evidence seems more likely or more specifically predicted in one world than the other (Historical example: During the Amanda Knox murder case, after many hours
of police interrogation, Amanda Knox turned some cartwheels in her cell. The prosecutor argued that she was celebrating the murder. Would you, confronted with this argument, try to come up with a way to make the same evidence fit her innocence? Or would you first try visualizing an innocent detainee, then a guilty detainee, to ask with what frequency you think such people turn
cartwheels during detention, to see if the likelihoods were skewed in one direction or the other?)
Date of last example: □ Never
□ Today/yesterday □ Last week □ Last month □ Last year
□ Before the last year
e) I try to consciously assess prior probabilities and compare them to the apparent strength of evidence. (Recent example from Eliezer: Used it in a conversation
about apparent evidence for parapsychology, saying that for this I wanted p < 0.0001, like they use in physics, rather than p < 0.05, before I started paying attention at all.)
Date of last example: □ Never
□ Today/yesterday □ Last week □ Last month □ Last year
□ Before the last year
f) When I encounter evidence that's insufficient to make me "change my mind" (substantially change beliefs/policies), but is still more likely to occur in world X than world Y, I try to update my probabilities at least a little. (Recent
example from Anna: Realized I should somewhat update my beliefs about being a good driver after someone else knocked off my side mirror, even though it was legally and probably actually their fault—even so, the accident is still more likely to occur in worlds where my bad-driver parameter is higher.)
Date of last example: □ Never
□ Today/yesterday □ Last week □ Last month □ Last year
□ Before the last year
3. Handling inner conflicts; when different parts of you are pulling in different directions, you want different things that seem incompatible; responses to stress.
a) I notice when I and my brain seem to believe different things (a belief-vs-anticipation divergence), and when this happens I pause and ask which of us is right. (Recent example from Anna: Jumping off the Stratosphere Hotel in Las
Vegas in a wire-guided fall. I knew it was safe based on 40,000 data points of people doing it without significant injury, but to persuade my brain I had to visualize 2 times the population of my college jumping off and surviving. Also, my brain sometimes seems much more pessimistic, especially about social things, than I am, and is almost always wrong.)
Date of last example: □ Never
□ Today/yesterday □ Last week □ Last month □ Last year
b) When facing a difficult decision, I try to reframe it in a way that will reduce, or at least switch around, the biases that might be influencing it. (Recent example
from Anna's brother: Trying to decide whether to move to Silicon Valley and look for a higher-paying programming job, he tried a reframe to avoid the status quo bias: If he was living in Silicon Valley already, would he accept a $70K pay cut to move to Santa Barbara with his college friends? (Answer: No.))
Date of last example: □ Never
□ Today/yesterday □ Last week □ Last month □ Last year
□ Before the last year
c) When facing a difficult decision, I check which considerations are
consequentialist - which considerations are actually about future consequences. (Recent example from Eliezer: I bought a $1400 mattress in my quest for
sleep, over the Internet, and hence much cheaper than the mattress I tried in the store, but non-returnable. When the new mattress didn't seem to work too well once I actually tried sleeping nights on it, this was making me reluctant to spend even more money trying another mattress. I reminded myself that the $1400 was a sunk cost rather than a future consequence, and didn't change the importance and scope of future better sleep at stake (occurring once per day and a large effect size each day).
Date of last example: □ Never
□ Today/yesterday □ Last week □ Last month □ Last year
□ Before the last year
4. What you do when you find your thoughts, or an argument, going in circles or not getting anywhere.
a) I try to find a concrete prediction that the different beliefs, or different people, definitely disagree about, just to make sure the disagreement is real/empirical. (Recent example from Michael Smith: Someone was worried that rationality
training might be "fake", and I asked if they could think of a particular prediction they'd make about the results of running the rationality units, that was different from mine, given that it was "fake".)
Date of last example: □ Never
□ Today/yesterday □ Last week □ Last month □ Last year
□ Before the last year
b) I try to come up with an experimental test, whose possible results would either satisfy me (if it's an internal argument) or that my friends can agree on (if it's a group discussion). (This is how we settled the running argument over what to
call the Center for Applied Rationality—Julia went out and tested alternate names on around 120 people.)
Date of last example: □ Never
□ Today/yesterday □ Last week □ Last month □ Last year
□ Before the last year
c) If I find my thoughts circling around a particular word, I try to taboo the word, i.e., think without using that word or any of its synonyms or equivalent concepts. (E.g. wondering whether you're "smart enough", whether your partner is "inconsiderate", or if you're "trying to do the right thing".) (Recent
example from Anna: Advised someone to stop spending so much time
wondering if they or other people were justified; was told that they were trying to do the right thing; and asked them to taboo the word 'trying' and talk about how their thought-patterns were actually behaving.)
Date of last example: □ Never
□ Today/yesterday □ Last week □ Last month □ Last year
□ Before the last year
5. Noticing and flagging behaviors (habits, strategies) for review and revision.
a) I consciously think about information-value when deciding whether to try something new, or investigate something that I'm doubtful about. (Recent
example from Eliezer: Ordering a $20 exercise ball to see if sitting on it would improve my alertness and/or back muscle strain.) (Non-recent example from Eliezer: After several months of procrastination, and due to Anna nagging me about the value of information, finally trying out what happens when I write with a paired partner; and finding that my writing productivity went up by a factor of four, literally, measured in words per day.)
Date of last example: □ Never
□ Today/yesterday □ Last week □ Last month □ Last year
□ Before the last year
b) I quantify consequences—how often, how long, how intense. (Recent example
from Anna: When we had Julia take on the task of figuring out the Center's name, I worried that a certain person would be offended by not being in control of the loop, and had to consciously evaluate how improbable this was, how little he'd probably be offended, and how short the offense would probably last, to get my brain to stop worrying.) (Plus 3 real cases we've observed in the last year: Someone switching careers is afraid of what a parent will think, and has to consciously evaluate how much emotional pain the parent will experience, for how long before they acclimate, to realize that this shouldn't be a dominant consideration.)
Date of last example: □ Never
□ Today/yesterday □ Last week □ Last month □ Last year
□ Before the last year
6. Revising strategies, forming new habits, implementing new behavior patterns
a) I notice when something is negatively reinforcing a behavior I want to repeat. (Recent example from Anna: I noticed that every time I hit 'Send' on an email,
I was visualizing all the ways the recipient might respond poorly or something else might go wrong, negatively reinforcing the behavior of sending emails. I've (a) stopped doing that (b) installed a habit of smiling each time I hit 'Send'
(which provides my brain a jolt of positive reinforcement). This has resulted
in strongly reduced procrastination about emails.)
Date of last example: □ Never
□ Today/yesterday □ Last week □ Last month □ Last year
□ Before the last year
b) I talk to my friends or deliberately use other social commitment mechanisms on myself. (Recent example from Anna: Using grapefruit juice to keep up
brain glucose, I had some juice left over when work was done. I looked at Michael Smith and jokingly said, "But if I don't drink this now, it will have been wasted!" to prevent the sunk cost fallacy.) (Example from Eliezer: When I was having trouble getting to sleep, I (a) talked to Anna about the dumb reasoning my brain was using for staying up later, and (b) set up a system with Luke where I put a + in my daily work log every night I showered by my target time for getting to sleep on schedule, and a — every time I didn't.)
Date of last example: □ Never
□ Today/yesterday □ Last week □ Last month □ Last year
c) To establish a new habit, I reward my inner pigeon for executing the habit. (Example from Eliezer: Multiple observers reported a long-term increase in
my warmth / niceness several months after... 3 repeats of 4-hour writing sessions during which, in passing, I was rewarded with an M&M (and smiles) each time I complimented someone, i.e., remembered to say out loud a nice thing I thought.) (Recent example from Anna: Yesterday I rewarded myself using a smile and happy gesture for noticing that I was doing a string of low-priority tasks without doing the metacognition for putting the top priorities on top. Noticing a mistake is a good habit, which I’ve been training myself to reward, instead of just feeling bad.)
Date of last example: □ Never
□ Today/yesterday □ Last week □ Last month □ Last year
□ Before the last year
d) I try not to treat myself as if I have magic free will; I try to set up influences (habits, situations, etc.) on the way I behave, not just rely on my will to make it so. (Example from Alicorn: I avoid learning politicians’ positions on gun
control, because I have strong emotional reactions to the subject which I don’t endorse.) (Recent example from Anna: I bribed Carl to get me to write in my journal every night.)
Date of last example: □ Never
□ Today/yesterday □ Last week □ Last month □ Last year
□ Before the last year
e) I use the outside view on myself. (Recent example from Anna: I like to call
my parents once per week, but hadn't done it in a couple of weeks. My brain said, "I shouldn't call now because I'm busy today." My other brain replied, "Outside view, is this really an unusually busy day and will we actually be less busy tomorrow?")
Date of last example: □ Never
□ Today/yesterday □ Last week □ Last month □ Last year
□ Before the last year
Annotated Schedule
Friday - How to Use Your Brain
On the first day of the workshop, you’ll learn about the basic building blocks of human reasoning (and some of the biases that turn up when they’re misapplied). You’ll learn how to draw on your best habits and heuristics more often, and start adding to your arsenal of cognitive tools.
Opening Session
What does it mean to be rational? Popular culture shows us a Spock-like figure – a narrow powerhouse, unable to deal with nuance or emotion. At CFAR, we train thinkers who work well across many
domains, as ready to use quick and dirty heuristics as careful, deliberate reasoning.
We’ll also cover the logistics for the weekend: introducing the instructors, how to get the most out of classes and conversations, and the betting game that runs during the workshop.
Building Bayesian Habits
Probability theory shows us how to best update our picture of everyday reality in response to evidence. So how can you remember how to do that in everyday life? Learn the habits and heuristics to apply Bayesian reasoning to everything from qualitative data (“I had a good feeling about that interview”) to information that seems too small-scale for statistics (“My friend liked this book, should I read it?”).
Your Inner Simulator
You can trust your reflexes to dodge a thrown ball, but when else is your intuition likely to be reliable? How can you frame problems in the language and images you brain is most prepared to parse? You might use “pre-hindsight” as you imagine a message from the future saying a project has failed, and watching as your brain instinctively fills in “It didn’t work because…”
Emotional Re-Association
Our emotions distill data quickly, powerfully, unreliably, and often in strange ways. We’ll teach you how you can use our understanding of neuroscience to learn from your instinctive, emotional reactions and how to change emotional gears (e.g. if you’re in a situation where it would be more useful to be curious than angry).
Goal Factoring
Do you ever find yourself saying, “Unfortunately, I have to… X?” Goal factoring teaches you to systematically break down everything obvious and non-obvious you’re accomplishing, and ask about ways you could achieve those factors separately and more effectively – a new perspective on
everything from reading habits to email etiquette to outings with friends.
Attention
What pulls your concentration away from work you'd like to do, or experiences you'd like to enjoy? How can you maintain your focus and even expand the amount of attention you can place on your
Saturday - Try Things!
On day one, we taught you about how you think and reason, and gave some tips on exploiting this knowledge to achieve your goals. On day two, we shift into exploration mode so you can start
expanding your ability to act, imagine, and create. Overcome aversions, try out a new internal story, and find valuable opportunities to step outside your comfort zone.
Aversion Factoring and Calibration
We daily shy away from risks and opportunities that aren’t really harmful. What are some possibly-valuable things you’ve “never gotten around” to trying? Would it really be that painful to try them? Order new foods, break your commute routine, speak to an intimidating coworker, and learn not to take ‘never’ for an answer.
Againstness
Tensing up can win a physical fight, but it won’t win a debate or make a good life decision. Learn to notice and control your body’s instinctive fight-or-flight response; in stressful situations, remain calm and open to new information. Use your understanding of your body to be able to redirect an unhelpful kneejerk reaction.
Implementation Intentions
If you could make only one change to your planning habits, what would you expect would have the largest effect? Give yourself the chance to follow through on plans more successfully and increase your sense of having opportunities to seize as you go through your day.
Curating Your Emotional Library
A piece of art, whether highbrow or low, can transfix us. Learn to make deliberate use of images, audio, places, and people that have a strong emotional effect on you, so you can change your disposition or just experience life more richly.
Comfort Zone Expansion (CoZE)
It’s tempting to stick to the habits you trust. And it’s hard to estimate the cost of the opportunities you miss. In this class, figure out how to experiment socially and see where your fear is limiting your fun. This class is followed by a real life practicum in a public shopping mall, where you can practice with real, live people. Use improv exercises to get some practice thinking on your feet and experimenting with new ideas under pressure.
Sunday - Compound Returns
How will you make use of your new skills and habits? On the final day of classes, learn how to make a new behavior routine, stick to plans, aquire skills efficiently and other tools for helping you get the most out of what you know faster.
Delegating to Yourself
How can you make plans (from the everyday to the ambitious) that you can trust yourself to carry out? Lower your strain by thinking socially when you plan, so you keep yourself accountable in the future without becoming a hectoring taskmaster.
We’ll also teach you a way to batch process the way you learn from experience, so you can recursively improve your strategy for planning or anything else you want to adjust.
Propagating Urges
Use some of the best-tested principles in experimental psychology to connect the intermediate steps toward your goal to your natural enthusiasm for the result. Instead of actively reinforcing the behaviors you want to want, learn how to run that reinforcement on autopilot, using your brain’s natural facility for shaping behaviors.
Offline Habit Training
Do you want to remember to plug in your cellphone each time you get home for work, or stop yourself from interrupting others? Learn how to use visualization, associations, and practice to break destructive habits and create beneficial ones. Kick the habit of checking email in bed by setting aside five minutes to practice your ideal morning routine.
Turbocharging Training
You’re probably wondering how to budget time to practice and internalize all these wonderful new techniques. Gain new skills much faster by understanding how you learn and find the most effective ways of expanding your competencies.
Value of Information
Quantify the expected value of new information and revamp your guesses about the relative importance of evidence you can gather and predictions you can test. Learn to try the easy and affordable
experiments that ‘probably won’t work’ and search for $20 expenditures that might return $2,000 of value. Finding a way to save 10 minutes on one leg of your commute buys you five extra days per year, and that’s worth a minute to consider carefully or half an hour to test.
Closing
Attendees and instructors gather to share their reflections on how it all fits together and what they plan to do when they get home to start making use of these tools.
Party
Play rationality games, write satiric songs, or just hang out with your fellow attendees as you unwind. Attendees of past workshops will stop by to welcome you to the CFAR alumni network of people, businesses and ideas.
Monday – Practice/Troubleshooting
On Monday, you’ll start rotating through discussion groups to identify and extend the underlying and unifying themes of the classes. After the recap, you can meet with instructors one on one or in small groups to practice the techniques you’ve been learning and/or to get advice on how to use these new tools for a specific problem or project in your life.
What is Applied Rationality?
(Part I)
The sphex is a small digger wasp that seems to exhibit complicated, deliberate behavior. When it catches a cricket, it drags the body back to the entrance of its burrow, leaves the cricket outside, goes into the burrow to makes sure there are no lurking enemies, and then drags the cricket inside and lays eggs.
But, it turns out there’s a funny way to interfere with the sphex’s routine. While the sphex is inspecting the burrow, if you pull the cricket an inch or two to the left, then, when the sphex comes out, it pulls the cricket back to the entrance, and then goes in and inspects the burrow all over again. And if you move the cricket again, the sphex will tug it back into position and then go inspect the burrow a third time. Essentially, you can keep running a sphex through its loop indefinitely, without it ever noticing that it should break the pattern. That inspired Douglas Hofstadter to coin the word sphexish.
Sphexish (adj) – following patterns by rote, even when maladapted to the situation at hand.
Although the word is inspired by a wasp, its applications are unfortunately broad. When humans stay stuck in a loop, we’re behaving like the sphex. Here are some loops that some of us get stuck in:
1. Read something wrong on the internet → write reply
2. Parent brings up touchy subject → give the same cached response as in the last fight 3. Sit down at computer → open facebook
The opposite of being sphexish is being agenty. An agent is an ideal at the other end of the spectrum: someone who could analyze beliefs, habits, and repeated behaviors and choose which ones to reinforce and which ones to drop.
In order to make those choices, we’re going to give you a better view of your inner workings. You’ll get a clearer sense of how your mind works and what heuristics it uses. That way you can notice what your habits are, revise them, or add new ones.
Becoming more agenty doesn’t happen all at once. Making a small amount of progress can make it easier to keep going and improving. You can bootstrap larger changes from small ones.
The more you practice applied rationality, the more like an agent you can be – using automatic and analytic processes deliberately, and being able to course-correct when an old strategy no longer works.
What is Applied Rationality?
(Part II)
When we make analyze information or make decisions, we tend to talk in terms of two modes of thought. In his book Thinking Fast and Slow, Nobel Prize winner Daniel Kahneman described them as
System 1 and System 2.
System 1 System 2
Comes first in evolutionary history Developed later, more unique to humans Processes information quickly Processes information slowly
Sometimes called ‘intuition’ Sometimes called ‘analytic thinking’ Process of thinking is not transparent Often a verbal mode of thinking
Doesn’t use up working memory Limited by available working memory
Neither system is perfect. System 1 can fail by making the wrong connections – if a new acquaintance resembles an old enemy, you may find yourself feeling anxious without knowing why. System 2 can fail by limiting the information you tag as relevant – if you can’t put a feeling of unease into words, you may be tempted to leave it out of your calculations.
Sometimes, people think applied rationality is the process of muting System 1 and just relying on System 2. After all, the System 2 parts are the bits that are most unique to humans; wouldn’t it be great to do our best thinking all the time?
Applied rationality is about using the best tools at hand to achieve your goals, and turning off the bulk of your brain is seldom helpful. In the classes at this workshop, we’ll talk about how to better
understand System 1 and System 2, so you can play to their strengths and be more efficient. The aim is to make deliberate, thoughtful use of all the skills you have, not to only use the skill of thinking slowly and deliberately.
What is Applied Rationality?
(Part III)
You can imagine all possible actions you could take being laid out according to (x,y,z) coordinates, mapped out according to three dimensions:
• How much you enjoy the action • How well the action serves your goals • How often you take the action
So you might find breathing at something like (not very much fun, enormously necessary, constantly doing), because you don't particularly enjoy breathing, but must do it to achieve any other goal, and are always doing it.
Browsing through your RSS reader might be a bit more like (moderately fun, relatively unneeded, frequently doing) if you like checking your reader, but don't feel very strongly, tend not to make much use of what you're reading, but keep finding yourself hitting refresh.
You can imagine all of your possible actions being graphed within some kind of agency cube, that represented graphically all the ways you're currently choosing to spend your time.
You can think of this cube as having two very valuable regions. First, there's the corner (on top in the image above) where you are doing useful, pleasurable things frequently. And then there's the
diametrically opposed corner where you don't do things that you hate and that frustrate your goals. Part of applied rationality is trying to concentrate your possible actions as densely as possible in these
Getting the Most out of the Workshop
Digestion
As a child, you might have thought that when you ate food, it just became part of you without
alteration. Eat a hotdog, and there would be little tiny pieces of hot dog studded throughout your body. Eventually, we learned that food isn’t just incorporated into us, it’s digested. We break it down into something we can use, that may look very different than what we originally ingested, even if it has the same nutrients.
In a similar way, when we learn, we don’t just take up the instructor’s opinions and material wholesale. We adapt it and process it and try to extract what we need most. So it doesn’t help too much to just memorize what you hear in class. The goal is to internalize it and make it your own.
When you digest food, you use stomach enzymes to break it down into building blocks that you can use to become stronger. When you digest content and ideas, you have a number of tools to break down what you’re hearing:
• Paraphrase the instructor, make sure you can put the idea in your own words • Try teaching the material to a friend
• Look for new applications for the material (especially in your own life) • Look for predictions, if this theory is true, what would you observe • Ask questions
• Talk about the content with others • ...and many more
Over the course of the workshop, look for opportunities to digest the material in the classes, so it becomes a part of you, not just something you can describe having happened to you. Deliberately digesting content helps you get the most useful, personal material out of a lesson, and the information will be a lot easier to retain.
Getting the Most out of the Workshop
Try Things!
When you’re exploring new ideas and habits, trying them out is a great way to gather data. It can be faster and simpler to just try out a new practice and see whether it works for you than to spend a lot of time trying to anticipate whether or not you’ll find it useful.
When you try something out, and it works, you get to keep doing it, which can be really valuable over the long run. For example, if friends kept inviting you to ok-to-mediocre-sounding leisure activities, you might keep declining their offers. But what if you agreed to try them out? Maybe your list of attempts would look something like this:
Activity T1 T2 T3 T4 T5 T6 T7 T8 T9 Yoga X Ultimate Frisbee X Meditation Classes √ √ √ √ √ √ √ √ √ Salsa X
The failed trials (represented by X) are more than counterbalanced by the sustained run of a now successful habit (shown with √). So it can be worth your time to try out a number of lower probability trials, as long as a couple pay off as long-term habits.
So when you listen and participate in class, look for ways to turn lessons into actions that you can try out and test drive. The act of turning class content into practical activities is a great way to digest material (you’re paraphrasing theory into practice) and helps you decide which material you want to prioritize when you return from the workshop.
Getting the Most out of the Workshop
Capture
At the end of the workshop, what will you remember? What will you need to be able to take action? How do you make sure you retain the lessons that are the most valuable to you?
There are two common failure modes when taking notes or trying to retain information:
• You might trust your brain too much. Because something feels important in the moment, you
figure you’ll retain it. Has this strategy worked in the past? How many important-feeling things can you be exposed to before your working memory is strained?
• You might just try transcription. Instead of experiencing and digesting the class, you might
put most of your mental energy into just capturing everything. This means you’ll still have to filter the useful information later, and you won’t have access to the instructor or the other participants while you do it.
We try to preserve the basic material of each class in the workbook, so you don’t have to worry about transcription. Instead, try capturing the actions and insights that pertain to you, that aren’t already in the workbook.
When it comes to these ideas, err on the side of inclusion. This are the content you won’t be able to find anywhere else, because you’re the one who created it.
We’ve included a capture strip at the bottom of each page in the workbook, so it’s easy to retrieve what you captured. If you already have a working capture system for idea, plans, and actions, you may want to use it instead.
Don’t forget, you won’t only think of interesting things in class! Have a plan to capture important actions and insights in conversation or during digestion of material. Why not write that plan down in the ‘Action’ section at the bottom of this page? Make the commitment effect work for you!
But don’t let capture drive you crazy. You’re storing up notes for you whole post-workshop life -- don’t panic about trying to put everything into practice the week or the month after you get home. If you just make one small change (using pomodoros, doing a few VOI calculations, etc) you’re making it easier to bootstrap to other changes in the future.
Prediction Markets
Overview
Scattered throughout the camp on various walls, you’ll see large sticky sheets with predictions, probabilities, and times on them. These are prediction markets, and they offer a playful way to help refine your predicative accuracy. To play in a market, just pick up a pen, write a probability below the last one already written, and write your name and the time next to your probability.
When the prediction is settled, you will receive a (possibly negative) number of points - called
centibits, or hundredths of a bit - according to one of two rules. For the sake of describing the rules,
let’s say your bet is X% and the one before yours is Y%. If the claim turns out to be true, then the number of centibits you receive will be:
If the claim turns out to be false, however, the number of centibits you receive will be:
The House (i.e., an assigned staff member) will keep track of how many points you have. At the end of the camp, there will be a lottery for prizes and your chance of winning will be proportional to the number of points you've won in the betting markets! (But the real point of this is to engage in the practice of putting your degree of confidence on the table. The point is the process, not the product!)
Example
Suppose there’s a market with the claim, “It will be raining on Monday at noon.” At noon on Monday the market is automatically closed, at which point the following bets have been made:
P(true) P(false) Sun 12pm 50% 50% House Sun 9pm 40% 60% Alice Mon 11am 90% 10% Bob Mon 11am 55% 45% Alice
At noon on Monday, let’s suppose that it is in fact raining. That means we use the first column to calculate the payoffs: those were the probabilities the players were assigning to the true state of the world. Alice would lose some centibits for causing the market to update in the wrong direction on two
occasions. Specifically, she would “gain” -32 centibits for her first bet:
and “gain” -71 centibits for her second bet:
Bob, on the other hand, would gain 117 centibits, reflecting that he caused the market to update strongly in the correct direction:
For contrast, let’s pretend that it turns out that it’s not raining on Monday at noon. That means we use the second column of numbers to calculate the payoffs: those were the probabilities the players were assigning to the true state of the world. Here, Alice would be rewarded 26+217 centibits for causing the market to update in the correct direction on two occasions:
Bob would lose 259 centibits for his significant overconfidence in causing the market to update in the wrong direction:
In Sum
The following theorem summarizes game play:
● When you bet, your score is the amount of information, in hundredths of a bit, that your probabilities have provided as an update to the previous probability.
● Your best strategy is to honestly report your credence at any time. That is, at any moment, if your credence is not exactly the same as the last credence written on the board, then you expect to win centibits by writing up your own.
● Over time, being better calibrated (being right X% of the time that you're X% sure) will make you better at the game.
(For an explanation of why we use this scoring rule, look up Eliezer Yudkowsky’s online essay “A Technical Explanation of Technical Explanation.”)
Proposing betting markets
Anyone can propose a betting market. However, the House may choose to ignore any given market that they did not put up themselves. (We’ll avoid ignoring markets this way as much as possible. This is in
place primarily to avoid breaking the system with overly clever markets.) Also, the House has to know about a market for it to count.
Every proposed market should have:
● a statement whose truth or falsity can be and will be determined to everyone’s satisfaction during the camp,
● a condition for closing (ideally one that happens automatically), and ● an initial bet assigned to the creator of the market.
Well-formed bets
The House will ignore any bet of 100% or 0% when calculating points, treating such lines as though they were never written. (This is to prevent you from losing infinitely many points, or from allowing the person following your bet to potentially gain infinitely many points.)
You need to specify a percentage that the House’s computer can score. Variable notation like “ ” will cause the line in question to be completely ignored when calculating points. If you want to specify a very large or very small probability, use something fairly standard like scientific notation. Clever uses of unusual notation like tetration (e.g., 1/(3^^^3)) might force the House to ignore your bet, unless you care to explain how to enter your bet into the computer (and offer to write the program if entering the score requires programming!).
Bear in mind that you don’t gain all that much by being extremely confident and correct, but you can lose an arbitrary number of centibits for being overconfident and wrong. If you offer a severely overconfident bet and lose huge numbers of centibits, you are in effect paying those centibits to whomever follows your bet with anything more reasonable. (But if you actually, honestly have a gargantuan amount of confidence, your expected score is still maximized by correctly stating your confidence!)
Opening Session – Further Resources
Sphexishness and Agency
In a 1982 Scientific American article (later reprinted in his book Metamagical Themas), Douglas Hofstadter coined the term “sphexish” to refer to repetitive, pre-programmed behavioral patterns, and identified the ability to reflect on one’s own patterns as essential to breaking out of these loops.
Hofstadter, D. R. (1985). “On the seeming paradox of mechanizing creativity.” In Metamagical
Themas, 525-546.
http://amzn.com/0465045669
In his book The Robot’s Rebellion, cognitive scientist Keith Stanovich (2004) connects the issue of sphexishness to recent psychology research on human judgment and decision making.
Stanovich, Keith. (2004). The Robot's Rebellion: Finding Meaning in the Age of Darwin. http://amzn.com/0195341147
System 1 and System 2
Psychologists distinguish between “System 1” cognitive processes (which are fast, intuitive,
associative, and parallel) and “System 2” cognitive processes (which are slow, reflective, deliberate, and serial). Daniel Kahneman’s (2011) book, Thinking, Fast and Slow, elaborates on this distinction (and the field of judgment and decision making research which he co-founded) in depth; Kahneman’s (2003) review article provides a briefer summary.
Kahneman, D. (2011). Thinking, Fast and Slow. http://goo.gl/5J0zj
Kahneman, D. (2003). A perspective on judgment and choice: Mapping bounded rationality.
American Psychologist, 58, 697-720.
http://tinyurl.com/kahneman2003
System 1 processes share many similarities to the human perceptual system. When they are
functioning well, they can draw on a large body of low-level data to identify patterns, which come to mind readily without any explanation attached. However, there is a risk that the variable which System 1 reports will be subtly different from the one which you are attempting to assess. This type of error (called “attribute substitution”) is analogous to a visual illusion where you attempt to assess the two-dimensional size of a drawing on paper, but your perceptual system reports the three-two-dimensional size
that the depicted object would have (Kahneman, 2003; 2011).
Getting the most out of the workshop
Education research emphasizes that students learn more when they engage with the material in a way that goes beyond simply listening to the words in a lecture or reading the words in a book. Active learning focuses on getting students involved in activities which lead them to think through and make use of the material that they are learning about.
http://en.wikipedia.org/wiki/Active_learning
A review article of research on the benefits of active learning:
Prince, M. (2004). Does active learning work? A review of the research. Journal of Engineering
Education, 93, 223-231.
http://goo.gl/omHuJ
Cognitive scientist Roger Schank (1995) argues that people have a set of learned “scripts” for how to interact with the world, such as a script for how to order food at a restaurant. Doing new things allows a person to learn new scripts (or variations on the scripts that they already know), which increases the opportunities available to them and reduces their sphexishness.
Schank, R. C. (1995). What we learn when we learn by doing. (Technical Report No. 60). Northwestern University, Institute for Learning Sciences.
http://cogprints.org/637/1/LearnbyDoing_Schank.html
Capture is the first step of David Allen’s “Getting Things Done” organization system. Capture involves writing things down (in particular, things that you want to do) and putting them into your system. Having a functioning capture system ensures that this information will come back to your attention later on when you have time to process it and act on it, and frees your attention in the moment to do something other than memorizing things-to-do.
Allen, David (2001). Getting Things Done: The Art of Stress-Free Productivity. http://en.wikipedia.org/wiki/Getting_Things_Done
Knowledge tends to decay over time, as people gradually forget information that they have learned. Regularly accessing knowledge helps a person retain it. For a given amount of studying, a person is more likely to retain long-term memories of the material if the studying is spread out over time rather
by more recent studies (e.g., Cepeda et al., 2006). http://en.wikipedia.org/wiki/Spacing_effect
Spaced repetition software is designed to help people retain long-term memories by providing reviews that are timed efficiently based on the human forgetting curve. Programs such as Anki and Supermemo use a flashcard-based model that repeats a given flashcard less and less often over time as long as you remember it, so that you are reminded of the information before you forget it but do not spend much time reviewing information that you already know well.
An in-depth review of research on spaced repetition, including summaries of published research as well as advice on how to use spaced repetition software:
Gwern. “Spaced repetition.”
http://www.gwern.net/Spaced%20repetition
An online article with tips on how to design spaced-repetition cards effectively: Wozniak, Piotr. “The 20 rules of formulating knowledge in learning.” http://www.supermemo.com/articles/20rules.htm
A meta-analysis, providing a quantitative review of research on the spacing effect (also known as “distributed practice”):
Cepeda, N. J., Pashler, H., Vul, E., Wixted, J. T., & Rohrer, D. (2006). Distributed practice in verbal recall tasks: A review and quantitative synthesis. Psychological Bulletin, 132, 354-380. http://goo.gl/KgP0n
Predictions
People’s intuitive sense of probability tends not to be properly calibrated; among events that someone thinks are 80% sure to happen, typically only 60% actually take place (Russo & Schoemaker, 1992).
http://en.wikipedia.org/wiki/Overconfidence_effect
Repeated rounds of prediction and feedback allow a person to calibrate their expectations to reality. Sufficient training and experience in a domain with clear feedback (like weather forecasting) can lead to more accurate, unbiased estimates in that domain (Kahneman & Klein, 2009). Calibration practice can also lead to less biased estimates across many domains (Russo & Schoemaker, 1992).
A review article of research on expertise, with an emphasis on which areas of subject matter allow people to develop expertise and which do not:
Kahneman, D., & Klein, G. (2009). Conditions for intuitive expertise. American Psychologist,
64, 515-526.
http://goo.gl/lGIPZ
A review article of research on calibration and calibration training:
Russo, J. E., & Schoemaker, P. J. H. (1992). Managing overconfidence. Sloan Management
Review, 33, 7-17.
http://goo.gl/C98as
A website where you can make, share, and track probabilistic predictions about any event: http://predictionbook.com/
The Credence Game, developed by CFAR, provides calibration training by giving you immediate feedback on probabilistic judgments.
Friday – How to Use Your Brain
Building Bayesian Habits
Building Bayesian Habits - Overview
Using Bayes' Theorem to update your beliefs involves four steps:
1. Determine the hypothesis being considered and what the alternative hypothesis is 2. Determine your prior
3. Determine the strength of the evidence you’re considering
4. Combine the prior and the strength of evidence to produce a conclusion (“posterior”)
Each of these steps has a mathematical formalism. For instance, the strength of evidence is technically defined to be a ratio of two conditional probabilities that are related in a particular way. However, we will also practice some "quick-and-dirty" heuristics to engage parts of your intuition (like your "inner simulator") that don't really operate in terms of numbers.
Depending on how important a decision is, there are varying amounts of time you can spend using Bayes Rule in various ways to update your beliefs:
• 10-minute Bayes --- Doing careful research into priors and strength of evidence, spanning
minutes to hours. Useful for:
◦ High-stakes decisions --- In situations where it's worth taking some time to think and
investigate, Bayes Rule can help you evaluate conflicting sources of evidence, and identify useful questions to ask next.
◦ Settling disagreements --- Bayes can help you navigate a disagreement with someone, by
isolating whether you have different background knowledge or have differing impressions of what evidence means.
• 10-second Bayes --- A quick-and-dirty mathematical estimate, taking around 5-20 seconds.
Useful for:
◦ Emotional situations --- Sanity-checking your judgments when you feel tired, stressed out,
or otherwise emotionally compromised.
◦ Confusing situations --- when you notice your previous experience is at odds with
something you're seeing or that someone is saying.
• 1-second Bayes --- Thinking in terms of calibrated intuitions instead of numbers, which takes
around 1-3 seconds.
◦ Following arguments --- Deciding from moment to moment whether an argument or
reasoning process makes sense, as it is being presented.
This last 1-second version requires having "Bayesian habits" installed as automatic mental reflexes, taking almost no conscious reflection time at all. This means training your "System 1" as Kahneman or Stanovich would say. The types of exercises we will focus on are designed to help you hone and accelerate those reflexes.
A note on engaging in exercises:
Many exercises will involve reading a claim (e.g. “This restaurant can’t be good, or it would be more crowded”) and then reasoning about it. Rather than reasoning verbally about the claim, please make up
and imagine (i.e. mental simulate) a specific scenario fitting the description, and answer the claim
about the scenario you made up. (This will causes our answers to be different, but ensures you are practicing something closer to a real-life decision.)
Building Bayesian Habits – Drill #0
Accessing Anticipations
You have an inner simulator that lets you check how surprising something would feel if it happened. It’s a way to query your System One mind with a specific useful question written up by System Two. For example, imagine a friend of mine had RSVP’d “yes” to a party but didn’t show up. If I imagine how I would feel if this happened, I would be, at least, a bit surprised – it’s not what I anticipated happening that night. But my surprise falls along a spectrum – I might be a good deal more surprised if she showed up on time but had shaved off her hair.
And I can categorize those experiences as qualitatively different before either happens. I can use my surprise as a crude measure of how likely I expect something to occur, and get a clearer intuitive answer than if I immediately try to ascribe a numerical probability to it.
When to use Accessing Anticipations:
• A question or decision is high stakes.
• You have a lot of emotional bias clouding your response. • You’re in a disagreement.
How to use Accessing Anticipations:
• Imagine the situation as vividly as possible.
• Be specific, don’t just imagine a vague friend, think of a particular one. • Think about where your surprise falls on a spectrum.
• Don’t argue with your System 1 until you acknowledge its answer.
• Then feel free to probe at any discomfort or confusion about how surprised you were and decide which part you want to adjust.
• Notice if System 1 was responding to data you hadn’t consciously factored in.
• You can try to translate back and forth from intuitions to probabilities to tune calibration. Are your surprise reactions synched up to how unlikely something actually is? Can you use some known events as markers on your surprise thermometer?
Practicing Accessing Anticipations:
Try to imagine each of these experiences vividly; notice how surprising it seems. Then, you can try and place it along a spectrum of experiences. You can use the letters for each example to label the
spectrum, with more surpising (less likely) things near the bottom.
A. Your friend doesn’t turn up for a party s/he RSVP’d for.
B. You do badly on an exam that you felt good about while taking. C. The cashier greets you by name at the grocery store.
D. You receive a parking ticket in the mail. E. Your parent dyes her/his hair purple. F. You find a bear cub in your living room. G. You wake up naked in the woods. H. A sunny day turns to rain.
I. You receive a summons for jury duty.
J. Your name is misspelled on your reissued driver’s license.
K. Your bookcase is missing, but all the books are neatly piled on the floor. L. Your bookcase and books are both missing.
M. You turn a corner with a friend in a city and meet a tiger. N. You can’t find your car keys in the usual spot.
O. It snows on Christmas Day, where you live.
A thought experiment:
Imagine, while walking across a large public university’s campus, you meet a guy named Tom who seems very reserved and introverted. Would you guess that Tom is a business major or a math major?
Building Bayesian Habits – Drill #0
Parsing hypotheses and evidence
It takes effort and practice to change casual speech into word problems. In a conversation (or an argument), your interlocutor may not clearly identify what evidence she’s considering and what hypothesis she intendeds it to support.
The quickest way to ensure we mean the same thing by "hypotheses" and "evidence" will be to work through some examples below.
Evidence: This is what you observe to be true.
Hypothesis: Your guess about what is true, often based on some evidence. It may be a prediction,
or a speculation for what could have caused or explained the evidence you saw.
Useful times to parse colloquial language into hypotheses and evidence:
When you hear any of these phrases (or similar ones) come out of your mouth or someone else’s: ● “X happened because of Y.”
● “I believe X because Y happened.” ● “Well, I know Y, because X!” ● “Y is another reason to believe X.” ● “X can’t be true because of Y!”
Exercises / examples of parsing casual language into hypotheses and evidence:
Put a circle around the evidence (observation) the speaker is presenting and a box around the hypothesis (prediction or speculation) she's using it to support. Then, as a precursor to using Bayes later, give a short name or "handle" for the hypothesis, an alternative hypothesis, and the evidence. Note: There may be extraneous data and/or background information.
1. This restaurant can’t be good, or it would be more crowded.
Hypothesis: "bad" Alternate hypothesis: "good" Evidence: "not crowded"
2. A volunteer from the campaign visited me; so they're probably well organized.
Hypothesis: Alternate hypothesis: Evidence:
3. I’m very good at this subject, so my low test score must mean there was a Scantron error. Hypothesis: Alternate hypothesis: Evidence:
4. My car keys are missing, because my kids hid them.
Hypothesis: Alternate hypothesis: Evidence:
5. I believe she’s telling the truth, because she didn’t look away when she told me.
Hypothesis: Alternate hypothesis: Evidence:
6. My friend hasn’t mentioned the fight again, so he’s not angry.
Hypothesis: Alternate hypothesis: Evidence:
7. This candidate graduated from an Ivy League school; she’s probably qualified for the job. Hypothesis: Alternate hypothesis: Evidence:
8. This candidate stumbled over a question in the interview; she’s probably not qualified. Hypothesis: Alternate hypothesis: Evidence:
9. This candidate has previously worked in this field; she’s probably qualified for the job. Hypothesis: Alternate hypothesis: Evidence:
Building Bayesian Habits – Drill #1
Anchoring off the Prior
Many studies have shown that people tend to systematically neglect their own background information when presented with evidence, especially if the background information is less salient in their intuition or "System 1". So, before you start to make sense of a new piece of evidence, it’s important to check your baseline assumptions about the possible explanations: that is, what you would have expected before seeing the evidence. This is called your prior.
Our brains do not do this automatically. This phenomenon is called Base Rate Neglect, or Prior
Insensitivity. When your prior is lopsided in favor of a particular hypothesis, it needs to be
intentionally weighed in against new evidence, otherwise your brain is liable to ignore it. To do this, you can anchor on your prior, and use evidence to adjust away from it. When your prior doesn't favor any hypothesis in particular, this step is less important, as the new evidence alone is enough to
determine your new belief.
When to Anchor on your the Prior:
• Not that often! Only if there’s a big disparity between the two options.
• When you see strong evidence favoring a theory and want to check if you should shift your belief sharply.
Note: You don’t always have to come up with a plausible prior yourself, if Googling is faster.
How to use your Surprisometer to check whether a prior is lopsided:
● Imagine sampling from a bag of outcomes --- what do you expect to pull out?
● How many people do you think you’d need to survey to find five people in whatever category you’re talking about?
● In a big city (say, NYC – 8 million people), how many people do you expect to fall into this category?
Calibrating your sense of confidence
http://acritch.com/credence/ --- This is a simple game to give you practice assigning probabilities to your guesses and getting feedback about your actual success rate. After playing for a while, you can develop the property that when you say you are 70% sure, you are actually correct 70% of the time... for most people, this is not true by default!
Practicing using your Surprisometer to construct a prior odds ratio:
In each of these examples, write down a short name for the hypothesis being considered and an
alternative, as in Drill #0. Your prior is what you would have expected before seeing the evidence, and what odds you'd give to that expectation. You may use the techniques listed above to help you decide your prior.
1. Example: The restaurant can’t be good, or it would be more crowded. ( good : not good )
Prior odds = (3 : 1)
(This means you think that good restaurants are about 3 times more common than bad ones.) 2. [A specific friend] enjoyed this book, therefore I will, too.
Prior odds =
3. A volunteer from the campaign visited me; that’s evidence that they’re well-organized. Prior odds =
4. I’m very good at this subject, so my low test score must mean there was a Scantron error. Prior odds =
5. My car keys are missing, because my kids hid them. Prior odds =
6. I believe she’s telling the truth, because she didn’t look away when she told me. Prior odds =
7. My friend hasn’t mentioned the fight again, so he’s not angry. Prior odds =
8. This candidate graduated from an Ivy League school; she’s probably qualified for the job. Prior odds =
Building Bayesian Habits – Drill #2
Strength of Evidence
You can use your Surprisometer to help check whether evidence actually supports a given hypothesis, and to gauge how strongly it supports that theory.
Evidence supporting a given explanation X means that it’s more likely to occur in a world where X is true than where X is false. It doesn’t mean that X is necessarily the most plausible cause of whatever you’ve observed.
Techniques for using your Surprisometer to judge Strength of Evidence as a likelihood ratio:
• Imagine the same situation repeats a number of times (e.g. you go on 20 job interviews, 10 of which you’re a good match for and 10 of which are a reach). In how many of the interviews for which you’re well qualified do you expect to stumble over at least one question? How many of the 10 for which you’re a little less qualified?
• Your internal simulator should flag some results as more surprising than others. So you can home in on a fuzzy estimate of an odds ratio (e.g. [4-5] : [6-7]), which can give you a sense of how strongly the evidence (flubbed a question) supports a given hypothesis (can do this job capably)
Other tips:
• You’re trying to trigger a strong intuitive, System One reaction, so remember to be vivid and specific. • Estimates can be helpful, even if they’re fuzzy. You can use ranges here, just as you used orders of
magnitude when estimating priors.
• If there’s a large disparity between your tallies in the two cases, the evidence is strong; if they’re close to even, the evidence is weak (q.v. the section on Bayes without numbers).
Practicing using your Surprisometer to judge Strength of Evidence as a likelihood ratio:
Go back over the list of propositions, and try to estimate an odds ratio by checking how often you expect to see this piece of evidence appear when the hypothesis is true and when it’s false. After you finish the list, compare to the previous page to see if the magnitude of your ratings varied, depending what method you used.
1. Example: The restaurant can’t be good, or it would be more crowded. Evidence: “not crowded”
Qualitative strength
of evidence: weak, for “not good” OR
( good : not good) Likelihood ratio = (60% : 80%)
= (3 : 4)
2. [A specific friend] enjoyed this book, therefore I will, too.
3. A volunteer from the campaign visited me; that’s evidence that they’re well-organized.
4. I’m very good at this subject, so my low test score must mean there was a Scantron error.
5. My car keys are missing, because my kids hid them.
6. I believe she’s telling the truth, because she didn’t look away when she told me.
7. My friend hasn’t mentioned the fight again, so he’s not angry.
8. This candidate graduated from an Ivy League school; she’s probably qualified for the job.
9. This candidate stumbled over a question in the interview; she’s probably not qualified.
Building Bayesian Habits – Drill #3
Bayes, altogether
Finally, we put the first two steps of Bayes together --- the prior and the likelihood ratio --- for form our new belief, called the posterior.
1. Example: The restaurant can’t be good, or it would be more crowded. ( good : not good)
Prior odds: = (3 : 1)
Evidence: "not crowded"
Likelihood ratio = (60% : 80%) = (3 : 4) Posterior: = (3*3 : 1*4)
= (9 : 4) ~ (2 : 1)
2. [A specific friend] enjoyed this book, therefore I will, too.
3. A volunteer from the campaign visited me; that’s evidence that they’re well-organized.
4. I’m very good at this subject, so my low test score must mean there was a Scantron error.
5. My car keys are missing, because my kids hid them.
6. I believe she’s telling the truth, because she didn’t look away when she told me.
7. My friend hasn’t mentioned the fight again, so he’s not angry.
8. This candidate graduated from an Ivy League school; she’s probably qualified for the job.
9. This candidate stumbled over a question in the interview; she’s probably not qualified.
Optional: Using your Surprisometer for Bayes without numbers
Surprise behaves a lot like the logarithm of probability. The advantage of thinking entirely in terms of surprise affects is that this process can become very instinctive and take on the order of a second or so. This is the kind of 1-second Bayes discussed in the outline. Here's what the steps look like when combined, for the restaurant example above:
Practicing using your Surprisometer to construct a prior as a surprisal difference:
In each of the following examples, write down a short name for the hypothesis being considered and an alternative, as in Drill #0. Your prior is that you would have expected before seeing the evidence, and what odds you'd give to that expectation You may use the techniques listed above to help you decide your prior.
1. Example: The restaurant can’t be good, or it would be more crowded.
(This means your Surprisometer thinks good restaurants are more
common than bad ones.)
2. A volunteer from the campaign visited me; that’s evidence that they’re well-organized.
3. I’m very good at this subject, so my low test score must mean there was a Scantron error.
4. My car keys are missing, because my kids hid them.
5. I believe she’s telling the truth, because she didn’t look away when she told me.
6. My friend hasn’t mentioned the fight again, so he’s not angry.
How to use your Surprisometer to judge Strength of Evidence as a surprisal difference
1 Imagine you knew that your hypothesis was true (you like the book). Would it be surprising to find out that you saw the evidence that you observed (that your friend liked it, too)?
2 Imagine you knew that your hypothesis was false (you don’t like the book). Would it be surprising to find out that you saw the evidence that you observed (that your friend liked it, too)?
3 Compare how far out on your surprise-o-meter each of these hypotheticals fall. a If there’s a large disparity, the evidence is stronger. b If the gap is small, the evidence is weaker.
4 It could be that you’d be very surprised to see this evidence in either case. Whether or not intelligent aliens exist, you’d be surprised to be personally contacted. But the gap is large, even though both scenarios are unexpected.
