Calibrated Estimates
The methods we have talked about so far require subjective evaluation of quantitative probabilities. For example, the cyber security expert will need to assess a probability that an event will occur or how much will be lost if it does. 2 Things are clear though: (1) most people are bad at assigning probabilities, but (2) most people can also be trained to be very good at it. That is the premise of the lab this week.
Deliverable 1: Determine whether the event is Discrete Binary or Continuous for each of the items in the list below
Discrete Binary: An either/or type of event, it happens or doesn’t; expressed as a probability the event will occur.
Continuous: A quantity with a range of possible values; expressed as a confidence interval.
| Example | Type of Probability Distribution |
|---|---|
| A coin flip | Discrete |
| A data breach happens in a given time period | Continuous |
| The duration of a future system outage | Continuous |
| A system goes down | Discrete |
| The size of a future data breach | Continuous |
| “The 90% confidence interval for the duration of a system courage in case X is 30 min to 4 hours” | Continuous |
| “There is a 5% chance of a data breach of PHI that will be required to be reported on the HHS Website” | Continuous |
| The change in sales due to a past data breach | Continuous |
Deliverable 2: Determine the Lower Bound and Upper Bound for a question with a 90% Confidence Interval
**Note: Try to answer these questions without looking up the answers. The intention is to see how accurate your subjective estimates are.
| Question | Lower Bound | Upper Bound |
|---|---|---|
| In 1939, a British steam locomotive set a new speed record by going how fast (Mph) | 100 | 120 |
| In what year did Sir Isaac Newton bullish the Universal law of gravitation? | 1683 | 1700 |
| How many inches long is a typical business card? | 3 | 4.25 |
| The Internet (then called “Arpnet”) was established as a military communications system in what year? | 1960 | 1973 |
| In what year was William Shakespeare born? | 1500 | 1650 |
| What is the air distance between New York and Los Angeles (miles)? | 2000 | 2500 |
| What percentage of a square could be covered by a circle of the same width? | 75 | 80 |
| How old was Charlie Chaplin when he died? | 70 | 90 |
| What is the weight, in pounds, of the fist edition of How to measure anything? | 1 | 4 |
| The TV show Gilligan’s Island first aired on what date? | 1954 | 1980 |
Deliverable 3: Determine Whether the event is True/False and the confidence percentage you think you are right.
**Note: Try to answer these questions without looking up the answers. The intention is to see how accurate your subjective estimates are.
The questions will vary in difficulty. Some will seem easy while others may seem too difficult to answer. No matter how difficult the question seems, you still know something about it. Focus on what you do know.
| Statement | Answer (True/False) | Confidence That you are Correct (50%, 60%, 70%, 80%, 90%,100%) |
|---|---|---|
| The ancient Romans were conquered by the ancient Greeks. | False | 80% |
| There is no species of three-humped camels | False | 90% |
| A gallon of oil weighs less than a gallon of water. | False | 100% |
| Mars is always farther away from Earth than Venus. | False | 50% |
| The Boston Red Sox won the first world series. | True | 75% |
| Napoleon was born on the island of Corsica. | True | 60% |
| “M” is one of the three most commonly used letters | False | 85% |
| In 2002, the price of the average new desktop computer purchased was under $1,500 | False | 60% |
| Lyndon B. Johnson was a governor before becoming vice president. | False | 50% |
| A kilogram is more than a pound. | False | 100% |
Deliverable 4: Now Go back and check your answers to see if you were correct. Then, use the Equivalent Bet test on 3 of your questions to determine if you are under or over confident.
Since the range questions you answered were asking for a 90% Confidence interval, you are, in effect, saying that you expect 9 out of 10 of the true answers to be within your ranges. All you need to do is compare how many answers.
_Example: _Let’s say the one about when Newton published the Universal Laws of Gravitation. Suppose you were offered a chance to win $1,000 in one of these two ways:
- A: You win $1000 if the true year of publication of Newton’s book turns out to be between the dates you gave for the upper and lower bound. If not, you win nothing. Choosing this options means you think there is more than a 90% chance your range contains the answer, even though you are representing yourself as being 90%. This would be UNDERCONFIDENT.
- B: You spin a dial divided into two unequal “pie slices,” one comprising 90% of the dial and the other just 10%. (SEE SLIDES for Visual) If the dial lands on the large slice, you win 1,000. _Choosing this option means that you think the dial has a higher chance of payoff than the range. The conclusion from this is that your confidence might be 50%, 65% or 80%. This would be OVERCONFIDENT.
- C: Indifferent between options A and B. This means you believe you have a 90% chance, not more or less, and that the answer is within your range.
Now use the same example, but a different question, to determine your actual confidence level.
| Question | Over/Under Confident/Neutral | Which Equivalent Bet Option would you pick A, B, C |
|---|---|---|
| In 1939, a British steam locomotive set a new speed record by going how fast (Mph) | Over Confident. | A |
| Lyndon B. Johnson was a governor before becoming vice president. | Under Confident | B |
| A kilogram is more than a pound. | Neutral | A |
Deliverable 5: Reflection
Reflect on how accurate your answers were and strategies you can use to make better informed estimates.
Provide feedback on the lab and how well you understood the material from this week like the slides and lab.
For Deliverable 4 the first question I was extremely confident in my answer but the trains was 6 mph faster than my high range. I feel overall I was pretty good with the ranges just because I tried to think of a center point to base my information off of. For example the Isaac Newton Question I knew he died in the early 1700s so I just thought it had to be 1700. My main strategy for the questions was trying to find a middle point or taking an educated guess that I was confident in. I will say I do like to take risks and chances so even with my educated guesses I would still have a very high confidence rating.