Book Review: Who’s In Charge — Michael Gazzaniga

Philosophers, theologians, and scientists have debated the concept of free will for centuries.  The very likable concept of free will is at odds with the very nature of our observations — things tend to have causes.  And so the question remains, how much decision-making freedom do we really have?  And what’s this conscious “we” concept, while we’re at it?

In the 1800s, a rather well known mathematician made a bold statement with some long-ranging consequences.  He said that if we knew the position and momentum of every particle in the universe, then we could calculate forward in time and predict the future.  This built on the foundations of the physical laws that Isaac Newton observed and was the start of what became known as determinism.  Since the brain is subject to physical laws (in essence, its functionality is a complex set of chemical reactions), this leaves no room for free will, which pretty much everyone believes they have.

“We may regard the present state of the universe as the effect of its past and the cause of its future.  An intellect which at a certain moment would know all forces that set nature in motion, and all positions of all items of which nature is composed, if this intellect were also vast enough to submit these data to analysis, it would embrace in a single formula the movements of the greatest bodies of the universe and those of the tiniest atom; for such an intellect nothing would be uncertain and the future just like the past would be present before its eyes.”

—Pierre Simon Laplace, A Philosophical Essay on Probabilities

As modern neuroscience has developed, our understanding of the brain and of the mind has progressed.  But as with all good research, every good answer leads us to more questions.  Michael Gazzaniga is a neuroscientist and a professor at one of my alma maters.  In Who’s In Charge he takes you on a journey through the concepts of emergence and consciousness, the distributed nature of the brain, the role of the Interpreter, and ultimately how these might change what we think about free will.

The conscious mind has considerably less control over the human body than it would like to believe.  This is the underlying theme of the book.  Gazzaniga’s personal career with split-brain patients (a treatment for severe epilepsy), and his review of modern neuroscience are convincing to that effect.  While it is nice to think that “we” call the shots, it becomes clear that “we” aren’t always in charge and who this “we” is has some interesting properties.

“The brain has millions of local processors making important decisions.  It is a highly specialized system with critical networks distributed throughout the 1,300 grams of tissue.  There is no one boss in the brain.  You are certainly not the boss of the brain.  Have you ever succeeded in telling your brain to shut up already and go to sleep?”

What our conscious self thinks is largely the result of a process that takes place in the left hemisphere of our brain.  Gazzaniga calls it “the interpreter.”  This process’s job is to make sense of things, to paint a consistent story from the sensory information that enters the brain.  Faced with explaining things that it has no good data for, the interpreter makes things up, a process known as confabulation. There is a story of a young woman undergoing brain surgery (for which you are often awake).  When a certain part of her brain was stimulated, she laughed.  When asked why she laughed, she remarked, “You guys are just so funny standing there.”  This is confabulation.

“What was interesting was that the left hemisphere did not say, “I don’t know,” which truly was the correct answer.  It made up a post hoc answer that fit the situation.  It confabulated, taking cues from what it knew and putting them together in an answer that made sense.” 

But the brain is even stranger than that.  If you touch your finger to your nose, the sensory signals from the finger and from the nose take measurably different times to reach the brain.  Different enough that the brain receives the signal from the nose well before it receives the signal from your finger.  It is the interpreter that alters the times and tells you that the two events happened simultaneously.

This is where neuroscience’s contribution to the sensation of free will comes into play.  Gazzaniga says, “What is going on is the match between ever-present multiple mental states and the impinging contextual forces within which it functions.  Our interpreter then claims we freely made a choice.”  This is supported by Benjamin Libet’s experiments which demonstrated that the brain is “aware” of events well before the conscious mind knows about them.  Libet even goes so far as to declare that conscousness is “out of the loop” in human decision making.  This is still hotly debated, but fascinating.

Gazzaniga argues that consciousness is an emergent property of the brain.  Emergence is, in essence, a property of a complex system that is not predictable from the properties of the parts alone.  It is a sort of cooperative phenomenon of complex systems.  Or, as Gazzaniga more cleverly puts it, “You’d never predict the tango if you only studied neurons.”  Emergence is a part of what’s known as complexity theory, which has increased in popularity in the last decade or so.  But at this point, designating something as an emergent property is still really just a way to say you don’t know why something happens.  And despite all the advances that have been made in neuroscience, we still fundamentally don’t understand consciousness.

Gazzaniga makes the case that the development of society likely had much to do with the development of our more advanced cognitive abilities.  That is, as animals developed more social behavior, that increased cognitive skills were necessary, and that this was probably the driving force in the evolution of the neocortex.

“Oxford University anthropologist Robin Dunbar has provided support for some type of social component driving the evolutionary expansion of the brain.  He has found that each primate species tends to have a typical social group size; that brain size correlates with social group size in primates and apes; that the bigger the neocortex, the larger the social group; and that the great apes require a bigger neocortex per given group size than do the other primates.”

There is some physiological evidence to support the relationship between society and neocortical function in the case of mirror neurons.  First discovered in macaque monkeys, when a monkey grabbed a grape, the same neuron fired in both the grape-grabbing monkey and one who watched him grab the grape.  Humans have mirror neurons too, though in much greater numbers.  They serve to create a sympathetic simulation of an event which drives emotional responses.  That is, the way we understand the emotional states of others is by simulating their mental states.  So when Bill Clinton told Bob Rafsky that he felt his pain, perhaps he really did.

This is not Gazzaniga’s first book and it shows.  The work is well planned and executed.  He uses clear language to describe some of the wonderful discoveries of modern neuroscience, and makes them available for laymen to learn and enjoy.  He discusses his own fascinating research, for which he is well known in his field, and also discusses other hot topics in neuroscience and their implications on modern society and also on the free will debate.  He ends the book discussing how modern neuroscience can and should be used in regards to the legal system, which caught me somewhat by surprise.  It is a fine chapter, but it doesn’t read like the rest of the book, feeling like a separate work that was added in as an afterthought.

I enjoyed Who’s In Charge? immensely.  It is an excellent read and will undoubtedly challenge some of your thoughts and enlighten you about how we think about the mind and the brain today.

Bubble Markets, Burst Markets

Wall Street forecasters are notoriously bad at predicting what the markets are going to do.  In fact the forecasts for 2001, 2002, and 2008 were actually worse than guessing.  Granted, predicting the future is a hard job, but when it comes to stock markets, there are some things you can count on.  Disclaimer:  This is a look at the numbers; it is not investment advice.

Let’s take the Standard & Poor’s 500.  It is an index of 500 large US companies stock values, much broader than the Dow Jones’s 30 company average.  It isn’t a stock, and you can’t buy shares in it.  But it is a convenient tool for tracking the overall condition of the stock market.  It may also reflect on the state of the economy, which we’ll look at in a bit.  Below are the monthly closing values of the S&P 500 since 1950.  It’s value was about 17 points in January of 1950 and it closed around 2100 points here in June of 2015.  It’s bounced around plenty in between.

Closing values of the S&P 500 stock index.

Closing values of the S&P 500 stock index.

One of the questions to ask is whether the markets are overvalued or undervalued.  Forecasters hope to predict crashes, but also to look for good buying opportunities.  Short term fluctuations in the markets have proven to be very unpredictable.  But longer term trends are a different story, and looking at them can give huge insights into what’s currently going on.

But first we have to look at the numbers in a different way.  The raw data plot above makes things more difficult than they really need to be because it fails to let you clearly see the trend in how the index grows.  Stocks have a tendency to grow exponentially in time.  This is no secret, and most of the common online stock performance charts give you a log view option.  Exponential growth is why advisors recommend most working people to get into investments early and ride them out for the long haul.

The exponential growth in the S&P is easy to see in the plot below, where I plotted the logarithm of the index value.  For convenience I also plotted the straight line fit to these data — this is its exponential trend.  Note that these data span six and a half decades, so we have some bull and bear markets in there — and whatever came in between them.  And what you see is that no matter what short term silliness was going on, the index value always came back down to the red line.  It didn’t necessarily stay there very long, but the line represents the stability position.  It is a kind of first order term in a perturbation theory model, if you will.  The line shows the value that the short term fluctuations move around.

Here I've taken the logarithm (base 10) of the index values to show the exponential growth trend.  The grey area represents the confidence intervals.

Here I’ve taken the logarithm (base 10) of the index values to show the exponential growth trend. The grey area represents the confidence intervals.

This return to the line is a little bit clearer if we plot the difference between the index and the trend.  This would seem to be a reasonable way to spot overvalued or undervalued markets.  Meaning, that in 2000, when the S&P was some 800 points over its long term model value, the corresponding rapid drop back down to the line should have caught no one by surprise.

Differences between the S&P 500 index value and the exponential trend model value.

Differences between the S&P 500 index value and the exponential trend model value.

But this look at the numbers is a bit disingenuous.  That’s because the value of the index has changed by huge amounts since 1950, so small points swings that we don’t care about at all today were a much bigger deal then.  This makes more recent fluctuations appear to be a bigger deal than they may really be.  So what we want to see is the percentage of the change, not the actual change.

And on top of this, let’s mark recession years (from the Federal Reserve Economic Database) in red.  From this view we can see the bubble markets develop and the resulting panics that result when they burst (hello 2008).  And that every recession brought a drop in the index (some bigger than others), but not every index drop represented a recession.  In the tech bubble of the late 1990s the market was 110% overvalued at its peak.  The crash of 2008 had it drop to about 45%, which is considerably undervalued.  All that in 8 years.  I think it’s safe to call that a panic.  I know it made me panic.

Deviations in the S&P 500 index value from the exponential model are shown as a percentage of the index values.  And recession years (from FRED) are shown in light red.

Deviations in the S&P 500 index value from the exponential model are shown as a percentage of the index values. And recession years (from FRED) are shown in light red.

What we see is that the exponential model does a good job at calculating the baseline (stable position) values.  If it didn’t, the recession-related drops in the index wouldn’t line up with the FRED data, and things like the 1990s bubble and the 2008 financial meltdown wouldn’t match the timeline.  But they do.  Quite well, actually.  So this is a useful analysis tool.

It is also enlightening to take the same looks at the NASDAQ index since it represents a different sector of the stock market.  NASDAQ started in 1971 and is more of a technology focused index.  The NASDAQ composite index is created from all of the stocks listed on the NASDAQ exchange, which is more than 3000 stocks.  So more companies in the index means this is a broader look, but it is focused on tech stocks.

So, as with the S&P above, here are the raw data.  It looks similar to the S&P, and the size of the tech bubble is more clear.  The initial monthly close of the index was 101 points, and it is over 5000 today.

Closing values of the NASDAQ stock index.

Closing values of the NASDAQ stock index.

Not surprising to anyone, this index also grows with an exponential trend.  The NASDAQ was absolutely on fire in the late 1990s.  I wonder if this is what Prince meant when he wanted to party like it was 1999.  Maybe he knew that would be the time to cash out?

Here I've taken the logarithm (base 10) of the index values to show the exponential growth trend.  The grey area represents the confidence intervals.

Here I’ve taken the logarithm (base 10) of the index values to show the exponential growth trend. The grey area represents the confidence intervals.

The size of the dot-com bubble is clearer if we look at the deviation from the model, as we did with the S&P.  At the height of the tech bubble, the NASDAQ was about 3500 points overvalued.  Considering that the model puts its expected value at about 1300 points in 2000, I have to ask myself, what were they thinking?

Differences between the NASDAQ index value and the exponential trend model value.

Differences between the NASDAQ index value and the exponential trend model value.

The percent deviation plot shows this very clearly.  At the height of the tech bubble, the NASDAQ was some 275% overvalued, almost three times that of the S&P 500’s overvalue.  Before the late 1990s the NASDAQ had never strayed more than about 50% from the model value.  Warren Buffet has said that the rear view mirror is always clearer than the windshield, but maybe Stevie Wonder shouldn’t be the one doing the driving.

Deviations in the NASDAQ index value from the exponential model are shown as a percentage of the index values.  And recession years (from FRED) are shown in light red.

Deviations in the NASDAQ index value from the exponential model are shown as a percentage of the index values. And recession years (from FRED) are shown in light red.

From this perspective, the NASDAQ today actually looks a few percentage points undervalued, so tech still seems to be a slightly better buy than the broader market (this is not investment advice).

Not only that, but the growth model of the NASDAQ, based on its 45 years of data, shows that it grows considerably faster than the broader market.  If you go back and look at the raw data for either of the two indices, you’ll notice something special about the nature of exponential growth.  The time it takes to double (triple, etc.) is a constant.  As these are bigger numbers and because it is convenient, let’s look at the time it takes to grow by a factor of ten (decuple).  The S&P 500 index decuples every 33.3 or so years.  The NASDAQ composite, on the other hand, decuples every ~24 years (about 23 years and 11 months, give or take).  This has huge implications for growth.  That’s nine fewer years to grow by the same factor of 10.

Now comes the dangerous part.  Let’s take the both of these indices and forecast their model values out thirty years.  Both of the datasets contain more than thirty years worth of data, so forecasting this far out is a bit of a stretch, but not without some reasonable basis.  Still, this is an exercise in “what if,” not promises, and certainly not investment advice.

Since we started with the S&P, let’s look at that first.  If the historic growth trends continue, the model forecasts that the S&P 500 (currently around 2000 points) should be bouncing around the 10,000 point mark some time in the middle of 2038.

S&500 data, along with its exponential model fit, extended out thirty years.  The grey area represents the confidence intervals.

S&500 data, along with its exponential model fit, extended out thirty years. The grey area represents the confidence intervals.

The NASDAQ, on the other hand, which is currently around 5000 points, should average around 10,000 in late 2021, and 100,000 near the end of 2045.  (Note: the S&P should be around 16,000 points at that time).  Today the ratio of the NASDAQ to the S&P is about 2.4.  But in 2045 it could reasonably be expected to be more than 6.  Depending on the number of zeroes in your investment portfolio (before the decimal point…), that could be significant.

NASDAQ data, along with its exponential model fit, extended out thirty years.  The grey area represents the confidence intervals.

NASDAQ data, along with its exponential model fit, extended out thirty years. The grey area represents the confidence intervals.

This forecasting method will not predict market crashes.  But that’s OK, because the professionals who try to forecast them can’t do that either.  (Now if only Goldman-Sachs would hire me.)  What it can do is give us a very clear idea of the market is over or under valued.  By forecasting the stable position trend, we can easily spot bubbles, identify their size, and perhaps make wise decisions as a result.

The Adjacent Possible and the Law of Accelerating Returns

A concept that inventor and futurist Ray Kurzweil drives home in his books is what he calls the Law of Accelerating Returns.  That is, the observation that technology growth (among other things) follows an exponential curve.  He shows this for no small number of pages for varying technologies and concepts.  Most famous is Moore’s Law, in which Gordon Moore (one of the founders of Intel Corporation) observed that the number of transistors on a die doubled in a fixed amount of time (about every two years).  Kurzweil argues that this exponential growth pattern applies to both technological and biological evolution. In other words, that progress grows exponentially in time.  It should be clear that this is an observation rather than something derived from fundamental scientific theories.

What makes this backward looking observation particularly interesting is that in spite of our observation of it as generally true over vast periods of time, humans are very linear thinkers and have a difficult time envisioning exponential growth rates forward in time.  Kurzweil is a notable exception to that rule.  Because of exponential growth, the technological progress we make in the next 50 years will not be the same as what we have realized in the last 50 years.  It will be very much larger.  Almost unbelievably larger — the equivalent of the progress made in the last ~600 years.  This is the nature of exponential growth (and why some people find Kurzweil’s predictions difficult to swallow).

Interestingly, when a survey of scientific literature was done by Derek Price in 1961, an exponential growth in scientific publications was readily observed, but dismissed as unsustainable.  This unsustainability in the growth rate was understood to be obvious by Price.  The survey was revisited in 2010 (citing the original work), with the exponential growth still being observed 39 years later.  So this linear forecasting is a handicap that seems to exist even when we have the data to the contrary staring us in the face.

On the other hand we have biologist Stuart Kauffmann.  He introduced the concept of the Adjacent Possible which was made more widely known in Steven Johnson’s excellent book, Where Good Ideas Come From.  The Adjacent Possible concept is another backwards-looking observation that describes how biological complexity has progressed through the combining of whatever nature had on hand at the time.  At first glimpse this sounds sort of bland and altogether obvious.  But it is a hugely powerful statement when you dig a little deeper.  This is a way of defining what change is possible.  That combining things that already exist is how things of greater complexity are formed.  Said slightly differently, what is actual today defines what is possible tomorrow.  And what becomes possible will then influence what can become actual.  And so on.  So while dramatic changes can happen, only certain changes are possible based on what is here now.  And thus the set of actual/possible combinations expands in time, increasing the complexity of what’s in the toolbox.

Johnson describes it in this way:

“Four billion years ago, if you were a carbon atom, there were a few hundred molecular configurations you could stumble into.  Today that same carbon atom, whose atomic properties haven’t changed one single nanogram, can help build a sperm whale or a giant redwood or an H1N1 virus, along with a near infinite list of other carbon-based life forms that were not part of the adjacent possible of prebiotic earth.  Add to that an equally list of human concoctions that rely on carbon—every single object on the planet made of plastic, for instance—and you can see how far the kingdom of the adjacent possible has expanded since those fatty acids self-assembled into the first membrane.” — Steven Johnson, Where Good Ideas Come From

Kauffmann’s complexity theory is really an ingenious observation.  Perhaps what is most shocking is that, given how obvious it is in hindsight, no one managed to put it into words before.  I should note that Charles Darwin’s contemporaries expressed the same sentiments.

What is next most shocking is that Kauffman’s observation is basically the same as Kurzweil’s.  We have to do a little bit of math to show this is true.  I promise, it isn’t too painful.

The Adjacent Possible is all about combinations.  So first let’s assume we have some set of n number of objects.  We want to take k of them at a time and determine how many unique k-sized combinations there are.  This is popularly known in mathematics as “n choose k.”  In other words, if I have three objects, how many different ways are there to combine them two at a time?  That’s what we’re working out.  There is a shortcut in math notation that says if we are going to multiply a number by all of the integers less than it, that we can write the number with an exclamation mark.  So 3x2x1 would simply be written as 3!, and the exclamation mark is pronounced “factorial” when you read it.  This turns out to be very helpful in counting combinations.  Our n choose k counting problem can then be written as:

Math5

You can try this out for relatively small numbers for n and k and see that this is true.

The pertinent question, however, is what are the total number of combinations for all possible values of k.  That is, if I have n objects, how many unique ways can I combine them if I take them one at a time, two at a time, three at a time, etc., all the way up to the whole set?  To find this out you evaluate the above equation for all values of k from 0 all the way to n and sum them all up.  When you do this you find that the answer is 2^n. Or written more mathematically:

Math1

So as an example, let us take 3 objects (n=3), let’s call them playing cards, and count all of the possible combinations of these three cards, as shown in the table below.  Note that there are exactly 2^3=8 distinct combinations.  Here a 1 in the row indicates a card’s inclusion in that combination.  We have no cards, all combinations of one card, all combinations of two cards, and then all three cards, for a total of 8 unique combinations.

Card 3 Card 2 Card 1
0 0 0
0 0 1
0 1 0
0 1 1
1 0 0
1 0 1
1 1 0
1 1 1

You can repeat this for any size set and you’ll find that the total number of unique combinations of any size for a set of size n will always be 2^n.  If you are familiar with base 2 math, you might have recognized that already.  So for n=3 objects we have the 2^3 (8) combinations that we just saw.  And for n=4 we get 2^4 (16) combinations, for n=5 we have 2^5 (32) combinations, and so on.

So in other words, the number of possible combinations grows exponentially with the number of objects in the set.  But this exponential growth is exactly what Kurzweil observes in his Law of Accelerating Returns.  Kurzweil simply pays attention to how n grows with time, while Kauffman pays attention to the growth of (bio)diversity without being concerned about the time aspect.

Kauffman uses this model to describe the growth in complexity of biological systems.  That simple structures first evolved, and that combinations of those simple things made structures that were more complex, and that combinations of these more complex structures went on to create even more complex structures.  A simple look at any living thing shows a mind-boggling amount of complexity, but sometimes it is obvious how the component systems evolved.  Amino acids lead to proteins.  Proteins lead to membranes.  Membranes lead to cells.  Cells combine and specialize.  Entire biological systems develop.  Each of these steps relies on components of lower complexity as bits of their construction.

Kurzweil’s observation is one of technological progress.  That the limits of ideas are pushed through paradigm after paradigm, but still it is the combination of ideas that enable us to come up with the designs, the processes, and materials that get more transistors on a die year after year.  That is to say, semiconductor engineers 30 years ago had no clues how they would get around the challenges they faced in reaching today’s level of sophistication.  But adding new ways of thinking about the problems lead to entirely new types of solutions (paradigms) and the growth curve kept its pace.

Linking combinatorial complexity to progress gives us the modern definition of innovation.  That innovation is really the exploring and exploiting of the Adjacent Possible.  It is easy to look back in time and see the exponential growth of innovation that has brought us to the quality of life we have today.  It is much easier to dismiss it continuing on because we are faced with problems that we don’t currently have good ideas about how to solve.  What we see from Kurzweil’s and Kauffman’s observations is that the likelihood of coming up with good ideas, better ideas, life-changing ideas, increases exponentially in time, and happily, we have no good reason to expect this behavior to change.

The USPS and You

Every day but Sunday, a government employee comes to that place you call home and leaves you with any number of items.  Packages perhaps, but certainly letters, bills, advertisements, or magazines.  Most of these are sent to you by complete strangers.  Is there something interesting or valuable that can be learned by paying attention to what arrives in the mailbox?

Questions we might want to ask:  “How much mail do I get?”, “Who sends me mail?”, “How often do they send it?”, and “What kinds of mail do I get?”  Advertisers certainly have each one of us in their databases.  I’m sort of curious to know something about what they think they know about me.  But I’m also eager to explore what can be learned by simply paying attention to something that goes on around me with a high degree of regularity.

I’ve mentioned this before, but my methods here are to record the sender and the category of each piece of mail I receive daily.  This is for mail specifically directed to me, or not specifically directed to anyone (i.e. “Resident”).  I’ve been doing this since the end of July 2014, so I have a fair amount of data now.

Let’s start with quantity.  On average I’m getting about 100 pieces of mail per month.  This is pretty consistent over 8 months, but note that things picked up at the beginning of November and then dropped back in January.  The rate (i.e., slope) didn’t really change, there was just a shift in the baseline.  The November shift is undoubtedly from election related mail.  The January shift is the post-Christmas dropoff that we’ll see later.

Cumulative amount of delivered mail.

Cumulative amount of delivered mail.

One of the more interesting observations is the breakdown of the mail by category.  It should come as no surprise these days that the majority of mail is advertising.  If you include political adverting (a category I break out separately), this overall advertising category accounts for more than half of the mail I get in my letterbox.  Considering that the USPS’s own numbers suggest about 52% of the mail was advertising in 2014, it looks like my dataset is representative.  Interestingly, the percentage of mail that was advertising in 2005 was only about 47%, so the percentage of mail that is advertising is on the increase.  This is not unexpected.  The NY Times published a piece in 2012 indicating that the Postal Service had announced their plan for addressing the huge decreases first class mail.  It was to focus on increasing the amount of advertising mail that they carry.  The Wall Street Journal has a piece from 2011 showing that the advertising percentage was only about 25% in 1980 and has been increasing steadily ever since.  Mission accomplished.

Categorical percentages of delivered mail.

Categorical percentages of delivered mail.

The next largest category, “Information”, is communications from people that I know or businesses that I deal with.  In other words, mail I want or care about in some fashion.  This is about 22% of the total.  Bills are a separate category as I think they are different enough to track separately.  Yes I still get magazines.  No I don’t wish to convert to a digital subscription.  But thank you for asking.

I find it interesting to look at the breakdown of the composition of the mail over time.  Judging from the sharp changes in color in the largest category (bottom bar), you can probably guess when the last state primary and general election took place.  But note that in general, each week is dominated by advertisements.  Notable times that this is not true are the week leading up to an election, when political advertisements dominate (note that these are still advertisements), and the weeks leading up to Christmas.  This last week shows an increase in “Information” mail largely because of Christmas Cards.

Weekly mail by category.  Note that 2015 began mid-week.

Weekly mail by category. Note that 2015 began mid-week.

Let’s look more closely at the advertisement mail numbers all by themselves.  October was the peak month, which is somewhat surprising given the frenzy over the Black Friday shopping.  Predictably, direct mail fell off in January after the end of the Christmas shopping season.  But somewhat surprisingly it climbs back without too much delay.

Amount of advertising mail received each month.

Amount of advertising mail received each month.

So who exactly is it that sends me so much junk mail?  Good question.  Redplum is the biggest of them all by far.  Also known as Valassis Communications, Inc., they provide media and marketing services internationally, and they are one of the largest coupon distributors in the world.  In other words, they’re a junk mail vendor.  You can count on them, as I’m sure the USPS does, for a weekly delivery of a collection of ads contained inside of their overwrap.  After that I have Citibank, Bank of America, SiriusXM, and Geico, in that order.  I would not have expected Geico to show up this high on the list, but there they are.

The amount of advertising mail sorted by sender.

The amount of advertising mail sorted by sender, restricted to those with 2 or more pieces of mail being delivered.

Another question to consider is when does all this mail come?  We looked before at the monthly advertisement mailings numbers, but we can dig a little deeper and look at how mail deliveries vary by weekday.  If we look at raw numbers, we notice that Friday is by far the biggest mail day in terms of the number of items received.  This has been consistently true for the entire time I have been analyzing my mail.  I don’t have a good explanation for this observation.

The quantity of mail, by category, with the day of the week it was delivered.

The quantity of mail, by category, with the day of the week it was delivered.

But there’s more to it than just that.  We don’t get mail every weekday.  Lots of federal holidays fall on Mondays where there is no mail delivery.  What we really want to do is to look at how much mail we get for every day that mail was actually delivered.  This lets us compensate for an uneven amount of delivery weekdays.  When we do this, we find things even out quite a bit.  Big Friday is still the king, but the other days even out quite nicely.  Understanding what is going on with Friday deliveries is something I’m interested in.

Mail by category each weekday, normalized to the number of days mail was delivered each weekday.

Mail by category each weekday, normalized to the number of days mail was delivered each weekday.

What you can see from all this is that you are (or I am, in any case) more likely to get certain types of mail on some days than on others.  This is somewhat easier to see if we plot each category by itself.  I find it remarkable to see that I basically don’t get bills on Wednesdays.  Credit card applications come primarily Saturdays.  Charities don’t ask me for money on Mondays.  And political ads come on Thursdays and Fridays.  I’ll bet that if I further broke down the advertisement category into senders that more weekday specificity would emerge.

Normalized daily mail categories per weekday.

Normalized daily mail categories per weekday.

In the interest of completeness, we finish up by looking at the statistics of the daily mail delivery.  That is, how often do we get some particular number of pieces of mail in the letterbox?  Here we don’t concern ourselves with the category, only the quantity and how many times that quantity shows up.  We can see from the plot that we most often find three pieces of mail and have never found more than thirteen.  This distribution in quantities approximately follows what is known as a Poisson distribution.  It has nothing to do with fish, but rather was named after a French mathematician Siméon Denis Poisson.  The red line fit is a scaled Poisson distribution with the average (lambda) equal to 3.5.  This indicates that, on average, I get 3.5 pieces of mail daily.  This is slightly lower than the mean value from the plots above of 3.9, but they’re calculated in slightly different ways and have somewhat different meanings.

The distribution of mail quantities follows a Poisson distribution.

The distribution of mail quantities follows a Poisson distribution.

The most unexpected things that I have observed are the Big Friday effect, and the amount of regularity in the weekly of delivery of some specific types of mail.  As they have endured over eight months of data collection, I am inclined to think they are real, but it will be interesting to watch and see if they exist after an entire year of mail collection.  It is also interesting that the Wikipedia article on the Poisson distribution specifically mentions that it should apply to mail, seemingly appropriately, but I can find no record anywhere that anyone has actually done this experiment.

Followup: The Effect of Elections on Gasoline Prices

My intention for the last post, The Effect of Elections on Gasoline Prices, was to be as thorough and quantitative as possible.  A friend who is properly trained in statistics pointed out the need to run significance tests on the results.  This is good advice and the analysis will be complete with its inclusion.

That last post ended with a visualization of the non-seasonal changes in gasoline prices in the months leading up to the election (August to November) for election years (Presidential or midterm), and used the same data in the same timeframe in non-election years as a control.  We used inflation-adjusted, constant 2008 dollars to properly subtract the real seasonal changes and discover real trends in the analysis.  That final figure (below) clearly showed that there is no trend of election-related price decreases.  In fact, prices have tended to increase somewhat as the election nears.  But the question that I failed to adequately address last time is:  Are the price changes in election years significantly different from those of non-election years?  This is the definitive question.

Non-seasonal, August to November changes in U.S. regular unleaded gasoline prices from 1976 to 2013.  The comparison is made for election and non-election years.  Original data source is the U.S. Bureau of Labor Statistics.

Non-seasonal, August to November changes in U.S. regular unleaded gasoline prices from 1976 to 2013. The comparison is made for election and non-election years. Original data source is the U.S. Bureau of Labor Statistics.

Because any sampled data set will suffer from sampling errors (it would be extremely difficult for every gas station in the country to be included in the BLS study each month), the sampled distribution will differ somewhat from the actual distribution.  This is important because we frequently represent and compare data sets using their composite statistical values, like their mean values.  And two independent samplings of the same distribution will produce two sets with different mean values; this makes understanding significant differences between them an important problem.  What we need is a way to determine how different the datasets are, and if these differences are meaningful or if they are simply sampling errors (errors of chance).

Fortunately we are not the first to need such a tool.  Mathematicians have developed a way to compare datasets to determine if their differences are significant or not.  These are “tests of significance.”  The t-test is one of these tests and it determines the probability that the differences between the means of the two distributions are due to chance. The first thing we should do is look at the distributions of these price changes.  The two large election-year price drops (2006, 2008) are very clearly seen to be outliers, and the significant overlap of the distribution of price changes is readily visible.

Distributions of non-seasonal, August to November changes in U.S. regular unleaded gasoline prices from 1976 to 2013. Original data source is the U.S. Bureau of Labor Statistics.

Distributions of non-seasonal, August to November changes in U.S. regular unleaded gasoline prices from 1976 to 2013 for both election and non-election years. Original data source is the U.S. Bureau of Labor Statistics.

It is clear that were it not for the outliers in the election year data, these distributions would be considered to be very nearly identical.  But to characterize the significance of their differences, we’ll run an independent t-test.  The primary output of the test that we are concerned with is the p-value.  This is the probability that differences between the two distributions are due to chance.  Recall that the maximum value of a probability is 1.  If it matters, I’m using R for data analysis.

Welch Two Sample t-test

data:  electionyear$changes and nonelectionyear$changes
t = -0.6427, df = 21.385, p-value = 0.5273
alternative hypothesis: true difference in means is not equal to 0
95 percent confidence interval:
  -0.2530637 0.1334810
sample estimates:
  mean of x mean of y 
-0.02367507 0.03611627

This p-value tells us that there is a 52.7% probability that differences between these two distributions are chance.  The alternative hypothesis is then rejected and the difference in means is the same as 0.  This answers the question that we posed and indicates that the changes in gas prices in election years are not significantly different from those of non-election years.

The Effect of Elections on Gasoline Prices

A quick web search shows that I’m not the only one who’s heard the talk about how gasoline prices always decline before an upcoming election.  Of course, this always gets mentioned when local gas prices are declining and an election is coming up.  But is this actually true?  Do gas prices in the United States decrease leading up to an election?  There are lots of articles written about this topic, and some even use numbers and statistics to back up their position, but I intend to be a bit more thorough here.

To get started, we’ll use the inflation-adjusted price of gasoline that we get from the U.S. Bureau of Labor Statistics.  We’ve looked at these data in a previous post, and if you’re at all interested in how constant-dollar costs are calculated, you should go read that post first.  This dataset goes back to 1976, so it includes a sizable number of election years.  Nineteen, to be precise.  It is important to use inflation adjusted data in this analysis because it compensates for the price changes from the changing buying power of the dollar.  Ten cent changes in the price of gas in 1980 and in 2010 aren’t the same, and inflation-adjusted prices account for this.

Unleaded regular gasoline prices from 1980 to 2014 in constant 2008 dollars.  Source:  U.S. Bureau of Labor Statistics.

Unleaded regular gasoline prices from 1980 to 2014 in constant 2008 dollars. Source: U.S. Bureau of Labor Statistics.

The first thing we should do is to look for annual trends.  It is entirely reasonable to expect that the price of gas shows some regular changes each year, so we should first understand if this happens and by how much so that we can account for it in our analysis.  To do this, we break the above graph up into its parts.  We separate the observed data into the parts that repeat on an annual basis (seasonal changes), the parts that change more slowly (long-term trend), and the parts that change more quickly (remainder).  If we add these all parts back together we will get the original observation.  For gasoline, this additive decomposition gives us the results plotted below.  Note the differences in the y-axis scales for the different plots.  The “trend” component is the largest.  Seasonal variations swing about seventeen cents from high to low, but the remainder, the fast-changing non-periodic fluctuations, can be in excess of a dollar, though they generally exist inside of a quarter of a dollar on either side of zero.

Decomposition of U.S. Gasoline prices into seasonal and other components.  Original data source:  U.S. Bureau of Labor Statistics.

Decomposition of U.S. Gasoline prices into seasonal and other components. Original data source: U.S. Bureau of Labor Statistics.

The seasonal component here is the part we’re interested in first.  There are a couple of ways to extract this component in a decomposition, but because we are using constant-dollar prices and looking for the true seasonal fluctuations, we don’t want to window the filter at all.  So let’s take a closer look at that seasonal component.  Recall that this is the component that occurs repeatably every year since 1976.

Seasonal change in U.S. gasoline prices (regular unleaded) in constant 2008 dollars.  Original source:  U.S. Bureau of Labor Statistics.

Seasonal change in U.S. gasoline prices (regular unleaded) in constant 2008 dollars. Original source: U.S. Bureau of Labor Statistics.

Not too surprisingly, we see there is a 7.7 cent increase in the summer, peaking in the June driving season, and a 9.2 cent decrease in gas prices in the winter, with the bottom arriving in December.  Interestingly, since elections are in November, they occur during the natural seasonal price decline.  This will not be a problem for us.

The key to determining whether or not non-seasonal conditions (i.e., elections) impact prices is simply to use the non-seasonal components of the prices for comparison.  That is, by considering only the trend and remainder components.  By excluding the seasonal fluctuations we can see how the non-seasonal prices changed, and from there the effect of elections on price changes will be able to be observed.

For the sake of simplicity, let’s define the time before the election that we are interested in to August through November.  This is an assumption on my part.  But the last quarter before the election seems to be the time we should pay the most attention to.  We can repeat the analysis using any other window of time if we desire.  Since we’ve already identified the seasonal changes in the decomposition, this is straightforward.  I’ve highlighted the pre-election time on the following graph for ease of viewing.

Non-seasonal changes in U.S. regular unleaded gasoline prices, in constant 2008 dollars during election years from 1976 to 2012.  Original source:  U.S. Bureau of Labor Statistics.

Non-seasonal changes in U.S. regular unleaded gasoline prices, in constant 2008 dollars during election years from 1976 to 2012. Original source: U.S. Bureau of Labor Statistics.

So when we do this we find the following for election year, non-seasonal gas price changes:

  • 1976:  +$0.08/gallon
  • 1978:  +$0.09/gallon
  • 1980:  -$0.02/gallon
  • 1982:  No Change
  • 1984:  +$0.10/gallon
  • 1986:  +$0.05/gallon
  • 1988:  +$0.02/gallon
  • 1990:  +$0.37/gallon
  • 1992:  +$0.09/gallon
  • 1994:  +$0.07/gallon
  • 1996:  +$0.10/gallon
  • 1998:  +$0.07/gallon
  • 2000:  +$0.14/gallon
  • 2002:  +$0.12/gallon
  • 2004:  +$0.21/gallon
  • 2006:  -$0.64/gallon
  • 2008:  -$1.40/gallon
  • 2010:  +$0.20/gallon
  • 2012:  -$0.10/gallon

So only four out of the last 19 election years have shown a drop in gas prices that were not part of the normal seasonal variation.  And only two of those were by more than a dime.  The average change here is a 2.3 cent drop, but that is very heavily influenced by the 2008 drop of $1.40/gallon, statistically an outlier.  The median value of an 8.1 cent increase is more in line with the typical behavior.  And of the years when the prices don’t drop, the average increase is 11.5 cents.  In other words, there is no election-year drop in gasoline prices using the BLS data.

We should ask ourselves how these election year results differ from those of non-election years.  This is also straightforward to answer.

Non-seasonal changes in U.S. regular unleaded gasoline prices, in constant 2008 dollars during non-election years from 1977 to 2013. Original source: U.S. Bureau of Labor Statistics.

Non-seasonal changes in U.S. regular unleaded gasoline prices, in constant 2008 dollars during non-election years from 1977 to 2013. Original source: U.S. Bureau of Labor Statistics.

And we find the following non-election year, non-seasonal gas price change results (August to November):

  • 1977:  +$0.06/gallon
  • 1979:  +$0.17/gallon
  • 1981:  +$0.04/gallon
  • 1983:  -$0.02/gallon
  • 1985:  +$0.04/gallon
  • 1987:  +$0.05/gallon
  • 1989:  -$0.01/gallon
  • 1991:  +$0.08/gallon
  • 1993:  +$0.11/gallon
  • 1995:  +$0.01/gallon
  • 1997:  +$0.04/gallon
  • 1999:  +$0.10/gallon
  • 2001:  -$0.09/gallon
  • 2003:  No Change
  • 2005:  -$0.09/gallon
  • 2007:  +$0.35/gallon
  • 2009:  +$0.13/gallon
  • 2011:  -$0.09/gallon
  • 2013:  -$0.20/gallon

And so we find that just six of the nineteen non-election years showed August to November gas price decreases, and only one of those was more than a dime drop.  The average price change in non-election years is a 3.6 cent increase with the median value of a 3.8 cent increase (it is nice when they agree).  And for the years that show an increase, the average change is a rise of 9.9 cents.  I think it is easier to grasp this visually.

Non-seasonal, August to November changes in U.S. regular unleaded gasoline prices from 1976 to 2013.  The comparison is made for election and non-election years.  Original data source is the U.S. Bureau of Labor Statistics.

Non-seasonal, August to November changes in U.S. regular unleaded gasoline prices from 1976 to 2013. The comparison is made for election and non-election years. Original data source is the U.S. Bureau of Labor Statistics.

This tells us that the non-seasonal median gas price change between August and November in an election year actually increases by 4.3 cents/gallon (in constant 2008 dollars) in an election year compared to the same time frame in a non-election year.  The caveat here is that we are dealing with national prices instead of local, but I think we can call this myth busted.

The Cost of Things: A Constant-Dollar Look at Common Goods

The other day I went to the grocery store to get some things to make tacos.  We often end up getting ground turkey these days, but I admit to preferring ground beef for this sort of thing.  But after I looked at the prices of ground beef, I believe I understand why we usually get turkey.  In case you haven’t noticed, the price for ground beef is on the rise, having crossed $4/lb in August of 2014 with no looking back.  This perplexes me somewhat as ground beef comes from cows, which are not dangerous, rare, or hard to kill.  Oh, for the cheap ground beef days of the 1980s, right?

Average U.S. prices for ground chuck (100% Beef) from 1980 to 2014.   Source:  U.S. Bureau of Labor Statistics.

Average U.S. prices for ground chuck (100% Beef) from 1980 to 2014. Source: U.S. Bureau of Labor Statistics.

Now any time you want to compare prices over some length of time you run into the same problem. The buying power of the dollar is not constant. It changes over time, and this makes it somewhat difficult to tell whether something is getting more or less expensive or if the buying power of the dollar is changing.  Or both.  The folks at the U.S. Bureau of Labor Statistics track the buying power of the dollar each month by finding out what it costs to purchase a basket of particular items in the marketplace. The result of their work is called the Consumer Price Index, or CPI.  So whether the cost of something actually changes depends on how it changes compared with the index.  To see this, we use the index to adjust past prices to reflect what they would have been if the dollar had kept a constant buying power.

This method isn’t perfect because it only reflects the buying power of the dollar at some point in time and doesn’t directly consider the change in the money supply.  You might have thought that the amount of money in the U.S. economy was a fixed number, and that would be a reasonable assumption.  But it would be a very wrong assumption.  An increase in the money supply will decrease buying power but takes some time to be recognized by the market.  Because of this, the CPI will lag somewhat behind actual values.  In any case, we won’t let perfection be the enemy of good enough.

In order to use the CPI to compare historic prices, we pick a year and normalize everything with respect to that year’s value.  Below is a plot of the CPI using 2008 as the reference year (where the CPI = 1).  To use the chart, we can look and see that in 1985 the CPI was one-half, and this indicates that prices doubled between 1985 and 2008.  Specifically, this means that the prices of the things in the BLS’s market basket doubled.  That’s another way of saying the buying power of the dollar was cut in half in that time period.  So much for your savings.

Consumer Price Index (CPI) from 1980 to 2014, using 2008 as the reference.

Consumer Price Index (CPI) from 1980 to 2014, using 2008 as the reference.

With this information in hand, we can have another look at the prices of ground beef in constant dollars.  And now we see that the cost of ground beef was surprisingly high in the 1980s but decreased until about 2000.  It then started its way up sharply around the middle of 2013 leading up to today’s prices.  So ground beef really is getting more expensive, and by no small amount.  Not good for taco lovers who prefer beef to turkey.  And before you ask, they don’t track ground turkey prices, and I don’t know why.

Historic pricing of ground chuck in the U.S. in constant 2008 dollars.  Source:  U.S. Bureau of Labor Statistics.

Historic pricing of ground chuck in the U.S. in constant 2008 dollars. Source: U.S. Bureau of Labor Statistics.

On a side note, who would have ever thought, at the time, that we would look back at the 1980s as the days of the strong dollar?

In any case, the BLS tracks the prices of a lot of other products.  We can get a feel for the health of the economy by looking at how the cost of things have changed using constant-valued currency.  So let’s continue with food.  Specifically, chicken.  The inflation adjusted cost of fresh, whole chicken in constant 2008 dollars shows that chicken prices have been stable since about 1990, which means the price at the register has increased at about the rate of inflation.  Hopefully your income has as well.

BLS data for U.S. fresh whole chicken prices in constant 2008 dollars. Source: U.S. Bureau of Labor Statistics.

BLS data for U.S. fresh whole chicken prices in constant 2008 dollars. Source: U.S. Bureau of Labor Statistics.

Milk is an interesting product to look at because it one of the items in the BLS market basket that is used to calculate the CPI.  Because of this, we should expect the cost of milk to move pretty much in line with the CPI and give us a flat cost curve, which it does—on average.  These numbers are nationwide averages, so some fluctuations should be expected.  But we also have (or have had in any case) minimum prices federally guaranteed at various times during this period.  Meaning there’s no limit to how high milk prices can go, but prices won’t fall below a set value.  And if that happens, your tax dollars are used to pay for the milk you didn’t buy in the store.

Milk prices from 1980 to 2014 in constant 2008 dollar prices. Source: U.S. Bureau of Labor Statistics.

Milk prices from 1980 to 2014 in constant 2008 dollar prices. Source: U.S. Bureau of Labor Statistics.

Sugar is the last of the food products that we’ll look at.  Sugar is another one of those agricultural products with price supports, but this one isn’t in the BLS market basket.  Interestingly, the supported prices don’t generally adjust in line with inflation.  So while the retail price of sugar (not shown) has been largely steady throughout the decades, the inflation-adjusted true cost of it has been coming down.  Interestingly, it was the production shortages in 1979 and and 1980 that lead to the soaring prices in the early 1980s that ultimately lead to the switch to high fructose corn syrup (HFCS) by food and beverage companies.  Other references show similar or larger spikes in sugar prices in the 1960s and 1970s.  It isn’t clear that it was worth it, but sugar hasn’t been quite so volatile since the switch.

Sugar prices from 1980 to 2014 in constant 2008 dollars. Source:  U.S. Bureau of Labor Statistics.

Sugar prices from 1980 to 2014 in constant 2008 dollars. Source: U.S. Bureau of Labor Statistics.

There are some other interesting products to look at. Electricity is one of them. I haven’t seen anything that shows seasonal price fluctuations quite as clearly as the electricity cost chart.  What is especially interesting here is that sometimes the retail price (not shown) of electricity increased while the cost in constant dollars came down.  Being a regulated public utility, the prices for electricity are generally not market-driven, but based on cost-recovery, so we can read from this that the costs of the electric utility business have been decreasing fairly consistently since the mid 1980s.  Here in New England they’re talking about 40% increases in electricity rates in the next year or so, which would take us back to the historically high 1980s costs.  There is no historic precedent for that kind of single year price increase.  At least, not in the last three decades.

The cost of electricity in the U.S. from 1980 to 2014 in constant 2008 dollars.  Source:  U.S. Bureau of Labor Statistics.

The cost of electricity in the U.S. from 1980 to 2014 in constant 2008 dollars. Source: U.S. Bureau of Labor Statistics.

Since we are on the topic of energy, it makes sense to look at gasoline.  Gasoline showed a fairly flat cost curve from the mid 1980s though the early 2000s before it went all to hell.  There was a significant correction in late 2008 which is curious.  And in spite of the fact that the prices are on the decline, they’re still historically high by around a dollar per gallon.  I always hear about how gasoline prices decline in election years.  That isn’t clear from this look, so that will be a topic I’ll dig into soon.

Unleaded regular gasoline prices from 1980 to 2014 in constant 2008 dollars.  Source:  U.S. Bureau of Labor Statistics.

Unleaded regular gasoline prices from 1980 to 2014 in constant 2008 dollars. Source: U.S. Bureau of Labor Statistics.

Staying with energy, for those who heat with natural gas, you’ll see that current rates are close to average lows.  But costs have clearly been on a roller coaster for the last 15 years.

Natural gas prices from 1980 to 2014 in constant 2008 dollars.  Source:  U.S. Bureau of Labor Statistics.

Natural gas prices from 1980 to 2014 in constant 2008 dollars. Source: U.S. Bureau of Labor Statistics.

So what does this tell us?  First, that someone has to be making some serious money in the ground beef industry.  The inflation-adjusted cost of ground beef is growing well above the rate of inflation, and I’ll bet that the difference is being pocketed by someone clever enough to pull it off.  Second, I was surprised to see how some of the inflation-adjusted prices have actually declined.  My instincts were to guess that I would see most costs on the rise, but flat or decreasing costs seem to be the rule and not the exception.  This is, of course, a simple survey rather than an all-encompassing study.  For all I know, every other product is skyrocketing.  Given the slope of the inflation curve, retail prices are doing a good job of increasing at the register.  May your income increase ever faster.

Feeding America: The Extraordinary Increase in US Farm Productivity (Part 2)

After my last post on the remarkable increase in field crop yields in American agriculture, I was interested in seeing where else yield improvements were observed and hopefully getting a better understanding of the causes.  I’m interested in understanding how much of the impact mechanization (i.e tractors and harvesters) had as opposed to improvements in seed, fertilizers, and pesticides.  Certainly all of these came together to boost productivity, but what had the greatest impact?

So on to (non-field crop) vegetables.  The same USDA website has statistics for vegetables as well as for field crops, but to get any data of any historical significance, we can’t afford to be choosy.  Only two crops have data going back to the pre-1970s days, and thankfully they go back to the 1860s.  So here we have the crop yield data in cwt/acre (cwt is a centum weight, or hundredweight—a one hundred pound increment) for potatoes and sweet potatoes.  Given what we have already seen, this isn’t terribly surprising.  It looks much the same as the graphs for corn or wheat.  That is, a relatively flat yield curve until the 1930s followed by a sharp upturn and a monotonic increase spanning seven decades.  I still find this to be remarkable.

Potatoes and Sweet Potatoes yields in centum weight/acre (1868-2014).

Potatoes and Sweet Potatoes yields in centum weight/acre (1868-2014).

And what about how the yield has grown?  Looking below we can see that potatoes are 8 times more productive than they were some seventy years ago.  Considering the crops we have looked at so far, that is the record (beating rice’s sevenfold yield increase).

Yield growth for Potatoes and Sweet Potatoes (1868-2014).

Yield growth for Potatoes and Sweet Potatoes (1868-2014).

Explaining this is somewhat of a problem.  Every crop we have looked at has shown the same yield curve behavior.  But certainly not every crop had a sudden, massive successful hybridization improvement at the same time.  And if pesticides and fertilizer were primarily responsible, then why wouldn’t we see a large step change in the yield curve instead of steady incremental growth?  We do know that the tractor and other mechanized equipment came to popular use in the 1930s, but isolating its impact on crop yields is difficult.

To understand the effect of mechanization, we have to look at something else.  We have to look at a product that doesn’t rely on fertilizers or pesticides.  We have to look at an agricultural product where hybrid seed isn’t a factor.  We have to look at a product where mechanization is the primary driver of yield growth.  We have to look at milk.

Milk yield in lbs/head from 1924-2014.

Milk yield in lbs/head from 1924-2014.

And milk shows the exact same behavior.  In 2014, a single dairy cow in the United States could be expected to produce close to 22,000 pounds of milk in a year.  That is no small feat considering that the same cow was producing only about one-fifth of that at just over 4,000 pounds annually in 1924.  (Well not the same cow…)  This suggests that mechanization, and the knowledge behind it, has been the primary driving factor in the increase in farm productivity over the last 70 years.

Milk yield growth from 1924-2014.

Milk yield growth from 1924-2014.

And by knowledge I mean that it isn’t enough to make a milking system that pumps faster or has larger storage tanks to increase the time between handling.  Today’s automated milking systems enable a cow to decide when she needs to be milked to enter a milking stall and have the system automatically attach and begin.  This is an interesting video.  I don’t know the language, but it shows we are a far cry from a three-legged stool and a bucket.

Now you might say, “Not so fast!  The dairy industry gives cows hormones to increase milk production!”  And while there is an element of truth there, it doesn’t explain what we see in the yield.  In the 1930s it was found that Bovine Growth Hormone (BGH, also bovine somatotropin, or BST) boosted milk production. But to get BGH they had to extract it from the pituitary glands of cow cadavers, so it wasn’t able to be widely used.  It wasn’t until the 1970s that the gene for producing it was identified.  Recombinant bovine somatotropin (rBST) didn’t receive FDA approval for usage until 1983, and wasn’t commercially available until 1994.  It also isn’t clear that it has always made great economic sense, and adoption has been less than commercial producer expectations, at least at times.  So hormones can’t explain this observation.

This takes us back to mechanization.  Knowledge, ingenuity, and the desire to improve have lead us to build equipment to do the job better on the farm.  Better tractors, seed planters, harvesters, milkers, irrigation systems, and so on have allowed us to do more with less.  And the most interesting thing of it all is that in most of the data that we’ve seen, there is no leveling off in sight.  We haven’t hit the maximum yet; production yields continue to increase each year, on average.

Predicting the future is hard, but it is very easy to look backward in time to see how we got to where we are today.  I don’t imagine a farmer alive in the 1920s would ever have imagined that it would be possible to be getting the production levels we are getting today.  Likewise, today it seems impossible that these numbers can continue to increase for another seven decades.  In spite of that, I’m going to bet on the impossible coming to reality when it comes to feeding America.

Feeding America: The Extraordinary Increase in US Farm Productivity

A few nights ago I was re-reading a P.J. O’Rourke book on terribleness.  In the chapter on famine, everyone’s favorite bedtime reading subject, something he said struck me.  “In most of the world, food production has well outpaced the growth of population.  In the 1930s American wheat growers had an average yield of thirteen bushels per acre.  By 1970 the yield was thirty-one bushels.  In the same period the corn yield went from twenty-six bushels per acre to seventy-seven.”  This was unexpected. I have the picture in my mind of the differences between arithmetic and geometric progressions when it comes to comparing food production and population.  This is courtesy of Rev. Malthus, whose treatise was nicely summarized in the same book on terribleness in the chapter on overpopulation: “…there’s no end to the number of babies that can be made, but you can only plant so much wheat before you run the plow into the side of the house.

According to data from the fine folks at the USDA, the story here is rather interesting.  Yes, the wheat yield in the 1930s was in the low teens.  But what is surprising is that it had been there at least since the USDA began recording wheat crop yields in 1866!  In other words, the amount of wheat an acre of farmland produced showed no significant improvements for at least some seventy years, until the early 1930s.  Since then, however, it has increased at a roughly constant rate, reaching its all time high of 47.1 bushels per acre in 2013.  And it hasn’t settled at that yield; it continues to increase.

U.S. wheat crop yields, in bushels per acre.  1866-2014.

U.S. wheat crop yields, in bushels per acre. 1866-2014.

It should come as no great surprise, then, that corn yield numbers tell essentially the same story.  But I was surprised by the fact that the upturn in the yield happens at about the same date.  Corn production was a flat 25 bushels per acre for decades and started its way upward at about 1930.  For the curious, it peaked at 173.4 bushels per acre in 2014 and continues to climb as well.  That is a huge number.  A corn farmer in 1900 working hard to get his 28 bushels per acre never in his most fantastic dreams thought yields like this were possible.

U.S. corn crop yields, in bushels per acre. 1866-2014

U.S. corn crop yields, in bushels per acre. 1866-2014.

We can look at this another way, by plotting the corn yield against the wheat yield.  And what we see is that the relationship is well behaved.  When wheat yields increase, so do corn yields, though not necessarily in the same proportion.  After a point, every 10 bushels/acre growth in wheat equals about a 40 bushel/acre growth in corn.  This suggests there is more to the story — that there is some common factor that drives this effect.

The crop yields for U.S. corn plotted against the yields of U.S. wheat for the years 1866-2014.

The crop yields for U.S. corn plotted against the yields of U.S. wheat for the years 1866-2014.

And there is more to the story.  The USDA doesn’t just measure wheat and corn production.  They determine the yields for all of the field crops.  So that we can compare like things, first are the crop yields that are measured in bushels per acre: barley, corn, flaxseed, rye, sorghum, soybeans, and wheat.   Of those, only rye and sorghum yields look as though they have stopped increasing.  The rest show this continuously increasing trend over time, starting around the same year — 1930.

USDA crop yields (bushels/acre) for barley, corn, flaxseed, rye, sorghum, soybeans, and wheat.

USDA crop yields (bushels/acre) for barley, corn, flaxseed, rye, sorghum, soybeans, and wheat.

And then we have the crop yields that are measured by weight (pounds per acre).  These are: beans, cotton, hay, hops, peanuts, peppermint oil, rice, spearmint oil, sugarbeets, and tobacco.  These also show the same yield growth since about 1930 behavior.  Interestingly hay and tobacco yields seem to have joined rye and sorghum yields in leveling off (showing classic error function behavior).

USDA crop yields (pounds/acre) for beans, cotton, hay, hops, peanuts, peppermint oil, rice, spearmint oil, and tobacco.

USDA crop yields (pounds/acre) for beans, cotton, hay, hops, peanuts, peppermint oil, rice, spearmint oil, sugarbeets, and tobacco.

If it wasn’t clear before, it is now.  Sometime around the 1930s, something quite dramatic began to happen in field crop production.  An agricultural revolution of sorts.  And while the effect is present for all of them, it varies in its impact depending on the crop.  To show this, we normalize the yield rate by the average of the first few years in the dataset for a given crop.  This shows us how the yield rate has grown in time.  And we can compare them all if we plot them all with the same y-axis scale (conveniently done below).  So the biggest crops in terms of yield rate growth are:  corn (~7x), cotton (~6x), peanuts (~5.5x), rice (~7x), sorghum (4-6x).

Yield growth in US field crops.  Plotted is the multiplier in the yield rate since the start of data collection for the given field crop.

Yield growth in US field crops. Plotted is the multiplier in the yield rate since the start of data collection for the given field crop.

I find it interesting that the curves for soybeans and cotton begin to increase in 1920, which is a few years before the others.  Peanuts seem to be among the last to join the group as their yield didn’t start taking off until about 1950.  Did some experimentation take place with soybeans and cotton crops and then once successful transition to other crops such as wheat, corn, and then peanuts?  Soybeans became quite important in the US around 1910, so this is plausible.  Though I haven’t found anything to suggest this is what actually happened.

We do know that hybrid seed became all the rage starting around 1930.  Gregor Mendel demonstrated plant hybridization in the 1860s with peas, but it wasn’t until the 1930s when hybrid seed was able to produce a corn crop that was well suited to mechanical harvesting.  One report says: “The tractor, corn picker, and hybrid seed corn came together to raise labor and land productivity in corn production in the late 1930s.”  Tractor development (power takeoff, rubber tires) and sales really started to take off in the 1920s, which fits the timeline perfectly.  And I’ll bet that improved chemical fertilizers and pesticides factor in as well.

In any case, I should go back to the original statement where O’Rourke said food crop yield rates were increasing faster than the population.  In 1927 the world’s population is estimated to have been about 2 billion people.  This is convenient for our comparison as it about coincides with our 1930 start of this agricultural revolution.  Corn and wheat yields both doubled by about 1960, when the global population was about 3 billion (a 1.5x increase).  So far, so good.  And we reached about 7 billion people in 2012, an increase of a factor of 3.5 since 1930.  This about matches the rate of increase in wheat production.  But corn and rice have managed to stay ahead by a considerable margin, so O’Rourke is right and Reverend Malthus has been wrong.  At least, since this “revolution” started.

But these crop yield numbers aren’t global numbers, they’re for domestic production.  How do they compare against the growth in the U.S. population?  Quite favorably, it turns out.  In 1930 the U.S. population was about 123 million.  It grew by about 50 percent to about 180 million in 1960, which is in line with global population increase we just considered, where crop yields for corn and wheat doubled.  But in 2012 the U.S. population reached 313 million, a growth factor of only 2.5x over 1930.  The slowest of the food crop yields shown above have at least grown at the same pace as the domestic population.  But the major food crops like corn and rice are out in front by a mile, with their yields growing some 2.8 times faster than the growth in domestic population.

This is startling, but in a good way.  It is a win for science and innovation and demonstrates how humans can manipulate their situation to work to their advantage.  The primary food crop yields show no signs of leveling off.  It makes sense that they probably will at some point.  But for now it is nice to know that the Reverend Malthus is wrong.

Book Review: Thinking Fast and Slow – Daniel Kahneman

There are people who spend their lives peeling an onion.  If they are lucky, it is a sweet bulb, and offers up its layers without too many tears.  If they are very lucky, what the peeling reveals is interesting enough that others are interested and pay attention.  And if they are very, very lucky, well, then the Royal Swedish Academy of Sciences honors their efforts with the most prestigious award that onion peelers can receive.

There were two people peeling this particular onion, but one passed away before their work was fully recognized.  In telling their story on their work on human decision making psychology, Daniel Kahneman recognizes his longtime collaborator Amos Tversky while sharing their work that was recognized with the 2002 Nobel Prize in Economics.

Since before the 1970s, it was well understood that humans were rational decision makers, and that when we strayed from this rational behavior, it was driven by some strong emotion.  Anger, fear, hatred, love — these were the things that pushed us into irrationality.  This makes perfect sense.  This “Utility Theory” was well known and not really challenged because it was, well, obvious.  Tversky and Kahneman, however, challenged its depiction of rational human decision making in their 1979 paper, “Prospect Theory: An Analysis of Decision Under Risk.”  We, it turns out, don’t behave very rationally at times (the Ultimatum Game is a very good example of this).  But what made this paper special was that they went beyond documenting the failure of Utility Theory and pointed their fingers at the design of the machinery of cognition as the reason, rather than the corruption of thought by emotion.  They argued that heuristics and biases were the key players in judgement and decision making.  This was a revolutionary idea.  The first layer of the onion had been peeled back, and as you might expect, it revealed more questions.

“Jumping to conclusions on the basis of limited evidence is so important to an understanding of intuitive thinking, and comes up so often in this book, that I will use a cumbersome abbreviation for it: WYSIATI, which stands for what you see is all there is.  System 1 is radically insensitive to both the quality and the quantity of the information that gives rise to impressions and intuitions.”

Kahneman describes the cognitive machine with a cast of two players.  These are, as he calls them, “System 1” and “System 2.”  It would be easy to call these the subconscious and the conscious; this would be a good first approximation, but it wouldn’t be entirely correct.  System 1 does operate quickly, automatically, and with no sense of voluntary control, as you would expect the subconscious to do.  But it is responsible for feeding System 2 with things like feelings, impressions, and intuitions.  System 2 generally takes what System 1 gives it without too much fuss.  But System 1 behaves in funny ways sometimes, producing some very interesting results.

Interestingly, you can give System 2 a specific task that taxes the cognitive resources and some interesting things happen.  For example, watch the video below and count the number of times the players wearing white shirts pass the ball.

Did you get the correct number?  Did you see the gorilla?  About half of the people who watch the video simply do not see the gorilla.  System 2 is busy counting, while System 1 is supporting that task and not distracting it with irrelevant extraneous information.  Sometimes, it would appear, we are quite blind to the obvious.  And blind to our own blindness.  This “tunnel vision” of sorts happens not only when we are cognitively busy, but also in times of stress or high adrenaline.

I experienced this personally a few years ago during an Emergency Response training exercise.  I was part of the two-man team entering a room where we had to assess the scene and respond accordingly.  The instructor running the exercise had taped a sign to the wall with information that would provide some answers to questions we would have in the room; things that would be obvious in a real situation.  Except it wasn’t obvious.  I didn’t see it.  At all.  A coworker was playing the part of an injured person and lying on the floor and safely removing him from the scene was all we could think about.  As we debriefed I was asked why I did not address the issues on the sign.  I had to go back to see the sign for myself to convince myself they were serious.  I was shocked.

After Prospect Theory, discerning the rules for the cognitive machine became a hot research area in the field of cognitive psychology.  And what they found is astounding.  A brief overview of the heuristics and biases can be found online, and these are discussed in detail in the text.  Some, like the affect heuristic, make perfect sense.  Emotion and belief or action are tied together.  This is a significant influence in how you create your beliefs about the world.  But priming, on the other hand, is downright scary because we have no conscious knowledge that it is taking place.

One of the interesting things that comes out of this research is that not only are humans are not rational thinkers, we aren’t very good statistical thinkers either.  Kahneman and Tversky’s first paper together was “Belief in the law of small numbers.”  The “law of small numbers” asserts, somewhat tongue in cheek, that the “law of large numbers” applies to small numbers as well.  There is much truth about this in how we build very lasting first impressions, quickly finding rules where random chance is a better explanation.  Kahneman’s insights into the day-to-day decisions and judgements that we make, with no thought into them, are priceless.

“The idea that large historical events are determined by luck is profoundly shocking, although it is demonstrably true…It is hard to think of the history of the twentieth century, including its large social movements, without bringing in the role of Hitler, Stalin, and Mao Zedong.  But there was a moment in time, just before an egg was fertilized, when there was a fifty-fifty chance that the embryo that became Hitler could have been a female.  Compounding the three events, there was a probability of one-eighth of a twentieth century without any of the three great villains and it is impossible to argue that history would have been roughly the same in their absence.  The fertilization of these three eggs had momentous consequences, and it makes a joke of the idea that long-term developments are predictable.”

It is pleasant to find an academic that can write a general interest book.  Too frequently the result of such an effort is a dense tome that is closer to a textbook.  Thinking Fast and Slow is enjoyably readable.  But it is more than that.  It is a very complete book—a dissecting of the machinery of the mind.  It pulls back, in 38 chapters, the covers to reveal in plain language the mechanisms that operate our minds every day.  The sorts of things that go on behind the scenes in every decision we make.  But also the myriad ways that advertising professionals can and do manipulate us.

This is not a weekend quick read.  The paperback version weighs in at 512 pages.  That shouldn’t hold you back.  After all, this is a review of an entire lifetime of Nobel Prize-winning research, in clear language without jargon, told with its historical perspective.  There is gold on every page and I’m grateful for every one of them.