Thursday, October 08, 2015

Estimating and minimizing consumer worry

The process of selling in general, and web commerce in particular, is often described or charted as a funnel. Prospective customers are poured in at one end, and a fewer number of paying customers come out at the other. The other prospects spill out through other holes or over the side of the funnel and don't bring you any revenue. The fraction of customers left, converted from prospects to customers, is called the conversion rate. As prospects proceed from initial interest to final sale, from initial entry page to clicking the final "I Agree" button, more and more of them become discouraged by various worries which beset the consumer. They drop out. The remaining prospects, those who have not dropped off, have been converted into customers or into an audience for your advertisers.

There are a variety of factors that cause drop-off, which vary from business to business. A common cause is forms.  Simplifying forms often greatly increases conversion rates.  For example, in one study cutting the number of lines on a form in half increase conversions by a third. As one web designer put it: "[i]s every field you're asking the visitor to submit absolutely necessary?  Can you trim the fat and make the process simpler?"

Besides the sheer tediousness and time consumed in filling out forms, rational consumers also worry about the potentials for privacy violation and identity theft from the information most e-commerce sites currently require them to divulge: physical and e-mail addresses, phone numbers used for cross-site behavioral tracking, insecure credit card numbers, and more.

The tinfoil-wallet crowd is now mainstream

Instances of regret that one has filled out a form, only to have one's trust violated -- or pride among the sophisticated that they refused to fill out such a form -- are on the rise.

The worst worry culprit is usually the step you most want your customers to complete -- paying you. "[T]he credit card form likely has the highest abandonment rate of any other part of the sign-up process." [Source].

If you don't require payments, you are probably funding your service through advertisements. Those also cause worries. Ads typically distract and delay from the content users are after, provide a low quality of entertainment or information, and are too often offensive. And sophisticated users are worried about the tracking that tends to go with ads. Ad blocking grew nearly tenfold between 2009 and 2014.

Replacing ads and identity-based payments with payments that don't require identity, such as bitcoin, can greatly reduce these worries, lowering the barriers and hesitations that currently prevent consumers from paying for your service.

But there remains a big worry that no payment system can reduce.  Consumers worry about whether they are getting their money's worth -- the mental transaction cost problem (see also this paper). If e-commerce were as worry-free as some of it could be, your customers would neither have to fill out forms, nor be bothered by ads, nor have to worry about repeated charges for content or services of variable value. They would be able to just insert a few digital coins into your online vending machine and then not have to worry about losing your service for another year. Eliminate forms and eliminate repeated payments -- both are key to worry-minimized e-commerce.

Many bitcoin startups are making the grave mistake of replacing one set of worries with another. The ability of cryptocurrency systems to facilitate small payments tempts many companies to nickel-and-dime their customers with pay-per-click micropayments and other such excruciating schemes. Don't follow the many lemmings who have already jumped off that cliff. Stick to long-term subscriptions for content (or other services of variable value) and pay-per-unit for fungible units of consistent value (as in phone minutes).  That way customers aren't saddled with having to constantly re-evaluate the amount and worthiness of recurring charges. The costs to your customers of having to finance a years' worth of low-cost subscription to a reputable brand is almost always far less than the mental transaction costs of recurring charges for content or services of variable value. The ideal worry-free commerce is to "stick the coin into the machine" once, and then never have to pay again for an entire year. A vending machine for subscriptions. Reduce your customers' worries across the board: eliminate forms and eliminate recurring charges.

Ideal worry-minimization can only be closely approached in some purely online forms of commerce, such as video streaming, remote storage, privacy services, and the like. The more physical and offline contract performances are -- a common example being physical delivery -- the more location, various kinds of identity (legal, social network, etc.) may need to come into play, adding, often greatly and necessarily, to the worry overhead, the mental transaction costs, of your relationship with your consumers.

I have previously called this worry-minimized commerce by a narrower label, "form-minimized commerce."  The complexity of the forms you make your users fill is indicative of the worries you are causing them, and thus the barriers you are putting up between your prospects and their decisions to purchase your services.

When you are a consumer, the tediousness of the forms you are filling out is not only a direct cost of your time, and your ability to enjoy that time, it is on top of that a decent proxy measure of the odds of your identity being stolen and of your privacy otherwise being violated. The fewer forms you fill out, the more the tediousness, worries, and risks in your life caused by interacting with the world's institutions will drop in proportion.

While such a proxy measure does not account in particular for the wide variety of information that can be disclosed, nor that some kinds of information (social security numbers) are more risky to divulge than others (throw-away email addresses), nor for the wide variety of risks in identity theft and privacy violation that are consequent, nevertheless consumers necessarily must bring to bear such sweeping rules-of-thumb in order to satisfactorily navigate the bizarre complexities of the digital world.  And when your users are using, whether consciously or implicitly, such estimates, you the service provider and the product designer must use them too.

Add to the forms your customers must fill out the repeated charges you make your customers make, and we get a rough proxy measure of the worry that you are causing your consumers:
Index of worry = number of lines of forms +  number of repeated charges for content or services of variable value
If you are funded through ads rather than consumer payments, you can substitute for the repeated charges the proportion of screen space covered by your ads, or any other reasonable estimate of the delay and distraction the ads on your pages cause.

The index of worry allows you to estimate and minimize the worries you are causing your users, and as a result to minimize the drop-off in your sales funnel and maximize the number of users coming back for more -- and willing to view your ads or pay for the privilege.

Friday, July 03, 2015

The Greek financial mess; and some ways Bitcoin might help

Many years of government debt buildup in Greece has ultimately resulted, in the last few days, in a political and financial maelstrom.  The political maelstrom includes demonstrations in the run up to a referendum on obscure debt-restructuring provisions to be held this Sunday (July 5th). This article focuses on  financial problems and some potential practical steps that can be taken to mitigate them. The imposition of capital controls is a disaster for a modern trade-driven economy, a catastrophe which however digital technology, and in particular the digital currency bitcoin (which given the Greek environment usually must involve direct use of the Bitcoin blockchain), has the potential to mitigate.  This article will explore some of the severe practical problems that capital controls are causing Greek individuals and businesses, and suggest some potential bitcoin solutions, many of which could also be applied to other countries with some similar financial problems and controls such as Argentina and Venezuela.  These aren't solutions that can be applied in time to help with the current 6 days of capital controls, but could substantially help some Greeks and some aspects of the Greek economy if some version of these controls is continued for months or years.

At root the Greek financial problem is that the Greek government has spent more, compared to the GDP generated by its economy, than the vast majority of other governments.  It has borrowed copious sums to do so, falling ever deeper into debt.  Here is its payment schedule:

And you thought your student loan debt was bad

The only place the Greek government has left to go for money to fund its ongoing expenditures and pay these debts is Greek banks.  Fearing capital controls and "haircuts" (government confiscation of certain fractions of bank deposits), many Greeks in recent months have, quite rationally, started withdrawing money out of their banks and sending it overseas.  More trusting Greeks kept their savings in their banks, with the result that, with the imposition of capital controls last Monday, they have been locked out of their savings, and plans for "haircuts" of 30% or more have been reported (If somebody lopped off 30% of your head, you’d have more than a haircut).

When capital controls were first rumored and then announced on Sunday, vast lines formed at ATMs as Greeks rushed to rescue what little of their life savings that they could:

ATM line, Thessaloniki

ATM line, Larissa City

On Tuesday, the Greek government defaulted on its scheduled debt payment to the International Monetary Fund (IMF).

Under capital controls, ATM withdrawals from Greek bank accounts are now limited to 60 euros a day.  Debit cards can still be used for payments within the country, but the money simply gets transferred from one frozen bank account to another.  As a result many businesses no longer accept debit cards, and many more are demanding a substantial premium price (in  at least one business, double) for debit cards (transferred bank balances) versus hard cash.  There is a growing shortage of such cash; as a result some stores are paying their suppliers in private "scrip", which can be used by the supplier's workers to purchase goods from the issuing store. (more on this below).

Use of credit and debit cards to pay out of the country is banned and effectively blocked, resulting in a near-complete freeze-out of Greeks from Internet commerce. This restriction, along with the controls resulting in Greeks being excluded from the pan-European money settlement system, means that Greek businesses can't pay for imports.  Many shipments into the country have been halted as a result. (The government plan is to create a whitelist of politically approved cases in which such payments for imports will be unblocked).

A crucial feature of store-issued scrip is that it literally circulates through a complete closed cycle: store --> supplier --> workers --> store.  Such specific cycles are a pattern that is commonly found when currencies are primitive or newly emerging, and every Bitcoin marketer and evangelist should be familiar with them.

The kula ring, two specific cycles (counter-circulating cycles of shell money) allowing exchange of seasonal goods in the precolonial South Pacific

It doesn’t help much to sell bitcoin to isolated individuals: as a mere store of value its volatility is much greater than most existing currencies; as an investment it only makes sense as a tiny high-risk fraction of one's portfolio.  Bitcoin does have some political-affinity and status value in developed countries; by contrast in many developing countries and in countries under financial crisis such as Greece, there are urgent needs bitcoin potentially can address.  In terms of these needs Bitcoin is mainly useful as a way to send money across borders for investment in more stable assets overseas, and to substitute for cash or other substitute currencies in a money-starved environment.

To have value as a medium of exchange, bitcoin must be taken up by a community of people who already frequently trade with each other,  and who have a strong need to use it in these trades. It is especially important to market to the links in the cycle that have the strongest negotiating leverage with the others (in the case of Greek the Greek store scrip cycle, the store and its larger suppliers).  The link in the cycle with the greatest incentives to switch to bitcoin here are likely the store's suppliers, because they don’t fully trust the store, nor the underlying currency, euro or post-euro, that is the “O” in an IOU, but are participating in the scrip because, sans bitcoin, they have no other choice.

In bitcoin specific cycles create other cost savings.  Almost everywhere they economize on the increasingly high KYC/AML (know your customer/anti-money-laundering) costs of going through a fiat-bitcoin exchange.  What's more, in a capital controls environment like Greece specific cycles avoid the capital controls that would be imposed on a Greek-based fiat-bitcoin exchange, and avoid the need nearly all Greek customers using out-of-country exchanges would have to futilely try to tap into their frozen bank accounts in order to purchase bitcoin. Bitcoin will not, contrary to some feverish news reporting, help Greeks get money out of their frozen bank accounts.

But bitcoin does have great potential to help in less obvious ways: for one thing, as a superior (not vulnerable to trust in an issuing store, and in any currency underlying an IOU) substitute for the emerging store scrips.  For another, it could help greatly with the severe cross-border commerce issues that are emerging.  Exporters, including freelancers working over the Internet, can bring bitcoin into the country, thereby avoiding earning wages that get deposited to frozen bank accounts (per Greek lore, be wary of a cave with many tracks coming in but few coming out). Importers can pay for goods with bitcoin while other electronic payment channels (European money settlements, Paypal, and credit & debit cards when paying foreign businesses, etc.) remain frozen. Again specific cycles must be set up: isolated marketing to just exporters or importers will be far less effective than organizing existing supply chains that involve both.

There are likely many other, mostly highly non-obvious, niches in which bitcoin, and other cryptocurrencies, and smart contract platforms could play a quite valuable role in capital-controlled and other financially handicapped countries.

Bitcoin is not easy to learn, either conceptually or in setting up businesses and individuals with the software (and preferably also the secure hardware) to accept it. This is especially the case in a capital controls climate where the traditional bitcoin exchanges and retail payment companies, with their consumer-friendly front ends, as they normally operate in developed countries, likely can't effectively operate. To take advantage of bitcoin many Greeks will have to use the Bitcoin blockchain directly. So it's too late for bitcoin to help much with the current 6 days of bank closure,  but once the learning curves have been surmounted, the participants in specific cycles educated, bitcoin has great potential to address likely many ongoing problems with capital control, in Greece as long as they continue in various forms, and in many other parts of the world where such financial restrictions designed for a pre-digital era have been imposed.

[Update: various minor edits: the first version was rather rough, sorry :-)]

Monday, May 25, 2015

Small-game fallacies

A small-game fallacy occurs when game theorists, economists, or others trying to apply game-theoretic or microeconomic techniques to real-world problems, posit a simple, and thus cognizable, interaction, under a very limited and precise set of rules, whereas real-world analogous situations take place within longer-term and vastly more complicated games with many more players: "the games of life".  Interactions between small games and large games infect most works of game theory, and much of microeconomics, often rendering such analyses useless or worse than useless as a guide for how the "players" will behave in real circumstances. These fallacies tend to be particularly egregious when "economic imperialists" try to apply the techniques of economics to domains beyond the traditional efficient-markets domain of economics, attempting to bring economic theory to bear to describe law, politics, security protocols, or a wide variety of other institutions that behave very differently from efficient markets. However as we shall see, small-game fallacies can sometimes arise even in the analysis of some very market-like institutions, such as "prediction markets."

Most studies in experimental economics suffer from small-game/large-game effects. Unless these experiments are very securely anonymized, in a way the players actually trust, and in a way the players have learned to adapt to, overriding their moral instincts -- an extremely rare circumstance, despite many efforts to achieve this -- large-game effects quickly creep in, rendering the results often very misleading, sometimes practically the opposite of the actual behavior of people in analogous real-life situations. A common example: it may be narrowly rational and in accord with theory to "cheat", "betray", or otherwise play a narrowly selfish game, but if the players may be interacting with each other after the experimenters' game is over, the perceived or actual reputational effects in the larger "games of life", ongoing between the players in subsequent weeks or years, may easily exceed the meager rewards doled out by the experimenters to act selfishly in the small game. Even if the players can somehow be convinced that they will remain complete strangers to each other indefinitely into the future, our moral instincts generally evolved to play larger "games of life", not one-off games, nor anonymous games, nor games with pseudonyms of strictly limited duration, with the result that behaving according to theory must be learned: our default behavior is very different. (This explains, why, for example, economics students typically play in a more narrowly self-interested way, i.e. more according to the simple theories of economics, than other kinds of students).

Small-game/large-game effects are not limited to reputational incentives to play nicer: moral instincts and motivations learned in larger games also include tribal unity against perceived opponents, revenge, implied or actual threats of future coercion, and other effects causing much behavior to be worse than selfish, and these too can spill over between the larger and smaller games (when, for example, teams from rival schools or nations are pitted against each other in economic experiments). Moral instincts, though quite real, should not be construed as necessarily or even usually being actually morally superior to various kinds of learned morals, whether learned in economics class or in the schools of religion or philosophy.

Small-game/large-game problems can also occur in auditing, when audits look at a particular system and fail to take into account interactions that can occur outside their system of financial controls, rendering the net interactions very different from what simply auditing the particular system would suggest. A common fraud is for trades to be made outside the scope of the audit, "off the books", rendering the books themselves very misleading as to the overall net state of affairs.

Similarly, small-game/large-game problems often arise when software or security architects focus on an economics methodology, focusing on the interactions occurring within the defined architecture and failing to properly take into account (often because it is prohibitively difficult to do so) the wide variety of possible acts occurring outside the system and the resulting changes, often radical, to incentives within the system. For example, the incentive compatibility of certain interactions within an architecture can quickly disappear or reverse when opposite trades can be made outside the system (such as hedging or even more-than-offsetting a position that by itself would otherwise create a very different incentive within the system), or when larger political or otherwise coercive motivations and threats occur outside the analyzed incentive system, changing the incentives of players acting within the system in unpredictable ways. Security protocols always consist of at least two layers: a "dry layer" that can be analyzed by the objective mathematics of computer science, and a "wet layer" that consists of the often unpredictable net large-game motivations of the protocols' users.  These should not be confused, nor should the false precision of mathematical economic theories be confused with the objective accuracy of computer science theories, which are based on the mathematics of computer architecture and algorithms and hold regardless of users' incentives and motivations.

A related error is the pure-information fallacy: treating an economic institution purely as an information system, accounting only for market-proximate incentives to contribute information via trading decisions, while neglecting how that market necessarily also changes players' incentives to act outside of that market. For example, a currently popular view of proposition bets, the "prediction markets" view, often treats prop bets or idea futures as purely information-distribution mechanisms, with the only incentives supposed as the benign incentive to profit by adding useful information to the market. This fails to take into account the incentives such markets create to act differently outside the market.  A "prediction market" is always also one that changes incentives outside that market: a prediction market automatically creates parallel incentives to bring about the predicted event. For example a prediction market on a certain person's death is also an assassination market. Which is why a pre-Gulf-War-II DARPA-sponsored experimental "prediction market" included a prop bet on Saddam Hussein's death, but excluded such trading on any other, more politically correct world leaders. A sufficiently large market predicting an individual's death is also, necessarily, an assassination market, and similarly other "prediction" markets are also act markets, changing incentives to act outside that market to bring about the predicted events.

Thursday, December 11, 2014

The dawn of trustworthy computing

When we currently use a smart phone or a laptop on a cell network or the Internet, the other end of these interactions typically run on other solo computers, such as web servers. Practically all of these machines have architectures that were designed to be controlled by a single person or a hierarchy of people who know and trust each other. From the point of view of a remote web or app user, these architectures are based on full trust in an unknown "root" administrator, who can control everything that happens on the server: they can read, alter, delete, or block any data on that computer at will.  Even data sent encrypted over a network is eventually unencrypted and ends up on a computer controlled in this total way. With current web services we are fully trusting, in other words we are fully vulnerable to, the computer, or more specifically the people who have access to that computer, both insiders and hackers, to faithfully execute our orders, secure our payments, and so on. If somebody on the other end wants to ignore or falsify what you've instructed the web server to do, no strong security is stopping them, only fallible and expensive human institutions which often stop at national borders.

The high vulnerability we have to web servers stands in sharp contrast to traditional commercial protocols, such as ticket-selling at a movie theater, that distribute a transaction so that no employee can steal money or resources undetected. There is no "root administrator" at a movie theater who can pocket your cash undetected.  Because, unlike a web server, these traditional protocols, called financial controls, can securely handle cash, you didn't have to fill out a form  to see a movie, shop for groceries, or conduct most other kinds of every-day commerce. You just plunked down some coin and took your stuff or your seat. Imperfect and slow as these processes often are (or were), these analog or paper-based institutions often provided security, financial control, and/or verifiability of fiduciary transactions in many ways far superior to what is possible on web servers, at much less hassle and privacy loss to customers. On the Internet, instead of securely and reliably handing over cash and getting our goods or services, or at least a ticket, we have to fill out forms and make ourselves vulnerable to identity theft in order to participate in e-commerce, and it often is very difficult to prohibitive to conduct many kinds of commerce, even purely online kinds, across borders and other trust boundaries. Today's computers are not very trustworthy, but they are so astronomically faster than humans at so many important tasks that we use them heavily anyway. We reap the tremendous benefits of computers and public networks at large costs of identity fraud and other increasingly disastrous attacks.

Recently developed and developing technology, often called "the block chain", is starting to change this. A block chain computer is a virtual computer, a computer in the cloud, shared across many traditional computers and protected by cryptography and consensus technology. A Turing-complete block chain with large state gives us this shared computer. Earlier efforts included state-machine replication (see list of papers linked below).  QuixCoin is a recent and Ethereum is a current project that has implemented such a scheme. These block chain computers will allow us to put the most crucial parts of our online protocols on a far more reliable and secure footing, and make possible fiduciary interactions that we previously dared not do on a global network 

Much as pocket calculators pioneered an early era of limited personal computing before the dawn of the general-purpose personal computer, Bitcoin has pioneered the field of trustworthy computing with a partial block chain computer. Bitcoin has implemented a currency in which someone in Zimbabwe can pay somebody in Albania without any dependence on local institutions, and can do a number of other interesting trust-minimized operations, including multiple signature authority. But the limits of Bitcoin's language and its tiny memory mean it can't be used for most other fiduciary applications, the most obvious example being risk pools that share collateral across a pool of financial instruments.

A block-chain computer, in sharp contrast to a web server, is shared across many such traditional computers controlled by dozens to thousands of people. By its very design each computer checks each other's work, and thus a block chain computer reliably and securely executes our instructions up to the security limits of block chain technology, which is known formally as anonymous and probabilistic Byzantine consensus (sometimes also called Nakamoto  consensus).  The most famous security limit is the much-discussed "51% attack".  We won't discuss this limit the underlying technology further here, other than saying that the oft-used word "trustless" is exaggerated shorthand for the more accurate mouthful "trust-minimized", which I will use here.  "Trust" used in this context means the need to trust remote strangers, and thus be vulnerable to them. 

Trust-minimized code means you can trust the code without trusting the owners of any particular remote computer. A smart phone user in Albania can use the block chain to interact with a computer controlled by somebody in Zimbabwe, and they don't have to know or trust each other in any way, nor do they need to depend on the institutions of either's countries, for the underlying block chain computer to run its code securely and reliably. Regardless of where any of the computers or their owners are, the block chain computer they share will execute as reliably and securely as consensus technology allows, up to the aforementioned limits. This is an extremely high level of reliability, and a very high level of security, compared to web server technology. 

Instead of the cashier and ticket-ripper of the movie theater, the block chain consists of thousands of computers that can process digital tickets, money, and many other fiduciary objects in digital form.  Think of thousands of robots wearing green eye shades, all checking each other's accounting. Individually the robots (or their owners) are not very trustworthy, but collectively, coordinated by mathematics, they produce results of high reliability and security.

Often block chain proponents talk about the "decentralized" block chain versus the "centralized" web or centralized institutions. It's actually the protocol (Nakamoto consensus, which is highly distributed) combined with strong cryptography, rather than just decentralization per se, that is the source of the far higher reliability and and much lower vulnerability of block chains. The cryptography provides an unforgeable chain of evidence for all transactions and other data uploaded to the block chain. Many other decentralized or peer-to-peer (P2P) technologies do not provide anything close to the security and reliability provided by a block chain protected by full Byzantine or Nakamoto consensus and cryptographic hash chains, but deceptively style themselves as block chains or cryptocurrency.

A big drawback is that our online and distributed block chain computer is much slower and more costly than a web server: by one very rough estimate, about 10,000 times slower and more costly, or about the same as it cost to run a program on a normal computer in 1985. For this reason, we only run on the block chain that portion of an application that needs to be the most reliable and secure: what I call fiduciary code. Since the costs of human ("wet") problems caused by the unreliability and insecurity of web servers running fiduciary code are often far higher than the extra hardware needed to run block chain code, when web server reliability and security falls short, as it often does for fiduciary computations such as payments and financial contracts, it will often make more sense  to run that code on the block chain than to run it less reliably and securely on a web server. Even better, the block chain makes possible new fiduciary-intensive applications, such as posting raw money itself to the Internet, securely and reliably accessible anywhere on the globe -  apps that we would never dare do with a web server.

What kinds of fiduciary code can we run?  We are still thinking up new applications and the categories will be in flux, but a very productive approach is to think of fiduciary applications by analogy to traditional legal code that governs traditional fiduciary institutions. Fiduciary code will often execute some of the functions traditionally thought of as the role of commercial law or security, but with software that securely and reliably spans the global regardless of traditional jurisdiction. Thus:

* Property titles (registered assets), where the on-chain registry is either the legally official registry for off-chain assets or controls on-chain ones, thus providing reliable and secure custody of them. One can think of a cryptocurrency such as Bitcoin as property titles (or at least custody enforced by the block chain consensus protocol) to bits recognized as being a fixed portion of a currency, or as controlling unforgeably costly bits, or both. Block chains could also control hardware which controls the function of and access to physical property.

* Smart contracts: here users (typically two of them) agree via user interface to execute block chain code, which may include transfer of money and other chain-titled assets at various times or under various conditions, transfer and verification of other kinds of information, and other combinations of wet or traditional (off-chain) and dry (on-chain) performance. A block chain can hold cryptocurrency as collateral (like an escrow) which incentivizes off-chain performance that can be verified on-chain, by the parties or by third parties. A full block chain computer can pool on-chain assets into a single chain-controlled risk pool spread among many similar financial contracts, reducing the amount of collateral that needs to be stored on-chain while minimizing the need for off-chain collateral calls. The block chain can also make the search, negotiation, and verification phases of contracting more reliable and secure. With on-chain smart contracts we will be able to buy and sell many online services and financial instruments by button and slider instead of by laboriously filling out forms that disclose our private information.

* On-chain treasuries, trusts, and similar, where money lives on the block chain and is controlled by multiple signature ("multisig") authority.  Putting a treasury with signature authority on a block chain computer is low-hanging fruit, but is often tied to more speculative efforts under the label "distributed autonomous organization (DAO)", which may include voting shares and other mechanisms to control the treasury like a corporation or other kind of of organization.

I hope to discuss these block chain applications, especially smart contracts, in future posts. While there is much futurism in many block chain discussions, including many trying to solve problems that aren't actually solved by the block chain, I will generally stick to low-hanging fruit that could be usefully implemented on Quixcoin, Ethereum, or similar technology in the near future, often interfacing to still necessary parts of traditional protocols and institutions rather than trying to reinvent and replace them in whole.


Here is a list of basic computer science papers describing the technology of block chains (including cryptocurrencies).

Wet vs. dry code

Thursday, October 16, 2014

Transportation, divergence, and the industrial revolution

After about 1000 AD northwestern Europe started a gradual switch from using oxen to using horses for farm traction and transportation.  This trend culminated in an eighteenth-century explosion in roads carrying horse-drawn carriages and wagons, as well as in canals, and works greatly extending the navigability of rivers, both carrying horse-drawn barges. This reflected a great rise in the use of cultivated fodder, a hallmark of the novel agricultural system that was evolving in northwestern Europe from the start of the second millennium: stationary pastoralism.  During the same period, and especially in the seventeenth through nineteenth centuries, most of civilized East Asia, and in particular Chinese civilization along its coast, navigable rivers, and canals, faced increasing Malthusian pressures and evolved in the opposite direction: from oxen towards far more costly and limited human porters. Through the early middle ages China had been far ahead, in terms of division of labor and technology, of the roving bandits of northern Europe, but after the latter region's transition to stationary pastoralism that gap closed and Europe surged ahead, a growth divergence that culminated in the industrial revolution.  In the eighteenth century Europe, and thus in the early industrial revolution, muscle power was the engine of land transportation, and hay was its gasoline. 

Metcalfe's Law states that a value of a network is proportional to the square of the number of its nodes.  In an area where good soils, mines, and forests are randomly distributed, the number of nodes valuable to an industrial economy is proportional to the area encompassed.  The number of such nodes that can be economically accessed is an inverse square of the cost per mile of transportation.  Combine this  with Metcalfe's Law and we reach a dramatic but solid mathematical conclusion: the potential value of a land transportation network is the inverse fourth power of the cost of that transportation. A reduction in transportation costs in a trade network by a factor of two increases the potential value of that network by a factor of sixteen. While a power of exactly 4.0 will usually be too high, due to redundancies, this does show how the cost of transportation can have a radical nonlinear impact on the value of the trade networks it enables.  This formalizes Adam Smith's observations: the division of labor (and thus value of an economy) increases with the extent of the market, and the extent of the market is heavily influenced by transportation costs (as he extensively discussed in his Wealth of Nations).

The early industrial revolution was highly dependent on bringing together bulk goods such as coal and iron ore.  Land transportation of such materials more than a dozen miles in most parts of the world was prohibitively costly, and they were only rarely located a shorter distance from navigable water (the costs per mile of water transport were generally orders of magnitude cheaper than the costs per mile of of land transport).  As a result, the early industrial revolution, and the potential for a region to be the first to industrialize, was very sensitive to small changes in land transportation costs.

Furthermore, land and sea-borne transportation were far more complements than substitutes.  Cheaper land transportation was a "force multiplier" for water transportation.  Decreasing the costs of getting to port from field or mine by a factor of two increased the number of fields and mines accessible by a factor of four, and increased the number of possible ways to divide labor, and thus the value, by an even greater factor via Metcalfe's law.  This in turn incentived greater investment in sea-borne transport. It's thus not surprising that, even before the industrial revolution, the leaders in global trade and colonization were European countries that could access the Atlantic.

By the dawn of the industrial revolution in northwest Europe the effects of horse haulage had already been dramatic: drop by a factor of two in the costs, and increase in speed by about the same factor, of transporting goods by land, the corresponding increase in commercial crop area and in area that could be economically lumbered and coal and metals that could be mined.   Multiply that factor of four by much more when we factor in (1) innovations in wheels, tires, shock absorption, and road building that followed on the heels, as is were, of the great increase in horse haulage, and (2) the great increase in mileage and inland penetration of navigable rivers and canals, especially in the 18th century, the barges again hauled by horses.  And as Metcalfe's Law suggests, the number of combinations, and thus the value, increased by a far greater factor still. Not only did northwestern European ports have access to far more land, but there were far more ports far more "inland" along rivers and canals, thanks again chiefly to the draft horses and the nutrient-rich cultivated fodder that fed them.

To enable the industrial revolution, mines and nutrient-dense fodder had to be colocated within efficient bulk transport distance of each other — which in the case of horses hauling coal or wood by rural road, was typically less than twenty miles, and for oxen and human porters far less still — to produce the low-cost bulk transportation networks needed to make industrial revolution scale use of most commercial crops and mines. Efficient bulk transportation is needed _all the way_ between the iron mine, the coal mine, and the smelter.  Because the cost per mile of water transport was so much smaller than the costs of land transport, this “last few miles to the mine” problem usually played a dominant role in transportation economics, somewhat analogous to the “last mile” problem in modern cable networks. That’s why stationary pastoralism with its efficient markets for nutrient-dense (because cultivated) fodder was such a huge win — it allowed horses to be housed at the mines, canals, roads, and factories where they worked, which no place in the world outside Europe could during that era do.  Nutrient-dense fodder created a virtuous recursion, enabling itself to be harvested (via horse-drawn mows and rakes) and transported to mine, factory, and stable at increasingly lower costs.

Industrialization came in many phases. Very roughly speaking, the first phase, in the latter half of the eighteenth century, involved the culmination and optimization of the use of horses, by northwestern Europe, and especially England, greatly expanding its horse wagon and carriage roads and horse-drawn barge canal networks.  Horses brought coke or charcoal and iron ore to the smelters. Horse-powered capstans performed some arduous farm tasks such as threshing. Along with primitive Newcomen steam engines they pumped coal mines. Horse gins also powered most of the early versions of innovative textile machinery (they switched to more power-efficient water mills when they later scaled up).  That classic carnival ride, the merry-go-round, was inspired by these perpetually circling horses.

Again roughly speaking, the second phase of industrial growth, after about 1830, was more scientific and far easier to copy than northwestern Europe's unique biology: steam engines came to replace horse gins and water mills for running industrial machinery, and the steam-powered railroad radically lowered transportation costs between major mines, factories, and urban centers. When non-European countries industrialized, such as Japan after the 1870s, they did it in a "leap-frog" style: they skipped over the long-evolved improvement in draft animals and went straight to mature steam engines and, soon thereafter, electrical motors.  Much as countries installing phone networks for the first time over the last few decades have leap-frogged over the land line era, going straight to cell phones. Starting early in the 20th century industrializing countries could replace all the remaining important functions of the horse with internal combustion engines.  England, which made the longest and most thorough use of the horse, and thereby had the transportation economies allowing it to pioneer the industrial revolution, had a less pressing need to use the internal combustion engine and thus lagged enough in that technology so that second-generation  industrializers like Japan, Germany, and the United States became leaders in internal combustion engine products.

Given the scientific nature of the second phase of the industrial revolution, which could be discovered by any culture full of literate craftsmen, this second phase was more technologically inevitable and didn't ultimately depend on northwestern Europe's unique biology.  At the same time, during the long evolution that culminate in the industrial revolution, and during its first phase, land transportation the world over was muscle powered and the unique system of stationary pastoralism, by breeding draft horses that ran on cultivated, nutrient-dense fodder, substantially lowered transportation costs. This allowed the value of northwestern Europe's bulk transportation networks to radically increase and made it very nearly as inevitable that that region would be the pioneers of the industrial revolution.

Hat tips and references: Edward Wright and Raymond Crotty among many other authors have explored some of these issues.
Raymond Crotty
Raymond Crottyamong many other authors have explored some of the issues.

Wednesday, July 02, 2014


Monday, November 18, 2013

European-Asian divergence predates the industrial revolution

Stephen Broadberry describes new estimates of per capita GDP which say that the economic divergence between Western Europe and other civilized parts of the world predates the industrial revolution.  (H/T Marginal Revolution).  This is more consistent with my own theories (linked below) than the idea that the Great Divergence magically appears from nowhere around the year 1800.  Nevertheless I feel compelled to point out shortcomings in these kinds of estimates, on any side of such debates.

There are the usual correctable, but sadly seldom corrected, problems with datasets comparing European economies over historical periods, for example using "Holland", and leaving out, presumably not only the rest of the modern Netherlands, but the entire area of the exceptional Low Country late medieval industry and wealth (Flanders, Brabant, Hainault, etc.), most of which migrated (along with most of the skilled craftsmen and merchants) to the Netherlands during the 16th century wars there.  The southern Low Countries, until those wars, were the leading centers of European textile manufacture and probably also had the most labor-productive agriculture.

Worse are these and all other attempts to compared historical European "wealth" or "income" to those of non-European cultures before the era of cheap global precious metals flows (initiated by the exploration explosion) allows comparison of prices.  How do you compare the “wealth” or “income” of rice-eating and cotton-wearing Chinese farmer to a milk-drinking, oat-eating, and wool-clad Scottish peasant? It it is neither very useful nor very reliable to try to reduce such cultural and even genetic differences to mere numerical estimates.

So it's no surprise to see such conjectural and subjective estimates subject to major revisions, and I'm sure we'll see many more such revisions, in both directions, in the future. That said, many of the economically important innovations in northwestern Europe long predate not only the industrial revolution, but also the Black Death (Broadberry's new date for the start of the Great Divergence), including the following biological bundle:

(1) heavy dairying

(2) Co-evolution of human lactase persistence and cow milk proteins

(2) delayed marriage

(3) hay

(4) greater use of draft animals

These innovations all long predate the Black Death, except that thereafter this biological divergence, especially in the use of draft animals, accelerated.  After a brief interruption the lactase persistent core resumed its thousand-year conversion of draft power from humans and oxen to horses, including super-horses bred to benefit from good fodder crops -- the Shire Horse, Percheron, Belgian, etc., and of course the famous Clydesdale of the beer ads.  Draft horses figured prominently in the great expansion of the English coal mines from the 14th to 18th centuries. They both pumped the mines and transported the coal to navigable water.  Due to lack of horsepower for pumping and transport, the Chinese use of coal, though already well established by the 13th century visit of Marco Polo, where both mine and consumer were within short human-porter distance to navigable water, failed to grow beyond that limit until the coming of the railroad.  Similarly draft horses, alongside the more famous water-mills, played a key role in the early (pre-steam) exponential growth of the English textile industry, the economically dominant feature of the early industrial revolution.

Greater use of draft animals led to higher labor productivity and larger markets for agricultural output, and thus to greater agricultural specialization. Higher labor productivity implies higher per capita income, even if it can’t be measured. For civilizations outside Western Europe by contrast, much less use was made of draft animals with the result that these effects were confined to within a dozen or less miles of navigable water.

Contrariwise, northern Europe has always been at a severe ecological disadvantage to warmer climates when it comes to growing rice, cotton, sugar, and most other economically important crops.  However these seem not to have had an anti-Malthusian effect in increasing labor productivity -- the increased efficiency of rice in converting solar power to consumable calories, for example, simply led to a greater population rather than a sustained increase in per capita income.

Sunday, August 04, 2013

Political relationships

In most political theories and ideologies, there is a preposterous oversimplification about what kinds of political relationships are desirable, common, or even possible. Given the irreduceable complexity of society, any summary of real-world political relationships is by necessity going to be greatly oversimplified, but most such movements neglect even very broad and common kinds of political relationships.  So herein, based on my extensive study of the legal relationships between political players that have existed in a very wide variety of polities, is a classification scheme:

Let's define a "polity" as any entity with some coercive powers. Polities can range in scale from the United Nations to the jail cell at the back of your local shopping mall.  By studying polities over many years, and borrowing from previous work on law and political science, I have identified three basic kinds of legal relationships between polities. The basic legal structure, or constitution, of a polity can also be characterized by how much and in what ways it is composed of each kind of relationship. The three basic kinds of political relationships are:

(1) Delegation: This includes any kind of delegation from a principal to an agent.  The principal authorizes the agent to act for him, e.g. by making a contract or treaty with a third party to which the principal will be bound. A principal can be a boss, a contractee, or voter; the corresponding agents being employee, contractor, or representative.  We can characterize principal/agent relationships by representation distance, with each extreme common:

(A) at a very short representation distance is the master/servant (in modern parlance, employer/employee) relationship. The master gives orders to the servant who is delegated to carry them out and closely supervised. A military dictatorship, for example the Roman Empire, is or was dominated by commander-subordinate relationships.  In such a system, to paraphrase the legal code compiled for the Emperor Justinian, the emperor's will is law.

(B) at the other extreme, an extremely long representation distance, is the relationship between millions of voters and the representatives they vote for in most modern governments.  Voters do not give orders, but rather are treated as having delegated most their coercive powers to their representatives.  Representatives in modern governments usually further delegate political and legal power to unelected bureaucracies themselves dominated by type A (boss/employee) relationships.

(2) Subsidiarity: for example the relationship in the United States between counties and states, or between the states and the federal government. Often these combine supremacy clauses (when in conflict the law of the encompassing jurisdiction trumps that of its subsidiaries) with typically enumerated powers (the subject matters of the encompassing power is typically limited relative to that of the subsidiary). We can characterize subsidiarity relationships by how much and what kinds of coercive power can be exercised by the encompassing jurisdiction.

In medieval England, the subject matter of the encompassing jurisdiction was very small, the Crown essentially having jurisdiction only regarding procedural laws for interactions between subsidiarity jurisdictions (which like the encompassing Crown were held as property by individuals or corporations), as well as some war-making powers. Substantive law was almost entirely in the hands of the encompassed jurisdictions, including the specialized merchant courts as peers enforced an international standard of business law, the lex mercatoria.

By contrast in the modern U.S., the substantive legal jurisdiction of the encompassing power has become vast in scope.  Nevertheless one can still find many examples of fine-grained subsidiarity, down to "stand your ground" laws, citizen's arrest, and those shopping mall jail cells.

In property law (which once was also procedural law and essential to defining political relationships), the landlord/tenant relationship is a subsidiarity relationship. The landlord is generally not the master of the tenant, and cannot issue the tenant arbitrary commands, but rather their relationship is governed on both sides by the constraints imposed by the tenancy.

(3) Peer-to-peer: these include any agreements made between polities where neither is a subsidiary of the other, or a standard law arrived at in parallel, either through parallel development of precedent (as in the lex mercatoria and many other bodies of law) or codification of a standard law (e.g. the Uniform Commercial Code, which is not federal or national law, but a standard set of laws separately enacted by 50 separate jurisdictions, the states of the United States, as peers).  Peer-to-peer relationships most commonly involve maintaining distinct sets of laws adapted to local conditions along with agreements or mutually evolved practices for resolving conflicts of laws. Conflict-of-laws law itself was primarily developed through parallel development of judicial precedent, through courts respecting each other in order to maintain their reputations for enforcing the rule of law. On a larger scale wars and treaties between nations are peer-to-peer relationships. In medieval Italy, a wide variety city-states that were often at war with each other nevertheless also developed through this process most of the body of modern conflict-of-laws law.

Since political theory developed in universities out of the study of Roman imperial law, it has been dominated by imputing to polities only one of the above kinds of relationships -- namely master/servant or commander/subordinate relationships, or at best delegation in general.  This is especially apparent in the quixotic search for a "locus of sovereignty", a search that typically amounts to conspiracy theory in search of a hidden commander-of-all when in fact far more sophisticated combinations of the above kinds of relationships are at play.

For further reading:
Conflict-of-laws law
Substantive vs. procedural law
Jurisdiction as property (subsidiarity and peer-to-peer relationships via property law)
Representation distance
Liberty of house (common law origins of stand-your-ground laws)

Saturday, July 20, 2013

A very underrated invention

Perhaps the most underrated invention in history is the humble hourglass.  Invented in Europe during the late 13th or early 14th century, the sand glass complemented a nearly simultaneous invention, the mechanical clock.  The mechanical clock with its bell was a centralized way of broadcasting the hours day and night; the sand glass was a portable way of measuring shorter periods of time.  These clocks were made using very different and independent techniques, but their complementarity function led to their emergence at the same time and place in history, late medieval Europe.

The sandglass was more portable than a water clock. Since its rate of flow is independent of the depth of the upper reservoir, it was also more accurate.  And, important in northern Europe, it didn't freeze in winter.
An advancing technology in 13th century western Europe very different from mechanics was glass-blowing. The origin of the sandglass is quite obscure, but its accuracy relies on a precise ratio between the neck width and the grain diameter. It thus required extensive trial and error for glass-blowers to arrive at hour glasses for sand, ground marble, eggshell, and other sized grains, and techniques for mass producing these precisely sized works of glass, besides a ready of market of users, which Europe turned out to be.

There are no demonstrated cases of sandglasses before the 14th century. Manufacture and use of the sand-glass was widespread in western Europe by the middle of the 14th century. In 1339 Ambrosio Lorenzetti painted a fresco in Siena, one of the commercial cities of northern Italy, which shows a sandglass as an allegory for temperance (self-control). Mariners in the Mediterranean were likely using sandglasses to measure time and velocity by 1313. By 1394 French housewives were using recipes to make, along with food, glue, ink, and so on, marble grains for an hour-glass:
"Take the grease which comes from the sawdust of marble when those great tombs of black marble be sawn, then boil it well in wine like a piece of meat and skim it, and then set it out to dry in the sun; and boil, skim and dry nine times; and thus it will be good."
Such a recipe presumably creates grains of a size in a precise ratio to a standard hour-glass neck size, thus producing an accurate time.

The sandglass, not the mechanical clock, became between the 13th and 16th centuries the main European timekeeper in activities as diverse as public meetings, sermons, and academic lectures. It was also the main navigational and scientific clock during that period. [*]
From the point of view of later engineers, the mechanical clock was the more important invention -- they were on the cutting edge of technology from the time of their invention until the industrial revolution.  However,
For contemporaries....the sandglass was equally or more important.   Until the widespread use of small table-top mechanical clocks, the sandglass was the primary means of fair timekeeping.    The sand glass was visible to all in a room, and it could only be dramatically and obviously “reset”, it couldn’t be fudged like a mechanical clock.   [*]
As I detail here,  the sand glass also played an essential role in the technique of dead reckoning for ocean navigation, also developed in late medieval Europe.  A strict regimen of turning the glasses was kept non-stop throughout a voyage:
During the voyage of Ferdinand Magellan around the globe, his vessels kept 18 hourglasses per ship. It was the job of a ship's page to turn the hourglasses and thus provide the times for the ship's log. Noon was the reference time for navigation, which did not depend on the glass, as the sun would be at its zenith.[8] More than one hourglass was sometimes fixed in a frame, each with a different running time, for example 1 hour, 45 minutes, 30 minutes, and 15 minutes. [*]
Arab and Chinese navigators lacked this crucial piece, and thus by the time of the exploration explosion had not developed navigation techniques that could rival those of Western Europe.

Sunday, October 28, 2012

Dead reckoning, maps, and errors

In my last post I introduced dead reckoning as used during the exploration explosion. In this post I will describe the errors these explorers (Dias, Columbus, da Gama, etc.) typically encountered in dead reckoning (DR) when sailing on the oceans, and why dead reckoning could be usefully accurate despite the fact that trying to map those dead reckoning directions onto a normal map would be very inaccurate.

To get a taste of the issue, first consider the following abstract navigation problem -- hiking in foggy hills:
  1. There are only two useful landmarks, 1F (the origin or "first fix") and 2F (the destination or "second fix").
  2. It’s very foggy, so you have no way to use the hills as recognizable features. But the dead reckoning directions are of sufficient accuracy to get you within sight of landmark 2F. (For simplicity assume 100% accuracy).
  3. You don’t know and can’t measure hill slope angles. Indeed there are only two things the hikers can measure: (a) magnetic compass direction, and (b) distance actually walked. Observe that this is not distance as the crow flies, nor is it distance projected onto the horizontal plane. If a hill happens to be a pyramid, and you happen to be walking straight up it (and thus walking up the hypotenuse of a triangle), the distance measured is the length of the hypotenuse, not the length of the horizontal leg of that triangle.
  4. The first person who discovered 2F, starting from 1F, recorded dead reckoning directions to there and back as a sequence of tuples { direction, speed, time }.
We can draw a useful head-to-tail diagram of these directions on a piece of paper. But we can’t use these directions to figure out the distance as the crow flies between 1F and 2F, because we don’t know the slopes of the hills traversed. And for the purposes of our loose analogy to long-distance ocean navigation, our hikes are short and could be in all steep terrain or all flat, so that over the course of our hike the slopes don’t converge on a knowable average.

Since we have insufficient information to determine "crow flight" distances, we don’t have enough information to accurately draw our dead reckoning itinerary on maps as we know them (i.e. Ptolemaic maps). Yet such faithfully recorded directions are sufficient to get any hiker (who can also exactly measure bearings and distances) from 1F to 2F and back.

Most maps as we know them – Ptolemaic maps -- are projections from a sphere to a Euclidean plane based on lines of latitude and longitude where lines of longitude converge at the celestial poles. Latitude is determined by measuring the altitude of a celestial object, and latitude is also ultimately defined by what navigators call the celestial sphere (although by "Ptolemaic map" I will refer to any map that shows actual earth surface distances proportionately on the map, i.e. "to scale"). There are also non-Ptolemaic maps, for example subway maps, which show the topological relationships between entities but not proportional distances. This chart of the kind Zheng He may have used, or was drawn using information from those or similar voyages, was of such a topological nature (the west coast of India is along the top and the east coast of Africa is along the bottom):

A set of dead reckoning directions can be diagrammed.  But although it contains more information than a subway map, it doesn’t contain enough information to plot on a Ptolemaic map. Thus like a subway map this dead reckoning "space" cannot be accurately projected, or "mapped" in mathematical terminology, onto a normal (Ptolemaic) map without further information.

A subway map is in no way "to scale": the distances on are not proportional to any measured values.  By contrast a dead reckoning map can be drawn "to scale" in its own distinct Euclidan plane.  But not only cannot this dead reckoning space without further information be accurately projected (i.e. projected with proportions intact or "to scale") onto a Ptolemaic map, but two different dead reckoning itineraries drawn on a  Euclidean plane will also generally be in error relative to each other, as I will describe below.  And now to the central point I want to get across in this article: these two kinds of errors -- from trying to Ptolemaically map a dead reckoning itinerary on the one hand and between two dead reckoning itineraries on the other hand -- are very different.  They are quite distinct in kind and usually produce errors of very different magnitudes.

The unknown values ignored in a dead reckoning itinerary, analogous to the hill slopes in the scenario above, can be any spatially variable but temporally constant distances, directions, or vectors that are unknown to the navigators writing and following the directions. The three most important spatially variable but temporally constant sets of vectors generally unknown to or ignored by dead reckoners on ocean and sea voyages from the 13th century through the era of the exploration explosion were were magnetic variation (shown below as green arrows), current (red arrows), and curvature of the earth (ignored in this post, but the same argument applies). Since these temporally constant but spatially variable factors (analogous to the slopes of our foggy hills) were unknown or ignored, they had no way to map such pure dead reckoning directions onto a Ptolemaic map. The information they rigorously measured and recorded for the purposes of dead reckoning was insufficient for that purpose. Yet that information was sufficient to enable navigators to retrace their steps (to get back on course if blown off course) or follow a previously recorded dead reckoning itinerary (or a nearby course, as I'll show below)
with usefully small error.

Temporally constant but spatially variable vectors shown on a diagram.  Only the dead reckoning (DR) vectors are shown added head-to-tail, since these are all the dead reckoning navigator  in the exploration explosion era usually measured. The vectors shown here are magnetic variation (green) and current (red). Since these vectors were unknown, dead reckoning directions could not be accurately plotted on a Ptolemaic map. Curvature of the earth, not shown here, is also temporally constant and can thus also be ignored for the purposes of dead reckoning.

However some kinds of dead reckoning errors were due to unknowns variables that changed over time. These produced small but cumulative errors in dead reckoning even for the purposes of specifying repeatable directions. Errors in measuring bearing, speed, and time were of this nature. Externally, different winds required different tacking angles, creating "leeway", where the boat moves not straight forward but at an angle. If the directions don’t account for this, or account for it imperfectly, there will necessarily be a cumulative error. It was thus important to "fix" on landmarks or soundings. The more accuracy needed (such as when approaching shorelines, much more hazardous than open-ocean sailing), the more often fixes had to be taken. I hope to say more about fixes and temporally variable errors in future posts. This post is about dead reckoning between two fixes and errors that vary spatially but can be reasonably treated as constant in time.

A dead reckoning diagram made on a chart, with "fixes" or adjustments (dashed line)s to a landmark or sounding (yellow "X") diagrammed on the chart. The start and end of points of the voyage are also landmarks, so there is also a fix for the final landmark. Note that the chart still does not have to be Ptolemaic for this purpose -- the fixes need not be shown with proportionally correct distances to each other. Indeed the Zheng He era chart above is roughly in this form, with only one crude dead reckoning vector between each fix: it labels each arc with a crude time or distance estimate along with a (much more accurate) bearing estimate, but like a subway map it doesn't care about showing distances as proportional.

When sailing over continental shelves, European mariners (and sometimes Chinese mariners) of that era took "soundings" that measured depth and sampled the bottom soil, creating a unique signature of { depth, soil type} that functioned like landmarks but on open ocean. Soundings could be taken when sailing over the relatively shallow areas of continental shelves. As you can see, most parts of the oceans are too deep for this, but most shorelines are fronted by at least a few miles of soundable shelf, and sometimes hundreds of miles. Soundings were very useful for navigating in clouds, fog, and at night far enough away from the shore to avoid the hazards of very shallow water, yet close enough for the water to be shallow enough to sound. Pilots that used soundings thus had a set of "landmarks" for fixing their dead reckoning directions that allowed them to avoid hazardous navigation too close to land.

Notice that these kinds of fixes still do not give Ptolemaic coordinates -- they simply map or "fix" a particular point in our dead reckoning "space" to a particular point on the earth's surface of unknown Ptolemaic (celestial) coordinates, and indeed of unknown distances relative to other fixes.

(Side note -- explorers between Cao and Magellan usually could not get a celestial "fix" on a rolling deck of sufficient accuracy to be useful, i.e. more accurate than their dead reckoning -- and even in the case of Magellan this was only useful because there was nothing better, dead reckoning errors having accumulated to very high levels by the time they were in the mid-Pacific.  So like them we will have to ignore this way, both more ancient and more modern, but generally unused during the exploration explosion, of correcting DR errors at sea).

It's all fine and good for dead reckoning to provide, as shown above, repeatable directions to a destination, despite being Ptolemaically unmappable, when the same itinerary is exactly repeated.  But the best itinerary over the oceans depends on the wind.  These winds vary, and the early explorers of new oceans searched for the best courses and seasons in order to catch the best winds.  So the early explorers usually did not exactly repeat dead reckonings recorded on prior voyages.  They usually took courses a few hundred miles away from the prior voyages' itinerary in order to catch more favorable winds.  So the question arises: if the navigator adjusts his course by a few hundred miles, roughly what amount of resulting error should the navigator generally expect.

(BTW, it us  important to note that dead reckoning directions, while they did not have to account for currents, magnetic variation, and the curvature of the earth, for the reasons given in this article, did have to account for variations in winds and the related leeway from tacking, since these reasons do not apply to vectors with substantial temporal variability.  So we assume, as the navigators themselves probably did in some fashion, that the velocity vectors in our dead reckoning itineraries aren't strictly those measured, but are those measurements adjusted for variations in wind).

To reiterate the most important point: this is a different question than the question of what the error is when plotted on a normal map.  Historians trying to recreate these voyages, in order to figure out where their landfalls were, or plot them on maps, or to estimate what navigational acccuracy of European navigators achieved in that era, usually haven't understood this crucial distinction. Indeed, because currents and magnetic variation don't in most places in the open ocean change in extreme or sudden ways, the resulting errors in dead reckoning navigation tended to be much smaller than the errors when plotting the dead reckoning directions on a Ptolemaic map. If you can scrutinize some more complicated diagrams I can demonstrate this by example here. First consider two dead reckoning itineraries, unadjusted for current and magnetic variation and thus plotted non-Ptolemaically:

Black = DR velocity in given time period

Red = average current velocity in given time period

Green = average magnetic variation in given time period

A, B = Two different DR itineraries as recorded (i.e. not adjusted for unknown magnetic variation and current). B has different first and third leg plus different currents on last two legs (only DR measurements added head-to-tail) – navigator would not actually plot these on a chart of geographic location, or at least would not consider such plotting accurate.

1F, 2F = first fix, next fix (same in each case, but their geographical location doesn’t need to be known)

For simplicity I am treating magnetic variation as uniform and spatially varying only the current, but the same argument I make here applies even more strongly to magnetic variation (and even more strongly to curvature of the earth, which can be treated as another set of vectors).  The second fix (2F) has a question mark in front of it to indicate that the second itinerary (B) won't actually arrive at the same spot as A arrives at -- due to the different currents it encounters, it will arrive at a different spot.  We assume, as was usually the case out of sight of shore, that our early explorer doesn't know the current.  But the explorer did want to know, as historians want to know: roughly how large can such errors in typical oceans be expected to be?  To demonstrate the mathematics of this, I've created a Ptolemaic map of the itineraries (dashed lines) by adding in the currents and magnetic variations head-to-tail.  I've also superimposed the original non-Ptolemaic diagram (just the dead reckoning vectors added up) to show the much larger error that occurs when trying to project that onto a Ptolemaic map.

A‘, B’ = A and B adjusted to show difference in geographic location (all vectors added head-to-tail). The navigator in Columbus’ day could not usally compute these, since he typically did not know the current and magnetic variation values.

NA, NB = net effect of spatially variable but temporally constant current on geographic (i.e. Ptolemaic or celestial) location. Error if unadjusted itineraries Ptolemaically mapped. Separate red arrow diagram shows the same net effect of the two separate sets of currents.

Dashed blue line next to 2F = actual navigator’s error of two DR itineraries against each other when neither set of itineraries adjusts for current or magnetic variation. The next fix lies somewhere on this line, assuming no other errors.

(BTW if you can copy and paste arrows it's easy to make your own examples).

As you can see, the errors (solid blue lines labeled NA and NB) from trying to superimpose the non-Ptolemaic dead reckoning itineraries (solid lines) on the Ptolemaic map are much larger than the actual error (dashed blue line labeled 2F) that occurs from following itinerary A instead of B or vice versa (shown on dashed lines when adjusted for current.  The magnetic variation is held constant, but the same argument applies to that, and to the curvature of the earth.

Note that the error in locating our second fix 2F is simply the same as the difference between the two separately added sets of current vectors:

It would be instructive to create a computer simulation of this which plugs in actual values (which we now know in excrutiating detail) for current, magnetic variation, and curvature of the earth.

Thursday, October 18, 2012

Dead reckoning and the exploration explosion

Navigation is the art or science of combining information and reducing error to keep oneself on, or return oneself to, a route that will get you where you want to go. Note what I did not say here. Navigation is not necessarily the art or science of locating where you are. While answering the latter question – i.e. locating oneself in a Euclidean space, or a space reasonably projectable onto a Euclidean space – can usefully solve the navigation problem, figuring out such a location often requires different, and often more, information than you need to answer the questions of how to stay on or return to your desired route. And indeed this is what dead reckoning does – it gets you where you want to go with different information than what you would need to draw or find yourself on a normal map. I hope to explain more about this important incompatibility between the pilots’ and cosmographers’ systems during most of the age of exploration in a future post, but for now I will give an overview of the historical development of dead reckoning.

Between Italy of the late 13th century and the advent of GPS, dead reckoning formed the basis of most modern navigation. Dead reckoning was in particular the primary method of navigation used during the exploration explosion of the late 15th and early 16th centuries – the startlingly unprecedented voyages across unknown oceans of Dias, da Gama, Columbus, Magellan, and so on.

Dead reckoning is based on a sequence of vectors. Each vector consists of two essential pieces of information: direction and distance. Distance is typically calculated from time and speed, so each vector typically consists of the tuple {direction, time, speed}. With only speed and time, we have only a scalar distance value – it could be in any direction. With time but not speed, or speed but not time, we don’t have enough information to determine the distance covered.

From the start of a voyage to the last docking at the home port, dead reckoning was a strict regimen that never stopped: day and night, in calm and in storm, its measurement, recording, and diagramming protocols were rigorously followed.

Measuring or estimating the speed of a ship was a craft mystery the nature of which is still debated today, so I’ll skip over that and focus on the two more straightforward innovations in measurement, both of which occurred in or reached Italy and were first combined there in the 13th century: in measuring direction and in measuring time.

For measuring time mariners used the sand glass, invented in Western Europe during that same century. I have discussed this invention here. A strict regimen of turning the glasses was kept non-stop throughout a voyage.

For measuring direction, the ships of the exploration explosion typically had at least two magnetic compasses, usually built into the ship to maintain a fixed orientation with the ship. Typically one compass was used by the helmsman, in charge of steering the ship, and the other by the pilot, in charge of ordering both the sail rigging and the general direction for the helmsman to keep.

The magnetic compass was probably first invented in China, used first for feng shui and then for navigation by the early 12th century. Somehow, without any recorded intermediaries, it appears in the writings of authors in the region of the English Channel in the late 12th century where it was quite likely being used for navigation in that often cloudy region. Its first use in Italy was associated with the then-thriving port city of Amalfi. As both Amalfi and the English Channel were at the time controlled by the Normans, this suggests to me either a Norman innovation, or arrival via Norse trade connections to the Orient via Russia combined with now unknown Chinese trade routes. This is conjectural. Neither the Norse sagas nor writings about the Normans during earlier periods mention a magnetic compass, nor do Arab sources mention it until the late 13th century in the Mediterranean. In any case, it is the Italians who made the magnetic compass part of a rigorous system of dead reckoning.

Green dots indicate, in the case of northern Europe, the location of authors who mention use of the magnetic compass for navigation in the late 12th and 13th centuries, and for Italy, the traditional Italian association of the invention of the compass with Amalfi in the 13th century. Red indicates areas controlled by the Normans.

A dead reckoning itinerary can be specified as a sequence of tuples { direction, speed, time }. It can be drawn as a diagram of vectors laid down head-to-tail. However, as mentioned above, this diagram by itself, for nontrivial sea and ocean voyages, contains insufficient information to map the arrows accurately onto a Ptolemaic map (i.e. maps as we commonly understand them, based on celestial latitudes and longitudes), yet sufficient at least in theory to guide a pilot following such directions to their destination.

For recording speed and direction for each sand glass time interval (e.g. half hour), pilots used some variation of the traverse board, in which these values were specified by the locations of pegs in the board.

Traverse board. Pins on the upper (circular) portion indicate compass heading and (via distance from the center) for each half hour. Pins on the lower (rectangular) portion indicate estimated speed during each hour. The board thus allows an a pilot on a wet deck unsuitable for a paper log to record an equivalent of a sequence of tuples { direction, speed, time } over four hours, after which time this information is transferred to the ship’s written log(normally kept indoors), the progress is plotted as a head-to-tail diagram on a chart (also kept indoors), and the traverse board is reset. Note that the direction is read directly off the magnetic compass: thus north (the fleur-de-lis) is magnetic north, not geographic (celestial) north.
In a future post I hope to discuss more about dead reckoning directions and explain how the errors that can accumulate in such directions over long distances were corrected. I will also explain why neither the directions nor even the corrections could be accurately drawn on a normal (Ptolemaic or celestial coordinate) map, and yet such dead reckoning directions are sufficient at least in theory for the pilot to guide his ship from the starting port to the intended destination port. In practice, pilots "fixed" errors in their dead reckoning using landmarks and sounding, which I will also describe. And I hope to describe how this resulted in two incompatible systems of “navigation” (broadly speaking) during exploration explosion -- the pilot’s dead reckoning methods versus the cosmographers’ maps and globes based on latitude and longitude.

I also hope to someday figure out just why the exploration explosion occurred when it did. The advent of rigorous dead reckoning -- combining the compass, the sand glass, and decent estimates of speed with rigorous log-keeping -- did not occur in Asia (where the Chinese, lacking the sand glass at least, made a less systematic use of the compass), nor with the Arabs (who seldom used either sand glass or compass), which along with naval superiority explains why the exploration explosion occurred from western Europe. The puzzle of why the explosion started specifically in the 1480s, and not sooner or later, however, remains a mystery to be solved.