I recently read the introduction to History of Civilization in England by Henry Thomas Buckle (1821-1861). In this long essay, he outlined his dissatisfaction of history study being a collection of facts and called for its transformation into a proper science, which generates knowledge in the form of causal connection. He made many insightful remarks and one that got me thinking is about free will vs. determinism. He Obviously took the determinist position when he claimed that if he has enough information about the circumstance which his well acquainted friend is in, he can predict with certainty what his friend will do next. He keenly pointed out a problem of free will. Before we can have a meaningful discussion, I want to make explicit the essential conception of a free will. We think of a free will as an active agent capable of dictating one's thoughts and actions independent of one's experience or the given circumstances. Buckle's insight is that in order to freely will, we need to have another authoritative agent independent of deterministic laws to inject such decision into our consciousness given that the physical universe evolves according to deterministic laws. But how does that agent operate? Given that it lives in a world also evolves according to some deterministic laws, since it has to be free as well, it would require yet another free active agent. (This kind of reasoning is not unlike Descartes's notion of mind-body duality, which is wittily summarized as "ghost in the machine" by Gilbert Ryle.) Thus we enter an infinite regression that unsuccessfully evades the contradiction between determinism and free will.
This is a tough question because one is reluctant to give in entirely to either side. Some people suggested that quantum physics might hold the answer due to the truly random events allowed in quantum physics. I initially felt unsatisfied by such proposal because we want to think of will actively, instead of randomly decides on things. But lately I came to realize that in observation of a single act, the two are indistinguishable! My worry was due to my subjective experience of activeness. Let's put aside the subjective content and imagine an experiment in which Buckle watches his well acquainted friend for a while and then predicts what his friend will do next. If free will exists, Buckle's friend will be able to decide on things independent of his experience and as a consequence, Buckle will not be able to predict with certainty even with all the information. In observation, that is precisely what you expect from a random event whose outcome is independent of the past. Although our subjective experience is too intricate to give us direct clues, the unexpectedness of a certain decision might have germinated from a seed of randomness allowed by quantum physics.
This resolution proposes that we give up the deterministic view, which arguably is suggested and supported by classical physics, to allow for some non-deterministic events, which is supported by quantum physics, our current best knowledge. This answer might still upset people believing in free will, because randomness seems to dismiss one's active control over his or her life. But the pursuit for the possibility of control only pulls us back to the infinite regression. While the moral implication in absence of an active agent is interesting to consider, I think I will spend more time on pondering about how quantum mechanics allows for randomness when all its laws, like those in classical physics, are deterministic (in probability). In quantum mechanics, randomness creeps in at the act of measurement whose outcomes' probabilities are predicted by input and laws, but not the outcome itself. I will subject further discussion on this topic to a future article. Let's revisit our thought experiment and introduce a modification: instead of predicting the exact action of his friend, Buckle will predict the probability of each possible action and we shall repeat the experiments many times (maybe with identical clones of his friend). If free will can actively control one's decision, Buckle's friend can decide how frequently he wishes to do a certain thing and thus he can manipulate the frequencies of each action rendering Buckle's prediction of probability incorrect. This is why the condition of "a single act" is needed in my statement of indistinguishability. However, this experiment is hard, if not impossible, to carry out because after each decision, the state of Buckle's friend changes and cloning the quantum state of Buckle's friend is impossible: we need to find some decision whose probability does not change when repeated. Hence a definite answer to whether free will exists, and if so whether it exists in the weak, random form or the strong, ghost form, is not known yet.