Computer Science, Off Course! Episode 6 - Computer Power and Human Reason

New episode just dropped, on Computer Power and Human Reason by Joseph Weizenbaum.

This book is interesting, because Joseph Weizenbaum (as might be well-known, by now) was the creator of the first chatbot, ELIZA. After building ELIZA, he immediately realized this was a mistake. The best quote from the book undoubtedly is this one, in times of LLMs of course so incredibly fitting (I put it in my bio on BlueSky):

What I had not realized is that extremely short exposures to a relatively simple computer program could induce powerful delusional thinking in quite normal people

Weizenbaum, as author of ELIZA, of course is in a perfect position to observe this. His account of his secretary who wanted him to leave the room when she interacted with it became the stuff of legends the last few years. It is also incredibly interesting to see that in 1976 he writes:

It has happened many times in the history of modern computation that some technological advance in computer hardware or programming (software) has triggered a virtually euphoric mania.

Sadly though, he kept warning, until shortly before his death at DAVOS. It is a beautiful panel, not only because he speaks German and is translated, but sometimes his real audio is used when he quotes German words (like listening and hearing), but also sad to see, he is so frustrated that everyone is so positive. In contrast, he is not, he says, a bit negative, he is very negative!

Anyway, on the the book!

What computers shouldn't do

Weizenbaum positions himself against Dreyfus's book What Computers Can't Do (episode 4 of CSoC), and which appeared in 1972 for the first time. Weizenbaum's book appears only a little bit later, in 1976. He says that while Dreyfus argues there are things computers can't do, he talks about what computers shouldn't do. People, he argues, have feelings, culture, history, and take all of that with them in their lives. Most succinctly, he puts it later in the book:

[T]here are problems which confront man but which can never confront machines

and

What could it mean to speak of risk, courage, trust, endurance, and overcoming when one speaks of machines?

And the most beautiful one:

Sometimes when my children were still little, my wife and I would stand over them as they lay sleeping in their beds. We spoke to each other in silence, rehearsing a scene as old as mankind itself.

What a contrast with John McCarthy whom Weizenbaum cites as having said:

[In order to s]ucceeded in formalizing every aspect of the real world is that we have been lacking a sufficiently powerful logical calculus. I am currently working on that problem. (emphasis mine)

Of course, McCarthy here does not mean that he plans to formalize the task of watching your baby sleep, but yet is says "every aspect of the real world".

A somewhat more concrete example of a task a computer should not do, Weizenbaum explains using the contrast of deciding versus choosing. Making a decision can be done with both these actions, but where deciding can be calculated, choosing can also be based on empathy, or love. Computers on the other hand:

[...] may even be able to arrive at "correct" decisions in some cases—but always and necessarily on bases no human being should be willing to accept.

He elaborates (a bit long) on different cultures and different mores, much also like the gift giving example of Bourdieu and Dreyfus, but Weizenbaum takes as an example to be a judge and to decide in a court case. So much depends on customs a an outsider could never really know. Can we ever really decide what is appropriate in a (literal) foreign setting?

There are vast areas of authentically human concern in every culture in which no member of another culture can possibly make responsible decisions. It is not that the outsider is unable to decide at all—he can always flip coins, for example—it is rather that the basis on which he would have to decide must be inappropriate to the context in which the decision is to be made.

Weizenbaum himself takes a quote from John McCarthy here, who asked once asked him: "what judges know that we cannot teach a computer?"

Here I think he partly falls in the trap McCarthy laid, because the culture's argument I suspect would be answered by McCarthy by saying that a computer of course could be trained in the logic of another culture. He makes a lot more arguments that boil down to a computer doesn't do it well, although very very prescient:

Imagine an adding machine that adds some but not all numbers correctly, and about which we can't even say what characterizes the numbers it can add. We would hardly call that a mechanization of arithmetic.

Does that not sounds like ChatGPT today, haha! The argument about humanity I find stronger but of course (as Weizenbaum himself also explains, more on that later) that might not resonate so well with computer people.

Who is this book for?

This brings us to the question who this book is for. I think it is mostly a book that Weizenbaum wrote for himself, as a penance for having build ELIZA and a warning for future generations. In one part of the book he talks about scientists that, during the Vietnam war, were working on a line of mines to separate the parts of Vietnam (rather than directly bombing people) Weizenbaum asks them whether instead they should have just refused to work on the mines, too. One of the replied:

"We could have taken a moral stand, but what good would that have done?"

Weizenbaum does not agree, and continues:

But the good of a moral act inheres in the act itself

This is how I see this book, as a moral act of Weizenbaum himself.

Implications for programming education

Weizenbaum also has some (strong) things to say about programming education. This quote I have put in my newsletter several times:

I am constantly confronted by students, some of whom have already rejected all ways but the scientific to come to know the world, and who seek only a deeper, more dogmatic indoctrination in that faith (although that word is no longer in their vocabulary).

Weizenbaum connects CS to the broader role of the university:

Just because so much of a computer-science curriculum is concerned with the craft of computation, it is perhaps easy for the teacher of computer science to fall into the habit of merely training. But, were he to do that, he would surely diminish himself and his profession. He would also detach himself from the rest of the intellectual and moral life of the university. The university should hold, before each of its citizens, and before the world at large as well, a vision of what it is possible for a man or a woman to become. [...] The mere teaching of craft cannot fulfill this high function of the university.

I can stress this enough for CS people, especially those teaching in the university, how bad our programs are, how intellectually meagre, even compared to other natural sciences like physics (see also my recent paper that presents a detailed comparison of two international curricula).

What is understanding?

One thing I will take with me from this book is Weizenbaum's two meanings of understanding. He says that understanding a physics equation is not the same as understanding a piece of art. These things can't be put into the same category, it is a category mistake [[1]].

Weizenbaum also has a foreshadowing of the work of Naur (as we talked about in episode 1), whom of course came years later (and whom Naur does not cite, interestingly).

One programs, just as one writes, not because one understands, but in order to come to understand

And related, he talks about how abstraction also has downsides:

Science can proceed only by simplifying reality. The first step in its process of simplification is abstraction. And abstraction means leaving out of account all those empirical data which do not fit the particular conceptual framework within which science at the moment happens to be working, which, in other words, are not illuminated by the light of the particular lamp under which science happens to be looking for keys.

Yes! So much of programming (instilled by programming education) is getting rid of details, of the vagueness we talked about last week as well!

Contrast with Emily Bender

This quote really made me think of Emily M. Bender, who often says that it is dehumanizing to be seen as an LLM, for example in the Great Chatbot Debate. I felt her words a lot, I am not a machine, I have an inner language and feelings and grief and joy. Weizenbaum though makes a nice argument against that:

Seeing man as an information-processing system does not in itself dehumanize him, and may very well contribute to his humanity in that in that it may lead him to a deeper understanding of one specific aspect of his human nature.

I love that I can agree with both of them at the same time (vagueness, hooray!)

The (lack of) theory behind LLMs

One of the deeper points, and a point, I think will be harder to grasp for CS people is that people building AI often lack a strong theory for what they are building.

This must be understood in the context of the times. Some people in the early days of AI where trying to model thinking with the computer not so much as a means of simulating it, but as a means of understanding it. That I think is an excellent and interesting goal, and to a certain extent also succeeded. Over time though, this goal drifted, he says. He distinguishes two modes of making AI, which I think are very insightful for today also: simulation mode and performance mode. Performance mode is when:

workers in AI is to build machines that behave intelligently, whether or not what they produce sheds any light on human intelligence.

He criticizes both laypeople understanding of computers, which is so simple that they can only imagine the computer working as their own mind does, but also the understanding of intelligence itself:

I shall argue that an entirely too simplistic notion of intelligence has dominated both popular and scientific thought, and that this notion is, in part, responsible for permitting artificial intelligence's perverse grand fantasy to grow.

(you see how he was not very popular among his CS peers, haha)

However, there is always the risk that building such a theory is limited by computer thinking. As Weizenbaum says:

[T]he only conceptual structures it admits as legitimate are those that can be represented in the form of computer-manipulatable data structures. These are then simply pronounced to constitute all the conceptual structures that underlie all of human thought.

This of course is visible in current discourse about LLMs too; people simply declare that they can think whereas they in reality are ultimately "represented in the form of computer-manipulatable data structures".

Homework

There are a lot of fun things to do!

• As I already say in the episode, specifically for teachers it would be very good to reflect on where you are showing your students "a vision of what it is possible for a man or a woman to become".
• Think about the quote on abstraction, think of situations where it is bad, what is the stuff you have to leave out to fit your abstraction.
• Build an ELIZA clone and interact with it!

Hanna's very epic homework can be found here.

Notes

[[1]]: A term I recently learned was coined by Gilbert Ryle who strongly influenced Peter Naur in Programming as theory building which we covered in episode 1!