Intro. [Recording date: June 27, 2023.]
Russ Roberts: Today is June 27th, 2023, and my guest is Zvi Mowshowitz. His Substack is Don’t Worry About the Vase. It is a fantastic, detailed, in-depth compendium every week, and sometimes more than once a week, about what is happening in AI [Artificial Intelligence] and elsewhere.
Our topic for today is what is happening in AI and elsewhere, particularly a piece that you wrote recently and we will link to, called “The Dial of Progress,” which by itself, regardless of its application to AI, I found very interesting. We’re going to explore that in our conversation.
Zvi, welcome to EconTalk.
Zvi Mowshowitz: Honored to be here.
1:14
Russ Roberts: First, on just the technical capabilities of where we are right now with AI, where do you think we are?
Zvi Mowshowitz: So, I think it’s still very early days. Right? So, AI has been advancing super-rapidly in the last few years as OpenAI and others have thrown orders of magnitude more compute, orders of magnitude more data, and superior algorithms continuously at the problem, including many more people working on how to improve all of these things.
The results of this recently with the giant breakthrough of ChatGPT [Chat Generative Pre-trained Transformers] and GPT-4, which is also used in Microsoft’s Bing search, which is a tremendous jump in our ability to just talk with it like we would talk to a human, to have it be a better way of learning about the world, getting your questions answered, exploring issues, than, say, a Google search, or in many cases going to a textbook or other previous information sources. It’s amazing at things like editing, translation, creating images for things like Stable Diffusion and Midjourney. It’s very, very good at allowing us to do things like perform class, to translate styles, to allow us to understand things that we’re confused by.
And it’s continuously learning. Right? Every month, we learn about new developments. Every week, I have this giant list introducing these–there are people who compile The 100 New AI Tools That You’ll be Able to Use This Week, and mostly they’re slight variants on things that happened last week or the week before. But, iteratively, all these things improve.
And so, now we’re starting to see multimodal approaches where not only can you use text, you can use pictures, and soon it will also be video. AIs are starting to generate voices more and more accurately. They can now match human voices very accurately on almost no data. They’ll soon be able to be generating videos.
Their context windows, their amount of information they can hold in their storage and react to at any one time, it usually expands. They’re now up to the length of books like The Great Gatsby in some cases, or at least by Anthropic and a model called Claude.
And, the sky’s the limit in many of these ways, and it’s all very exciting. I am substantially more productive in many ways than I would have been a few months ago because–like, when I see something without a reference, I’ll say, ‘Oh, okay, where’s that reference?’ I’ll just ask Bing. And, Bing searches the Internet for me without me having to think about the query and finds the reference, explains the information. I can ask for details; I can ask for summaries. I can ask about details of papers. Whenever I’m confused about something, I can ask it about what that’s about. These things are just scratching the surface of what I can do. And my coding ability has gone through the roof as well.
4:03
Russ Roberts: So, that’s where we are now. And, it’s interesting: I think the world is divided up into people who are using it frantically–frenetically would be a better word, not frantically–frenetically, like you or Marc Andreessen, who we recently interviewed, and those who have never heard of it, don’t know how to use it, think it’s something weird. And, I’m in the in-between group. I’m somebody who thinks: ‘I bet if I use this more often, I’d be more productive.’ But, I don’t think to use it: It’s not my habit yet. I don’t rely on it in any sense whatsoever.
And, I love it as a novelty item. But it’s much more than a novelty item.
And the question–when we made this original leap from 3.5 to 4, there was this view that we were now soon going to take off. Then very quickly, shortly after that–I don’t know if it was strategic or just accurate–Sam Altman said: We’ve kind of exhausted some of the range of stuff we can do with bigger datasets, more data for AI to be exposed to, for ChatGPT to be exposed to. Where do you think we’re headed relatively soon? And where do you think we’re headed relatively further down the road from now?
Zvi Mowshowitz: Yeah, I think that’s an important distinction to draw and also to keep in mind that what ‘soon’ means is constantly changing.
If you had told me five years ago about the pace of developments in the last six months or so, where, like, every week I have this giant array of things to handle just in terms of practical things you can use today, even if you exclude all the speculations about what might happen years from now, it just would’ve blown my mind. And, it’s a really scary pace of development.
But what’s going on, is that as the promise of transformers and the uses of stacking more layers of transformers–which is the method of implementing AI and doing calculations within artificial intelligence–has caused them to spend orders and orders of magnitude more money and gather orders of magnitude more data, and use orders of magnitude more to compute, and more hardware and more electricity and so on, to do all of these problems, they’re starting to hit walls. Right?
So, we’ve had Moore’s Law operating for many years. We’ve had hardware getting more capable and manufacturing more of it. And, what happened was, we weren’t using anything like the maximum amount of capacity that we physically had available–that we knew how to build and we knew how to use and that was available to purchase.
And now, with GPT-4, we’re starting to get close to the limits of what we know how to do, such that we hit to a 4.5 style level, it’s plausible to say that you’re going to have to get more creative. You can’t simply throw an extra zero on your budget, assemble 10 times as much stuff, and get the next performance jump just on its own, because you’re starting to run into issues where everything is more duplicated than it used to be. And, in order to get that next order of magnitude of jump in effective compute, you need to be more creative than that, or you have to wait for our technologies to improve some more.
So, I do think that, like, we’re not going to see that much more time of the same level of underlying jumps in capabilities as rapidly as we saw from 2 to 3 to 4, where we saw orders-of-magnitude jumps that were not like the progress we make in hardware–that were vastly faster than the progress we make in hardware.
But, over time, we will still make progress on the hardware. And, we’re seeing jumps in algorithmic progress, especially often coming from open source models that are starting to figure out how to imitate the results that we did get, from GPT-4 and similar models, more and more effectively using less and less compute and using more and more tricks.
And, we’re only just now beginning to figure out what we can do with GPT-4. Right?
So, like, we have this amazing nuke[?] idea: We have a companion, we have an assistant, we have a tremendous knowledge base, we have new interface for using computers. We have a new way of structuring information, we have a new way of coding, we have so many other things.
And, we’ve only had this thing around for a few months. And, even the people who are just focusing on how to use it for productivity, who are just building apps on top of it, just haven’t had the human time necessary to unpack what it can do and to progress the capabilities you can build on top of what we have. So, I think that even if we don’t see a more advanced model for several years, we’re still going to be very impressed by the pace of what we can do with it.
In particular, I think things like the integration into Microsoft 365 Copilot and into the Google suite of products where the machine starts to look at, ‘Okay, here are your emails and your documents,’–in a way that feels secure and safe for people and which they know how to implement without having to go through a lot of technical details that are harder for people even like me–and say, ‘Okay, given that context, I now know the things you know that you have written down. I know who these people are that you’re talking to. I have all of this context.’ And now I can address what you actually need me to address in this place that’s seamlessly integrated into your life. And, this becomes a giant boost to the effective capabilities of what you can do. Plugins are an area where we’re just exploring–like, what can you attach?
And then, the idea of: If every website that starts building up–okay, I now have a chat interface with an LM [Language Model] that’s trained particularly for the questions that are going to be asked on my website to help people with my products to help me get the most out of this thing and to help me have the best customer experience. We’re just starting to get into those things. We’re just starting to get into applications for AR [augmented reality] and VR [virtual reality]. We’re just starting to get into the ideas of: just what do people want from this technology?
And, we’re also seeing penetration. Like, the majority of people still haven’t even tried this, as you pointed out.
And, we’re going to see what those less technical people, what those less savvy people actually can benefit from. Because in many ways, they’re the people who most need a more human, less technical way of interacting with these systems. And in some ways they can benefit the most. So, just getting started basically.
Russ Roberts: So, AR and VR are augmented reality and virtual reality.
10:01
Russ Roberts: When Google Search came along, it was really exciting. I’ve used the example a few times of my grandfather who remembered a phrase, ‘The strong man must go.’ He knew it was from a poem, he couldn’t figure out, couldn’t remember. And then, one day, years after it been bothering him, he yelled out in a crowded restaurant, ‘It’s Browning! it’s Robert Browning.’
And, poor guy: Google finds that in a fraction of a second. And that’s really–it’s a wonderful thing on so many dimensions. Google Search is, quote, “smarter than I am,” in the very narrow sense, but not trivial, that it knows more than I do. By an unimaginable amount, obviously.
So, ChatGPT understandably is only a particular generation of artificial intelligence. It, quote, “knows more than I do.” It can do many things that I can do: write poetry, write a memo, code quicker than I can, sometimes better than I can. And, in some dimension it’s smarter than I am–in a similar way to Google Search, but a more interesting way, I would say. And therefore it’s much more productive potentially in making my life better. Google Search helps me find things I can’t find. This is going to do many things beyond that.
But, in what sense would you say the current generation of models–as they improve and we get more plugins and we get more websites that are optimized for having them built in–in what sense is it going to be smart? And, I ask that question to head us, of course, to the question of sentience.
Now, we can talk all we want about Google being smart, or Siri being smart on my iPhone. It’s not smart. It just has access to more stuff than I can access. And, my hard drive is much smaller. Is ChatGPT really different or is it kind of the same thing but more so?
Zvi Mowshowitz: I think it’s somewhere in the middle. I think that when you see someone say, ‘I just had an IQ [Intelligence Quotient] test of 155.’ That just shows you the IQ test is not measuring what you thought you were measuring, when you go out of distribution and you see a very different thing that’s being tested.
Similar to how you’ve noted–you know, Bryan Caplan gave an examination in economics. Some of the questions were, ‘What did Paul Kirkman say?’ And of course, you just has the answer memorized. So, it just regurgitates. It doesn’t mean that you’re smart. It doesn’t mean that you understand economics.
But other questions, it shows that it actually has some amount of understanding.
And, the AI is going to have a natural–basically, I think of it this way. You have this thing that I like to think of as being smart, being intelligent, ability to think and apply logic and reason and figure unique things out. And, I think of that as distinct from certain other aspects of the system like memory and what knowledge you have and processing speed.
And so, there are certain abilities that the system just doesn’t have. And, no matter how much data you fed into it would not be able to do these things unless it simply had so many close facsimiles in its training data that it was just doing so in a kind of imitative way–that wasn’t the same thing as doing it the way that a person who actually understood this thing would do it. And, often people actually are in fact in this imitative style-way themselves.
You can make it in some sense smarter by giving what’s called prompt engineering. So, what you can do is you can ask it in a way that makes it think that it is trying to imitate a smarter person–that it is trying to act in a smarter way, that it’s dealing with a smarter interaction–and to frame the questions in the right way and guide it. And, it will give you much smarter answers to that.
And, that’s one area where I feel like not only have we generally not scratched the surface on this, but that I’m definitely under-investing in this. And, almost everyone who uses the system is sort of giving up too early. When the system just doesn’t give it what you wanted it to give you thought it maybe had the ability to do. And then, you just don’t try. And then, it ends up, like, you get disappointed and you move on and then you don’t realize that you could have put in more work.
The same way with a human. If you ask stupid questions, or you frame it in a way that makes them think you’re stupid or that you don’t want a smart answer, they’re going to give you a stupid answer. Right? And, you have to ask the right questions in an interview if you want to get thoughtful responses. And, it’s the exact same thing.
So, I think that the current version is not so smart, but that it’s not zero smart and that we will see them get smarter as we see them expand over time.
14:38
Russ Roberts: So, smart’s complicated. And, I feel like I should tell my listeners, over the last few weeks I’ve thought to myself, ‘Well, this is the last episode we’ll do on AI for a while.’ And, I’ve been wrong. I find them–they still are very interesting to me, and as long as I learn something and I hope you learn something, we’ll continue to do them because I believe it’s the most exciting technology that’s out there that’s come along in a long, long time. So, I think it’s quite important that we understand it.
But, one of the topics I haven’t spent much time on with my guests is this question of intelligence.
So, we gave an example earlier of intelligence having a big memory. It helps. Having a big memory, whether you’re human or a search engine, really helps–or ChatGPT. Having an accurate memory really helps. ChatGPT is famous now in its early days for making things up.
But, it’s really the next step that we would call creative, synthesizing–applications that didn’t immediately come to mind, that weren’t in the prompts–those are the things that are both exhilarating and potentially scary. And, you think they’re coming?
Russ Roberts: Or that they’re already here?–
Zvi Mowshowitz: They’ve given GPT-4 various tests of curiosity. And, sometimes the results come back, ‘Oh, GPT-4 is actually more creative than the average human,’ because the type of creativity they were measuring wasn’t the type of creativity that you’re thinking about. It’s this sort of more narrow, like, ‘There’s a thousand household uses for this piece of string. How many of them can you name?’ And, GPT-4 does vastly better than the average human at being creative in this strange sense.
That’s not the thing that we care about. That’s not the thing that we want. And, I think that a lot of what we think of as human creativity is just someone else sort of has different training data and different connections in their brains and thinks about different things; and then output is something that to them is not necessarily especially creative in that way, but that seems creative in that way to you. And because they’ve been exploring a different area of the space. And, I think with better prompt engineering, you can get what seemed like much more creative answers out of the system than you would normally get, the same way you can do so with a person.
But, I think that creativity in that sense, it’s definitely a relative weakness of the system. If you almost by definition say, ‘Okay, this is system that’s training on this data’, find things that are maximally different from that data and ask it to produce good quality things that are maximally different from that thing. So, it’s going to lag behind other capabilities if we continue to use this particular architecture and set of algorithms to train the systems, which we might continue to do so for a while or we might not.
But, by any definition of creativity they put together, there’s not zero creativity in what ChatGPT does. It’s just not as good as its other aspect. And, I think we will see it improve over time.
17:33
Russ Roberts: Well, let’s take a couple of examples. I have an upcoming interview with Adam Mastroianni about how we learn, and why is it that when I tell you something, you don’t really absorb it. You’re younger than I am, Zvi, and I say, ‘Look, Zvi, I’m 68, I’ve lived a long time. Here’s an insight that’s really valuable to you. I wish I’d known it when I was your age.’ And, you listen, and you hear it; it goes in one ear out the other, very rarely changes your life. And, even if I care deeply about you, as I do about my own children, for example, they’re either not interested because they’re my children–that’s a tricky relationship there–but you don’t have any of that baggage that my kids have. You’re just a thoughtful, curious person; and I have wisdom for you. But strangely enough you don’t always get it or maybe rarely get it.
And so, Adam wrote a very thoughtful essay–that’s what I’m going to an interview about–about why that is. Now I’ve thought about this problem a lot. And, in theory–I’m not expert on it–but I’ve thought about it. It intrigues me. And, when I read his essay, I thought, ‘Wow. Oh, that’s cool. I’ve learned something.’
Similarly, you wrote an article that we’re going to get to in a little bit about why certain people are unafraid of ChatGPT. And, you created a metaphor: it’s called The Dial of Progress. When we get to it, listeners will understand why it’s a metaphor; and whether it’s interesting to them or not, I don’t know. But, I find it extremely interesting. It’s the kind of thing a human comes up with–the kind of human I like to hang around with–where you hear that idea and you go, ‘Wow, I haven’t thought about that. That’s intriguing.’
And, it causes other connections in your brain, as we’ll see, and you connect it to other things that a little bit about, not as much as ChatGPT knows. But, I don’t know if ChatGPT could come up with those kind of metaphors yet. Do you think it could? To change my way of seeing the world? Not: coming up with a bunch of stuff I haven’t encountered. Sure, it’s better than me, any human maybe, in that kind of area.
But, this kind of area is what I think of as creativity. There’s other kinds of creativity–artistic, poetic, musical, or it’s visual–but this idea of, ‘Here’s a thought no one’s ever written about it.’ No one’s ever written about the Dial of Progress. You’re the first person. And, I found it interesting. That’s why we’re talking. Could ChatGPT do that? [More to come, 19:58]