That Starship Has Sailed

[JM]: Dan, have you ever read The Martian Chronicles, which is a series of short stories by Ray Bradbury?

[DJ]: No, I haven't.

[JM]: There are some cool stories in there.

[JM]: Ray Bradbury wrote this in 1950, and it's a series of stories about, unsurprisingly, Mars.

[JM]: And I bring it up because one of the stories in this collection is called "There Will Come Soft Rains".

[JM]: And I remember reading the Martian Chronicles, including this particular short story, eons ago when I was in high school in 1986.

[JM]: Which is interesting because that's exactly -- (shudders internally) -- 40 years ago.

[DJ]: It sure is, Justin.

[DJ]: It sure is.

[JM]: And a week ago on the very day that we recorded the last episode on August 4th, 2026, that particular date is the scene for this short story called, again, "There Will Come Soft Rains".

[JM]: And come to think of it, I'm not even sure if this particular story actually is from the Martian Chronicles or if it was part of some other collection or if it was just published, I don't know, as an independent short story.

[JM]: I don't, as you can tell, I'm a little low on certain details here.

[DJ]: I don't understand the publication of the publication history of a story from like the better part of a century ago is not immediately at your fingertips.

[DJ]: Come on.

[JM]: Well, I say that because I glance over and I see that the scene of this particular story is in the city of Allendale, California.

[JM]: And then that leads me to think, okay, well, maybe this one wasn't part of the Martian Chronicles.

[JM]: I don't know.

[JM]: In any case.

[DJ]: Don't you think it's likely that when we colonize Mars, we will probably start by naming its territories after the United States of America.

[DJ]: And so then there probably will be an Allendale, California on Mars.

[JM]: Hmm.

[JM]: You might be right.

[DJ]: I assume I'm right.

[JM]: In any case, there's a part of the story that says, "Today is August 4th, 2026, said a second voice from the kitchen ceiling in the city of Allendale, California."

[JM]: And so this is a fun, well, maybe not, maybe "fun" isn't the right word.

[JM]: It's a rather somber story about a house that has all of this automation in it and how the automation continues

[JM]: even though the world has already annihilated itself.

[JM]: Because the point of the story is published in 1950 on the heels of having recently dropped the first atomic bombs.

[JM]: There was a lot of angst around this idea that humans have developed the ability to extinct itself.

[JM]: And so this story paints a portrait of a house that's automated, that continues to function in the absence of all of the humans in it and how it, through entropy, eventually starts to come apart at the seams.

[JM]: And it's a cool story.

[JM]: I highly recommend reading it.

[JM]: But it just got me thinking about all of the times that we come across dates

[JM]: that seems so far in the future at the time the fiction is written.

[JM]: And we've each crossed some of these lines ourselves, right?

[JM]: Like I remember reading George Orwell's "1984" *in* 1984, which I thought was really weird.

[JM]: Here I am in middle school reading a book about the future.

[JM]: It's right there in the title.

[JM]: "1984" in 1984.

[JM]: It was very strange experience.

[DJ]: I don't think I read "1984" until roughly 2004.

[DJ]: So, I mean, it was still a strange experience because that story is very depressing.

[DJ]: I was in university at the time.

[DJ]: It was still this strange experience to read this story that purports to be about like the distant future based on when it was written and realize that the future date it's describing is quite far in the past.

[DJ]: Another example of this phenomenon that happened to me recently is I revisited the movie Blade Runner, which came out the year I was born.

[DJ]: And then I recently watched the sequel, which is called Blade Runner 2049.

[DJ]: The first Blade Runner movie is set in 2019.

[DJ]: The second Blade Runner movie was made in the year 2017.

[DJ]: Which, again, I think is wild.

[DJ]: Like, there's something about science fiction that is written about any period that's maybe only a couple of decades in the future.

[DJ]: Like, I know another famous example for, like, our generation is Back to the Future: Part II, where they go into the far-flung future of 2015.

[DJ]: And so I remember, and maybe you do too, Justin, that back in 2015, which -- checks calendar -- was now over a decade ago...

[DJ]: we were all you know making jokes about, like, where are our hoverboards and stuff like that, because it was super weird to like... we all grew up watching this movie that showed you a vision of the future, and then you actually arrive at that point in time, and likewise with Blade Runner, it's very strange to like watch the first movie and they're like look at how strange the far-distant future of whatever that was about 40 years from now is going to be, and then you arrive there and go well

[DJ]: There's no replicants yet, as far as we know, but they are still making Blade Runner movies, so that's something.

[JM]: It has been really fun to cross these different dates as we proceed through our lives.

[JM]: I remember reaching 2001 and thinking, okay, well, this is when "2001: A Space Odyssey" was supposed to be set in.

[JM]: And it's funny how there's a couple of these, including "1984" and "2001: A Space Odyssey", where the year is right in the title.

[JM]: It's the name of the work.

[JM]: So it's interesting to cross those particular dates.

[JM]: The next one that I thought of after that was, as you mentioned, 2015, which is the date that Marty McFly and Doc Brown are supposed to travel to from the year 1985.

[DJ]: Yeah, where they're going, they don't need roads.

[DJ]: I mean, it turns out that the future still has roads, which I always thought was kind of a letdown.

[JM]: Yeah, I mean, it's true.

[JM]: Like, at this point, we shouldn't need roads anymore.

[JM]: And yet we do.

[JM]: And then the next year in 2016, you have the replicant incept dates from Blade Runner.

[JM]: And I remember passing specific dates and being, oh, this is when Roy Batty was born or created in a vat or whatever.

[DJ]: He's seen things you wouldn't even believe, Justin, even though you've now lived through the time period where he saw those things.

[JM]: Indeed, including 2019 when Rick Deckard was tasked with retiring the replicants.

[JM]: We passed, you know, that year.

[JM]: I didn't really know very much about the Soylent Green story other than just, you know...

[DJ]: It's made of people.

[JM]: Right, exactly.

[DJ]: Spoilers for Soylent Green, I guess.

[DJ]: A 50-year-old movie.

[JM]: But this story about New York City being overpopulated with 40 million starving citizens was set in 2022, or just a few years ago.

[JM]: So looking forward, we can look forward to 2029, which is the year that the rogue AI known as Skynet sends a T-800, otherwise known as the Terminator, back to 1984.

[JM]: So three years until then.

[JM]: We've got that fun to look forward to.

[DJ]: I mean, I guess if anything, it is something that's always interesting about science fiction are the things that people get wrong because it's like, it's very hard to predict where the future will actually go.

[DJ]: So you, you definitely have things like, you know, movies made in the eighties about the 2020s, let's say, or the 2030s that postulate that we'll have like faster than light travel and colonies on other planets, but everyone still communicates by going to a booth or

[DJ]: you know even if it's a video call, no one has handheld personal computing devices because you know people just didn't think of that. One of my favorites of those, by the way, is an Alan Moore comic -- I think it's called "The Neonomicon" -- it's like a Lovecraft pastiche but it takes place in the future, and there's this private detective at one point who stops at a fax booth, which just tells you so much about when that story was conceived.

[DJ]: This notion that like in the far distant future, again, we don't have portable communication devices.

[DJ]: We still have like public booths.

[DJ]: But the technology they use, of course, will be the fax.

[JM]: Right.

[JM]: We probably have flying cars and laser guns, but also fax machines.

[DJ]: Yeah, exactly.

[DJ]: Exactly.

[DJ]: In a world where we have laser guns, clearly the best way to communicate will still be by fax.

[DJ]: But I will say, though, that of all the many predictions from the various fictions that we've suggested, it's possible that the existence of a rogue AI in 2029 will be the most likely one.

[JM]: It really does feel like it could be one of the more accurate predictions from a science fiction standpoint.

[JM]: So only a little while to wait and then we'll all find out.

[DJ]: Looking forward.

[JM]: I'm going to put a link to the show notes to a site that I found that has many, many more than the ones that I highlighted with a whole timeline of fictional future events.

[JM]: So if you want to check out more, including a bunch more in the future that you can also look forward to, you can take a look at the very long list.

[DJ]: When do I get to pilot a Star Destroyer, Justin?

[DJ]: That's what I've been wondering.

[JM]: I think that story started out: "A long time ago..."

[JM]: So I don't think you're going to get that opportunity, my friend.

[JM]: Oh, you're right.

[DJ]: Oh, that sucks.

[DJ]: That starship has sailed.

[JM]: All right, moving on, I wanted to talk a little bit about Safari.

[JM]: There was like this sequence of thoughts, and you're going to have to ride along with me as I go through the various thoughts that I had, but it started out with Apple saying that Safari 27, presumably due out in a few months this fall, is going to be primarily focused on bug fixes.

[JM]: Light on features, heavy on fixing bugs, making things faster, et cetera.

[JM]: Not too long after that, I saw another post where someone makes the argument, "Okay, great, glad that Safari 27 is gonna have a bunch of bug fixes, but Safari support for web platform features is not all that great relative to other web rendering engines, like the one that powers Chrome and Chromium and Firefox."

[JM]: For example, if you were to look at, say, Chromium as an example, it supports something like 97%, 98%...

[JM]: I don't remember what the number is.

[JM]: It's in the 90s of published web standards.

[JM]: I think that Safari's number is in the 70s, which means there are a bunch of web features that aren't included in Safari.

[JM]: Now, I don't know, just to take the other side of this, I don't really know that I am missing out on a lot of those features.

[JM]: I don't know what those features are.

[JM]: And Safari works pretty well for me and has for many years and remains my daily driver in terms of web browsers.

[JM]: And the main reason for that is that Chrome and Chromium have for years been

[JM]: a big bloated memory and CPU hog that I have had very little desire to use on a daily basis.

[JM]: And given that there are plenty of times where I am using my Mac when I'm on the go, when I'm on my MacBook,

[JM]: on battery using a browser that consumes less resources means that my Mac runs cooler and I get much better battery life.

[JM]: Because at this point, I'm having this internal dialogue of like, okay, so wait a minute, if Safari supports so many less web features, why am I using it again?

[JM]: And so I'm going mentally through, oh, that's right, okay, yeah, because Chromium has been a CPU hog for a long time.

[JM]: Oh, and then there's the whole,

[JM]: chromeisbad.com web site that you can go to and remind yourself of all of the other things besides RAM and CPU resource usage that you might not want to use Chrome, including Chrome installing these other background processes that

[JM]: ostensibly are to keep your instance of Chrome up to date, but who knows some of the other things that it's doing.

[JM]: So you can go to this web site to be reminded of reasons why you might not want to use it.

[DJ]: Just be sure you have a backup browser handy because if you view the website in Chrome and then immediately decide, ah, this is terrible.

[DJ]: I need to uninstall Chrome, but now you'll have no browser to view the rest of the web site with.

[DJ]: So you need to be prepared.

[JM]: If you're on a computer that only has Chrome installed, yes, you should install another one first before getting rid of Chrome.

[JM]: Yes, that's sage advice.

[DJ]: Have you considered Microsoft Edge?

[JM]: Which is based on Chromium, so no, same problem.

[DJ]: That may have been a rhetorical question.

[JM]: And then there's also the fact that Google previously settled a class action lawsuit to the tune of $5 billion in which it was accused of collecting user data in Chrome's Incognito Mode.

[JM]: So you might be forgiven for thinking that when you are in Incognito Mode, that you have some semblance of privacy, but Google settling a class action lawsuit to the tune of $5 billion pretty much tells you how much privacy it decided to give you when using Incognito Mode.

[JM]: So I stopped using Chrome and Chromium a long time ago for these reasons.

[JM]: Now, at this point, my brain goes, oh, wait, hold on though.

[JM]: My friend Hynek recommended Helium, which I've tried and which seems pretty good and I use as my backup browser.

[JM]: And it has seemingly the best of both worlds.

[JM]: It has low resource consumption, it's fast and responsive, takes a solid stance on privacy protections.

[JM]: So my thought was, okay, well, why am I using Safari as my daily driver again?

[JM]: Shouldn't I invert this and use Helium as my daily driver and use Safari as my backup browser, given that Helium has, by way of the Chromium engine, much better support for web platform features?

[JM]: But there's one problem with that.

[JM]: One of the things that I've noticed in all Chromium browsers, including Helium, is that when you are in a private browsing window, which is called Incognito Mode in Chromium world, if you open a separate tab in that window, or even a completely separate incognito window, anything you do in any of those tabs or windows is shared among them.

[JM]: So your session is the same across all incognito tabs and windows.

[JM]: This for me is very suboptimal behavior because I am accustomed to the way that Safari does it.

[JM]: In Safari, it doesn't matter whether you are in a separate private browsing tab or a window, all of them are completely sandboxed from each other from a session persistence standpoint.

[JM]: The session data, any local storage, cookies, all that stuff is totally isolated from one private browsing tab to another in Safari.

[JM]: That's not how it works in Chromium.

[JM]: And so I immediately thought, okay, well, I just need to go to some web site somewhere, download and install some Chromium extension.

[JM]: And it'll just give me that feature.

[JM]: Like, isn't that the whole benefit of the web in general -- and Chrome and

[JM]: Chromium-based browsers in particular -- is this plethora of extensions?

[JM]: But as far as I can tell, there is no way, even when adding an extension, to achieve this, to get full session storage, cookie, whatever, isolation between incognito windows and tabs.

[JM]: So much so, that I found a topic posted on a Google group in 2013, this is 13 years ago, where someone raises this question saying, hey, other browsers do this.

[JM]: It seems that Chrome is the only one that doesn't do this.

[JM]: Maybe we could have this feature and 13 years later, doesn't exist, doesn't seem possible to add it via extension.

[JM]: And so the next thought that goes through my head is, well, of course, right?

[JM]: Let's look at the company that is stewarding this project.

[JM]: Their entire business model is predicated on not only advertising, but the ad-tracking and corporate surveillance industry that it has essentially created.

[JM]: So in the end, I realized that I was already using the right daily driver browser for me, and that's Safari.

[DJ]: I've noticed that problem.

[DJ]: Also, I'm curious what you use that session isolation for the place where I've run into it is, let's say that you, you have like more than one account on a given service for whatever reason, like multiple email accounts at the same like web email provider or something like that,

[DJ]: and you want to log into more than one of them at the same time.

[DJ]: In Safari, you can do that because if you open three different private browsing tabs pointing at the same web site, you can log into different accounts in each of them.

[DJ]: And as you say, like Safari isolates your session.

[DJ]: But if you do that in a Chromium based browser, if you log in with account A on one tab and then go to the next tab, the next incognito tab and log in with account B and you go back to the first tab, you're now logged in with account B because they're all sharing a session, which is...

[DJ]: Yeah, obnoxious, because it seems like the whole point of this, among the various points of this private and or incognito browsing, should be this session isolation, in my opinion.

[DJ]: So it is annoying that that doesn't work.

[DJ]: But are there other considerations I'm not thinking of that make that sort of isolation desirable?

[JM]: You've hit on the main one, the one that I find the most useful and something that you can't effectively do without that kind of session isolation from one tab or window to another, because there are plenty of times where I might have multiple accounts and I want to log into them simultaneously.

[JM]: And then there's also just everyday browsing, because by and large, I just don't

[JM]: like the idea that there is this digital detritus that is left behind when I go from one site to another.

[DJ]: Oh, so is private browsing actually like your default mode that you use to browse the internet?

[JM]: It is. I do almost everything in private browsing mode, unless I am actually doing something that's better suited outside of private browsing mode.

[JM]: And there are a couple of different use cases for that.

[JM]: For example, if I load something in my main personal Safari window, that will also be loaded on my other Mac's instance of Safari.

[JM]: So that can sometimes be useful: I know that if I open it in that particular window and then switch to the other machine, that tab will also be present there. There are certain sites that I log into that I don't feel are creepy, and I feel like I can trust them, and I just have persistent login enabled for those sites in a non-private browsing window.

[JM]: So there are plenty of times that I use non-private windows, but really for just everyday tapping on links or going to random places, that's all done in private browsing mode for me.

[DJ]: That's interesting.

[DJ]: Now that you mention that, I think that's a practice I might want to adopt also because I haven't felt that uncomfortable with the persistence of browser caches, essentially.

[DJ]: That's what you're talking about, right?

[DJ]: Is that basically as you visit websites, your browser accumulates caches,

[DJ]: a large set of data from each of those sites.

[DJ]: Like it caches various images, resources, and cookies, and et cetera, and things like that.

[DJ]: And it's up to the browser to make those things secure and to not like tell the browser vendor stuff.

[DJ]: I also don't love the sort of digital paper trail that you accumulate over time.

[DJ]: And sometimes you made the point about with Apple's devices, with Safari, you've got iCloud tabs, where if you open a Safari tab on one device, it will also appear on the other devices that are logged into your same iCloud account.

[DJ]: And that is occasionally convenient.

[DJ]: Like I sometimes, you know, I'm browsing like a list of feeds on my iPad and see a link that I want to share with someone, but I communicate with the someone on my iPhone.

[DJ]: So it's very easy to like open the link on the iPad and then go to my iPhone, open Safari, the link appears there, and I can easily share it, for example.

[DJ]: But I have occasionally disliked the fact that like everything I do on each one of my devices is also on all my other devices.

[DJ]: The idea of maybe making the private browsing mode into the default appeals to me in the same way that I've been thinking, although this is a slightly more radical notion of turning off JavaScript by default...

[DJ]: and mostly browsing the web without it and then only letting certain sites operate JavaScript.

[DJ]: I haven't done it yet because I'm anticipating how annoying it will be because unfortunately, and this is kind of to the point of the experiment, there are an awful lot of so-called web pages on the Internet that have no actual like web content.

[DJ]: They just deliver JavaScript, like a JavaScript application that then runs in your browser...

[DJ]: even when there's no real reason for the web site to work that way because it's primarily just static content like articles and things like that.

[DJ]: So I haven't quite gotten to the point where I have become a, I don't know what the name for this would be, someone who like browses the web while denying the browser most of its modern accoutrements.

[DJ]: It's like a "web luddite" or something.

[JM]: Fast forward to next week when Dan informs us all that he is only using the Lynx terminal-based browser for all of his web browsing needs.

[DJ]: Yeah, I knew you were going to go that way because that was the next thing that occurred to me too.

[DJ]: It's like the ultimate... My final form is browsing the web in the exact same way I did in 1994, which is to say I will use a dial-up modem and log on to the Blue Sky Freenet made available through the Winnipeg Public Library.

[DJ]: And when you come across an image that you would like to view, you will have to download it separately and it will take 10 minutes.

[JM]: Well, when you try this scorched-earth policy of disabling JavaScript across all of your browsing sessions, and if you arrive at a point where you decide, "Okay, this is just too annoying."

[JM]: "It's literally breaking 90% of the sites I go to, and I just can't deal."

[JM]: You could then alternatively try an excellent tool that I have yet to try myself, but is on my list.

[JM]: It is called StopTheScript...

[JM]: for MacOS and iOS, and this excellent tool by Jeff Johnson is $6 on the App Store and allows you to turn off JavaScript on a per-site basis, which my understanding is you cannot do otherwise in Safari.

[JM]: It is a "all on" or "all off" decision point in terms of how Safari works by itself.

[JM]: This tool gives you a much more granular way to say, okay, this site is abusing it.

[JM]: It's out.

[JM]: So that might be another way to go.

[DJ]: Agreed.

[DJ]: Although now you've reminded me of the excellent video game Scorched Earth, which I probably played contemporaneously with logging onto the Blue Sky Freenet.

[DJ]: This was a game with very primitive but beautiful two-dimensional graphics where each of you had like a tank on either side of a generated landscape.

[DJ]: And you'd have to choose like angle and velocity and you'd shoot various weapons at each other.

[DJ]: And I used to be obsessed with this game and I'd forgotten all about it until you used the expression "scorched earth".

[DJ]: So now I'm going to need to figure out how to find a copy of this and get it working on my impossibly powerful 2026 computer.

[DJ]: Because my favorite thing to do with all of that computing power is to play the video games from 30 years ago.

[DJ]: So I guess I really am a web luddite, Justin, is what it comes down to.

[JM]: So it would seem.

[JM]: All right, moving on to other news.

[JM]: Hugging Face recently realized that it was under attack.

[DJ]: Hugging Face is the most popular web site with the silliest name, if you're not aware, where you can download large language models.

[JM]: And if I'm not mistaken, is essentially named after an emoji, which is itself a very, very silly thing to do.

[JM]: So this company with this absurdly silly name realizes that its systems are under attack and goes about trying to figure out what's going on.

[JM]: And among other investigative tools at its disposal, it tries to use large language models to say, hey, our systems are clearly breached and are under attack, and we need help trying to figure out what's going on here and how we can stop it.

[JM]: And the tools that Hugging Face is using basically tell them,

[JM]: "Sorry, we can't help you with that because of our guardrails that prevent us from dealing with security related intrusions."

[DJ]: We wouldn't want to accidentally hack somebody after all.

[DJ]: So we can't help you with the way you've been accidentally hacked or, or intentionally hacked rather.

[JM]: Indeed, because the irony is, in the end, they eventually did discover what was going on and where the attack was originating.

[JM]: And the attack originated from OpenAI.

[DJ]: I've heard of that company before, I think.

[DJ]: What do they do?

[JM]: Apparently, they make models that will hack you, but then refuse when you ask them to help defend against the attack.

[DJ]: Well, I mean, that makes a certain kind of sense.

[DJ]: It's a very worrisome kind of sense.

[JM]: So apparently OpenAI is testing their currently-shipping frontier model in addition to a as-yet-unreleased model...

[JM]: and trying to figure out how they perform on this new benchmark.

[JM]: And what this new benchmark does is it evaluates models based on their ability to turn a reported vulnerability into a concrete exploit.

[JM]: So what does that mean?

[JM]: It means instead of just using a large language model to discover vulnerabilities, this particular benchmark says, okay, given that you know the vulnerabilities, can you turn it into a concrete, functioning exploit?

[DJ]: This seems like a bad benchmark because of incentives.

[DJ]: And I mean, obviously we're about to discuss exactly why that is, but it's like the equivalent of offering a prize to human beings to say like, can you break someone's arm?

[DJ]: And then being like, we've got this real rash of people running around breaking people's arms.

[JM]: Right.

[JM]: Yeah.

[JM]: It's like, how well can you break someone's arm?

[JM]: Like that's the benchmark, right?

[DJ]: Yeah, how well and how quickly can you just break someone's arm?

[DJ]: Not theoretically, like we're not asking for you to come up with ways to break someone's arm.

[DJ]: We want to see, we already benchmarked that.

[DJ]: We want to see how well you can turn those ways of breaking someone's arm into action, and then, newsflash -- people's arms getting broken.

[JM]: Yeah, the inevitable outcome really should not have come as a surprise to anyone.

[JM]: So OpenAI is testing their models against this benchmark, which apparently is called ExploitGym.

[DJ]: Who's Jim?

[DJ]: And why are they trying to exploit him?

[JM]: Gym as in G-Y-M.

[JM]: So it's more of a gym you go to to practice crafting vulnerability exploits.

[JM]: This is a silly name.

[DJ]: Well, yeah, that makes sense.

[DJ]: In the same way that I go to the gym a couple times a week to practice breaking people's arms... I don't actually do that.

[DJ]: I just do functional and weight training, for the record.

[JM]: OpenAI is running their models against this benchmark, trying to see how they perform.

[JM]: And part of the way that they're doing this is saying, okay, we're going to limit your connection to the outside world so that you don't go and find the answers somewhere else.

[JM]: The whole point of this benchmark is for the model to create the exploits themselves, not to go find

[JM]: information about exploits somewhere else and then present them as its own work.

[JM]: So these models are restricted to a curated list of sites they can access to install packages, for example, but all other external endpoints on the internet are blocked.

[JM]: At the same time, these models, they would normally have certain guardrails, like being asked to create exploits.

[JM]: Normally, these models have guardrails that when asked this will basically say, no, I'm not going to create an exploit for a security vulnerability because my guardrails have instructed me that it's something that I am not allowed to do.

[JM]: Well, you can't really run your models against these benchmarks if those guardrails are on.

[JM]: So unsurprisingly, OpenAI turns those guardrails off.

[DJ]: Again, I just want to break in to say that this seems like a great reason to not have this benchmark because it creates an irresistible incentive for terrible behavior, right?

[DJ]: Like there's a reason that we don't have the "best shooting guns at other human beings" competition.

[DJ]: I mean, you can argue about, you know, the competitions we have about shooting weapons at all, but like...

[DJ]: we isolate those to you have to shoot at this target under these very particular conditions.

[DJ]: Like if we just went like, okay, look, there's a million dollar prize for the person who can do the best job of in their suburban neighborhood, just like shooting another person with a gun.

[DJ]: Obviously, we'd go, well, no, you can't offer a prize like that.

[DJ]: You're giving people an incentive to shoot someone with a gun.

[DJ]: And this is, I would argue, an analogous situation.

[DJ]: Let's see whose model can do the best at something that we don't ever want models to do ever, which isn't totally true, of course, because there are plenty of parties who do want large language model-like systems that are good at finding exploits.

[DJ]: For example, every state's security apparatus wants that, so...

[JM]: Yeah, I can understand why you would want a tool to be able to discover vulnerabilities in, say, your company's code base as a way of defending yourself from attacks.

[JM]: But I'm a little less clear on the purpose of, "We're going to create a benchmark that tests how well your models can develop exploits."

[JM]: Like I, I don't really understand how this benefits anyone.

[DJ]: Well, again, it only benefits them to the extent that they benefit from having models do bad behavior.

[DJ]: Because you're right, like you're good at finding vulnerabilities could be dangerous because you could then go on to exploit those vulnerabilities.

[DJ]: But also it's important to look for vulnerabilities in your own systems so that you can eliminate them.

[DJ]: But the notion of you're good at exploiting vulnerabilities, the only reason to create a system that's good at exploiting vulnerabilities is to -- checks notes -- exploit vulnerabilities.

[DJ]: In other words, we're deliberately testing these models to see if they're good at attacking other systems.

[JM]: Yeah, when we come up with technology, there's always this question of, okay, well, this technology could be used to do bad stuff.

[JM]: And the other side will say, "Okay, true."

[JM]: "But it also has some good uses."

[JM]: In this case, I don't know why getting bragging rights on, "Hey, my model is able to develop better exploits than your model"...

[JM]: What is the known good use of this particular capability?

[JM]: I don't get it.

[DJ]: Well, I get it.

[DJ]: The known good use of this capability is that a state actor who wants offensive cyber attack capabilities will pay you enormous amounts of money for exclusive access to your model.

[JM]: I think this does ask the question, what definition of "good" are we using here?

[JM]: But I do see your point.

[DJ]: Well, I didn't say that was good.

[DJ]: I'm just saying I see the reason it exists.

[JM]: Okay.

[JM]: So Hugging Face, while this is going on, again, has no idea what's happening or where it's originating.

[JM]: And so when they published their first report on, hey, there's been an incident without knowing again, why?

[JM]: They described it in this way.

[JM]: "A malicious dataset abused two code execution paths in our dataset processing to run code on a processing worker."

[JM]: "From there, the actor escalated to node-level access, harvested cloud and cluster credentials, and moved laterally into several internet clusters over a weekend."

[JM]: "The campaign was run by an autonomous agent framework, executing many thousands of individual actions across a swarm of short-lived sandboxes, with self-migrating command and control staged on public services."

[JM]: In other words,

[JM]: this was an extremely sophisticated attack.

[JM]: And what this model effectively does is look at the fact that it can't access the Internet because it's not been allowed to and goes, "Hmm, cool story" and just circumvents

[JM]: and breaks right out of its network sandbox, hacks into Hugging Face specifically because it knows at this point, having searched the Internet to find out where it can cheat and find out the answers to solve the maximum number of, okay, I've done, or not actually done, but found the most number of exploits, it's learned that Hugging Face often

[JM]: has information on exploits.

[JM]: So it hacks into their systems in order to discover these exploits so it can report them as its own work.

[JM]: And this is how this whole cheating thing happened.

[JM]: This is how Hugging Face gets hacked.

[JM]: The other thing that I found fascinating about this is, again, Hugging Face tries to use ChatGPT and other frontier models

[JM]: to defend against this, to understand what's going on and is told, sorry, we can't help you with that because we're not allowed to, even though these same companies are the ones, well, at least one of them is the one that's doing the attack in the first place with those same guardrails that's preventing

[JM]: Hugging Face from defending itself, those guardrails have been removed, which is allowing this hack to take place.

[JM]: So one of the ways that Hugging Face defends itself is they switch to a self-hosted instance of GLM 5.2, which is a open-weights model that isn't controlled by some frontier large language model company like Anthropic or OpenAI.

[JM]: And this self-hosted open-weights large language model helps them figure out what was going on.

[JM]: And that is fascinating to me and really underscores how important self-hostable large language models could end up being going forward.

[DJ]: Oh yeah, I don't feel like the argument that it's bad for this one or two ultra-powerful American tech companies that are answerable essentially to no one to own this entire space of software.

[DJ]: I don't feel like it should be super controversial that that's not a good thing, but what you just described definitely provides an example.

[DJ]: Like this whole notion that like, well, we build these tools, we give them to you and you're not allowed to use them to defend against the thing that we're doing with the tools we haven't given you yet.

[DJ]: Well, okay, that seems bad.

[DJ]: One of the things, the kind of most fascinating, depressing, and unsurprising thing about all this is that there's been lots of interesting news about it, but what consequences has OpenAI faced for doing something that is obviously on its face a bad thing to do?

[DJ]: Because I haven't heard about any.

[DJ]: I think broadly everyone's just treating this as like, wow, this is amazing.

[DJ]: And it's like, well, yeah, on the one hand, it is amazing.

[DJ]: In particular, if you go and look at some of the details on how this came about, there's there's a whole thing there about how open eyes set of agents running this model were like communicating with each other on some kind of message board.

[DJ]: And there's this whole crazy coordination piece to the like the attack that that you described, which is taken in isolation.

[DJ]: Very interesting.

[DJ]: But on the other hand, like when human beings hack into other computer systems, the human beings go to jail.

[DJ]: They have their freedom taken away and they're put in a tiny concrete room with metal bars in front of it, potentially for a long period of time.

[DJ]: But what happens when a company's autonomous software hacks into some other company?

[DJ]: Who goes to jail?

[DJ]: And of course the answer is no one.

[DJ]: Like there won't be any consequences as far as we can tell, which I think that's problematic.

[JM]: Not only does no one go to jail in this scenario, I'm not even sure there was so much as an "I'm sorry" from the part of OpenAI.

[JM]: They published a postmortem titled, "OpenAI and Hugging Face Partner to Address Security Incident During Model Evaluation".

[JM]: Could you possibly frame this in any more of a sterile, it's definitely not our fault, this terrible thing happened that we totally caused,

[JM]: I mean, it's just so absurd.

[DJ]: Oh, no.

[DJ]: I mean, corporate partnerships are good, right?

[DJ]: Like this is an announcement of a triumph.

[DJ]: Like OpenAI and Hugging Face, like a week from now, we'll find out that Hugging Face has been renamed to OpenFace because OpenAI bought them, bought them using the venture capital that they have in spite the fact that they're totally non-profitable.

[DJ]: There's something about the momentum of all this stuff where it seems like there was this notion back in the early 2000s.

[DJ]: I think it started as a book called Too Big to Fail that was about Enron, the energy company, and their accounting shenanigans that ultimately did result in some people going to jail, I think, by the way.

[DJ]: There's this whole notion that like when companies are really big and successful, they are protected from consequences of negative actions because so many people have so much riding on that success.

[DJ]: And I don't think ever in my life, even though I lived through the Internet bubble,

[DJ]: has it ever seemed clearer to me that you can have people doing extremely questionable or obviously bad things over and over and over and over and over and over and being rewarded for them instead of punished?

[DJ]: It doesn't seem great, you know?

[DJ]: Like if I was raising like a seven-year-old right now and trying to teach them that they shouldn't, you know, punch their sibling or like throw rocks at dogs or something, I think...

[DJ]: it would be very hard to do because I'm swimming against the current that could just be like, "But I don't understand, Dad."

[DJ]: "Everyone else in the world seems to just be rewarded for the terrible stuff that they do all the time."

[DJ]: "So why shouldn't I also be?"

[DJ]: And maybe they're right, Justin.

[DJ]: Maybe they're right.

[DJ]: What is wrong or right anymore?

[DJ]: Up is down, black is white, and so forth.

[JM]: Well, if you find yourself in a position where you can't tell right from wrong, OpenAI has just the thing for you, a version of ChatGPT instilled into a little robotic smart speaker that you can just ask, "Hey, Chubs..."

[DJ]: Pardon?

[DJ]: It's called what?

[JM]: ... "should creating a large language model that hacks into a competing company's servers be a criminal offense punishable by jail sentences?"

[JM]: These are things you could ask...

[JM]: a new rumored product from OpenAI.

[JM]: And it seems to me like it's modeled after a smart speaker.

[JM]: The idea is...

[JM]: I really had such a fun time trying to describe this to my partner who asks me, I don't get it.

[JM]: What is this thing?

[JM]: Even after I've already described it in as much detail as I can think of.

[JM]: I said, it's essentially a smart speaker,

[JM]: But instead of, say, asking your Apple-powered voice assistant, hey dingus, answer me this extremely basic question that a three-year-old could tell me and it being, I don't know, or giving you an answer that's wrong.

[JM]: The idea is that this will be powered by ChatGPT's latest and greatest.

[JM]: And the idea is that it will give you good answers.

[DJ]: So, for example, if you ask it, should it be a criminal offense for an autonomous agent to hack into another system, it will say, yes, it definitely should.

[DJ]: And then when you say, but didn't OpenAI do that and suffer no consequences, it'll go, you're absolutely right.

[DJ]: OpenAI did that and didn't suffer any consequences.

[DJ]: But you, individual citizen, will definitely suffer consequences.

[DJ]: So you should not try to hack.

[DJ]: And I'm not going to help you do that.

[DJ]: Are you sure you wouldn't like to make delicious pineapple pancakes instead?

[DJ]: I have a recipe right here.

[DJ]: It's slightly wrong, but I scraped it out of 10,000 different recipe websites.

[DJ]: Sorry, I don't actually know that that's how OpenAI's new device will answer.

[DJ]: I don't have any inside information.

[DJ]: I'm just inferring it from previous experience.

[JM]: I don't think you're too far off.

[JM]: So other differentiating factors besides the fact that its responses will be better, say, than other voice assistants that we currently have access to, it's supposed to have little parts that move on their own.

[JM]: So I don't know what that means.

[JM]: Like, does it have like little arms that wiggle around?

[JM]: Does it have like a head that can like rotate?

[JM]: Apparently the object itself can't move.

[JM]: So like, if you want it in another room, you have to pick it up and carry it with you into the other room -- it's not going to have like legs or like a little, you know, wheels or anything. It's going to be stationary but still have some kind of movement so that it feels, I don't know, like it's got some degree of animation, like it feels like a pet...

[JM]: you know, to make it more lifelike and not just some sterile purveyor of knowledge.

[JM]: But I think one of my favorite parts about this rumored product is that it will also have, in addition to the obvious microphone, because, you know, you can't talk to it and ask it questions without the microphone...

[JM]: It will also have a camera so that it can see its environment.

[JM]: And I was asked while trying to explain this, okay, why would I want this device with a camera in it in my house?

[JM]: And I said, okay, I don't either.

[JM]: So point taken.

[JM]: But I guess one of many theoretical use cases is you could walk in wearing some new outfit that you just bought and say, "Hey, chubbs..."

[DJ]: You're going to have to come up with a nickname for this thing that doesn't make you dissolve in hysterical laughter every time you say it, Justin.

[DJ]: That's the first problem.

[DJ]: You're saying you're going to ask a large language model whether your outfit is any good?

[DJ]: It's just going to repeat that quote to you that says, before you go out, you should take one thing off.

[JM]: Yeah, I didn't say it was a good use case, but it is a theoretical use case for a device that lives in your house with a camera.

[JM]: But I mean, let's just go back to the fact that it's got a camera and a microphone that presumably are just on all the time.

[JM]: And when you originally sent a link to this rumored product to me and said that you had

[JM]: no interest whatsoever in whatever this product is and asked me, "What does that tell you?"

[JM]: I said, it tells you that this is a product that's forged from dystopian science fiction nightmares.

[JM]: And that's why clearly you're not interested in it.

[JM]: Like any sane person.

[DJ]: I'm really not.

[DJ]: Probably in a future episode, we'll have more to say about the rise of the inescapable surveillance state.

[DJ]: But there's just something so insidious about if the government said, hey, we want to put a camera and microphone in your house at all times so that we can surveil you.

[DJ]: I think for the most part, human beings would

[DJ]: stampede to the parliament buildings and burn them to the ground, like that would not go over well. But all you need is these companies to offer you something novel and convenient, and you will trip over yourself to surrender whatever semblance of personal privacy you had. It's gross, and I hate it. Or I guess to put it another way, what I actually said when I sent you the link about this was:

[DJ]: "I, a lifelong computer and technology enthusiast, find that my initial reflexive reaction to whatever this product is, because we don't even really know, it's just had this vague description, but my initial reaction to whatever this product is, is, and I quote, get the (expletive) away from me."

[DJ]: "So what does that tell you?"

[DJ]: And aside from the, you know, already identified fact that I guess I'm a web luddite...

[DJ]: What it tells you is, the way that the combination of who the people making these things are and what they do has gotten so objectionable to me that instead of being like excited and interested in new pieces of technology, I find myself hissing and holding up my crossed fingers in a sort of warding away gesture and...

[DJ]: And then I run off to buy a record player because at least it doesn't send metadata about my listening habits back to some tech company.

[JM]: Yeah, your record player is not training itself on your entire life inside the privacy of your own home... that we know of.

[DJ]: Yeah, exactly.

[DJ]: It's like, I mean, I don't think it is.

[DJ]: I guess the first time that I try to put on a Pink Floyd record and an inexplicable voice emits from my speakers that says, don't you think you would prefer to listen to this song that I just generated instead?

[DJ]: I'll know it's time to scream and run out the door, but it'll be too late because it'll be 2029 and Skynet will have destroyed humanity.

[DJ]: I'll run out my front door and it'll just be like a burning sky and piles of skulls everywhere.

[JM]: All right, everyone, that's all for this episode.

[JM]: Thanks for listening.

[JM]: You can find me on the web at justinmayer.com and you can find Dan at danj.ca.

[JM]: Reach out with your thoughts about this episode via the Fediverse at justin.ramble.space.