Inspired by an exchange with Jeremy Gillen: I think there might be a bias on here to assume that if humanity goes extinct as the result of something an AI does, it will specifically be something the AI did on purpose, with the intent of causing humanity's extinction or the unrecoverable collapse of industrial civilization, or at least with the knowledge that that was a likely outcome. But it seems to me that one obvious way we "accidentally" end up in such a situation is 1. AI systems become capable of training successor systems which are more capable of doing particular dangerous actions than they are, and better at training later successors than they are 2. The successors are not enough better at risk assessment than their ancestors to outweigh the increase in ability to accidentally cause harm 3. Iterate I notice that, if you look at the history of near misses with civilization-ending events, they were not ones where someone tried and failed to end civilization. People have tried and failed, but they're not thr ones that got close. Anyway, on that note, happy day-of-Solstice everyone :)

In Jan 2023[1] I ran through the bio anchors notebook[2] which lets you set a wide variety of variables that go into the model, and changed a bunch of things to what seemed like first pass reasonable conclusions (e.g. setting the "evolution anchor" aka it will take as much compute to train AGI as was used across all brains in all of evolutionary time to zero, along with dozens of other minor parameters). It seemed like almost all of the ones which seemed off were off in the direction of making timelines look longer. When everything settled, I got: But, I noted that I don't remember exactly what the timelines i had in mind were, but probably like 1-3 years years sooner mean. I think bio anchors is a kinda interesting and at least vaguely informative framework[3], and mostly think the wildly long timelines look like they were the result of picking many free variables in individually mildly biased ways in a complex model? 1. ^ Discord message link, if you're on Rob Miles's discord. 2. ^ https://docs.google.com/spreadsheets/d/1gbcJjSN1_E7UTSngIqcvueGDAULjjOajzzft6wX5vBA/edit#gid=505210495 if anyone wants to see how I got to that, though most of the work was going through https://colab.research.google.com/drive/1YRf0AA6x57rk3xwcMCE1SfkvWZ11YbJ_?authuser=1#scrollTo=5qd4DJ2y-X60 3. ^ as a good rationalist, i must earn the 'disagree with yudkowsky on at least one thing' badge and this is how i'm earning it, damnit. my old one of 'emergence is a real concept with any use at all but people often misuse it as a curiosity stopper' has expired as he hasn't emphasised that one since like 2007.

Overconfidence from early transformative AIs is a neglected, tractable, and existential problem.  If early transformative AIs are overconfident, then they might build ASI/other dangerous technology or come up with new institutions that seem safe/good, but ends up being disastrous. This problem seems fairly neglected and not addressed by many existing agendas (i.e., the AI doesn't need to be intent-misaligned to be overconfident).[1] Overconfidence also feels like a very "natural" trait for the AI to end up having relative to the pre-training prior, compared to something like a fully deceptive schemer. My current favorite method to address overconfidence is training truth-seeking/scientist AIs. I think using forecasting as a benchmark seems reasonable (see e.g., FRI's work here), but I don't think we'll have enough data to really train against it. Also I'm worried that "being good forecasters" doesn't generalize to "being well calibrated about your own work." On some level this should not be too hard because pretraining should already teach the model to be well calibrated on a per-token level (see e.g., this SPAR poster). We'll just have to elicit this more generally.  (I hope to flush this point out more in a full post sometime, but it felt concrete enough to worth quickly posting now. I am fairly confident in the core claims here.) 1. ^ For example, you might not necessarily detect overconfidence in these AIs even with strong interpretability because the AI doesn't "know" that it's overconfident. I also don't think there are obvious low/high stakes control methods that can be applied here. 

Largest present advantage of claude code over codex is that I can actually tell what's going on

GPT = Generative Pre-Training? Everyone thinks GPT stands for "Generative Pre-trained Transformer". (For example, Wikipedia.) Does it really? The earliest mention of "GPT" is in the GPT-2 paper, which refers to "the OpenAI GPT model" and cites the GPT-1 paper. That paper does not contain the phrase "generative pre-trained transformer". But it does contain the phrase "generative pre-training", in the title and in the body, italicized.

been trying to decide if janus's early updates on LLM personhood were a very surprising successful prediction far, far in advance of public availability of the evidence, or a coincidence, or some third category the simulators essay in 2022 is startling for its prescience and somewhat orthogonal to the personhood claims. i'm starting to strongly feel that... i mean... that's 2 separate "epistemically miraculous" events from the same person i think we're underexamining LLM personhood claims because the moral implications might be very very big. but this is the community that took shrimp welfare seriously... should we not also take ai welfare seriously?

10+ years ago, I expected that self-driving trucks would be common on US highways by 2025, and self-driving would be having a large effect on the employment of long-haul truckers. In reality, self-driving trucks are still in testing on a limited set of highways and driving conditions. The industry still wants to hire more human long-haul truckers, and is officially expected to keep doing so for some time. I expected that long-distance trucking would have overtaken passenger cars as the "face" of self-driving vehicles; the thing that people argue about when they argue whether self-driving vehicles are safe enough, good or bad for society, etc. This has not happened. When people argue about self-driving vehicles, they argue about whether they want Waymo cars in their city. I expected that the trucking industry would shed a lot of workers, replacing them with self-driving trucks that don't need sleep, breaks, or drug testing. I expected that this would be an vivid early example of mass job loss to AI; and in turn that this would motivate more political interest in UBI. This, too, has not happened. (I certainly did not expect that the trucking industry in 2025 would be much more disrupted by anti-immigrant politics than by self-driving technology.)