Inspired by an exchange with Jeremy Gillen: I think there might be a bias on here to assume that if humanity goes extinct as the result of something an AI does, it will specifically be something the AI did on purpose, with the intent of causing humanity's extinction or the unrecoverable collapse of industrial civilization, or at least with the knowledge that that was a likely outcome. But it seems to me that one obvious way we "accidentally" end up in such a situation is 1. AI systems become capable of training successor systems which are more capable of doing particular dangerous actions than they are, and better at training later successors than they are 2. The successors are not enough better at risk assessment than their ancestors to outweigh the increase in ability to accidentally cause harm 3. Iterate I notice that, if you look at the history of near misses with civilization-ending events, they were not ones where someone tried and failed to end civilization. People have tried and failed, but they're not thr ones that got close. Anyway, on that note, happy day-of-Solstice everyone :)

Largest present advantage of claude code over codex is that I can actually tell what's going on

Overconfidence from early transformative AIs is a neglected, tractable, and existential problem.  If early transformative AIs are overconfident, then they might build ASI/other dangerous technology or come up with new institutions that seem safe/good, but ends up being disastrous. This problem seems fairly neglected and not addressed by many existing agendas (i.e., the AI doesn't need to be intent-misaligned to be overconfident).[1] Overconfidence also feels like a very "natural" trait for the AI to end up having relative to the pre-training prior, compared to something like a fully deceptive schemer. My current favorite method to address overconfidence is training truth-seeking/scientist AIs. I think using forecasting as a benchmark seems reasonable (see e.g., FRI's work here), but I don't think we'll have enough data to really train against it. Also I'm worried that "being good forecasters" doesn't generalize to "being well calibrated about your own work." On some level this should not be too hard because pretraining should already teach the model to be well calibrated on a per-token level (see e.g., this SPAR poster). We'll just have to elicit this more generally.  (I hope to flush this point out more in a full post sometime, but it felt concrete enough to worth quickly posting now. I am fairly confident in the core claims here.) 1. ^ For example, you might not necessarily detect overconfidence in these AIs even with strong interpretability because the AI doesn't "know" that it's overconfident. I also don't think there are obvious low/high stakes control methods that can be applied here. 

GPT = Generative Pre-Training? Everyone thinks GPT stands for "Generative Pre-trained Transformer". (For example, Wikipedia.) Does it really? The earliest mention of "GPT" is in the GPT-2 paper, which refers to "the OpenAI GPT model" and cites the GPT-1 paper. That paper does not contain the phrase "generative pre-trained transformer". But it does contain the phrase "generative pre-training", in the title and in the body, italicized.

Starting a shortform to keep track of tweets that I want to refer back to. You should not think of METR task length as the task length for all software engineering tasks, but "well-defined software tasks that could make it into such a study" (Dec 21, 2025) Unbounded task complexity (i.e., there are actually just a lot of things for AIs to learn) & low sample efficiency in AIs -> long timelines. (Nov 19, 2025 & Dec 21, 2025) (See also my commentary on task complexity when the METR study was first released on Mar 25, 2025) 59 page Neel Nanda MATS paper (Oct 30, 2025) 50 page Neel Nanda MATS application (Mar 3, 2025) Agents that are capable of solving hard problems need not be consequentialist & reflectively stable (Mar 3, 2025) Givewell girl summer (July 14, 2024) My thesis on Fox New's Effects on Social Preferences (June 13, 2023)

The post I originally wrote:    But then I went looking for examples, and it seemed a little more complicated. It was harder to find the sneering comments than I expected. At least some people claim that, for the high-quality knock-off brand I was considering, it's about respect for intellectual property. That definitely could be cover for status-policing but I 🤨 at knock-off legos so I think there's some legitimacy to it.  In this odyssey I also learned that "discount resellers" like Marshalls, and TJ Maxx[2] market themselves as selling overstock from other stores, but in fact most of their stock was always intended for their stores. Name brands will license their branding to the discounters to produce cheaper products with the designer label, so you can't trust their quality. For TJ Maxx and Marshalls you can check this by looking at the last digit in the ID number (1= produced for discounter, 2 = genuine overstock). 1. ^ Normally I'm fine with Amazon fast fashion but they save money on warm things by knitting them out of tissue paper. I suspected this problem was inherent in cheap sweaters but was afraid paying more would just get me more expensive tissue paper. I didn't feel like going through 10 return cycles, so I tried out spending real money on well regarded brands.  It worked out okay, but a week later I found some delightful sweaters for $13 at Ross Dress For Less. One snagged a thread while I was in the store and I'll be shocked if they're wearable next winter, but for now they are quite warm[3]. 2. ^ Ross is more genuine overstock, although does still sell its own merchandise. 3. ^ As long as there's no wind- but that's a property of cable knit, not cheapness.

10+ years ago, I expected that self-driving trucks would be common on US highways by 2025, and self-driving would be having a large effect on the employment of long-haul truckers. In reality, self-driving trucks are still in testing on a limited set of highways and driving conditions. The industry still wants to hire more human long-haul truckers, and is officially expected to keep doing so for some time. I expected that long-distance trucking would have overtaken passenger cars as the "face" of self-driving vehicles; the thing that people argue about when they argue whether self-driving vehicles are safe enough, good or bad for society, etc. This has not happened. When people argue about self-driving vehicles, they argue about whether they want Waymo cars in their city. I expected that the trucking industry would shed a lot of workers, replacing them with self-driving trucks that don't need sleep, breaks, or drug testing. I expected that this would be an vivid early example of mass job loss to AI; and in turn that this would motivate more political interest in UBI. This, too, has not happened. (I certainly did not expect that the trucking industry in 2025 would be much more disrupted by anti-immigrant politics than by self-driving technology.)