My impression (primarily from former representative Justin Amash) is that individual congresspeople have almost no power. Bills are crafted and introduced at the party level, and your choices are to vote with them or not. If you don't vote with the party and are in a safe district (which most are), the party will support candidates who will go along with the party line in the primary. The only scenario in which an individual congressperson has power is a close vote in a contested district where the party can't take the risk of a primary challenger. First, I want to know if which parts of this are true, and in more detail. One guy tweeting does not constitute proof. Are elected officials sitting around watching Netflix all day, or do they still have the power to privately horse trade to support their goals. Second, assuming it is true, where did the power go? Parties aren't real, the power must be in specific humans or incentive systems. Does it all rest in the top three elected officials of each party? Their staff? Unelected party officers? I'm very interested in evidence and arguments in either direction. 

As I was looking through possible donation opportunities, I noticed that MIRI's 2025 Fundraiser has a total of only $547,024 at the moment of writing (out of the target $6M, and stretch target of $10M). Their fundraising will stop at midnight on Dec 31, 2025. At their current rate they will definitely not come anywhere close to their target, though it seems likely to me that donations will become more frequent towards the end of the year. Anyone know why they currently seem to struggle to get close to their target? 

Inspired by an exchange with Jeremy Gillen: I think there might be a bias on here to assume that if humanity goes extinct as the result of something an AI does, it will specifically be something the AI did on purpose, with the intent of causing humanity's extinction or the unrecoverable collapse of industrial civilization, or at least with the knowledge that that was a likely outcome. But it seems to me that one obvious way we "accidentally" end up in such a situation is 1. AI systems become capable of training successor systems which are more capable of doing particular dangerous actions than they are, and better at training later successors than they are 2. The successors are not enough better at risk assessment than their ancestors to outweigh the increase in ability to accidentally cause harm 3. Iterate I notice that, if you look at the history of near misses with civilization-ending events, they were not ones where someone tried and failed to end civilization. People have tried and failed, but they're not thr ones that got close. Anyway, on that note, happy day-of-Solstice everyone :)

Overconfidence from early transformative AIs is a neglected, tractable, and existential problem.  If early transformative AIs are overconfident, then they might build ASI/other dangerous technology or come up with new institutions that seem safe/good, but ends up being disastrous. This problem seems fairly neglected and not addressed by many existing agendas (i.e., the AI doesn't need to be intent-misaligned to be overconfident).[1] Overconfidence also feels like a very "natural" trait for the AI to end up having relative to the pre-training prior, compared to something like a fully deceptive schemer. My current favorite method to address overconfidence is training truth-seeking/scientist AIs. I think using forecasting as a benchmark seems reasonable (see e.g., FRI's work here), but I don't think we'll have enough data to really train against it. Also I'm worried that "being good forecasters" doesn't generalize to "being well calibrated about your own work." On some level this should not be too hard because pretraining should already teach the model to be well calibrated on a per-token level (see e.g., this SPAR poster). We'll just have to elicit this more generally.  (I hope to flush this point out more in a full post sometime, but it felt concrete enough to worth quickly posting now. I am fairly confident in the core claims here.) 1. ^ For example, you might not necessarily detect overconfidence in these AIs even with strong interpretability because the AI doesn't "know" that it's overconfident. I also don't think there are obvious low/high stakes control methods that can be applied here. 

GPT = Generative Pre-Training? Everyone thinks GPT stands for "Generative Pre-trained Transformer". (For example, Wikipedia.) Does it really? The earliest mention of "GPT" is in the GPT-2 paper, which refers to "the OpenAI GPT model" and cites the GPT-1 paper. That paper does not contain the phrase "generative pre-trained transformer". But it does contain the phrase "generative pre-training", in the title and in the body, italicized.

I'm fanatically in favor of creating new ways to test (& thereby develop) rationality in general and scientific capabilities in particular. However, any such resources would necessarily be dual-use, providing AI developers evals (or eval paradigms) which could help accelerate AI development along the axes on which it's currently most lacking. This seems like an obviously insane thing to worry about but I can't figure out why; soliciting other opinions.

I talked to someone at an AI company [1] who thought safety would be net unaffected by timelines. The argument was: 1. Assuming x-risk is real, a higher ratio of safety to capabilities progress is good for safety 2. AI safety's growth is largely caused by increased AI hype, new models to study, and other factors downstream of AI capabilities 3. Slowing down or speeding up capabilities will slow/speed safety investment by roughly the same amount 4. Therefore, there will be no net safety impact. However, even if 1-3 are true, I claim 4 doesn't follow. * Suppose AI capabilities progress happens by people noticing capabilities improvements and making them, and safety progress happens by people noticing safety issues and solving them, both with a time lag of 1 year. Suppose further that each capability improvement causes 1 safety issue. * If there are 10 capabilities improvements per year, then the steady state is 10 outstanding safety issues. * If everything doubles, AI labs make 20 capabilities improvements per year but the time lag is still 1 year, so there will be a steady state of 20 outstanding safety issues. 4 only holds if the time lag decreases in lockstep with increasing investment. However, this seems false because most ways capabilities would speed up involve more parallel labor/compute or faster algorithmic progress, not faster feedback loops from the real world that would transfer to safety. By analogy, suppose that the speed of cars increased from 20mph to 100mph in one year in 1910. Even if the same technology sped up the invention of airbags, ABS, etc. to 2010 levels, more people would die in car crashes because they couldn't be installed until the next model year [2], unless companies proactively did real-world testing and delayed the release of the super fast car until such safety features could be invented and integrated, which would be a huge competitive disadvantage. Likewise, if cyberattacks/power-seeking/etc are solvable but require re