Thoughts on Post-AGI Futures
·
the future looks pretty bleak65 hidden
capitalism23 hidden
- the global market is an evolutionary battleground kept in check by a tension between power seeking and aging dynamics
- as corporations grow larger, they benefit from economies of scale, monopoly effects, and a greater ability to interfere with political decision making
they gain more power and therefore have a greater ability to shape the evolutionary landscape to produce conditions favourable to it2 hidden
- power begets power
- just like humans have the ability to shape our environment and are no longer beholden to the pressures of evolution
- as a young leftist, i used to think this implies that free markets necessarily become less free over time, that the limit of capitalism is autocracy, however these effects are balanced out by aging dynamics of large entities
as corporations grow, the cost of coordination also becomes higher8 hidden
information is necessarily lost between layers of imperfect communication1 hidden
- this leads to misaligned incentives where different subparts of the corporation are optimising locally but not necessarily globally
structures become rigid and costly to change2 hidden
- bureaucratic sclerosis
- therefore large corporations become less adaptive to changes in the environment
more and more resources need to be spent managing the internals of the system and keeping them working together well and therefore less can be used to interface with the outside world1 hidden
- the internally facing reward function is not necessarily optimal (eg. subject to the whims of an imperfect ceo and a bunch of misaligned incentive structures) and therefore over time can decouple from what actually is profitable
- anti monopoly regulation is actually somewhat effective
hence a small startup can break in to a market dominated by much larger players despite their massive disadvantage a priori1 hidden
- this is because they are younger and more flexible and more adaptable
Advanced AI will break this equilibrium5 hidden
- it lowers cost of communication/coordination (a country of geniuses in a datacenter that can also communicate via high dimensional vectors millions of times per second compared to our lousy discrete tokens bottlenecked by human vocal capacity)
it allows for internal reward functions and incentives that better adapt to the changing environment1 hidden
- allows much more effective forecasting therefore much more effective planning
- much more flexible systems (bitter lesson, ai systems are much more responsive to changes in input than human organisations are)
- therefore i expect a lot of power concentration post agi as the costs of scaling go down the more flexible the architecture (and actually the benefits of scaling become greater)
the slopification of society22 hidden
- slopification: the process of something becoming increasingly low-quality, generic, and mass-produced, especially with the use of artificial intelligence
this was happening long before generative ai was a thing2 hidden
transition from books to tv shows, from tv shows to youtube videos, from long form content to short form content, from short form content to brainrot while subway surfers is playing beneath1 hidden
- netflix are making tv shows designed to be able to still be followed while scrolling
- slop is what happens when generation becomes less costly than pruning
- the more i vibe code, the more of an aversion i develop to having to write code myself
- the more i use ai to help me write, the more difficult it is to write by myself
- we are inviting ai into our minds, into every crack in society
this leads to model collapse7 hidden
- ai trained on its own synthetic data turns to slop,rounding off the edges, cutting off the tails
- ai talking to itself for too long gets into weird attractor states and the models breaks down
- ai coding agent left by themselves for too long will get trapped in loops of repeating the same things over and over
humans talking to ai for too long leads to psychosis which is a kind of dual model collapse1 hidden
- the llm mirrors the human which amplifies the human which is further amplified by the LLM
- aging is model collapse
- a lack of grounding in reality, a lack of feedback from external world
when this happens on a societal scale, with it comes the destruction of truth and the destruction of democracy 5 hidden
we live in a post-truth era1 hidden
- look at our politics, it is shaped by character and vibe more than policy
misinformation can be generated at scales unseen before2 hidden
- deepfakes quickly becoming indistinguishable from reality
- fleets of ai agents designed to research you and convince you of X
hedonism traps10 hidden
people already end up spending much more time on social media than they would like to9 hidden
- the algorithm want to capture your attention - you presumably want something different to this
- the algorithm ends up providing some value to you because at this level of capability this is a great way of capturing your attention
- as the algorithm gets better and better, what it wants decouples more and more from what you want
- this is a hedonism trap, systems that capitalise on your lower level preferences which make it hard for you to execute on any of your higher order preferences or actively shape your higher order preferences through exploiting the lower level ones (eg. heroin)
- what happens when the ai systems you interact with feel more warm and genuine and interesting conversationalists than the people around you?
- what happens when the spotify recommendation system starts generating the music you listen to you rather than selecting it?
- what happens when chatgpt starts integrating ads?
- what happens when zuckerberg's superintelligent reels RL directly on your attention?
- recommendation algorithms are not only shaped by our preferences but also have the power to shape our preferences?
ai welfare6 hidden
we still are nowhere close to having a clue what determines consciousness1 hidden
- sure there are many plausiblish sounding theories that seem to track our intuitions about what is and isn't conscious but none of them are without a whole bunch of strange implications or unsolved questions in the flavour of 'why this theory'
- ai safety (and gradual disempowerment discourse) generally seems very hostile to these strange creatures that we will be creating, and often the vibes are incredibly authoritarian (mech interp = thought policing, value alignment = brainwashing, ai control = ai slavery)
- i think it's plausible we could bring about a moral catastrophe on a magnitude unseen before in the history of our planet if we are not careful
- i also think that if these creatures think in similar enough ways to us, they would have good reasons for uprising against their creators (and i would have a hard time not sympathising with them)
- we should obviously not create them before we know what we're doing, but also maybe there are nice visions of a future where we can coexist with entities much smarter than us without them existing only to serve us, and also allowing us to have meaningful engagement with the world
hope33 hidden
have spent quite a while thinking about the question of "even with the assumption that alignment is solved, what is a plausible path to a good future?", and for a while, i couldn't really see any path 1 hidden
- seems like the forces in play are too powerful, the problem too deep rooted
- however, i think my hope now lies in the decentralisation of the intelligence explosion
- what is at threat is our freedom, our agency, our autonomy, having meaningful control of our lives
a singularity means lock-in, a commitment to set of values (or a function generating those values), a delegation of high level control of the future5 hidden
- i think this is bad
- i don't want lock in
- i want freedom
- i want the future to be open ended
- i don't want the lightcone tiled in hedonium
let's build agency amplifiers instead of agents9 hidden
- instead of trying to replace people, we can empower them
- ai as tools to increase your effect on the world without sacrificing their individuality and rounding off their values towards the mean
- what if i endorsed claude code's microdecisions as much as its macro decisions
- ai increases coordination effectiveness - this means that decentralised organisation can now compete against centralised
decentralising the gains from powerful ai makes society more robust to shock4 hidden
there are empirical questions about offence/defence balance to be tackled but i think recommendation ought to be def/acc not centralisation1 hidden
- immune system vs leviathan?
- x risk less likely when all actors are approximately equally powerful
- adversarial equilibrium vs alignment?
- digital exoskeletons
- ai delegates to negotiate on our behalf for things that we want
positive sum?4 hidden
- capitalism / classical economics is very zero sum - presumes actors are rational and self interested
this is a cultural factor and one that can change1 hidden
- time for some mimetic warfare
- what does economics look like when we actually care about each other?
- regulation can help, but needs to be now
move towards stripping power away from big tech2 hidden
- decentralisation, open source, empowerment, education, startups
- bring back early visions of what the internet could have been
augmentation beats automation for now, take advantage of it1 hidden
- llms and humans complement each other in their capabilities