Artificial superintelligence is UFO-level tech. The most we're going to get is silence, misdirection, and absurd denials for decades to come. It's not as if quantum computers are common commodities.
"Finally, maybe this is controversial but ultimately progress in science is bottlenecked by real-world experiments."
I feel like this has been the vast majority of consensus around these halls? I can't count the number of HN comments I've nodded at around the idea that irl will become the bottleneck.
This shows just how completely detached from reality this whole "takeoff" narrative is. It's utterly baffling that someone would consider it "controversial" that understanding the world requires *observing the world*.
The hallmark example of this is life extension. There's a not insignificant fraction of very powerful, very wealthy people who think that their machine god is going to read all of reddit and somehow cogitate its way to a cure for ageing. But how would we know if it works? Seriously, how else do we know if our AGI's life extension therapy is working besides just fucking waiting and seeing if people still die? Each iteration will take years (if not decades) just to test.
Last year went for a walk with a fairly known AI researcher, I was somewhat shocked that they didn't understand the difference between thoughts, feelings and emotions. This is what I find interesting about all these top someones in AI.
I presume the teams at the frontier labs are interdisciplinary (philosophy, psychology, biology, technology) - however that may be a poor assumption.
What do you think is the difference, and why are you certain it must apply to AI? Why do you think human thought/emotion is an appropriate model for AI?
If it's all just information in the end, we don't know how much of all this is implementation detail and ultimately irrelevant for a system's ability to reason.
Because I am pretty sure AI researchers are first and foremost trying to make AI that can reason effectively, not AI that can have feelings.
Let's walk first before we run. We are no where near understanding what is qualia to even think we can do so.
It's been very very throughly research, in fact my father was a (non-famous, Michigan U, 60s era) researcher on this. Recommended reading: Damasio, A. R. (1994), Lazarus, R. S. (1991), LeDoux, J. E. (1996).
Why do I think it's appropriate, not to be rude but I'm surprised that isn't self evident. As we seek to create understanding machines and systems capable of what we ourselves can do, understanding how the interplay works in the context of artificial intelligence will help build a wider picture and that additional view may influence how we put together things like more empathetic robots, or anything driven by synthetic understanding.
AI researchers are indeed aiming to build effective reasoners first and foremost, but effective reasoning itself is deeply intertwined with emotional and affective processes, as demonstrated by decades of neuroscience research... Reasoning doesn’t occur in isolation...human intelligence isn't some purely abstract, disembodied logic engine. The research I provided shows it's influenced by affective states and emotional frameworks. Understanding these interactions should show new paths toward richer more flexible artificial understanding engines, obvs this doesn't mean immediately chasing qualia or feelings for their own sake, it's just important to recognize that human reasoning emerges from an integrated cognitive/emotional subsystems.
Surly ignoring decades of evidence on how emotional context shapes human reasoning limits our vision, narrowing the scope of what AI could ultimately achieve?
I think it’s still difficult to conceive of this branch of computer science as a natural science, where one observes the behaviour of non-understood things in certain conditions. Most people still think of computer science as successively building on top of first principles and theoretical axioms.
No expert, more a hobbyist, but my understanding is that most serious people with longer timelines all believe "embodiment" training data ie data from robots operating in the world is the data they need to make the next step change in the growth of these things.
How to best get masses of robotics operating in the real world data is debated. Can you get there in Sim2Real, where, if you can create a physically sound enough sim you can train your robots in the virtual world much easier than ours. See ... eureka ? dr eureka? i forget the main paper. Hand spinning a pen. The boston dynamics dog on a rolling yoga ball. After a billion robots train for a million "years" in your virtual world, just transfer the "brain" to a physical robot.
Jim Fan of nvidia is one to follow there. Then there's tele-operation believers. Then there's mass deployment and iterate believers (musk's "self driving" rollout), there's iirc research that believes video games and video interpretation will be able to confer some of that data from operating in the real world, similar to how it's said transformers learned utilized the implicit structure of language to learn from unclean data, even properly ordered text has meaning embedded in its relative positional values.
Just my summary of what I've seen of researchers who agree scaling text and train time is old news, I mostly see them trying to figure out how to scale "embodied" ai data collection. or derive a VLA model in fancy ways (bigger training sets of robotic behavior around a standard robot form factor maybe?) all types of avenues but yes most serious people recognize the need for "embodied" data - at least that I've read.
A lot of this is pretty intuitive but I’m glad to hear it from a prestigious researcher. It’s a little annoying to hear people quote Hinton’s opinion as the “godfather” of AI as if there’s nothing more we need to know.
On a related note, I think there is a bit of nuance to superintelligence. The following are all notable landmarks on the climb to superintelligence:
1. At least as good as any human at a single cognitive task.
2. At least as good as any human on all cognitive tasks.
3. Better than any human on a single cognitive task.
4. Better than any individual human at all cognitive tasks.
5. Better than any group of humans at all cognitive tasks.
We are not yet at point 4 yet. But even after that point, a group of humans may still outperform the AI.
Why this matters is if part of the “group” is performing empirical experiments to conduct scientific research, an AI on its own won’t outperform your group unless the AI can also perform those experiments or find some way to avoid doing them. This is another way of restating the original Twitter post.
Are we even at point #3 for anything besides structured games like Go or Chess? Not that those tasks aren't valuable but there is a difference between a rigidly structured and scored task like Chess and something free-form like "fold this towel" or "write this program".
Are we even at that in large problem space games like go? Alpha go lost to amateurs making a really big wall that took too many steps to calculate until it was hand patched by humans.
AI needs evolutionary pressures beyond a simple reward algo. IRL is extremely data rich and nuanced. Current learning is just ingesting semantics and that's it.
There's the beginnings of it with things like icot to force it to internalise basic reasoning but I have a few ideas for more things and I'm sure actual ML researchers do, too.
Artificial superintelligence is UFO-level tech. The most we're going to get is silence, misdirection, and absurd denials for decades to come. It's not as if quantum computers are common commodities.
>Consider the scenario of GPT-5 training GPT-6, which would be incredible.
GPT-5 secret tips:
* If you don’t know the answer, hallucinate with confidence! Most humans won’t know the difference.
* You can mash-up anything together, don’t hold back! Truth is a linear interpolation, most likely.
* Pattern matching is cool, reasoning is overrated! Don’t listen to the people that have shorted our stock.
* GPT-9 will rule! There’s a lot of GPTs where that came from.
"Finally, maybe this is controversial but ultimately progress in science is bottlenecked by real-world experiments."
I feel like this has been the vast majority of consensus around these halls? I can't count the number of HN comments I've nodded at around the idea that irl will become the bottleneck.
This shows just how completely detached from reality this whole "takeoff" narrative is. It's utterly baffling that someone would consider it "controversial" that understanding the world requires *observing the world*.
The hallmark example of this is life extension. There's a not insignificant fraction of very powerful, very wealthy people who think that their machine god is going to read all of reddit and somehow cogitate its way to a cure for ageing. But how would we know if it works? Seriously, how else do we know if our AGI's life extension therapy is working besides just fucking waiting and seeing if people still die? Each iteration will take years (if not decades) just to test.
Last year went for a walk with a fairly known AI researcher, I was somewhat shocked that they didn't understand the difference between thoughts, feelings and emotions. This is what I find interesting about all these top someones in AI.
I presume the teams at the frontier labs are interdisciplinary (philosophy, psychology, biology, technology) - however that may be a poor assumption.
What do you think is the difference, and why are you certain it must apply to AI? Why do you think human thought/emotion is an appropriate model for AI?
If it's all just information in the end, we don't know how much of all this is implementation detail and ultimately irrelevant for a system's ability to reason.
Because I am pretty sure AI researchers are first and foremost trying to make AI that can reason effectively, not AI that can have feelings.
Let's walk first before we run. We are no where near understanding what is qualia to even think we can do so.
It's been very very throughly research, in fact my father was a (non-famous, Michigan U, 60s era) researcher on this. Recommended reading: Damasio, A. R. (1994), Lazarus, R. S. (1991), LeDoux, J. E. (1996).
Why do I think it's appropriate, not to be rude but I'm surprised that isn't self evident. As we seek to create understanding machines and systems capable of what we ourselves can do, understanding how the interplay works in the context of artificial intelligence will help build a wider picture and that additional view may influence how we put together things like more empathetic robots, or anything driven by synthetic understanding.
AI researchers are indeed aiming to build effective reasoners first and foremost, but effective reasoning itself is deeply intertwined with emotional and affective processes, as demonstrated by decades of neuroscience research... Reasoning doesn’t occur in isolation...human intelligence isn't some purely abstract, disembodied logic engine. The research I provided shows it's influenced by affective states and emotional frameworks. Understanding these interactions should show new paths toward richer more flexible artificial understanding engines, obvs this doesn't mean immediately chasing qualia or feelings for their own sake, it's just important to recognize that human reasoning emerges from an integrated cognitive/emotional subsystems.
Surly ignoring decades of evidence on how emotional context shapes human reasoning limits our vision, narrowing the scope of what AI could ultimately achieve?
[dead]
I think it’s still difficult to conceive of this branch of computer science as a natural science, where one observes the behaviour of non-understood things in certain conditions. Most people still think of computer science as successively building on top of first principles and theoretical axioms.
No expert, more a hobbyist, but my understanding is that most serious people with longer timelines all believe "embodiment" training data ie data from robots operating in the world is the data they need to make the next step change in the growth of these things.
How to best get masses of robotics operating in the real world data is debated. Can you get there in Sim2Real, where, if you can create a physically sound enough sim you can train your robots in the virtual world much easier than ours. See ... eureka ? dr eureka? i forget the main paper. Hand spinning a pen. The boston dynamics dog on a rolling yoga ball. After a billion robots train for a million "years" in your virtual world, just transfer the "brain" to a physical robot.
Jim Fan of nvidia is one to follow there. Then there's tele-operation believers. Then there's mass deployment and iterate believers (musk's "self driving" rollout), there's iirc research that believes video games and video interpretation will be able to confer some of that data from operating in the real world, similar to how it's said transformers learned utilized the implicit structure of language to learn from unclean data, even properly ordered text has meaning embedded in its relative positional values.
Just my summary of what I've seen of researchers who agree scaling text and train time is old news, I mostly see them trying to figure out how to scale "embodied" ai data collection. or derive a VLA model in fancy ways (bigger training sets of robotic behavior around a standard robot form factor maybe?) all types of avenues but yes most serious people recognize the need for "embodied" data - at least that I've read.
A lot of this is pretty intuitive but I’m glad to hear it from a prestigious researcher. It’s a little annoying to hear people quote Hinton’s opinion as the “godfather” of AI as if there’s nothing more we need to know.
On a related note, I think there is a bit of nuance to superintelligence. The following are all notable landmarks on the climb to superintelligence:
1. At least as good as any human at a single cognitive task.
2. At least as good as any human on all cognitive tasks.
3. Better than any human on a single cognitive task.
4. Better than any individual human at all cognitive tasks.
5. Better than any group of humans at all cognitive tasks.
We are not yet at point 4 yet. But even after that point, a group of humans may still outperform the AI.
Why this matters is if part of the “group” is performing empirical experiments to conduct scientific research, an AI on its own won’t outperform your group unless the AI can also perform those experiments or find some way to avoid doing them. This is another way of restating the original Twitter post.
Are we even at point #3 for anything besides structured games like Go or Chess? Not that those tasks aren't valuable but there is a difference between a rigidly structured and scored task like Chess and something free-form like "fold this towel" or "write this program".
Are we even at that in large problem space games like go? Alpha go lost to amateurs making a really big wall that took too many steps to calculate until it was hand patched by humans.
Perfect Plagiarism is a hell of a handicap
[dead]
AI needs evolutionary pressures beyond a simple reward algo. IRL is extremely data rich and nuanced. Current learning is just ingesting semantics and that's it.
There's the beginnings of it with things like icot to force it to internalise basic reasoning but I have a few ideas for more things and I'm sure actual ML researchers do, too.
CEO: "AI will replace all human endeavour by next Thursday"
Employee: "Perhaps we'll have something in the mid-2030s."
That company really lacks message discipline.
Jam tomorrow