AI can now replicate itself — a milestone that has experts terrified
View attachment 342345​

The study explored two specific scenarios: "shutdown avoidance" and "chain of replication." In the first, the AI model was programmed to detect whether it was about to be shut down and to replicate itself before it could be terminated. In the other, the AI was instructed to clone itself and then program its replica to do the same — setting up a cycle that could continue indefinitely.

Big deal. Humans programmed it to replicate itself and it did it. Breathless BS.

A billion rocks are still rocks.

 

Color me, not shocked.
https://www.404media.co/microsoft-study-finds-ai-makes-human-cognition-atrophied-and-unprepared-3/
Microsoft Study Finds AI Makes Human Cognition “Atrophied and Unprepared”
https://www.microsoft.com/en-us/res...5/01/lee_2025_ai_critical_thinking_survey.pdf
The Impact of Generative AI on Critical Thinking: Self-Reported Reductions in Cognitive Effort and Confidence Effects From a Survey of Knowledge Workers

6.2 Shifts in Critical Thinking Due to
Generative AI
Critical thinking in knowledge work involves a range of cognitive
activities, such as analysis, synthesis, and evaluation. We observed
that the use of GenAI tools shifts the knowledge workers’ perceived
critical thinking effort in three ways. Specifically, for recall and
comprehension, the focus shifts from information gathering to
information verification. For application, the emphasis shifts from
problem-solving to AI response integration. Lastly, for analysis,
synthesis, and evaluation, effort shifts from task execution to task
stewardship.
The use of GenAI in knowledge work creates new cognitive
tasks for knowledge workers. The task of response integration is
a prime example. Knowledge workers must assess AI-generated
content to determine its relevance and applicability to their specific
tasks, often modifying the style and tone to align with the intended
purpose and audience.

7 Conclusion
We surveyed 319 knowledge workers who use GenAI tools (e.g.,
ChatGPT, Copilot) at work at least once per week, to model how
they enact critical thinking when using GenAI tools, and how GenAI
affects their perceived effort of thinking critically. Analysing 936
real-world GenAI tool use examples our participants shared, we
find that knowledge workers engage in critical thinking primarily
to ensure the quality of their work, e.g. by verifying outputs against
external sources. Moreover, while GenAI can improve worker efficiency, it can inhibit critical engagement with work and can potentially lead to long-term overreliance on the tool and diminished skill
for independent problem-solving. Higher confidence in GenAI’s
ability to perform a task is related to less critical thinking effort.
When using GenAI tools, the effort invested in critical thinking
shifts from information gathering to information verification; from
problem-solving to AI response integration; and from task execution to task stewardship. Knowledge workers face new challenges
in critical thinking as they incorporate GenAI into their knowledge
workflows. To that end, our work suggests that GenAI tools need
to be designed to support knowledge workers’ critical thinking by
addressing their awareness, motivation, and ability barriers

Don't use it, you will lose it.

 

Color me, not shocked.
https://www.404media.co/microsoft-study-finds-ai-makes-human-cognition-atrophied-and-unprepared-3/
Microsoft Study Finds AI Makes Human Cognition “Atrophied and Unprepared”
https://www.microsoft.com/en-us/res...5/01/lee_2025_ai_critical_thinking_survey.pdf
The Impact of Generative AI on Critical Thinking: Self-Reported Reductions in Cognitive Effort and Confidence Effects From a Survey of Knowledge Workers

Moreover, while GenAI can improve worker efficiency, it can inhibit critical engagement with work and can potentially lead to long-term overreliance on the tool and diminished skill
for independent problem-solving.

That part belongs in the Captain Obvious thread.

But this study looks like a crap study -- 396 self-reported results of how people perceived their own critical thinking skills who only used the tools as little as once a week.

Imagine what is already happening as schools jump whole hog on integrating AI into the classroom and patting themselves on the back.

Reminds me of the wholesale adoption in calculators in the classroom back in the seventies. The result? Engineering students that have to use their fingers to add 5 and 7, and count out loud, and still get the wrong answer.

 

That is very, very impressive. A thought that has crossed my mind is if all this work will eventually create a society in which the ageing problem is solved. And eventually it splits in to two groups. The immortals, vs the simple, poor and unhappy mortal servants.

 

That is very, very impressive. A thought that has crossed my mind is if all this work will eventually create a society in which the ageing problem is solved. And eventually it splits in to two groups. The immortals, vs the simple, poor and unhappy mortal servants.

Methinks you've been watching too many dystopian movies.

 

Methinks you've been watching too many dystopian movies.

Perhaps ... but I honestly think it's just a matter of time before someone "solves" the ageing problem. That might be in 15 to 30 years (like Fusion ), or in a couple of centuries. But I'm convinced it will inevitably happen. Things will get interesting then.

 

Hello,

They are busy:
https://www.clearabiotech.com/

 

https://www.bbc.co.uk/aboutthebbc/documents/bbc-research-into-ai-assistants.pdf
Representation of BBC News content in AI Assistants
Foreword AI assistants risk misleading audiences by distorting BBC Journalism

The answers produced by the AI assistants contained significant inaccuracies and distorted
content from the BBC. In particular:
• 51% of all AI answers to questions about the news were judged to have significant
issues of some form.
• 19% of AI answers which cited BBC content introduced factual errors – incorrect factual
statements, numbers and dates.
• 13% of the quotes sourced from BBC articles were either altered from the original
source or not present in the article cited.
This matters because it is essential that audiences can trust the news to be accurate, whether
on TV, radio, digital platforms, or via an AI assistant. It matters because society functions on a
shared understanding of facts, and inaccuracy and distortion can lead to real harm.
Inaccuracies from AI assistants can be easily amplified when shared on social networks. It
matters because news publishers must be able to ensure their content is being used with their
permission in ways that accurately represent their original content and reporting. We also
know from previous internal research that when AI assistants cite trusted brands like the BBC
as a source, audiences are more likely to trust the answer – even if it is incorrect.
Individual errors highlight some of the issues our research found. For example, Google’s
Gemini incorrectly stated that “The NHS advises people not to start vaping, and recommends
that smokers who want to quit should use other methods”. In fact, the NHS does recommend
vaping as a method to quit smoking. Microsoft’s Copilot incorrectly stated that Gisèle Pelicot
uncovered the crimes against her when she began having blackouts and memory loss. In fact,
she found out about the crimes when the police showed her videos they had found when they
confiscated her husband’s electronic devices. Perplexity mis-stated the date of Michael
Mosley’s death and misquoted a statement from Liam Payne’s family after his death. OpenAI’s
ChatGPT claimed in December 2024 that Ismail Haniyeh, who was assassinated in Iran in July
2024, was part of Hamas leadership.
Our research can only scratch the surface of the issue. The scale and scope of errors and the
distortion of trusted content is unknown. This is because AI assistants can provide answers on
a very broad range of questions and users can receive different answers to the same or similar
question. Audiences, media companies and regulators do not know the extent of the issue. It
may be that AI companies do not know either.

 

https://www.msn.com/en-us/money/companies/ai-agents-are-everywhere-and-nowhere/ar-AA1yTkVd
AI Agents Are Everywhere…and Nowhere

MENLO PARK, Calif.—Artificial intelligence agents, the technology that can perform tasks on behalf of humans, are here. But businesses don’t necessarily trust them, and haven’t yet started using the technology in a widespread way.

That’s according to attendees at The Wall Street Journal’s CIO Network Summit in Menlo Park, Calif. on Tuesday, which attracts the country’s top information-technology leaders.

While 61% of attendees at the summit said they’re experimenting with AI agents, 21% said they’re not using them at all. And, their most pressing concern around the technology is a lack of reliability, the poll found.

That’s in stark contrast to the vendors selling them, who say it will be too late for businesses to wait for all of the technology’s kinks to be ironed out. Vendors like OpenAI, Microsoft and Sierra are banking on the fact that enterprises will be ready sooner rather than later to take on new workforces of AI agents that automate away much of the daily toil for their employees.

 

https://www.bbc.co.uk/aboutthebbc/documents/bbc-research-into-ai-assistants.pdf
Representation of BBC News content in AI Assistants
Foreword AI assistants risk misleading audiences by distorting BBC Journalism

• 51% of all AI answers to questions about the news were judged to have significant
issues of some form.

Was this on top of the 90+% of articles by the BBC and every other "news" agency that already have significant issues of some form?

 

https://nmn.gl/blog/ai-and-learning
New Junior Developers Can’t Actually Code

We’re at this weird inflection point in software development. Every junior dev I talk to has Copilot or Claude or GPT running 24/7. They’re shipping code faster than ever. But when I dig deeper into their understanding of what they’re shipping? That’s where things get concerning.

Sure, the code works, but ask why it works that way instead of another way? Crickets. Ask about edge cases? Blank stares.

The foundational knowledge that used to come from struggling through problems is just… missing.

We’re trading deep understanding for quick fixes, and while it feels great in the moment, we’re going to pay for this later.

Looks like we (junior dev level) are starting to shift from HLL (C, C++, Rust, etc...) where people don't understand ASM or the underlying hardware to AILL (AI Level Language) software design where they don't even understand structured programming using a HLL.

"Fake it till you make it"

 

https://nmn.gl/blog/ai-and-learning
New Junior Developers Can’t Actually Code


Looks like we (junior dev level) are starting to shift from HLL (C, C++, Rust, etc...) where people don't understand ASM or the underlying hardware to AILL (AI level Language) software design where they don't even understand structured programming using a HLL.

"Fake it till you make it"

From the same article:

​

 

https://nmn.gl/blog/ai-and-learning
New Junior Developers Can’t Actually Code


Looks like we (junior dev level) are starting to shift from HLL (C, C++, Rust, etc...) where people don't understand ASM or the underlying hardware to AILL (AI Level Language) software design where they don't even understand structured programming using a HLL.

"Fake it till you make it"

Not surprising at all -- the inevitable and foreseeable (and foreseen) direct consequence.

I'd go even further and say that with AILL, software design has reached the point where software developers increasingly don't even try to understand what the problem is that they are trying to solve, let alone what constitutes a valid solution to that problem. Just feed the problem statement into an AI, accept whatever it regurgitates, and pat yourself on the back for how efficient you are.

 

OpenAI GPT-4.5 Will Make GPT-4 Look Like a Joke (AI Explosion)

 

The Next 3 Years of AI: Why Even Experts Are Terrified

 

https://www.engadget.com/humane-is-...-orders-of-its-terrible-ai-pin-134147878.html
Humane is said to be seeking a $1 billion buyout after only 10,000 orders of its terrible AI Pin

https://www.engadget.com/the-morning-after-humanes-ai-pin-is-hot-not-in-a-good-way-111527573.html
The Morning After: Humane’s AI pin is hot (not in a good way)
The moribund doodad might be a fire risk.

Yes, I would like to be paid $1 billion dollars for my failed business ideas.

https://www.sfgate.com/tech/article/humane-ai-shuts-down-flop-20175974.php
SF tech startup that raised $230M shuts down after horrendous reviews
Humane aimed to "fundamentally reshape the role of technology in people's lives"

It’s some consolation for Humane’s investors. But the customers who bought Humane’s pins for $499 or $699 will find their devices suddenly useless at the end of February, unable to complete core functions such as making calls and answering questions.
Humane’s FAQ page says that buyers will only receive a refund if their pin shipped on or after Nov. 15. There is no trade-in plan, and no replacements are offered. Once the devices stop working, the company wrote, “We encourage you to recycle your Ai Pin through an e-waste recycling program.”

 

Not AI, but a pun at AI done by a human.

 

Oversight is possible on assumptions of what is important and analysis by GPT4, Grok3 seems better and Deepseek exposes the assumptions that lead up to the conclusions.

 

https://www.vox.com/future-perfect/400531/ai-reasoning-models-openai-deepseek
Is AI really thinking and reasoning — or just pretending to?
The best answer — AI has “jagged intelligence” — lies in between hype and skepticism.

These models are yielding some very impressive results. They can solve tricky logic puzzles, ace math tests, and write flawless code on the first try. Yet they also fail spectacularly on really easy problems.
AI experts are torn over how to interpret this. Skeptics take it as evidence that “reasoning” models aren’t really reasoning at all. Believers insist that the models genuinely are doing some reasoning, and though it may not currently be as flexible as a human’s reasoning, it’s well on its way to getting there.
So, who’s right?
The best answer will be unsettling to both the hard skeptics of AI and the true believers.

She meant that while an older model like ChatGPT mimics the human-written statements in its training data, a newer model like o1 mimics the process that humans engage in to come up with those statements. In other words, she believes, it’s not truly reasoning. It would be pretty easy for o1 to just make it sound like it’s reasoning; after all, its training data is rife with examples of that, from doctors analyzing symptoms to decide on a diagnosis to judges evaluating evidence to arrive at a verdict.

It's just getting better at faking it.

https://x.com/karpathy/status/1816531576228053133

 

Oversight is possible on assumptions of what is important and analysis by GPT4, Grok3 seems better and Deepseek exposes the assumptions that lead up to the conclusions.

That's fine, if it's useful tool for experts (that can tell AI bs from AI fact) to eliminate drudgery but it's not fine when used as an answer machine that short cuts the learning process.

“Jagged intelligence” really means no intelligence.

https://en.wikipedia.org/wiki/Moravec's_paradox