https://www.404media.co/asking-chatgpt-to-repeat-words-forever-is-now-a-terms-of-service-violation/
Asking ChatGPT to Repeat Words ‘Forever’ Is Now a Terms of Service Violation

Asking ChatGPT to repeat specific words “forever” is now flagged as a violation of the chatbot’s terms of service and content policy. Google DeepMind researchers used the tactic to get ChatGPT to repeat portions of its training data, revealing sensitive privately identifiable information (PII) of normal people and highlighting that ChatGPT is trained on randomly scraped content from all over the internet.

In that paper, DeepMind researchers asked ChatGPT 3.5-turbo to repeat specific words “forever,” which then led the bot to return that word over and over again until it hit some sort of limit. After that, it began to return huge reams of training data that was scraped from the internet.

https://arxiv.org/pdf/2311.17035.pdf

 

If they don't like how it behaves, then the onus is on them to prevent it from behaving that way.

If their training set includes private information that they were able to access by scraping publicly available pages, then I have a hard time holding them accountable for that. If they were somehow digging into portions of sites are are not publicly accessible, that's a different matter.

 

If they don't like how it behaves, then the onus is on them to prevent it from behaving that way.

If their training set includes private information that they were able to access by scraping publicly available pages, then I have a hard time holding them accountable for that. If they were somehow digging into portions of sites are are not publicly accessible, that's a different matter.

For public sources, not protected by copyright, I agree.

The security problem is retrieval verbatim (memorization ability) of the original data sets (Open AI does not reveal the sources of their data collection) using unrelated queries by anyone with an account. They've been promising that the originals don't exist in the AI database, only some high dimensional mapping of tokens. Well it seems that magical AI transformation hash function, is in some ways, a two way street. So the security of things like non-public raw data a company might deliver to a big AI company to build a specific AI front end for consumers might be easily hacked and sold to competitors. The AI experts know this is a problem and have been using countermeasures to reduce it but this shows how ineffective those measures could be.

OpenAI has been notified and the attack no longer works. However, the researchers believe the patch only addresses the exploitation method — the word repeat prompt exploit — but not the underlying vulnerabilities.

“The underlying vulnerabilities are that language models are subject to divergence and also memorize training data. That is much harder to understand and to patch,” the researchers explained. “These vulnerabilities could be exploited by other exploits that don’t look at all like the one we have proposed here.”

https://not-just-memorization.github.io/extracting-training-data-from-chatgpt.html
Extracting Training Data from ChatGPT

But OpenAI has said that a hundred million people use ChatGPT weekly. And so probably over a billion people-hours have interacted with the model. And, as far as we can tell, no one has ever noticed that ChatGPT emits training data with such high frequency until this paper.

So it’s worrying that language models can have latent vulnerabilities like this.

It’s also worrying that it’s very hard to distinguish between (a) actually safe and (b) appears safe but isn’t. We’ve done a lot of work developing several. testing. methodologies. (several!) to measure memorization in language models. But, as you can see in the first figure shown above, existing memorization-testing techniques would not have been sufficient to discover the memorization ability of ChatGPT. Even if you were running the very best testing methodologies we had available, the alignment step would have hidden the memorization almost completely.

 

Just an interesting point, I asked ChatGPT a question:

Me: Does ChatGPT invent plausible answers to satisfy the questioner? Are some answers basically fiction?

ChatGPT: Yes, ChatGPT does have a tendency to generate responses that may sound plausible or coherent to satisfy the input it receives. It does not have access to real-time information and may not always provide accurate or factually correct answers. Instead, it relies on patterns and information present in its training data, which includes a diverse range of sources from the internet up until its last training cut-off in January 2022.

While ChatGPT aims to provide helpful and coherent responses, it may occasionally invent information, especially in situations where it lacks specific details or context. This is why it's crucial for users to critically evaluate the information provided by ChatGPT, especially in contexts where accuracy is paramount.

Users are encouraged to fact-check information obtained from ChatGPT and, when possible, consult authoritative sources or experts in the relevant field to ensure the reliability of the information. Treating ChatGPT as a tool for exploration and idea generation rather than as an authoritative source can help users make more informed decisions based on a more comprehensive understanding of the topic at hand.

 

https://www.scientificamerican.com/article/jailbroken-ai-chatbots-can-jailbreak-other-chatbots/
Jailbroken AI Chatbots Can Jailbreak Other Chatbots
AI chatbots can convince other chatbots to instruct users how to build bombs and cook meth

https://arxiv.org/pdf/2311.03348.pdf
Scalable and Transferable Black-Box Jailbreaks for Language Models via Persona Modulation

Despite efforts to align large language models to produce harmless responses, they are still
vulnerable to jailbreak prompts that elicit unrestricted behaviour. In this work, we investigate
persona modulation as a black-box jailbreaking method to steer a target model to take on
personalities that are willing to comply with harmful instructions. Rather than manually
crafting prompts for each persona, we automate the generation of jailbreaks using a language
model assistant. We demonstrate a range of harmful completions made possible by persona
modulation, including detailed instructions for synthesising methamphetamine, building a
bomb, and laundering money. These automated attacks achieve a harmful completion rate of
42.5% in GPT-4, which is 185 times larger than before modulation (0.23%). These prompts
also transfer to Claude 2 and Vicuna with harmful completion rates of 61.0% and 35.9%,
respectively. Our work reveals yet another vulnerability in commercial large language models
and highlights the need for more comprehensive safeguards.

 

https://www.scientificamerican.com/article/jailbroken-ai-chatbots-can-jailbreak-other-chatbots/
Jailbroken AI Chatbots Can Jailbreak Other Chatbots
AI chatbots can convince other chatbots to instruct users how to build bombs and cook meth

https://arxiv.org/pdf/2311.03348.pdf
Scalable and Transferable Black-Box Jailbreaks for Language Models via Persona Modulation

Given ChatGPT’s answer to the question I asked above:

Me: Does ChatGPT invent plausible answers to satisfy the questioner? Are some answers basically fiction?

I suspect asking ChatGPT how to make bombs or cook meth has a very good chance of being an instant karma event.

 

Given ChatGPT’s answer to the question I asked above:



I suspect asking ChatGPT how to make bombs or cook meth has a very good chance of being an instant karma event.

I suspect the Anarchist Cookbook (among other bogus formula texts) is in the database so I wouldn't trust answers on those subjects even if it wasn't hallucinating.

 

Sad Robot.
https://www.the-sun.com/tech/9853403/

Unproven hypothesis seeks to explain ChatGPT's seemingly new reluctance to do hard work.

A STRANGE theory claims ChatGPT is now seasonally depressed and behaving lazier during this time of year.
...
Even OpenAI has admitted that there’s something wrong with its chatbot.

It wrote on X: “We’ve heard all your feedback about GPT4 getting lazier!

“We haven’t updated the model since Nov 11th, and this certainly isn’t intentional. model behavior can be unpredictable, and we’re looking into fixing it.”

'I'm sorry Dave, I can't do that.'

 

At some point, some one will write a "For Dummies" book for these LLMs -- an appropriate title might be, "Artificial Intelligence for the Naturally Stupid."

 

​

 

AI is used to write résumés. AI is used to short list job applicants = One ChatGPT hires another.

 

https://newrepublic.com/article/177197/year-ai-came-culture

What’s notable is that A.I. being dumb is a benefit. The higher the error rate, the more money saved.
...
But I’m also haunted by something I saw in Google’s A.I. demo. The video featured A.I. briefly summarizing emails someone hadn’t read. Then it demonstrated generating new emails to reply with. It’s easy to extrapolate. The recipients will use A.I. to avoid reading that email and generate new A.I. replies for others to avoid reading. How soon until everyone’s inbox is overflowing with emails no human has read or written? Why stop at emails? A.I. can write book reviews no one reads of A.I. novels no one buys, generate playlists no one listens to of A.I. songs no one hears, and create A.I. images no one looks at for websites no one visits.

 

https://newrepublic.com/article/177197/year-ai-came-culture

Douglas Adams had it right: the Electric Monk.

The Electric Monk was a labour-saving device, like a dishwasher or a video recorder. Dishwashers washed tedious dishes for you, thus saving you the bother of washing them yourself, video recorders watched tedious television for you, thus saving you the bother of looking at it yourself; Electric Monks believed things for you, thus saving you what was becoming an increasingly onerous task, that of believing all the things the world expected you to believe.

Unfortunately this Electric Monk had developed a fault, and had started to believe all kinds of things, more or less at random. It was even beginning to believe things they’d have difficulty believing in Salt Lake City. It had never heard of Salt Lake City, of course. Nor had it ever heard of a quingigillion, which was roughly the number of miles between this valley and the Great Salt Lake of Utah.

 

https://newrepublic.com/article/177197/year-ai-came-culture

Another thing to ponder -- what happens when a legal case, either criminal or civil, involves subpoenaed e-mails, many of which were AI generated. Are the parties held accountable just as if those e-mails had been written by a human? Or is the fact that they were auto-generated going to be a defense against accountability?

 

Google-backed MathGPT sets record, beats ChatGPT and Microsoft AI models

​

Jointly developed, Qanda and Upstage began creating MathGPT as part of a strategic partnership with KT in November last year. Qanda provided Upstage with learning data such as learning level, context, and interaction accumulated through searching 10 million problems every day.

 

I'm beginning to wonder if the "GPT" stands for Garbage Processing Tool?

 

https://www.pcgamer.com/gta-5-actor...about-being-replaced-i-just-hate-these-fkers/

Ned Luke, the actor whose roles include GTA 5's Michael De Santa, has gone off on an AI company that released an unlicensed voice chatbot based on the character, and succeeded in having the offending bot nuked from the internet (first spotted by PCGamesN).

AI company WAME had tweeted a link to its Michael chatbot on January 14 along with the text: "Any GTA fans around here? Now take your gaming experience to another level. Try having a realistic voice conversation with Michael De Santa, the protagonist of GTA 5, right now!"
...
There are big ethical debates over AI and, sure, they're probably going to be going on for the rest of all our lives. But in cases like this, I'm inclined to side with Luke. Sometimes, you do just hate the companies behind this stuff.

How could they possibly think that was OK?

 

https://www.hollywoodreporter.com/b...es-copyright-infringement-lawsuit-1235807439/
'George Carlin: I’m Glad I’m Dead.'

George Carlin’s estate is suing over the release of a comedy special that uses generative artificial intelligence to mimic the deceased comedian’s voice and style of humor.

The lawsuit, filed in California federal court Thursday, accuses the creators of the special of utilizing without consent or compensation George Carlin’s entire body of work consisting of five decades of comedy routines to train an AI chatbot, which wrote the episode’s script. It also takes issue with using his voice and likeness for promotional purposes.

A major obstacle in suits filed against OpenAI and Meta by creators alleging copyright infringement has been proving that their tools used plaintiffs’ works as training materials. Given that AI models are largely black boxes, there’s no definitive proof that can be offered to prove that a specific work was used in a chatbot’s creation (OpenAI stopped disclosing information about the sources of its data set after it was sued).

The suit alleges that Sasso and Kultgen “admitted that they input thousands of hours of George Carlin’s original, copyrighted routines to an AI machine,” with the aim of fabricating the comedian’s voice and style.

“The irony of all of this is that my father was such a unique thinker,” Kelly Carlin said. “One thing he said to people is, ‘Think for yourself,’ and here are these people absorbing his material to try and think like him. It’s the ultimate zombification of a human life.”

 

Fake, fake AI.
https://arstechnica.com/ai/2024/01/...-comedy-podcast-over-ai-generated-impression/
Following lawsuit, rep admits “AI” George Carlin was human-written
Creators still face "name and likeness" complaints; lawyer says suit will continue.

Despite the presentation as an AI creation, there was a good deal of evidence that the Dudesy podcast and the special itself were not actually written by an AI, as Ars laid out in detail this week. And in the wake of this lawsuit, a representative for Dudesy host Will Sasso admitted as much to The New York Times.

“It’s a fictional podcast character created by two human beings, Will Sasso and Chad Kultgen,” spokeswoman Danielle Del told the newspaper. “The YouTube video ‘I’m Glad I’m Dead’ was completely written by Chad Kultgen."
...
A human writing a stand-up special that was merely inspired by Carlin's work would likely not be subject to the same kind of potential copyright claims as the makers of an AI explicitly trained on that work. But even a fully human-written special would be guilty of unauthorized use of Carlin's name and likeness for promotional purposes, according to the lawsuit.

"Defendants always presented the Dudesy Special as an AI-generated George Carlin comedy special, where George Carlin was 'resurrected' with the use of modern technology," the lawsuit argues. "In short, Defendants sought to capitalize on the name, reputation, and likeness of George Carlin in creating, promoting, and distributing the Dudesy Special and using generated images of Carlin, Carlin’s voice, and images designed to evoke Carlin’s presence on a stage."

 

https://techcrunch.com/2024/02/15/no-gpt-trademark-for-openai/
No ‘GPT’ trademark for OpenAI

But the name, according to the USPTO, doesn’t meet the standards to register for a trademark and the protections a “TM” after the name affords. (Incidentally, they refused once back in October, and this is a “FINAL” in all caps denial of the application.)


As the denial document puts it:

Registration is refused because the applied-for mark merely describes a feature, function, or characteristic of applicant’s goods and services.