ChatGPT

nsaspook

Joined Aug 27, 2009
16,330
https://arstechnica.com/tech-policy...ed-write-california-bar-exam-sparking-uproar/

AI secretly helped write California bar exam, sparking uproar
According to the LA Times, the revelation has drawn strong criticism from several legal education experts. "The debacle that was the February 2025 bar exam is worse than we imagined," said Mary Basick, assistant dean of academic skills at the University of California, Irvine School of Law. "I'm almost speechless. Having the questions drafted by non-lawyers using artificial intelligence is just unbelievable."

Katie Moran, an associate professor at the University of San Francisco School of Law who specializes in bar exam preparation, called it "a staggering admission." She pointed out that the same company that drafted AI-generated questions also evaluated and approved them for use on the exam.
Seems they should disbar the bar for using 'AI'.
 
Last edited by a moderator:

WBahn

Joined Mar 31, 2012
32,871
https://arstechnica.com/tech-policy...ed-write-california-bar-exam-sparking-uproar/

AI secretly helped write California bar exam, sparking uproar


Seems they should disbar the bar for using 'AI'.
The real question, in my opinion, is whether or not the questions, individually and taken as a whole, were reasonable and constituted an acceptable bar exam. The source of those questions is really irrelevant (provided no laws or rules were broken, such as plagiarism or reusing questions in ways that aren't allowed).

The same thing for the evaluation and approval process. If Company A is deemed worthy of evaluating and approving for use the questions that it developed, then it doesn't matter how those questions were developed. By the same token, if allowing a company to evaluate and approve questions that it developed using AI is unacceptable, then allowing a company to evaluate and approve questions that it developed is unacceptable regardless of how it developed those questions. If the state wants questions to be evaluated and approved by a different entity than the one that developed them, the require that (and it would seem to me to be a reasonable requirement).
 

MrChips

Joined Oct 2, 2009
34,820
One area of AI I find intriguing and shocking is in job applications and vetting.

When hundreds of applications (many undoubtedly composed by ChatGPT) are received for a job opening, HR does not have the time to review each and every single application.

Hence, presumably they use AI to short list potential candidates.

In effect, you have a ChatGPT program hiring another ChatGPT for the job based on fantasy, no matter what true credentials the applicant might have.
 

WBahn

Joined Mar 31, 2012
32,871
One area of AI I find intriguing and shocking is in job applications and vetting.

When hundreds of applications (many undoubtedly composed by ChatGPT) are received for a job opening, HR does not have the time to review each and every single application.

Hence, presumably they use AI to short list potential candidates.

In effect, you have a ChatGPT program hiring another ChatGPT for the job based on fantasy, no matter what true credentials the applicant might have.
This is an aspect that has myriad variations. Scholarship and university admissions essays that are written by an LLM and then evaluated by an LLM, for instance.

You would hope that, at least to some degree, that there is a sanity step in there somewhere. If companies are hiring people based solely one what some AI says, then something is seriously wrong (the same with all those other areas). But even if that's the case, it's a troubling situation because it puts at a disadvantage the people you should most want to be taking a serious look at, namely the people that are willing and able to do their own work.

And, having said that we should hope that there is a sanity check in place, that's not the same thing as there actually being one, even if policy demands it. People are lazy, so there is going to be the tendency to just accept some AI's recommendations and pencil-whip any review process that is supposed to take place.
 

nsaspook

Joined Aug 27, 2009
16,330
The real question, in my opinion, is whether or not the questions, individually and taken as a whole, were reasonable and constituted an acceptable bar exam. The source of those questions is really irrelevant (provided no laws or rules were broken, such as plagiarism or reusing questions in ways that aren't allowed).

The same thing for the evaluation and approval process. If Company A is deemed worthy of evaluating and approving for use the questions that it developed, then it doesn't matter how those questions were developed. By the same token, if allowing a company to evaluate and approve questions that it developed using AI is unacceptable, then allowing a company to evaluate and approve questions that it developed is unacceptable regardless of how it developed those questions. If the state wants questions to be evaluated and approved by a different entity than the one that developed them, the require that (and it would seem to me to be a reasonable requirement).
I agree in most part (so why hide the fact) when the results are certified by human experts (that should be independent from the developer) in the field. The problem is the process that was used really messed up the actual bar exam to the point many view that test and testing system with having dubious quality.

And the new test is specific to only CA and accepted only in CA instead of the previous National Conference of Bar Examiners' Multistate Bar Examination that could be used in several states after taking one test.
 

WBahn

Joined Mar 31, 2012
32,871
I agree in most part (so why hide the fact) when the results are certified by human experts (that should be independent from the developer) in the field. The problem is the process that was used really messed up the actual bar exam to the point many view that test and testing system with having dubious quality.

And the new test is specific to only CA and accepted only in CA instead of the previous National Conference of Bar Examiners' Multistate Bar Examination that could be used in several states after taking one test.
It sounds to me like most of the issues with the test had nothing to do with whether AI was used to craft questions, but rather that the company they hired just did a really poor job all the way around. The fact that some test takers complained that some of the questions were confusing is neither knew nor unexpected, regardless of where the questions came from.

But I certainly agree that there should have been no reason to hide that some of the questions originated via AI -- though I can't ascertain the degree to which that fact was actively hidden with the intent of hiding it, versus just simply not disclosed as being deemed irrelevant.
 

nsaspook

Joined Aug 27, 2009
16,330
It sounds to me like most of the issues with the test had nothing to do with whether AI was used to craft questions, but rather that the company they hired just did a really poor job all the way around. The fact that some test takers complained that some of the questions were confusing is neither knew nor unexpected, regardless of where the questions came from.

But I certainly agree that there should have been no reason to hide that some of the questions originated via AI -- though I can't ascertain the degree to which that fact was actively hidden with the intent of hiding it, versus just simply not disclosed as being deemed irrelevant.
Lawyers being lawyers.1745459959806.jpeg
 

BobTPH

Joined Jun 5, 2013
11,523
I hate that quote!

It is normally attributed to Shakespeare, with no further explanation, implying that is his belief.

It is actually spoken by “Dick the Butcher”, one of the villains in a plot to overthrow the King and impose tyrannical regime in one of his plays.

Attributing a line from a work of fiction to the author without giving the proper context is at best misleading, and more likely dishonest.

https://lithub.com/what-did-shakespeare-mean-when-he-wrote-lets-kill-all-the-lawyers/#:~:text=Approximately four hundred years after,the crooked, overpriced, counselor.
 

WBahn

Joined Mar 31, 2012
32,871
I hate that quote!

It is normally attributed to Shakespeare, with no further explanation, implying that is his belief.

It is actually spoken by one of the villains in a plot to overthrow the King and impose tyrannical regime in one of his plays.

Attributing a line from a work of fiction to the author without giving the proper context is at best misleading, and more likely dishonest.
I've been hearing Shakespeare quotes all of my life and it has always been understood that it is from one of his works. Sometimes a written quote will indicate which work, but often it is simply attributed to Shakespeare. For instance:

1745483367632.png
The quote (albeit in a different phrasing) is from Julius Caesar and is actually being used by Cassius to move Brutus in the direction of going along with Caesar's assassination. But that's almost never the type of intent that the quote is intended to convey, and it is almost never attributed to a work, but merely to Shakespeare.

In fact, I don't know that I've ever heard a quote that was claimed to be from Shakespeare himself, though I would imagine that they must be some around.
 

nsaspook

Joined Aug 27, 2009
16,330
I hate that quote!

It is normally attributed to Shakespeare, with no further explanation, implying that is his belief.

It is actually spoken by “Dick the Butcher”, one of the villains in a plot to overthrow the King and impose tyrannical regime in one of his plays.

Attributing a line from a work of fiction to the author without giving the proper context is at best misleading, and more likely dishonest.

https://lithub.com/what-did-shakespeare-mean-when-he-wrote-lets-kill-all-the-lawyers/#:~:text=Approximately four hundred years after,the crooked, overpriced, counselor.
I don't hate that quote. To be upset about it seems pretty selective in the world of possibilities of mis-attributions on lines used in the arts and literature over the centuries. Obviously it's not used as a fact of who actually said it but of the work's that used it. It's used as an association of feelings long felt about the law as a human condition of love and hate of those in the business of law as a necessary evil IMO.

1745487121358.jpeg
1745487675357.jpeg
 
Last edited:

Futurist

Joined Apr 8, 2025
764
AI headache? just now, copilot:

1745537832953.png

1745537900510.png

Reminds me of "Open pod bay doors, HAL"..."I'm sorry, Dave, I'm afraid I can't do that."
 
Last edited:

cmartinez

Joined Jan 17, 2007
8,768
Despite all the negative things being said about LLMs, it think it'll definitely be a very useful tool, if used correctly:

The chatbot told Bannon that she may have Hashimoto's disease, an autoimmune condition where the body's immune system mistakenly attacks the thyroid gland, causing it to become inflamed and eventually underactive, according to Kennedy News and Media.

Despite reservations from her doctor, Bannon insisted on being tested for the condition in Sept. 2024 — and was shocked to discover that ChatGPT was correct, despite the absence of any family history.
 

nsaspook

Joined Aug 27, 2009
16,330
Despite all the negative things being said about LLMs, it think it'll definitely be a very useful tool, if used correctly:

I've always said it's a useful tool as a interface for human intelligence. The machine didn't wake her up one night while worried and say, you have cancer. Her intelligence told her something was wrong, so she crafted a question for it. The machine then used human intelligence to craft a response.

The machine represents and displays the expertise of humans in the response, so human experts are more likely to believe it, than a woman without that experience.

She saved her life, not the machine.
 

BobTPH

Joined Jun 5, 2013
11,523
An anecdote is not evidence. I would be astounded if no AI response had uncovered a missed diagnosis. Random guessing would also do so occasionally.
 

nsaspook

Joined Aug 27, 2009
16,330
There is selection bias in the media for these these types of AI stories. It's called click-bait and it's a money maker for ad trackers and cookies.

The modern version of this:
"A woman claims to have caught the ghost of her dead aunt on camera moments after it was 'revealed' her mum had cancer through a Ouija board"

ChatGPT and the massive computer power is posses IMO represents the modern supernatural Oracle to many people. The term Oracle is used in AI research as an all knowing entity (An oracle is a definition for an entity who knows the correct answer to all questions in ML), it's used in mundane things like BLDC motor control theory as the predictor (say the motor will be here in X time, if we use X sets of transformation vectors for commutation calculations) of energy flow in the control system, etc ...
1745584928656.png
If the term and idea is pervasive in science and engineering then it should be shocking to nobody the same applies to the general public when they use these machines.

Just remember this:
In Greek mythology, anyone who contacts the Oracle seems to suffer a terrible fate.
 
Last edited:

nsaspook

Joined Aug 27, 2009
16,330
No understanding, just the next likely set of words, even if the words are psychobabble to a toddler from embedded errors or intentionally placed there by various "actors".

1745597368258.png
Screenshot of a command line program showing the term 'vegetative electron microscopy' being generated by GPT-3.5 (specifically, the model gpt-3.5-turbo-instruct). The top 17 most likely completions of the provided text are 'vegetative electron microscopy', and these suggestions are 2.2 times more likely than the next most likely prediction. (OpenAI)
 
Top