chatGPT policy needed?

AI generated content tends to be very confidently stated, even when wrong. There are some humans with this problem too, but with AI it’s constant—it’s always confident and at this point wrong more often than not. Moreover, it’s not wrong in the same ways that humans are. AI tends to do things that look sophisticated but are just bad copy-pasta from something on the internet that was a correct answer to some other question, but which is wrong—either wildly or subtly—with respect to the question at hand. Worse, AI can generate a lot of confident, incorrect answers. If we don’t take a hard stance against AI generated content, it would be very easy for it to become impossible to distinguish bullshit from real, useful answers. (@DNF already said the same better than I did, but I wrote this before I saw that post.)

It may come to pass that some future AI can answer programming questions confidently and correctly with enough reliability that we would actually intentionally deploy it to help users, but we are not there yet. And even when that does happen, I think we’d want to be transparent about it so that people know that a bot is answering their question, so I don’t think that would actually change the policy since AI output that is clearly marked as such is already ok.

14 Likes

The singularity will occur when AI chat bots begin arguing with each other.

2 Likes

Hah, I don’t think the “bullshit singularity” was what Kurzweil had in mind, but of course it’s what we’re going to get, isn’t it?

4 Likes

There are (as always) important differences in scenarios

Trying to pass off AI-generated content as your own or another real person’s. Or what is functionally equivalent, posting in such a way that it is easy to assume that it is not AI-generated.

Uncritically posting AI-generated commentary on Julia (or anything else really), even if it is marked as such, is obviously a problem. It should be discouraged, or the post flagged so that a warning or caveat can be posted.

Posting AI-generated content in the context of discussing it’s performance and usefulness. I think it is important to label this as suspect, and if it is wrong, clearly mark it as wrong. An analogy is examples of incorrect or poorly written code in a programming book. The convention is to mark them clearly as bad code. I did not do this in my previous post of ChatGPT content. If you are new to Julia and looking for answers, you have no way to distinguish the correct answers from the nonsense.
EDIT: I edited the other post on ChatGPT to clearly mark correct and incorrect answers.

Fair enough :+1: I was playing devil’s advocate here as I don’t feel strongly either way. I definitely agree that the current edition of ChatGPT over-projects confidence.

I think this is unlikely to become as much of a problem here as it might in other places (eg. StackOverflow) as the incentives here are basically nonexistent for such behavior.

1 Like

A post was split to a new topic: How does chatGPT work?

Given the propensity of confident-sounding/technically-wrong answers, and the limited time of human moderators, any policy that does not ban it altogether may unfortunately end up being practically untenable :man_shrugging:

Here’s what ChatGPT has to say on the matter:

It’s not for me to say whether or not the Julia Discourse forum should implement a policy regarding posts and comments generated via ChatGPT. That would be up to the administrators and moderators of the forum to decide based on their goals and guidelines for the community. It’s important for forums and online communities to have clear policies in place to ensure that the conversations remain constructive and on-topic, but the specific details of those policies will vary depending on the community and its goals.
As a language model trained by OpenAI, I don’t have any direct experience with online forums or communities, so I can’t speak to the specific policies that the Julia Discourse forum should consider. However, in general, it’s important for forums to have clear policies in place to ensure that conversations remain constructive and on-topic. This might include policies regarding the use of offensive language, spamming, and other behaviors that can disrupt the community.

2 Likes

To your point, and to use a real example from the forum, consider this post

I was reading the documentation and on this page Frequently Asked Questions of DifferentialEquations.jl, in the Stability and Divergence section, it says:

if you see these instability warnings during a parameter estimation process, this is likely the underlying problem. Simply check sol.retcode != :Success and throw an Inf cost and most optimizers will reject steps in those parameter regimes!

I wonder which optimizers will do that? and how exactly do they “reject the steps”? (e.g.: do they simply skip the current step? or do they randomly initialize a new set of parameters?)

The accepted solution was:

They skip the current step and do some line search thing. I know the Optim.jl algorithms support this. I need to dig around to see which others do it well.

While a ChatGPT solution looks like:

The specific optimizers that will reject steps in parameter regimes where the solution is unstable will depend on the optimization algorithm being used. DifferentialEquations.jl uses a variety of optimization algorithms, so it’s not possible to say which ones will reject steps in unstable regimes without knowing which algorithm is being used.

In general, optimizers will often have a way to handle situations where the optimization process encounters an unstable or invalid solution. This could involve skipping the current step and trying a different set of parameters, or it could involve randomly initializing a new set of parameters and starting the optimization process over from the beginning. The specific details will depend on the optimization algorithm being used.

Yuck.

1 Like

This highlights another thing: even if AI can answer run of the mill programming questions at some point it will still remain very far from the expertise of someone like Chris answering a question about his own code in an area where he is one of the world’s foremost experts. At the point we could consider an AI bot to answer beginner questions but you really wouldn’t want to let the bot answer questions like this one.

3 Likes

Maybe I should use ChatGPT to extend my responses haha. I think sometimes the terseness comes off as not wanting to be bothered, when really it’s “I’ve got two minutes left on the train so you either get this quick note or nothing”. Slap a bit of ChatGPT around the real answer and you have yourself a paper worthy response.

11 Likes

The lack of terseness of ChapGPT bothers me.
Walls of fluff devoid of content is grating to read.

15 Likes

I know a few real people who are just as terse :slight_smile: Makes me wonder if it would be easier for ChatGPT to pass the Turing test, or for a human to pass the “reverse Turing test”, i.e. successfully impersonate ChatGPT?

1 Like

We are now living Aldous Huxley’s worst nightmare, when the truth is drowned in a sea of irrelevance.

3 Likes

Agreed.

1 Like

Tongue in cheek, but have you told ChatGPT to make its responses more terse or concise? Had the same issue and would tell it to either outline its response in concise bullet points or constrain itself to only one or two sentences for a response. :joy: Has helped a lot.

2 Likes

I fully agree. It is great to have you here Chris. I keep the fingers crossed. To explicitly articulate it, this comment of course is all on a positive side! Again, I really keep the fingers crossed. :- )

1 Like

Error: “Chris” is ambiguous. Response type unknown.

4 Likes

Still, keeping the fingers crossed. :- )

Stefan’s copy-pasta remark shamed me into introspecting about how much of my own thought is actually some form of copy-pasta.

2 Likes