Thread Scoring: Metrical QC for AAC Discussions

Thread Starter

Yaakov

Joined Jan 27, 2019
3,565
So I have a fanciful, ¾-formed idea about quantitative ranking of thread quality.

It’s not “objective“, because some of it requires opinion of an expert. It’s also not entire subjective because the heuristics are constrained. If it works, it will produce a numerical score for a thread. Right now, it’s an open-ended score but it could possibly be scaled to thread length or something. I also allow the score to go negative because… some threads are.

This is just a fun Gedankenexperiment, don’t take it too seriously.

So here are some of the proposed rules for scoring. This isn’t intended to be an exhaustive list, just some musings. The idea (for me) is to think about what I value in a thread and have some idea what to include and avoid, with weighting. I think most of us can get any of the scores on the list from day-to-day, positive and negative and I want to work out how to make my participation better. This is just a sort of brainstorming about that.

Do you have ideas?

RuleValue
TS post is written in a way that an expert can solve the problem based on just what is provided.10
TS post is written in a way that an expert can ask the right questions directly to solve it based on what is provided.10
Response welcomes new member to AAC.2
Response solves the problem for the TS without further interaction.10
Response asks clear questions which eventually lead to a solution.3
Response provides clear answers that would assist an expert to solve the problem.3
Response warns of genuine safety issues.4
Response warns of genuine regulatory issues.3
Response (not TS) provides information to helping expert that assists in solving the problem.5
Response thanks members for help.5
Response provides useful but tangentially related Information.2
Response makes people laugh (not at someone’s expense) without interrupting help.3
Response (from sidelines) confuses the TS and disrupts the help.-3
Response demonstrates poster did not read the TS post carefully enough to see all the information provided.-3
Response is a derogatory comment about the TS.-5
Response is a derogatory comment about another member.-5
Response is tangent into a non-working solution.-4
Response is an unhelpful criticism of the method being discussed, even though the current thread is on target to a solution.-3
 

atferrari

Joined Jan 6, 2004
4,438
From the many and varied forums I participate since long time, I have seen many, different "evaluation" systems. Call them, badges, stars, trophies, medals, or just points. In most of them, one or more items seem to become the way to implement silent personal wars triggered by who knows what.

There was a bunch of many sites (surprisingly, most dedicated to programming) frequented by a majority from a country in Asia where their obsession to collect points was kind of pathetic. Even they openly asked in their posts things like "if you found this post useful, click..."

Nothing of that I would like to see here.
 

Papabravo

Joined Feb 24, 2006
16,935
Two unrelated ideas:

1. What would be the point of scoring threads?

2. Whatever the reason I think this would have more unintended consequences than perceived benefits.
1. I don't think there is one. People seldom come here to be judged, and in fact I suspect more than a few have left because of it.

2. I'm at a loss to see any benefits.
 

Thread Starter

Yaakov

Joined Jan 27, 2019
3,565
So I have a fanciful, ¾-formed idea about quantitative ranking of thread quality.

It’s not “objective“, because some of it requires opinion of an expert. It’s also not entire subjective because the heuristics are constrained. If it works, it will produce a numerical score for a thread. Right now, it’s an open-ended score but it could possibly be scaled to thread length or something. I also allow the score to go negative because… some threads are.

This is just a fun Gedankenexperiment, don’t take it too seriously.

So here are some of the proposed rules for scoring. This isn’t intended to be an exhaustive list, just some musings. The idea (for me) is to think about what I value in a thread and have some idea what to include and avoid, with weighting. I think most of us can get any of the scores on the list from day-to-day, positive and negative and I want to work out how to make my participation better. This is just a sort of brainstorming about that.

Do you have ideas?

RuleValue
TS post is written in a way that an expert can solve the problem based on just what is provided.10
TS post is written in a way that an expert can ask the right questions directly to solve it based on what is provided.10
Response welcomes new member to AAC.2
Response solves the problem for the TS without further interaction.10
Response asks clear questions which eventually lead to a solution.3
Response provides clear answers that would assist an expert to solve the problem.3
Response warns of genuine safety issues.4
Response warns of genuine regulatory issues.3
Response (not TS) provides information to helping expert that assists in solving the problem.5
Response thanks members for help.5
Response provides useful but tangentially related Information.2
Response makes people laugh (not at someone’s expense) without interrupting help.3
Response (from sidelines) confuses the TS and disrupts the help.-3
Response demonstrates poster did not read the TS post carefully enough to see all the information provided.-3
Response is a derogatory comment about the TS.-5
Response is a derogatory comment about another member.-5
Response is tangent into a non-working solution.-4
Response is an unhelpful criticism of the method being discussed, even though the current thread is on target to a solution.-3
Please re-read the description of why I came up with this. It is for self-improvement, not for judging others.
 

Thread Starter

Yaakov

Joined Jan 27, 2019
3,565
1. I don't think there is one. People seldom come here to be judged, and in fact I suspect more than a few have left because of it.

2. I'm at a loss to see any benefits.
This was for self-evaluation, as I said in the introduction. It was not intended to be applied to anyone who doesn't do it for themselves.
 

MrSalts

Joined Apr 2, 2020
598
This sounds like a quality department trying to solve a problem that only exists in the quality departments mind. A make-work-program to justify their existence. It does not get anyone an answer faster. It assumes a single answer is the best way (even though not all people understand explanations in the same way). Your scoring system also seems to have a goal of a minimum of answers for each thread. Not likely the goal of the site owner! SEO and advertising rates depend on activity, not efficiency.

Also, who cares if a TS provides only half of the useful information in the first post? They are not experts - that is why they are posting. Not only doesn't the TS know the answer to the question they are asking, they don't even knowhow much they don't know. They come with certain assumptions (sometimes wrong assumptions) about how physics works. The whole exchange is better for ACC's activity and new posts to drive SEO scores and keep ACC at the top of the Google results vs other forums.

also, what is the quality system measuring or trying to improve? If you are trying to do social engineering of how you want people to behave instead of getting an answer to people, then I question your motives - it seems a lot like the Chinese government's Social Credit System (https://en.m.wikipedia.org/wiki/Social_Credit_System).

The goal of the site is to address TS question (from the TS point of view) and the goal of the site is to drive internet traffic to the site to keep ad rates and revenue high (from the ACC site owner's point of view). I don't see your scoring system achieving these goals. The only goal achieved is to fulfill some latent need to measure how much better one is than other members on the site. Faux competitive nonsense.
 

MrSalts

Joined Apr 2, 2020
598
Please re-read the description of why I came up with this. It is for self-improvement, not for judging others.
That is the appearance on the surface but how do you improve yourself when
- the improvement measurements are nonsense and don't drive behavior to the goals of the site owner
- without comparing your number to others' numbers.
 

strantor

Joined Oct 3, 2010
5,660
I explained my motivation in the original post.
You refer to this?
The idea (for me) is to think about what I value in a thread and have some idea what to include and avoid, with weighting. I think most of us can get any of the scores on the list from day-to-day, positive and negative and I want to work out how to make my participation better. This is just a sort of brainstorming about that.
When I read this the first time I may not have paid a due amount of attention to that. I assumed this would be like other ranking/scoring systems on other forums, and publicly visible. But maybe that is a bad assumption, as you indicate it is for personal evaluation of your own contributions, it would only be visible to you.

But one initial assumption I made that I think is still valid: this feedback comes from others, right? It's peer review, others grading your work? I think it would have to be, in the interest of internal improvement, from an external source. Otherwise it would be undermined by Dunning Kreuger and a waste of time.

So assuming it is private feedback from others, I think it would suffer the same fate as online product reviews- that [phenomenon I don't know the name for] where feedback is unduly skewed to the negative because most people don't take the time to leave a positive review when something meets expectations, they just move on. Most people only leave a review when they're pissed off.

And if people did actually get with your program, they would be spending a lot of time filling out these score cards, which time they wouldn't be spending answering questions.

I assume it's honest feedback you want on your proposed idea, so here's mine: I wouldn't participate. I don't have a lot of time to spend here, and that which I have, I would not spend grading papers.
 

MrSalts

Joined Apr 2, 2020
598
Let's grade this thread...

RuleValueGrade
TS post is written in a way that an expert can solve the problem based on just what is provided.100 - too many follow up questions TS claims he was misunderstood
TS post is written in a way that an expert can ask the right questions directly to solve it based on what is provided.100 - why would the expert need to ask questions? This line item is pretty weirdly written and unclear.
Response welcomes new member to AAC.2N/A
Response solves the problem for the TS without further interaction.100 - obviously not
Response asks clear questions which eventually lead to a solution.30 - obviously not the case
Response provides clear answers that would assist an expert to solve the problem.3
Response warns of genuine safety issues.4
Response warns of genuine regulatory issues.3
Response (not TS) provides information to helping expert that assists in solving the problem.5
Response thanks members for help.5
Response provides useful but tangentially related Information.2
Response makes people laugh (not at someone’s expense) without interrupting help.3
Response (from sidelines) confuses the TS and disrupts the help.-3
Response demonstrates poster did not read the TS post carefully enough to see all the information provided.-3
Response is a derogatory comment about the TS.-5
Response is a derogatory comment about another member.-5
Response is tangent into a non-working solution.-4
Response is an unhelpful criticism of the method being discussed, even though the current thread is on target to a solution.-3
[/QUOTE]
 

Thread Starter

Yaakov

Joined Jan 27, 2019
3,565
You refer to this?

When I read this the first time I may not have paid a due amount of attention to that. I assumed this would be like other ranking/scoring systems on other forums, and publicly visible. But maybe that is a bad assumption, as you indicate it is for personal evaluation of your own contributions, it would only be visible to you.

But one initial assumption I made that I think is still valid: this feedback comes from others, right? It's peer review, others grading your work? I think it would have to be, in the interest of internal improvement, from an external source. Otherwise it would be undermined by Dunning Kreuger and a waste of time.

So assuming it is private feedback from others, I think it would suffer the same fate as online product reviews- that [phenomenon I don't know the name for] where feedback is unduly skewed to the negative because most people don't take the time to leave a positive review when something meets expectations, they just move on. Most people only leave a review when they're pissed off.

And if people did actually get with your program, they would be spending a lot of time filling out these score cards, which time they wouldn't be spending answering questions.

I assume it's honest feedback you want on your proposed idea, so here's mine: I wouldn't participate. I don't have a lot of time to spend here, and that which I have, I would not spend grading papers.
This is really about self evaluation. I realized that while I had a nebulous idea of what makes a good thread, I couldn't qualify it well. By quantifying it, I am hoping to be able to look at my participation here through the lens of my aspirations.

Of course I would like others to have the same idea of what is "good", and I think that despite inevitable arguments around boundaries there would be a reasonable consensus on what makes a good thread. That's why I posted this rather than just making notes to myself.

It's not a proposal intended to be implemented anywhere but in the minds and habits of regulars. It's a sort of self evaluation framework and a codification of certain things I assume others value as I do. But I can't know that, so I thought people might add some ideas or critique the rules and weightings I produced.

There is no practical way to make this useful as a feature of the site, or to expect people to "fill out score cards". I just like this place and want to contribute as best as I can to making it a good place to get help and a community of people that like each other (most of the time).

Thank you for your thoughts.
 

Thread Starter

Yaakov

Joined Jan 27, 2019
3,565
This site needs more structure, etiquette and protocols...not another ranking system.
It's not a ranking system for the site, it's an evaluation system for self assessment—for anyone who might want to use it. It is also a way to see the aspirational culture of the community. You are right that most scoring systems are not very useful for a variety of reasons.
 

MrSalts

Joined Apr 2, 2020
598
It's not a ranking system for the site, it's an evaluation system for self assessment—for anyone who might want to use it. It is also a way to see the aspirational culture of the community. You are right that most scoring systems are not very useful for a variety of reasons.
And the overwhelming consensus is, this thread should be ranked poorly.
 
I totally get this point, as I thoroughly read another of Yaakov’s threads. The one about those “hard of learning.”

To me at least, I find this a helpful PERSONAL tool to determine whether or not take the time and effort to answer a thread, and/or risk a confrontation with a stranger.

I am Personally doing this already, but I find Yakov’s list much better structured than my train of thought.

So Yaakov, if you find this list useful to you, by all means use it.
Godspeed to you.
 
Top