Failed CSET world language subtest with a weird result

Discussion in 'Single Subject Tests' started by jyfyyy, Jun 1, 2019.

  1. jyfyyy

    jyfyyy Rookie

    Joined:
    Mar 9, 2019
    Messages:
    7
    Likes Received:
    3

    Jun 1, 2019

    I took CSET world language Chinese Subtest V in May. I got my test result today but it's really weird. There were 5 Constructed Response questions, no Multiple Choice in this test. I got 4 check marks on my 1,2,4,5 question, and an "s" on the 3rd question, but the overall performance was just "++" and I failed it. My subtest score was 212. Does anyone have any idea about this result or do I need to request a test rescore?
     
  2.  
  3. futuremathsprof

    futuremathsprof Aficionado

    Joined:
    Jun 27, 2014
    Messages:
    3,475
    Likes Received:
    937

    Jun 1, 2019

    That doesn’t make sense to me at all. I normally don’t recommend this, but I would request a rescore. Moderator @TeacherGroupie is much more knowledgeable than I am when it comes to teacher certification tests and so I would be interested in hearing her perspective on this.
     
  4. TeacherGroupie

    TeacherGroupie Moderator

    Joined:
    May 13, 2005
    Messages:
    29,431
    Likes Received:
    991

    Jun 1, 2019

    Evidence suggests that the plus marks reflect the numerical scores earned by responses, on a scale from + to ++++. It's plausible, then, that each + represents a quartile: ++++ would be fourth quartile, or having earned 75% or more of available points, + would be first quartile, or having earned 25% or fewer, and ++ would be second quartile, or having earned 26% to 50%. With a score of 212, jyfyyy, you were probably on the high end of the second quartile.

    Let me add here that each CSET constructed response is scored by two scorers; if the two scorers disagree by more than one point, a third scorer is brought in as tie-breaker. The raw score per question is the sum of the closest two scores, on a scale from 0 to 3 for "focused" or shorter responses or 0 to 4 for "extended" or longer responses; for CSET Chinese V, the total raw score would be the sum of the raw scores for all five questions. The raw score is then converted to a scaled score - but the impotant thing is that the numerical score is what determines whether one passes the subtest. And, given the ++ overall, a scaled score below passing makes sense to me.

    Precisely because multiple scorers are involved, it's rather rare that score verification results in a passing score.

    Letters and checkmarks seem to reflect someone's judgment as to WHY the response got the scores it got. Letters correspond to evident shortcomings in a response: p, for instance, generally means that the response failed to answer the question correctly (or fully), and s generally means that a response that should have been supported by evidence or reasoning, wasn't. But getting a checkmark doesn't necessarily mean that one scored perfectly on the question: at minimum, it just indicates that one's response wasn't inadequate enough to merit one or more of the dreaded p, k, or s letters.
     
    Last edited: Jun 1, 2019
  5. futuremathsprof

    futuremathsprof Aficionado

    Joined:
    Jun 27, 2014
    Messages:
    3,475
    Likes Received:
    937

    Jun 1, 2019

    Okay, but four checks and one s out of five equals fail? The test taker didn’t get a p or a k, but just an s and on a single response. And that one s warranted her receiving a non-passing score? It appears as if that one question was weighted much higher than the other ones.

    I just can’t wrap my head around how you theoretically, seemingly, can get five checks out of five (let’s just stay someone’s responses were all minimally adequate for the sake of argument) — and that’s all that’s being graded here — and still fail.

    That, to me, sees like an extremely poor grading metric because, at least to the outside observer, it appears as if they answered each question correctly to the grader’s satisfaction.

    Why can’t they just give a more detailed score report and reveal the percentile so there is no ambiguity?
     
  6. jyfyyy

    jyfyyy Rookie

    Joined:
    Mar 9, 2019
    Messages:
    7
    Likes Received:
    3

    Jun 1, 2019

    For the questions that I got check marks, even if I just met the minimal requirement of the relevant knowledge, I should have gotten a least 2 out of 3 points according to the score point description, which means I probably got at least four 2-points scores with one 1-point for all five questions. Does this still equal to a failed result? I also compared my answer with the 3-point answer of the sample practice and I think I gave an appropriate response with strong and detailed evidence so some of them should not be given only 2 points.
     
  7. jyfyyy

    jyfyyy Rookie

    Joined:
    Mar 9, 2019
    Messages:
    7
    Likes Received:
    3

    Jun 1, 2019

    Hi futuremathsprof, I feel the same way about you. I checked the rubrics on the sample practice, and it said "The "2 points" response reflects a general command of the relevant knowledge and skills as defined in the CSET subject matter requirements for World Language Subtest V. ", which was the exact same definition of getting a check mark on my result report. So I should get at least four 2-points. Does this just equal to ++ (26%-50% correct)??

    I found that it seems nothing will change on most people's rescore cases, so I don't know if I need to request it or retake the test with another $99.
     
    futuremathsprof likes this.
  8. TeacherGroupie

    TeacherGroupie Moderator

    Joined:
    May 13, 2005
    Messages:
    29,431
    Likes Received:
    991

    Jun 2, 2019

    To quote the Oracular Slug from Ursula Vernon's wonderful graphic novel/epic Digger (the episode starts at https://diggercomic.com/blog/2007/04/28/digger-87/, but the relevant quotation is in the last frame of https://diggercomic.com/blog/2007/05/05/digger-81/): "Don't salt the messenger." I'm doing my best to be accurate about the system, but clarity shouldn't be taken for advocacy: this system is what it is.

    You both seem to be assuming that the diagnostics drive the passing. As I've already noted, that's almost certainly not so. Not only is it possible not to pass with just one letter and four checkmarks out of five: it is possible to get nothing but checkmarks in a constructed-response-only subtest and not pass, as attested by more than one complainant concerning CSET English Subtest IV (four focused responses, each scored on a scale from 0 to 3).

    Scorers score holistically, just as they do with CBEST - I visualize this as a scorer surrounded by piles of paper marked 0, 1, 2, and 3, with a response in hand poised to drop on one of the piles. Let me also repeat that there are two scorers per response. If each scorer can give up to three points, the total score per response is thus 6, not 3, and a total score per response of 3 (2+1) is thus both possible and plausible. The diagnostician could split the difference or align with one or the other of the scorers; that being the case, a checkmark could well be the diagnostic result. Let me also repeat that the plus marks almost certainly reflect raw scores (that is, what the scorers assign): the people who scored those five constructed responses could award up to 30 raw points (3 points per response times 2 scorers times 5 constructed responses), but ++ means that they see fit to award fewer than half.
     
  9. futuremathsprof

    futuremathsprof Aficionado

    Joined:
    Jun 27, 2014
    Messages:
    3,475
    Likes Received:
    937

    Jun 3, 2019

    First off, beautifully articulated. Secondly, I want to apologize if my earlier responses appeared as if I was “shooting the messenger.” I was merely commenting on how unfair it is for graders to give the “highest” markers possible on a response, as there is no symbol that is superior to the checkmark. It is just like there is no symbol higher than ++++ for any one section cumulatively.

    To the test taker, who has no idea of how each section was graded, getting a ++++ seems as if they answered the majority of their responses correctly and very closely in accordance with the grading rubric. However, I think where OP and I are frustrated is where graders can give someone marks that appear as if they are high marks when in actual fact the opposite is true. I mean, what is the purpose of giving five check marks out of five if that is a fail?

    The point here is that there should be no subjectivity or as little subjectivity as possible where grading is concerned, especially on a standardized test. A +++ or ++++ should correspond to a pass and a + or ++ should correspond to a fail, IMO, on a constructed-response-only subtest, in concert with mostly checks and very few or no letters given. Five checkmarks out of five should NOT correspond to a fail because, visually, it doesn’t make any sense. Why would you utilize the highest marks possible to denote a fail?

    Grading should be very cut and dry and objective as humanly possible. That’s why we are frustrated. I’m sick of subjectivity when it comes to measurable standards and objectives — after all, objective is kind of in the name. They are not called measurable subjectives.
     
  10. TeacherGroupie

    TeacherGroupie Moderator

    Joined:
    May 13, 2005
    Messages:
    29,431
    Likes Received:
    991

    Jun 4, 2019

    You're still insisting that checkmarks and pluses MUST be giving exactly and only the same information. If they do and must, though, there'd really be no need to have both. And it's not as though grades aren't relative: the performance in prealgebra that gets an A doesn't look much like A work in AP Calculus B. In the context of ++, checkmarks across the board can't possibly mean top-notch responses, but what that combination of pluses and checkmarks tells me is that the responses are strong considered as ++ responses: that is, good as far as they go, but the test taker should have pushed them rather farther - deeper, broader, or both - than she did. In the context of +++ and not passing (and, yes, that can occur), checkmarks across the board mean the test taker is closer to the mark but still needs to do more.
     
  11. futuremathsprof

    futuremathsprof Aficionado

    Joined:
    Jun 27, 2014
    Messages:
    3,475
    Likes Received:
    937

    Jun 4, 2019

    This is a fair statement, but why not use a p, k, or s instead of a ✔️ if more work is needed? How else would the test taker know which response needed more work? Sort of how some grading rubrics give students a ✔️-, ✔️, or ✔️+. Here, a ✔️+ would correspond to a complete or at least a substantial response and the student would receive full marks or nearly full marks. If I just put a ✔️ without indicating where the deficiency is or numerically that the student had an incomplete response and give that student an F, would that seem logical to you?

    Likewise, it would make more sense if a +++ or ++++ corresponded with some combination of at least 3/4 ✔️, ✔️, ✔️ and one/two letter(s) OR ✔️, ✔️, ✔️, ✔️, ✔️, ✔️; respectively. That would seem like the way to go.
     
  12. TeacherGroupie

    TeacherGroupie Moderator

    Joined:
    May 13, 2005
    Messages:
    29,431
    Likes Received:
    991

    Jun 6, 2019

    p,k,s - I intend here all three letters applied to a single response - is in use to indicate that the response largely or entirely fails to get and/or deal with the point of the question, largely or entirely fails to show off the test taker's knowledge of the field and its terminologies, assumptions and modes of discourse, and largely or entirely fails to support its conclusions with appropriate reasoning and evidence. Your proposal, followed rigorously, would have given the OP four p,k,s indicators along with the s, thus inaccurately implying that the s response was the most successful of the lot. In addition, it assumes that there's no interesting difference between slightly below standard and massively below standard, or at least that it's unuseful for a test taker to know which of those conditions is responsible for a non-passing result on a test.
     

Share This Page

Members Online Now

  1. Ms.Holyoke,
  2. K1teach,
  3. evvv,
  4. dr.gator,
  5. Kelster95,
  6. TeacherNY,
  7. jadorelafrance
Total: 322 (members: 10, guests: 270, robots: 42)
test