Feature Wiki
Tabs
Anonymous Peer Feedback
1 Requirements
Even for small class sizes, random / anonymous peer feedback is a worthwhile effort, since it might improved the degree of honesty and critical, constructive contributions (honesty can sometimes collide with being gentle).
Didactically, users have to complete a workflow which looks somewhat like that:
- Hand in a solution to an excercise, ideally direct text input in to excercise item (see Matthias Kunkel's feature request Text Area for Solution )
- the the user receives the anonymous, random solutions of N (i.e. 2-4) of his peers in the course and gives feedback / "grades" to their work - again, ideally directly in a text area in the excercise item
- after he has handed in N feedbacks, and only after taht, he receives the feedback to his own solution from N randomly selected anonymous peers from his course
What we need isthe following:
- a Text Area for Solution in the excercise in order to be able to enter a solution directly into the excersice
- a randomised, anonymous distribution of of this solution to N members of the course
- a check whether a user has completed the N peers feedbacks he/she has to give
- a way to return the anonymous feedbacks from his/her peers to the user who has handed in his/her solution
2 Status
- Scheduled for: Release 4.4
- Funding: Funded by Universität Freiburg
- Development: Feature is to be developed by Leifos
3 Additional Information
- If you want to know more about this feature, its implementation or funding, please contact: Marko Glaubitz / Uni Freiburg / marko.glaubitz@rz.uni-freiburg.de
4 Discussion
JF 15 Apr 2013: We appreciate this request and schedule it for 4.4. The text editing component will be discussed on the Text Area for Solution page. One question: Would a simple text area be enough for the "grading"? Or would the ILIAS rating feature (five star) be appropriate? Or a real grading scheme needed?
07.06. 2013, Marko Glaubitz: Regarding the grading: the user should have the chance to give a written feedback which can be plain text, although a tinymce might be better. Being able to give a simple numerical grade (whatever the scale 1-3, 1-5, ..) or a "starred" rating in addition would be great from a didactic point of view.
24.06.2013, Tobias Welz, HfT-Leipzig: to maximize the possible types of exercices and types of peer-feedback:
Our university believes, there will be a much higher benefit, if this feature is opened for: "file"-based exercise-solutions and "file"-based peer-feedback.
e.g informatics: an programming-project isn't done in a text snippet, very often it is an archive of many files
e.g. mathmatics: the calculation of a solution is as much important as the solution itself, especially if the solution is wrong. The calculation way is possible to draw in a whiteboard-file like Microsoft OneNote, but a ilias-text-field would be not enough.
e.g. economy: there are a lot of exercises, where you have to draw something i.e. process management. A good tool would be powerpoint or a mindmap-tool like freemind. As container of the information we need a "file".
Is there a possibility to supplement this feature according to my description? How could our university help to get it sheduled for Ilias 4.4?
last note: I found this feature with the "search function". until this moment this festure is not listed in the feature wiki of Ilias 4.4.
http://www.ilias.de/docu/goto.php?target=wiki_1357_Release_4.4
Marko Glaubitz, Uni Freiburg:
Thank you Micheal for the very good implementation so far and thank you Hansjörg for your suggestions. We are currently assessing the state of the feature so far.
Right now, we have one thing that we'd like see changed: the "Commentary" text filed should be implemented as a TinyMCE as well.
09.08.2013, Tobias Welz, HfT-Leipzig:
Thank you for this implementation, i tested it and have some hints:
- peer-feedback-functionality as a simple Text-Commentary will probable be not sufficient in pratice
better would be TinyMCE to give structured feedback,
but the best possibility would still be annotations of file based feedback to show complex things (i mentioned above)
- The configuration of a peer review exercise is not robust enough against mistakes of the administrating Ilias-User. You can still change adjustments when the exercise is running / ended. So there could be the error message "no peer found" for some users.
- i think there is a real bug: It is not possible to get Feedback, if there is configured "Minimum number of feedback given required to see personal feedback" bigger then "1". That means, if it is configured with e.g. 2 and you gave 2 feedbacks, there is still no "Show Received Feedback"-button. Tested in a scenario with 6 Ilias Users / 6 papers.
19.08.2013, marko Glaubitz (Uni Freiburg)
I have some points that I would like to have changed in order for the anonymous user feedback function to reach maximum didactical potency.
Period for giving feedback needs "Ending time"
In order for the anonymous peer feedback to work, the period for giving feedback must be limitable. Since, all other functions / modi of the excercise terminate at some point, users will "expect" this behaviour. Furthermore: there is nothing more effective for getting things done than deadlines... :)
Usability of the actual feedback should be improved
Right now, I cannot look at the solution of another user while writing my feedback.
Thus, the table that shows the solutions that a user is supposed to evalutate should only do exactly that, ideally show a preview (first lines of the solution) and show the first part of the feedback that has already been entered, where right now the "Kommentar" text box is located.
Giving feedback (and rating for that matter) in a TinyMCE (thanks Tobias for pointing that out in your posting above) should be put on the same page that displays the actual text that is being evaluated.
These changes will make the feedback process much easier and clearer for course members, as the table will then only work as a read-only overview screen, while work (giving feedback) is being done on another page. As a side effect, users have to look at the actual solution before giving feedback, right now, I can give feedback without having had a single glance at the actual solution of my peer.
Notification on missing feedbacks to peers and notification that a new feedback from peers
The excersice should be able to send a reminder after the the hand-in time has elapsed telling the users that the have to give x feedbacks.
Besides the amount of obligatory feedbacks (a counter, as it were) should be displayed close to the overview table to indicated how many peer solutions have not been given feedback to. This solves another problem along the way: if the number of feedbacks per user does not divide smoothly with the number of available solutions, showing the amount of feedbacks needed would be an elegant way of showing the exceptions (is 4 feedbacks have to be given with , some/one user might not have enough peers to feedback, so he could onyl feedback 3 peers) - but that depends on how intelligent to distribution algorithm is (see below)
In case, a users gives feedback, but has not received feedback to his/her solution yet, because the peers have not handed it in yet, the user should receive a notifiation mail telling him/her that a new feedback is available. Otherwise, the motivation for giving and receiving feedback might founder because is is simply not noticed.
Random solution distribution not robust enough
Furthermore, I fully support Tobias Welz recommendations and comments. I can confirm the bugs that he has found. Especially, that random solution distribution process is not robust, since impossible combinations can be set and lead to errors, for example: 3 feedbacks to give with 3 users will lead to problems. The algorithm should check whether the number of feedback checks out numerically with the number of possible users. I think limitig the maximum number of feedbacks to {1,2,3,4} might be a feasible limitation to keep the math simple and giving feedback to more than 4 peers (regularly) is practically overkill anyway. I think I have found a good algorithm, which is quite lengthy to describe as text. I don't know how you do it now, but i think shuffeling the list of peers once and permutating the numbers for every round of feedback afterwards would do for a robust algorithm, which would even allow for modifying the number of reviews to give after the process has started.
p{i} is the shuffled array of peers, shuffle only once for every anon. peer feedback excercise unit, let 1 < i < n with n = #peers, let k < n be the number of reviews to write
example: n = 4, k = 3
p1 reviews p2 + p3 + p4
p2 reviews p3 + p4 + p1
p3 reviews p4 + p1 + p2
p4 reviews p1 + p2 + p3
JL 20 Aug 2013: I just fixed the bug with the minimum number of received feedback needed, it should work now as intended for values > 1. I cannot validate that number against the "maximum number of exercise participants", as this number can change at any moment up to the deadline. Regarding the distribution: I currently fail to see the weakness of the current approach and AFAIK it was implemented exactly as proposed above. The given mininum setting is compared to all participants (-1) and the minimum of both is used as basis for the distribution.
Please be aware that the peer feedback feature is available for all types of assignment. The text assignment is the only one where the "solution" is shown inside the browser (e.g. in give feedback list), all other types of assignments need to be downloaded.
Feedback deadline is now available. The text assignment display screen (coming from give feedback list) now has rating and comment, too. We decided against making the give feedback list read only as it would be a step back for all download-based assignment types.
"File-based peer feedback" is currently out of scope and would be an additional feature request (for 4.5+). Using an RTE (e.g. TinyMCE) for the feedback comments will be discussed during the next Jour Fixe.
Thank you Jörg for putting feedback entry "form" on the same page as the solution. That makes a huge difference. Is there really not way of putting a "number of feedbacks to give" above the overview table after the hand-in deadline has been reached? By then ILIAS knows how many users have submitted a solution and how many solutions have already been worked on (either by rating or by typing in some comment).
I have got another conceptual point and some small usability / UI remarks.
Final submission of feedback / "draft mode"
Right now, giving and receiving feedback is working fine. However, I can still edit the feedback that I have given after I have seen the feedback that I have received. I think, from a didactical point of view, this is not optimal, for example, if a user has got negative feedback him/herself and wants to pay something back from this.
My suggestion would be to add a "Submit Feedback" button beneath the overview table which sets the feedback that a user has given tom read-only and makes the received feedback visible. In a nutshell, feedback is still in a "draft mode" and only visible for the user who is giving it until he/she submits it for good. Another advantage of this would be that the instructor would only then (after the final submission) be abel to view the user feedback which might take some pressure for the feedbackers away.
UI / usability remarks
- After hitting "Save" in the overview table there is not link or button back to the main excercise screen. Users will mostly want to see whether they have already received feedback.
- Wording: "Comment" ("Kommentar") is IMHO not the best wording: what about: "My Feedback"
- "Comment" text box on the "solution page" is rather small. I would suggest to make it as broad as the solution text above. I have tried this with a long solution (1000 words) and then the current text box looks rather tiny :)
- the default value of "Minimum number of feedback given required to see personal feedback" on the excercise item edit screen should be set to 2 or 3.
JL 21 Aug 2013: "Number of feedbacks to give" has been added. "Comment" renamed to "My feedback", min number default value has been changed. As a more simple solution the peer review deadline now also controls the access to the received feedback. When a deadline is set the received feedback will only be accessible from then on (with sufficient given feedbacks). The final-/draft-mode would make this feature even more complicated and to me the exercise settings are complex enough already.
Btw, the "cancel" button on the overview list will bring you back to the assignment (accordion) list.
Again, Jörg, thanks for implementing so many of our wishes instantly. Great. I think, as a start and to keep complexity down, we are happy with the deadline being the trigger for releasing the feedbacks to the students.
I have discussed the current state of the feature with some co-workers from my team and we have come up with some additional ideas:
Optional Anonymity
We have tried out the feature with a small number of people and have notice that it is a great tool even for very small groups. However, for small groups anonymity is not really a factor - on the contrary, for small groups (n<20) knowing whom to feedback is being seen as an advantage in my team. So we wondered, whether it might be possible to make this configurable, since the ILIAS saves who gave feedback to whom either way right? Course admins could then choose whether to use it as an anonymous feedback tool or not, which would definitely broaden the potential application range.
Instructions / Instruction File for Feedback
Another thing, that we found missing was that course admins can upload an instructions file or type in instructions for the assignment, but not for the feedback. Since most students have never usered such a tool before and since instructors, who know this method, might have a pretty clear idea of how the feedback is supposed to be given and what it should contain, this might be a good idea.
Instructor can see feedback before deadline
We found, that it is not quite fair that the instructor can see the given feedback before the deadline has been reached. Students might feel "betrayed" if they find out that the safety they assumed is not given, in that unfinished feedback for or by themselves is visible to the person who gives them their final grade.
Small issues:
- When a peer assignment is created the label above the number of feedbacks to give reads "Mindestanzahl" / Minimum number...". This might be misleading, since it is acutally THE numbers of feedbacks to give if there are enough student hand-ins. So, why not change it to "Anzahl" / "Number"
- About getting back from the overview table: the "Cancel" button is not an option, since it suggests that changes won't be saved. Even if I had pressed save before, I would fear that hitting "Cancel" might undo my changes. Would it be possible to implement another save-button like "Save and Return"
JL 22 Aug 2013: I added a "back"-link to "Show Text", "Give Feedback" and "Received Feedback". "Minimum" has been removed.
"Instructions / Instruction File for Feedback" is an additional feature (see above), but should those instructions (of a core feature) not be a part of the online help? "Optional Anonymity" needs to be discussed in the Jour Fixe, this could lead to all kinds of privacy issues. "Instructor can see feedback before deadline" seems like a non-issue for me as ILIAS administrators are normally not bound to deadlines and limitations of "normal" users.
Marko Glaubitz, 22.08.2013
Regarding the work instructions:
These instruction are highly specific as they vary with the lecturer, the subject (compare feedback to a literary essay with feedback to a language class). Therefore lecturers need the opportunity to give work instructions either as text or as a file (just like with the assigment text / work instructions).
Instructor Feedback
If we can't change the fact that course admins can see premature feedback, would it then be possible to implement a function for them to enter their feedback through this feature. Right now, they can only read what is going on, but they cannot intervene if something goes wrong (by which I don't mean that they should be able to alter existing feedback, but simply add their own as a marked "instructor feedback") and they cannnot add their comments / feedback even if nothing bad happend.
JF 2 Sep: Giving feedbacks as files and optional anonymization should be postponed to 4.5. Filenames should be anonymized using the ID as well (if this is not already the case) "Assignment_Titel_ID.suffix".
Since we would like to keep the possibility to edit all feedbacks on one screen we object the TinyMCE use. Having multiple TinyMCEs in one table would be a mess.
For the future we see the need to have a simple formatting text area (especially for cases where learners edit text) in different object types and would like to have a solution for this. Matthias will start a new feature request Basic Rich Text Editing.
5 Implementation
JL 16 Jul 2013: This feature has been implemented for 4.4/trunk.
Last edited: 28. May 2024, 10:09, Kunkel, Matthias [mkunkel]