Introduction
History and Context
Starting circa 2015 the section sizes for the programming courses (at the time 1574,2574, and 3574) rose to the point that instructors could not give timely feedback to students. We required many graders to manually compile and run students submissions, grading the output. Most instructors provided ad-hoc grading scripts to do this to ensure consistency among the graders, but it still was common only to grade a final submission from students to any given assignment. This lead to much frustration as students often misinterpret (or do not read closely) specifications and would have points deducted, requesting a regrade. The person hours required to do all this grading was also limiting the number of assignments and their complexity we could give. Since we already had the grading scripts, the obvious next step was to automate this. Dave McPherson and Chris Wyatt had been using the WebCAT auto-grader that CS uses (used) in 1574 since about 2012, but it was really targeted at Java submissions and support was largely at the generosity of Stephen Edwards in CS. We, Chris in particular, wanted something more flexible and under our control.
So, Chris set up the first grader on an old machine in his office based on INGinious and used it in one section of 2574 in 2017. It worked reasonably well but the old desktop was not up to the number of students, so he moved it to a larger old machine from his lab, and started using it in 2574 and then in 3574. Eventually other instructors wanted to use it, and we ran up against resource limits and aging/rickety hardware, so Paul Plassmann convinced the department purchased three new machines for a new version of the grader in 2018. Chis set this up with the help of John Gra, and with only a few issues related to VT LDAP authentication, it has graded many hundred of thousand of student submissions since. It is now used in the revised software sequence 2514, 3514, and 3574.
In 2023 the version of INGinious was updated, as well as the host OS and grading containers. To help onboard new instructors and in case Chris was hit by a bus, this documentation was created.
Overview of the grader
The INGinious grader is very flexible. On a task submission by the student it starts, on a backend machine, a Docker container you specify, with a copy of the task directory contents as the current working directory and executes a file called run, either a shell or python script. An API is provided to get input from the grader task submission as well as provide output messages containing feedback, a numerical grade, and a pass/fail indicator back to the grader. What happens in between is entirely up to whomever is writing the grading scripts. Since it is based on Docker it is easy to setup just about any compiler, interpreter, library, etc. that will run on Linux.