Creating your own tester framework#

DO NOT DO THIS, UNLESS YOU KNOW WHAT YOU ARE DOING!

Testers are passed a set of run arguments that govern how it runs.

These are in order, for a compiled in test:

[OUTPUT_PATH] [SKIP_NUMBER] [SECRET_LOCATION]

And for an executable or Python test:

[TESTED_FILE] [OUTPUT_PATH] [SKIP_NUMBER] [SECRET_LOCATION]

There are 2 crucial concepts that the tester must implement: skipping and validation. - skipping is the ability to stich togeter results from multiple runs, if a question fails. To do this the tester must be able to skip a certain number of questions - validation is the method of assuring that the result was not modified by the student. This requires the generation of a validation file.

The tester must create a file to the [OUTPUT_PATH] location, with the result of the test encoded in a JSON format. At the start of the test, the tester must create the result file, with the appropriate number of results, question names and weights, but the marks and feedbacks left blank (""). These must be populated one by one, as the questions are executed. If the [SKIP_NUMBER] is specified, the first [SKIP_NUMBER] of questions must be skipped and not populated, but the feedbacks file must still have results for all questions, and leave the skipped ones blank. This is why it's important to have questions that can be executed independent of each other.

Each time the result file is update, a validation file must be created alongside it. It must be located in the same directory as the result file and it must be called [SECRET_LOCATION]. In order to generate the validation file the secret key must be obtained. The key is initially located in an environment variable called [SECRET_LOCATION]. Before executing any student code the variable must be read and replaced with an empty string. Student code is not allowed to access the key. The validation file must contain the sha256 hash of the result file and the secret key (combined in that order). If the validation file is missing, or it doesn't contain the expeted hash, the question result will not be considered, unless validation is disabled.