Home

testwhat?

A major part of DataCamp's interactive learning is centered around automated and meaningful feedback. When a student submits an incorrect answer, the system tells the student what he or she is doing wrong. This happens through so-called submission correctness tests, or SCTs. An SCT is a test script that compares the different steps in a student's submission to the ideal solution, and generates meaningful feedback along the way.

testwhat is an R package that can help you write these SCTs for interactive R exercises on DataCamp. It allows you to easily compare parts in the student's submission with the solution code. The testwhat package is a wrapper around Hadley Wickham's testthat package, the de facto standard of unit testing in R. testwhat provides a bunch of functions to test object definitions, function calls, function definitions, for loops, while loops, and many more. You can also use testwhat to write custom unit tests. testwhat automatically generates meaningful feedback that's specific to the student's mistake; you can also choose to override this feedback with custom messages.

Writing SCTs, for which testwhat is built, is only one part of creating DataCamp exercises. For general documentation on creating R courses on DataCamp, visit the Teach Documentation. To write SCTs for Python exercises on DataCamp, have a look at pythonwhat.

How does it work?

When a student starts an exercise on DataCamp, an R session is started and the pre_exercise_code (PEC) is run. This code, that the author specifies, initializes the R workspace with data, loads relevant packages etc., such that students can start coding the essence of the topics treated. Next, a separate solution environment is created, in which the same PEC and actual solution code, also coded by the author, is executed.

When a student submits an answer, his or her submission is executed and the output is shown in the R Console. Then, the correctness of the submission is checked by executing the Submission Correctness Test, or SCT. Basically, your SCT is an R script with calls to testwhat test functions. testwhat features a variety of functions to test a user's submission in different ways; examples are test_object(), test_function() and test_output_contains(). To do this properly, testwhat uses several resources:

• The student submission as text, to e.g. figure out which functions have been called.
• The solution code as text, to e.g. figure out whether the student called a particular function in the same way as it is called in the solution.
• The student environment, to e.g. figure out whether a certain object was created.
• The solution environment, to e.g. figure out whether an object that the student created corresponds to the object that was created by the solution.
• The output that's generated when executing the student code, to e.g. figure out if the student printed out something.

If, during execution of the SCT, a test function notices a mistake, an appropriate feedback will be generated and presented to the student. It is always possible to override these feedback messages with your own messages. Defining custom feedback will make your SCTs longer and they may be error prone (typos, etc.), but they typically give the exercise a more natural and personalized feel.

If all test functions pass, a success message is presented to the student. testwhat has some messages in store from which it can choose randomly, but you can override this with the success_msg() function.

Overview

To get started, make sure to check out the Quickstart Guide. This wiki also goes over every test function that testwhat features, explaining all arguments and covering different use cases. They will give you an idea of how, why and when to use them.

Basic functions

• Multiple choice exercises: test_mc()
• What student typed: test_student_typed()
• Which output is generated: test_output_contains()
• Value of object in ending scope: test_object()
• Usage of function: test_function()

Logic-inducing functions

• If some tests fail, do other tests to pinpoint the problem: test_correct()
• Specify several tests, only one of which should pass: test_or()

Advanced functions

• Value of data frame in ending scope: test_data_frame()
• If ... else ... constructs: test_if_else()
• For loops: test_for_loop()
• While loops: test_while_loop()
• Test user-defined functions: test_function_definition()

All these functions are also documented inside the testwhat source code itself. Follow the instructions in the README to install and load the package. Next, you can use ?function_name to read through its documentation.

For more full examples of SCTs for R exercises on DataCamp, check out the source files of the introduction to R course. In the chapter files there, you can can see the SCTs that have been written for several exercises.

After reading through this documentation, we hope writing SCTs for R exercises on DataCamp becomes a painless experience. If this is not the case and you think improvements to testwhat and this documentation are possible, please let us know!