INGInious Documentation

(1)

INGInious Documentation

Release 0.4

Author

Dec 10, 2017

(2)

(3)

CHAPTER 1 Installation, configuration and upgrade

1.1 Installation and deployment

1.1.1 Supported platforms

INGInious is intended to run on Linux (kernel 3.10+), but can also be run on Windows and macOS thanks to the Docker toolbox.

1.1.2 Dependencies setup

INGInious needs:

• Python(with pip) 3.5+

• Docker1.12+

• MongoDB

• Libtidy

• LibZMQ

RHEL/Cent OS 7.0+, Fedora 24+

The previously mentioned dependencies can be installed, for Cent OS 7.0+ :

# curl -fsSL https://get.docker.com/ | sh #This will setup the Docker repo

# yum install -y epel-release https://centos7.iuscommunity.org/ius-release.rpm

# yum install -y git mongodb mongodb-server gcc libtidy python35u python35u-pip

˓→python35u-devel zeromq-devel

Or, for Fedora 24+:

1

(6)

# dnf install -y git mongodb mongodb-server gcc libtidy python3 python3-pip python3-

˓→devel zeromq-devel

You may also add openldap-devel if you want to use the LDAP auth plugin and xmlsec1-openssl-devel libtool-ltdl-develfor the SAML2 auth plugin.

Danger: Due to compatibility issues, it is recommended to disable SELinux on the target machine.

You can now start and enable the mongod and docker services:

# systemctl start mongod

# systemctl enable mongod

# systemctl start docker

# systemctl enable docker

Ubuntu 16.04+

The previously mentioned dependencies can be installed, for Ubuntu 16.04+:

# apt-get install git mongodb gcc tidy python3 python3-pip python3-dev libzmq-dev

You may also add libldap2-dev libsasl2-dev libssl-dev if you want to use the LDAP auth plugin and libxmlsec1-dev libltdl-devfor the SAML2 auth plugin

You can now start and enable the mongod and docker services:

# systemctl start mongodb

# systemctl enable mongodb

# systemctl start docker

# systemctl enable docker

OS X 10.9+

We usebrewto install some packages. Packages are certainly available too via macPorts.

$ brew install mongodb

$ brew install python3

Follow the instruction of brew to enable mongodb.

The next step is to installDocker for Mac.

Windows

Danger: INGInious rely on Docker to run containers. While Docker is supported on Windows 10 (version 1607), INGInious does not provide support for Windows containers yet.

(7)

INGInious Documentation, Release 0.4

The recommended way to run INGInious under Windows is by using a Linux virtual machine, for much more simplicity. One can also only run the Docker agent under a Linux virtual machine and run the backend and selected frontend under Windows.

In the later case, you’ll need to install Python 3.5+, MongoDB, LibTidy and LibZMQ.

1.1.3 Installing INGInious

The recommended setup is to install INGInious via pip and the master branch of the INGInious git repository. This allows you to use the latest development version. This version is currently the supported one for issues.

$ pip3 install --upgrade git+https://github.com/UCL-INGI/INGInious.git This will automatically upgrade an existing version.

Note: You may want to enable the LDAP/SAML2 plugin or use (F)CGI instead of the web.py default webserver.

In this case, you have to install more packages: simply add [cgi], [ldap] or [saml2] to the above command, depending on your needs:

$ pip3 install --upgrade git+https://github.com/UCL-INGI/INGInious.git

˓→#egg=INGInious[cgi,ldap]

Some releases are also published on Pipy. However, no support is provided for these versions now. To install the latest previous release:

$ pip3 install --upgrade inginious

1.1.4 Configuring INGInious

INGInious comes with two frontends:

• The LTI frontend, which allows to interface with Learning Management System via theLTIspecification. Any LMS supportingLTIis compatible. This includes Moodle, edX and Coursera, among many others.

• The Web App, a mini-LMS made for on-site courses. It provides statistics, group management, and the INGIn- ious studio, that allows to modify and test your tasks directly in your browser.

You can use one, or both. Each of them have to be configured independently. This can be done automatically with the inginious-installCLI. To configure the LTI frontend:

$ inginious-install lti To configure the Web App frontend:

$ inginious-install webapp

This will help you create the configuration file in the current directory. For manual configuration and details, see Configuration reference.

The detailed inginious-install reference can be found atinginious-install.

1.1. Installation and deployment 3

(8)

1.1.5 Running INGInious

During the configuration step, you were asked to setup either a local or remote backend. In the former case, the frontend will automatically start a local backend and grading agents.

With local backend/agent

To run the frontend(s), please use the inginious-lti or inginious-webapp CLI. This will open a small Python web server and display the url on which it is bind in the console. Some parameters (configuration file, host, port) can be specified. Details are available atinginious-ltiandinginious-webapp.

If you use the LTI frontend, you have to add it to your LMS: follow the instructions inConfiguring the LTI frontend.

With remote backend/agent

To run INInious with a remote backend (and agents), do as follows:

1. On the backend host, launch the backend (seeinginious-backend) :

inginious-backend tcp://backend-host:2001 tcp://backend-host:2000

The agents will connect on tcp://backend-host:2001 and clients on tcp://backend-host:2000 2. Possibly on different hosts, launch the Docker and MCQ agents (see inginious-agent-dockerandinginious-

agent-mcq) :

inginious-agent-docker tcp://backend-host:2001 inginious-agent-mcq tcp://backend-host:2001

3. In you INGInious frontend configuration file (seeConfiguration reference), set backend to : backend: tcp://backend-host:2000

4. Run your preferred frontend usinginginious-ltioringinious-webapp(or both).

1.1.6 Webterm setup

An optional web terminal can be used with INGInious to load the remote SSH debug session. This rely on an external tool.

To install this tool :

$ git clone https://github.com/UCL-INGI/INGInious-xterm

$ cd INGInious-xterm && npm install You can then launch the tool by running:

$ npm start bind_hostname bind_port debug_host:debug_ports

This will launch the app on http://bind_hostname:bind_port. The debug_host and debug_ports parameters are the debug paramaters on the local (seeConfiguration reference) or remote (seeinginious-agent-docker) Docker agent.

To make the INGInious frontend aware of that application, update your configuration file by setting the webterm field to http://bind_hostname:bind_port (seeConfiguration reference).

(9)

For more information on this tool, please seeINGInious-xterm. Please note that INGInious-xterm must be launched using SSL if the frontend is launched using SSL.

1.1.7 Webserver configuration

Warning: In configurations below, environment variables accessible to the application must be explicitly repeated.

If you use a local backend with remote Docker daemon, you may need to set the DOCKER_HOST variable. To know the value to set, start a terminal that has access to the docker daemon (the terminal should be able to run docker info), and write echo $DOCKER_HOST. If it returns nothing, just ignore this comment. It is possible that you may need to do the same for the env variable DOCKER_CERT_PATH and DOCKER_TLS_VERIFY too.

Using lighttpd

In production environments, you can use lighttpd in replacement of the built-in Python server. This guide is made for CentOS 7.x.

Install lighttpd with fastcgi:

# yum install lighttpd lighttpd-fastcgi

Add the lighttpd user in the necessary groups, to allow it to launch new containers and to connect to mongodb:

# usermod -aG docker lighttpd

# usermod -aG mongodb lighttpd

Create a folder for INGInious, for example /var/www/INGInious, and change the directory owner to lighttpd:

# mkdir -p /var/www/INGInious

# chown -R lighttpd:lighthttpd /var/www/INGInious

Put your configuration file in that folder, as well as your tasks, backup, download, and temporary (if local backend) directories (seeConfiguring INGIniousfor more details on these folders).

Once this is done, we can configure lighttpd. First, the file /etc/lighttpd/modules.conf, to load these modules:

server.modules = (

"mod_access",

"mod_alias"

)

include "conf.d/compress.conf"

include "conf.d/fastcgi.conf"

You can then replace the content of fastcgi.conf with:

server.modules += ( "mod_fastcgi" ) server.modules += ( "mod_rewrite" ) alias.url = (

"/static/webapp/" => "/usr/lib/python3.5/site-packages/inginious/frontend/webapp/

˓→static/",

"/static/common/" => "/usr/lib/python3.5/site-packages/inginious/frontend/common/

˓→static/"

)

1.1. Installation and deployment 5

(10)

fastcgi.server = ( "/inginious-webapp" =>

(( "socket" => "/tmp/fastcgi.socket",

"bin-path" => "/usr/bin/inginious-webapp",

"max-procs" => 1,

"bin-environment" => (

"INGINIOUS_WEBAPP_HOST" => "0.0.0.0",

"INGINIOUS_WEBAPP_PORT" => "80",

"INGINIOUS_WEBAPP_CONFIG" => "/var/www/INGInious/configuration.yaml",

"REAL_SCRIPT_NAME" => ""

),

"check-local" => "disable"

)) )

url.rewrite-once = (

"^/favicon.ico$" => "/static/common/icons/favicon.ico",

"^/static/(.*)$" => "/static/$1",

"^/(.*)$" => "/inginious-webapp/$1"

)

Replace webapp by lti if you want to use theLTI frontend.

The INGINIOUS_WEBAPP or INGINIOUS_LTI (according to your config) prefixed environment variables are used to replace the default command line parameters. Seeinginious-ltiandinginious-webappfor more details.

The REAL_SCRIPT_NAME environment variable must be specified under lighttpd if you plan to access the application from another path than the specified one. In this case, lighttpd forces to set a non-root path /inginious-webapp, while a root access if wanted, in order to serve static files correctly. Therefore, this environment variable is set to an empty string in addition to the rewrite rule.

Finally, start the server:

# systemctl enable lighttpd

# systemctl start lighttpd

Using Apache

You may also want to use Apache. You should install mod_wsgi. WSGI interfaces are supported through inginious- webappand inginious-lti scripts. This guide is made for CentOS 7.x.

Install the following packages (please note that the Python3.5+ version of mod_wsgi is required):

# yum install httpd httpd-devel

# pip3.5 install mod_wsgi

Add the apache user in the necessary groups, to allow it to launch new containers and to connect to mongodb:

# usermod -aG docker apache

# usermod -aG mongodb apache

Create a folder for INGInious, for example /var/www/INGInious, and change the directory owner to apache:

# mkdir -p /var/www/INGInious

# chown -R apache:apache /var/www/INGInious

(11)

Put your configuration file in that folder, as well as your tasks, backup, download, and temporary (if local backend) directories (seeConfiguring INGIniousfor more details on these folders).

Set the environment variables used by the INGInious CLI scripts in the Apache service environment file (seelighttpd for more details):

# cat << EOF >> /etc/sysconfig/httpd

INGINIOUS_WEBAPP_CONFIG="/var/www/INGInious/configuration.yaml"

INGINIOUS_WEBAPP_HOST="0.0.0.0"

INGINIOUS_WEBAPP_PORT="80"

EOF

# rm /etc/httpd/conf.d/welcome.conf

Please note that the service environment file /etc/sysconfig/httpd may differ from your distribution and wether it uses systemd or init.

You can then modify your /etc/httpd/conf/httpd.conf file to apply the following rules:

LoadModule wsgi_module /usr/lib64/python3.5/site-packages/mod_wsgi/server/mod_wsgi-

˓→py35.cpython-35m-x86_64-linux-gnu.so

WSGIScriptAlias / "/usr/bin/inginious-webapp"

WSGIScriptReloading On

Alias /static/common /usr/lib/python3.5/site-packages/inginious/frontend/common/

˓→static/

Alias /static/webapp /usr/lib/python3.5/site-packages/inginious/frontend/webapp/

˓→static/

Alias /static/lti /usr/lib/python3.5/site-packages/inginious/frontend/lti/static/

Require all granted

</Files>

</Directory>

Require all granted

</DirectoryMatch>

Please note that the compiled wsgi module path may differ according to the exact Python version you are running.

1.2 Updating INGInious

1.2.1 Updating the sources

To fetch the latest updates on the Git repository master branch :

$ pip3 install --upgrade git+https://github.com/UCL-INGI/INGInious.git If you installed INGInious via Pipy, you can simply upgrade it using:

$ pip3 install --upgrade inginious

1.2. Updating INGInious 7

(12)

1.2.2 Updating your containers

The provided containers can be automatically updated using:

$ inginious-container-update

For your own or third-party containers, please refer toCreating a new container image.

1.2.3 Updating the configuration

Most of the time, you won’t need to update your configuration. If something goes wrong, backup your existing configuration file(s) and run inginious-install again. For further details, please refer toinginious-installor Configuration reference.

1.3 Configuration reference

Hint: The best way to configure INGInious is to use inginious-install. SeeConfiguring INGInious.

Configuring INGInious is done via a file named configuration.yaml or configuration.lti.yaml. To get started, files named configuration.example.yaml and configuration.lti.example.yaml are provided.

The different entries are :

tasks_directory The path to the directory that contains all the task definitions, grouped by courses. (seeCreat- ing a new task)

backend The link to the backend used. You can either set it to local or indicate the address of your manually- managed backend.

• local. In this mode, which is the default, you have to ensure the docker daemon is local to your machine, or, at least, share the same directory structure. This is typically the case if you use Linux and have a local Docker daemon, or if you use Docker for Mac/Windows, or even docker-machine with local machines.

This is the configuration described in this tutorial. You will need a running docker daemon on your machine for this to work. If you can use any Docker client command, like docker info, INGInious should run flawlessly.

In this mode, a supplementary config option is available, local-config.

• tcp://xxx.yyy.zzz.aaa:bbbb, udp://xxx.yyy.zzz.aaa:bbbb or ipc:///path/to/

your/sock, where the adresses are the ip/socket path of the backend you started manually. This is for advanced users only. See commands inginious-backend and inginious-agent for more information.

local-config These configuration options are available only if you set backend:local.

concurrency Number of concurrent task that can be run by INGInious. By default, it is the number of CPU in your host.

debug_host Host to which the users should connect in order to access to the debug ssh for containers. Most of the time, just do not indicate this option: the address will be automatically guessed.

debug_ports Range of port, in the form 64100-64200, to which INGInious can bind SSH debug containers, to allow remote debugging. By default, it is 64100-64200.

(13)

tmp_dir A directory whose absolute path must be available by the docker daemon and INGInious at the same time. By default, it is ./agent_tmp.

mongo_opt MongoDB client configuration.

host MongoDB server address. If your database is user/password-protected, use the following syntax:

mongodb://USER:PASSWORD@HOSTNAME/DB_NAME

database You can change the database name if you want multiple instances or in the case of conflict.

log_level Can be set to INFO, WARN, or DEBUG. Specifies the logging verbosity.

use_minified_js Set to true to use the minified version of Javascript scripts, false otherwise.

webterm Link to the INGInious xterm app with the following syntax: http[s]://host:port. If set, it allows to use in-browser task debug via ssh. (See _webterm_setup for more information)

plugins A list of plugin modules together with configuration options. SeePluginsfor detailed information on available plugins, including their configuration. Please note that the usage of at least one authentication plugin is mandatory for the webapp.

1.3.1 Webapp-specific configuration

superadmins A list of super-administrators who have admin access on the whole stored content.

maintenance Set to true if the webapp must be disabled.

backup_directory Path to the directory where are courses backup are stored in cases of data wiping.

smtp Mails can be sent by batch containers at the end of the job execution, or by plugins.

sendername Email sender name, e.g. : INGInious <[email protected]>

host SMTP server.

port SMTP port.

username SMTP username.

password SMTP password.

starttls Set to true if TLS is needed.

1.3.2 LTI-specific configuration

The LTI interface uses most of the same configuration options as the webapp as well as the following:

lti A list of LTI consumer key and secret values.

lti_user_name The LTI field used to identify the user. By default this is user_id, which for many LMS system would be the numeric ID of the user. It can be set to ext_user_username which is often a unique username.

download_directory The path to the directory where downloads are stored temporarily during the archive is being prepared.

1.3.3 Plugins

Several plugins are available to complete the INGInious feature set. Some of them are only compatible with the webapp, while others are compatible with both webapp and LTI application.

1.3. Configuration reference 9

(14)

Auth plugins (webapp)

You need at least one auth plugin activated.

demo_auth

Provides a simple authentification method, mainly for demo purposes, with username/password pairs stored directly in the config file.

To enable this plugin, add to your configuration file:

plugins:

- plugin_module: inginious.frontend.webapp.plugins.auth.demo_auth users:

username1: "password1"

Each key/value pair in the users field corresponds to a new username/password user.

ldap_auth

Uses an LDAP server to authenticate users.

plugins:

- plugin_module: inginious.frontend.webapp.plugins.auth.ldap_auth host: "your.ldap.server.com"

encryption: "ssl" #can be tls or none

base_dn: "ou=People,dc=info,dc=ucl,dc=ac,dc=be"

request: "uid={}", prefix: "",

Most of the parameters are self-explaining, but:

request is the request made to the LDAP server to search the user to authentify. “{}” is replaced by the username indicated by the user.

prefix a prefix that will be added in the internal username used in INGInious. Useful if you have multiple auth methods with usernames used in more than one method.

saml2_auth

Uses a SAML2-compliant identity provider (such as Shibboleth IdP) to authenticate users.

plugins:

- plugin_module: inginious.frontend.webapp.plugins.auth.saml2_auth strict: true

sp:

entityId: "<your_entity_id>"

x509cert: "<your_cert>"

(15)

privateKey: "<your_private_key>"

idp:

entityId: "https://idp.testshib.org/idp/shibboleth"

singleSignOnService:

url: "https://idp.testshib.org/idp/profile/SAML2/Redirect/SSO"

binding: "urn:oasis:names:tc:SAML:2.0:bindings:HTTP-Redirect"

x509cert: "<idp_cert>"

additionalX509certs:

- "<idp_cert>"

security:

metadataValidUntil: ""

metadataCacheDuration: ""

attributes:

cn: "urn:oid:2.5.4.3"

email: "urn:oid:1.3.6.1.4.1.5923.1.1.1.6"

uid: "urn:oid:0.9.2342.19200300.100.1.1"

Your IdP is required to provide at least attributes corresponding to the username, the complete name and the email address. Use the attributes entry for the mapping. The additionalX509certs is a plugin-specific entry to specify several certificates in case your IdP is able to use more than one.

This plugin mainly relies onpython3-samlpackage and configuration parameters are interoperable. Please refer to the package documentation for more detailed configuration parameters. The SP Attribute Consuming Service (ACS) is automatically configured by the plugin.

db_auth

Uses the MongoDB database to authenticate users. Provides a basic email-verification based registration and password recovery. It does not support manual user management yet. The superadmin has to register the same way other users do.

plugins:

- plugin_module: inginious.frontend.webapp.plugins.auth.db_auth allow_deletion: true

For mails to be sent by the registration module, you need to configure the smtp parameter of your configuration file.

allow_deletionparameter can be used to specify if users can delete their accounts from INGInious.

Scoreboard plugin (webapp)

This plugin allows to generate course/tasks scoreboards. To enable the plugin, add to your configuration file:

plugins:

- plugin_module: inginious.frontend.webapp.plugins.scoreboard

To define a new scoreboard, an additional field scoreboard must be defined in the course.yaml file associated to a course (SeeCreating a new course). For instance:

scoreboard:

- content: ["taskid1"]

- content: ["taskid2", "taskid3"] # sum of both score is taken as overall score name: "Scoreboard for task 2 and 3"

(16)

- content: {"taskid4": 2, "taskid5": 3} # overall score is 2*score of taskid4 +

˓→3*score of taskid5

reverse: True

This defines three scoreboards for the course. The first one will create a scoreboard for task id taskid1 and will be displayed as Scoreboard task 1. The second one will create a scoreboard for taskid2 and taskid3 where both scores are added. The last one is more complex and will create a reversed scoreboard for task taskid4 and taskid5where both scores are wieghted by factor 2 and 3, respectively.

The score used by this plugin for each task must be generated via a key/value custom feedback (seefeedback-custom) using the score key. Only the succeeded tasks are taken into account.

Contests plugin (webapp)

This plugin allows to manage an ACM/ICPC like contest inside a course between students. To enable the plugin, add to your configuration file:

plugins:

- plugin_module: inginious.frontend.webapp.plugins.contests

A new configuration page named Contest appears on the administration page. To enable the contest mode, check the Enable contest pluginbox on the appropriate course. Please note that the plugin will override the task accessibility dates.

Simple grader plugin (webapp, lti)

This simple grader allows anonymous POST requests without storing submissions in database.

To enable the plugin, add to your configuration file:

plugins:

- plugin_module: inginious.frontend.webapp.plugins.simple_grader courseid : "external"

page_pattern: "/external"

return_fields: "^(result|text|problems)$"

• courseid is the course id you want to expose to the simple grader.

• page_pattern is the URL at which you want to make the simple grader available.

• return_fields is a regular expression matching the submission fields that can be returned via the simple grader.

A demonstration POST form will be available at the page_pattern specified URL.

New synchronized job

External submissions must take the form of a POST request on the url defined by page_pattern. This POST must contains two data field:

• taskid: the task id of the task

• input: the input for the task, in JSON. The input is a dictionary filled with problemid:problem_answer pairs.

(17)

The return value will contains the standard return fields of an INGInious inginious.backend job plus a “status” field that will contain “ok”.

If an internal error occurs, it will return a dictionary containing {

"status": "error",

"status_message": "A message containing a simple description of the error"

}

New asynchronous job

This POST request allows new jobs to be treated asynchronously. It must contains three data fields:

• taskid: the task id of the task

• input: the input for the task, in JSON. The input is a dictionary filled with problemid:problem_answer pairs.

• async: field that indicate that the job must be launched asynchronously. Only have to be present, content is not read.

The return value will be a dictionnary containing:

{

"status": "done",

"jobid": "the jobid of the async job. Will be needed to get the results."

} or {

"status": "error",

"status_message": "A message describing the error"

}

Get status of asynchronous job

Given a jobid in input (as field of the POST request) and will return either:

{

"status": "waiting"

} or {

"status": "error",

"status_message": "A message describing the error"

} or {

"status": "done",

"...":"..."

}

(18)

where ... are the results of the job, as defined in the return_fields configuration value.

Git Repo plugin (webapp, lti)

This plugin allows saving submissions history in a Git repository, according to the following path pattern : courseid/taskid/username. The version kept in the head of branch is the latest submission made.

plugins:

- plugin_module: inginious.frontend.webapp.plugins.git_repo repo_directory: "./repo_submissions"

The repo_directory parameter specify the path to the repository that must be initialized before configuration.

Task file readers plugin (webapp, lti)

It is possible to store task files in other formats than YAML. However, these plugins are provided for retro- compatibility with previous supported formats, which are deprecated. You therefore use these plugins at your own risks.

JSON

To enable the JSON task file format:

plugins:

- plugin_module: inginious.frontend.webapp.plugins.task_file_readers.json_reader

RST

To enable the reStructuredText task file format:

plugins:

- plugin_module: inginious.frontend.webapp.plugins.task_file_readers.rst_reader

1.4 Configuring the LTI frontend

In order to use the LTI frontend with your favorite LMS, you first have to define your LTI keys (consumer key and secret), then installing them on your LMS.

1.4.1 Defining your LTI keys

The LTI keys are defined in the configuration file of the LTI frontend, usually configuration.lti.yaml. Here is an example of configuration:

lti:

consumer_key_1:

secret: a_very_secret_password consumer_key_2:

(19)

secret: wow_such_secret courses:

- my_course

This obviously defines two LTI keys, consumer_key_1 and consumer_key_2, with passwords a_very_secret_passwordand wow_such_secret.

By default, consumer keys allow accessing all courses data from the consumer. If you want a consumer to be restricted to only one course data, use the courses field and specify the list of courses to give access to. This is illustrated in the example with consumer_key_2.

1.4.2 Setting up your LMS

Setting up (Open) edX

edX providesa good tutorial on how to install LTI components.

When it asks for the LTI passport, you have to enter it in the format

an_id_that_you_define:consumer_key:password. A good example, taking values from the start of this document, would be

inginious:consumer_key_1:a_very_secret_password

The launch url is, if your server is located at https://HOST:PORT/, and you want to load the task task_id from the course course_id:

https://HOST:PORT/launch/course_id/task_id

Please note that, for now, official edX needs https. You also need to set the LTI activity to accept a score back from INGInious, without which the activity won’t launch.

Setting up Moodle

Under edition mode, select add an activity, choose external tool, and confirm.

Directly click on Show more.... Fill in the activity name.

The Launch URL is, if your server is located at https://HOST:PORT/, and you want to load the task task_id from the course course_id:

https://HOST:PORT/launch/course_id/task_id

For the field Launch Container, the best value is “Embded without block”. Consumer key and Consumer secretare the LTI key you defined earlier. In the Privacy fieldset, verify that accept grades from the toolis checked. Leave the other fields blank (or modify them as you want).

Save, and it should work.

Setting up other LMS

INGInious has only been tested with edX and Moodle, but it should work out-of-the-box with any LMS that respects LTI 1.1. You are on your own for the configuration, though; but with the LTI keys and the launch URL, it should be enough to configure anything.

1.4. Configuring the LTI frontend 15

(20)

1.5 Troubleshooting common problems

1.5.1 Solving problems hangs, on OS X with docker-machine or VirtualBox to run Docker

There is a known problem with VirtualBox shared folders: it is impossible for the VM to create a unix socket inside (for strange reasons). To solve this, you can mount instead your /Users directory using docker-machine-nfs:

1.5.2 LTI grades are not pushed back using LetsEncrypt

INGInious uses PyLTI which uses oauth, oauth2 and libhttp2. The list of certificate authorities known to libhttp2 may be out of day with your host operating system. In particular, as of August 2016, it does not include the LetsEncrypt CA, and thus websites protected with a LetsEncrypt certificate won’t work (you won’t be able to push grades back).

1.5.3 LTI frontend keeps on OAuth errors

LTI uses OAuth which uses time-based replay prevention. You need to insure that your webserver (LTI consumer) and LTI producer have reasonably synchronous clocks.

1.5.4 Impossible to get the LTI frontend work

You may find http://ltiapps.net/test/tp.php and http://ltiapps.net/test/t.php useful when debugging producers and con- sumers.

(21)

CHAPTER 2 Commands reference

2.1 Frontend commands

2.1.1 inginious-install

Assistant to create a complete configuration file for the specified frontend.

Read and follow screen instructions.

inginious-install [-h] [--file FILE] {lti,webapp}

--file

Specify the configuration file to use. By default, it is configuration.yaml or configuration.lti.yaml, depending on which backend you use

-h, --help

Display the help message.

lti, webapp

The frontend to install

2.1.2 inginious-webapp

Starts the Web App Frontend. This command can run a standalone web server (see --host and --port options), but also as a FastCGI backend.

inginious-webapp [-h] [--config CONFIG] [--host HOST] [--port PORT]

--config

Specify the configuration file to use. By default, it is configuration.yaml or configuration.json, depending on which is found first. This can also be specified via the INGINIOUS_WEBAPP_CONFIG environment variable.

17

(22)

--host HOST

Specify the host to which to bind to. By default, it is localhost. This can also be specified via the INGINIOUS_WEBAPP_HOSTenvironment variable.

--port PORT

Specify the port to which to bind to. By default, it is 8080. This can also be specified via the INGINIOUS_WEBAPP_PORTenvironment variable.

-h, --help

2.1.3 inginious-lti

Starts the LTI Frontend. This command can run a standalone web server (see --host and --port options), but also as a FastCGI backend.

inginious-lti [-h] [--config CONFIG] [--host HOST] [--port PORT]

--config

Specify the configuration file to use. By default, it is configuration.lti.yaml. This can also be specified via the INGINIOUS_LTI_CONFIGenvironment variable.

--host HOST

Specify the host to which to bind to. By default, it is localhost. This can also be specified via the INGINIOUS_LTI_HOSTenvironment variable.

--port PORT

Specify the port to which to bind to. By default, it is 8080. This can also be specified via the INGINIOUS_LTI_PORTenvironment variable.

-h, --help

2.2 Backend commands

2.2.1 inginious-agent-docker

Start a Docker grading agent. This is typically only used when running the INGInious backend remotely. If you configured INGInious to use a local backend, it is automatically run by inginious-webapp or inginious-lti.

inginious-agent-docker [-h] [--debug-host DEBUG_HOST]

[--debug-ports DEBUG_PORTS] [--tmpdir TMPDIR]

[--tasks TASKS] [--concurrency CONCURRENCY] [-v]

backend -h, --help

--debug-host DEBUG_HOST

The agent hostname for SSH debug. If not specified, the agent autodiscover the public IP address.

--debug-ports DEBUG_PORTS

Range of port for job remote debugging. By default it is 64120-64130 --tmpdir TMPDIR

Path to a directory where the agent can store information, such as caches. Defaults to ./agent_data

(23)

--tasks TASKS

The path to the directory containing the courses. Default to ./tasks.

--concurrency CONCURRENCY

Maximal number of jobs that can run concurrently on this agent. By default, it is the two times the number of cores available.

-v, --verbose

Increase output verbosity: logging level to DEBUG.

backend

The backend port, using the following syntax : protocol://host:port. E.g. tcp://127.0.0.

1:2001. The agent will connect to the backend listening on that port.

2.2.2 inginious-agent-mcq

Start a MCQ grading agent. This is typically only used when running the INGInious backend remotely. If you configured INGInious to use a local backend, it is automatically run by inginious-webapp or inginious-lti.

inginious-agent-mcq [-h] [--tasks TASKS] [-v] backend -h, --help

--tasks TASKS

-v, --verbose

backend

The backend port, using the following syntax : protocol://host:port. E.g. tcp://127.0.0.

1:2001. The agent will connect to the backend listening on that port.

2.2.3 inginious-backend

Start an INGInious backend. This is typically only used when running the INGInious backend remotely. If you configured INGInious to use a local backend, it is automatically run by inginious-webapp or inginious-lti.

inginious-backend [-h] [-v] agent client -h, --help

-v, --verbose

agent

The agents port, using the following syntax : protocol://host:port. E.g. tcp://127.0.0.1:2001.

The backend will listen for grading agents on that port.

client

The clients port, using the following syntax : protocol://host:port. E.g. tcp://127.0.0.

1:2000. The backend will listen for client frontend on that port.

2.2. Backend commands 19

(24)

2.3 Task tests commands

2.3.1 inginious-check-task

Verify the syntax and the parameters of a course/task file.

inginious-check-task [-h] [--taskdir TASKDIR] course task -h, --help

--taskdir TASKDIR

course

The course id to test --port PORT

The task id to test

2.3.2 inginious-test-task

Replay a task with the set of .test files available in the task folder and checks if results are identical. If results differ, detailed information are printed on the terminal.

The .test files can be obtained by downloading a valid submission and storing the .test file found in the downloaded archive in the task folder.

inginious-test-task [-h] [-c CONFIG] [-v] courseid taskid -h, --help

-c, --config

Specify the INGInious config file to use. If not specified, looks for a configuration file in the current directory -v, --verbose

Increase output verbosity: display entire stdout/stderr.

courseid

The course id corresponding to the task to test.

taskid

The task id corresponding to the task to test.

2.4 Utilities

2.4.1 inginious-synchronize

Synchronization tool for INGInious Git repos. Each repository is suppposed to content the files required for a course.

When run, the tool pulls the modifications done remotely and force-merge the the local version if conflicts cannot be resolved automatically.

(25)

A configuration file synchronize.json must be provided or specified using environment variable INGINIOUS_SYNC_CONFIG. This file contains the main task directory as well as the course identifier, private key for pulls and repo url, as follows:

{

"maindir":"../tasks",

"repos":

[ {

"course":"TEST0000",

"keyfile":"TEST0000.key",

"url":"[email protected]:user/TEST0000.git"

}, {

"course":"TEST0001",

"keyfile":"TEST0001.key",

"url":"[email protected]:user/TEST0001.git"

} ] }

For more compatibility, please run this command in an ssh-agent session.

Before adding in crontab, add the following lines to .ssh/config for user who runs the scripts : Host *

StrictHostKeyChecking no

This tells SSH not to check host keys, we always trust the remote servers

2.4.2 inginious-old-task-converter

Convert old tasks (in .json or .rst format) to the newer format, .yaml. You will probably not need this command.

inginious-old-task-converter [-h] [--delete-html] [--convert-yaml] tasks -h, --help

--delete-html

Delete*IsHTML fields. Default to false --conver-yaml

Force conversion to YAML. Default to false tasks

The path to the directory containing the courses.

2.4.3 inginious-containers-update

Update all the containers created and/or maintained by the INGInious team.

Takes no argument, but needs a properly configured Docker environment. If you use a remote Docker instance, please check the DOCKER_HOST environment variable.

2.4. Utilities 21

(26)

(27)

CHAPTER 3 Teacher’s documentation

Contents:

3.1 What is INGInious?

INGInious provides a simple and secure way to execute and test untrusted code. It has been developed by theINGI department (Université catholique de Louvain) to automatic grading of programming assignments. The whole tool is written inPython(version 3.5+) and relies onDockerto provide secure execution environments and onMongoDBto keep track of submissions.

INGInious is completely language-agnostic and is able to run anything. Currently, this is limited to Linux programs as only Linux containers are provided and supported.

INGInious also provides anLTImodule, allowing its integration to your existing (Open)edX,Moodle,... courses.

3.1.1 How does INGInious work?

INGInious is based on the concept of tasks (see Creating a new task). A task is a set of one or more related (sub)questions. For each task, an infinite number of submissions is allowed, but a user must wait for the result of its current submission before trying a new one.

For simplicity, tasks are grouped by courses (seeCreating a new course). Usually, an INGInious course has one task per assignment.

A submission is a set of deliverables (chunks of code, files, archives, etc.) that correspond each to one of the (sub)questions of the task. These files are made available to the run file (see Run file), a special script provided by the task. That script is responsible for providing feedback on the submission by compiling, executing or applying any form of checking and testing to the deliverables. In its simplest form, the feedback consists of either success or failed.

This run file is run inside a container (precisely, a grading container), that completely jails the execution of the script, because even teachers and assistants are never fully trusted. Grading containers are able to start sub-containers, called student containers, that runs the scripts that the students sent with their submission, in another jailed environment.

23

(28)

This separation in two step of the grading is mandatory to ensure a complete security for the server hosting INGInious and a complete security of the grading process, making impossible for the student to interact “badly” with the run script.

These containers are created/described by very simple files called Dockerfile. They allow to create containers for anything that runs on Linux. For details about to create new containers and add new languages to INGInious, see Creating a new container image.

Architecture

INGInious comes with three distinct parts, the backend (and its agent) and a frontend.

The backend (see ../dev_doc/backend) receives the code of the students and sends it to its agent (see ../dev_doc/agent), which is then responsible to send it to a Dockercontainer, and interact with the request made by the container.

That container then makes some verifications on the submission and returns one of the following four possible status : success, crash, timeout, or failed.

INGInious also provides a frontend (see ../dev_doc/frontend). Made with MongoDB as database, the frontend is in fact an extension of the backend and allows students to work directly on a website. This frontend also provides statistics and management tools for the teachers.

Most of these functionalities can be extended through plugins.

For a more advanced view of the architecture of INGInious, seeUnderstand INGInious.

Docker containers

Docker containers are small virtual operating systems that providesisolationbetween the processes and resources of the host operating system. Docker allow to create and ship any software on any free Linux distribution.

As there are no hypervisor, the processes launched in the container are in fact directly run by the host operating system, which allows applications to be amazingly fast.

Docker allow teachers to build new containers easily, to add new dependencies to the tests applied on the student’s code (seeCreating a new container image)

Isolation

Isolation allows teachers and system administrators to stop worrying about the code that the students provides.

For example, if a student provides a forkbomb instead of a good code for the test, the forkbomb will be contained inside the container. The host operating system (the computer that runs INGInious) won’t be affected.

The same thing occurs with memory consumption and disk flood. The running time of a code is also limited.

3.1.2 Compatibility

INGInious provides two compatibility layers with Pythia v0 and v1. Except the task description file which has to be updated, everything is 100% compatible with INGInious.

(29)

3.2 Creating a new course

Courses are defined by subdirectories found in the tasks directory, which has been specified in the configuration. See Configuration reference. These subdirectories are composed of a course.yaml file describing the course parameters and other subdirectories corresponding to tasks (SeeCreating a new task).

Here is an example of the content of a tasks folder:

tasks/

course_id_1/

course.yaml task_id_1/

task.yaml run ...

...

Ideally, you should only give permissions to a course folder to the course administrator if needed. The webapp task editor should not require you to give this access. If needed, several methods exist. Seeinginious-synchronizefor Git repository synchronization.

3.2.1 Tutorial

Creating courses is reserved to the super-administrators (SeeConfiguration reference). Course administrators are then able to configure the course by themselves.

Note: Demonstration tasks are made available for downloadhere. They can also be downloaded and installed automatically via theinginious-installscript.

Using the webapp

1. As a super-administrator, go to the bottom of the course list and enter a new course id, for instance demo, and click on Create new course. A newly created hidden course named demo appears on the list.

2. Click on that course, and then on Course administration to change the course parameters, add course administrators and tasks.

Please note that, if you give access to the course directory to course administrators, you still have to do some manual work for this to be effective.

Manually

The course description is a YAML file containing all the course parameters used by INGInious. Here is a simple course description. Put this file with the name course.yaml in a newly created demo folder in your tasks directory.

admins:

- demouser

This elementary course description file will make a new publicly visible course with id demo appear as [DEMO]

Demonstration courseon the course list.

3.2. Creating a new course 25

(30)

3.2.2 Course description files

Inside the task folder, courses are identified by subdirectories name by their course id and containing a course.yaml file. For instance, this file, for a course with id courseid1, should be placed in a courseid1 subdirectory.

course.yamlis a YAML file containing the course configuration.

admins:

- demouser

tutors: []

groups_student_choice: false use_classrooms: true

accessible: true registration: true

registration_password: null registration_ac: null

While the course.yaml file must be present at the course root dir, all the fields inside are actually only used by the webapp. Here are the possible fields to set:

• name Displayed name of the course on the course list.

• admins List of administrators usernames. These users will have complete administrations right on the course.

• tutors List of tutors usernames (restricted-rights teaching assistants). These users will have read-only rights on the course content. They cannot change course parameters nor tasks, cannot replay submissions, launch batch operations or wipe the course data. However, they can manage the classroom composition and download all the student submissions.

• accessible When this field is defined, the course is only visible if within the defined period. A course is always accessible to its admins, and is only hidden to normal users, even if they are registered to the course.

This field can contain theses values:

true the task is always accessible;

false the task is never accessible;

"<start>/<end>" where <start> and <end> are either empty or valid dates like “2014-05-10 10:11:12” or

“2014-06-18”. The task is only accessible between <start> and <end>. If one of the values is empty, the corresponding limit does not apply.

Dates are always considered as a precise instant (to te lowest resolution of the clock). For example, “2014- 05-21” is expanded to “2014-05-21 00:00:00”. This means that start limits are inclusive, while end limits are exclusive.

Some examples:

"2014-05-21 / 2014-05-28"

"/ 2014-01-01 " # (strictly) before january the first

"2030-01-01 /" # opens in 2030

"/" # Always open

"/ 2013-12-31 23:59:59" # closes one minute before "/ 2014-01-01"

• registration When this field is defined, users can only register to the course between the defined period. It takes the same arguments as accessible.

• allow_unregister If this field is defined and set to false, then students are not allowed to auto-unregister from the course.

• registration_password A password that is asked upon registration to the course. If empty or not defined, no password will be asked.

(31)

• registration_ac Access control (AC) method. Can be null (anyone can register), username (filter by username), realname (filter by real name) or email (filter by email address). If AC is activated, the allowed values for the filter should be set in the registration_ac_list key.

• registration_ac_list If AC is activated, registration_ac_list should contain a list of values for the filter.

• nofrontend If this field is defined and set to true, then the course won’t be displayed on the webapp course list.

• groups_student_choice If this field is defined and set to true and if collaborative work is activated for a given task, students will be invited to register by themselves for a group or team before submitting.

• use_classrooms If this field is set to true, the classroom model will be used, otherwise, the team model will be used. The default value for this field is true. (SeeClassrooms and teams)

3.3 Creating a new task

3.3.1 Tutorial

In this document we will describe how to create a simple task, that checks that a code in Python returns “Hello World!”.

Note: Demonstration tasks are made available for downloadhere. They can also be downloaded and installed automatically via theinginious-installscript.

Creating the task description Using the webapp

If you are using the webapp, this procedure can be done using the graphical interface:

1. Go to the Course administration/Tasks page, enter helloworld as a new task id and click on Create new task.

2. In the Basic settings tab, set the task name to Hello World! and put some context and author name. Con- tainer setup can be left with default parameters.

3. In the Subproblems tab, add a new code-type problem with problem id question1.

4. Set some problem name and context, and set language to python.

5. Save changes and go to Task files tab.

Manually

This is only possible if the administrator has given access to the course directory to the course administrator.

The task description is a YAML file describing everything that INGInious needs to know to verify the input of the student. Here is a simple task description. Put this file with the name task.yaml in a newly created helloworld folder in your course directory.

author: "The INGInious authors"

accessible: true name: "Hello World!"

context: "In this task, you will have to write a python script that displays 'Hello

˓→World!'."

3.3. Creating a new task 27

(32)

problems:

question1:

header: "def func():"

type: "code"

language: "python"

limits:

time: 10 memory: 50 output: 1000 environment: default

Most of the fields are self-explanatory. Some remarks:

• The field problems is a dictionary of problems. Each problem must have an unique id, for example here question1.

• Problem question1 have its type field that equals to code, which means the student must enter some code to answer the question. Other types exists, such as multiple-choice.

• The field limits are the limits that the task cannot exceed. The time is in seconds, and memory and output are in MB.

• The environment field is intended to change the environment where the tasks run. The available environments are those you downloaded during installation or those you created by creating a grading container. Please seeCreating a new container image.

More documentation is available here:Creating a new task.

Creating the run file

In your task folder, you will put every file needed to test the input of the student. This folder content can be shown in the webapp in the Task files tab of the Edit task page.

1. Create a template file template.py, where we will put the code of the student.

def func():

@ @question1@@

func()

The syntax is very simple: put a first @ on the line where you want to put the code of the student. Then indent the line and write a second @. Now write the problem id of the problem you want to take the input from (question1) then write another @, write a possible suffix (not used here), and then finish the line with a last

@.

2. Create the run file. This file will be the script that is launched when the task is started. Here we will create a bashscript, that parses the template and verifies its content.

#! /bin/bash

# This line parses the template and put the result in studentcode.py parsetemplate --output student/studentcode.py template.py

# Verify the output of the code...

output=$(run_student python student/studentcode.py) if [ "$output" = "Hello World!" ]; then

# The student succeeded

(33)

feedback-result success

feedback-msg -m "You solved this difficult task!"

else

# The student failed feedback-result failed

feedback-msg -m "Your output is $output"

fi

Here we use three commands provided by INGInious, parsetemplate, run_student and feedback.

The code is self-explanatory; just notice the usage of run_student that ask INGInious (precisely the Docker agent) to start a new student container and run inside the command python studentcode.py.

Please note that the run_student command is fully configurable: you can change the environment on which you run the task, define new timeouts, memory limits, ... Seerun_studentfor more details.

3. If not using the webapp, don’t forget to give the run file the execution rights:

$ chmod +x helloworld/run More documentation is available here:Run file.

3.3.2 Task reference

Task description files

Inside a course folder (seeCreating a new course), tasks are identified by subdirectories named by their task id and containing a task.yaml file. For instance, this file, for a task with id taskid1, should be placed in a taskid1 subdirectory.

task.yamlis a YAML file containing information about the task.

author: Your name context: |-

The context of this task. Explain here what the students have to do.

order: 1 groups: false

problems:

a_problem_id:

language: c limits

time: 30 memory: 128 environment: default network_grading: False

• author, context, order, name, language and header are only needed if you use the frontend.

contextand header are parsed using restructuredText.

• order is an integer, used by the frontend to sort the task list. Task are sorted in increasing value of order.

• weight is a decimal value indicating the weight of the task score to use to compute the total course score.

(34)

• accessible describes when the task is accessible to student. This field is not mandatory (by default, the task is visible) and can contain the following values:

true the task is always accessible false the task is never accessible

"START" where START is a valid date, like “2014-05-10 10:11:12”, or “2014-06-18”. The task is only accessible after START.

"/END" where END is a valid date, like “2014-05-10 10:11:12”, or “2014-06-18”. The task is only accessible before END.

"START/END" where START and END are valid dates, like “2014-05-10 10:11:12”, or “2014-06-18”. The task is only accessible between START and END.

• problems describes sub-problems of this task. This field is mandatory and must contain at least one problem.

Problem types are described in the following sectionProblem types. Each problem must have an id which is alphanumeric and unique.

• limits contains the limits that will be applied on the grading container. time is the CPU timeout in seconds, and hard_time is the timeout in real time.

By default, hard_time is defined to be to 3*‘‘time‘‘. This can leads to problems when INGInious is under heavy load, but allow to detect processes that do too much system interruptions (sleep calls or IO)

memoryis the maximum memory allowed to the container.

Please note that the limits of the student containers (container that you start inside the grading container) will use these limits by default.

• environment is the name of the Docker container in which the grading code will run. This field is only needed if there is code to correct; a multiple-choice question does not need it. This environment will be used by default for the student containers.

• groups allows to indicate if the submission is to be done individually or per groups/teams. (see Classrooms and Teams).

• network_grading indicates if the grading container should have access to the net. This is not the case by default.

• evaluate indicates the submission that must be used for evaluation. This can be either:

best This is the default value. In this case, the best submission is used.

last In this case, the last submission is used.

student In this case, the student can select the submission for evaluation. This allows student to select the submission for evaluation without submitting it again or if submission replays are planned. This feature is not available in the LTI module due to LTI specifications limitations, and will be considered as best submission.

• submission_limit indicates the amount of submissions a student can make within a certain period of time.

It is composed of two fields:

– amount is an integer value indicating the amount of submission. A value of -1 corresponds to an infinite amount of submissions.

– period is an integer value indicating the length of the submission period in hours. A value of -1 corresponds to an infinite period. At the end of this period, the student can submit amount submissions again during period hours.

• stored_submissions indicates the amount of submissions that must be saved in the submission history. A value of 0 keeps all the submissions.

(35)

Problem types Code problems

type: codeproblems allows students to submit their code. The code is then sent to a container where a script made by the teaching team corrects it.

Here is a simple example for a code problem type: code

language: c header: |-

Hello dear student!

I'm a multiline header!

header and language are only needed when using the frontend and are not mandatory. This description typically displays on the frontend a box where student can put their code.

optionalis an optional field, that defaults to false, that indicates if this problem is mandatory or not.

Code problem input’s are available in the run script (seeRun file) directly with the id of the problem.

Single code line problems

type: code-single-lineis simply a code box that allows a single line as input.

type: code-single-line language: c

header: |-

Hello dear student!

I'm another multiline header, parsed with *RST*!

Single line code problem input’s are available in the run script (seeRun file) directly with the id of the problem.

Advanced code problem

Advanced code problems are available:

type: code

header: some text

boxId1:

type: text

content: Some additional text boxId2:

type: input-text maxChars: 10 optional: true boxId3:

type: multiline

(36)

maxChars: 1000 lines: 8 language: java

Boxes are displayable (on the frontend) input fields that allows the student to fill more than one entry per problem. Different box types are available, all of them are demonstrated above. Every configuration in the boxes (max- Chars,*lines*,*language*) is not mandatory, except content if the box type is text, and the field optional (default to false), that indicates if the box is mandatory or not.

In the run file (seeRun file), boxes input are available with the name problem_id/box_id

Match problems

Match problem are input that allows a single-line input from the student and that returns if the student entered exactly the text given in the “answer” field.

header: some text describing this problem answer: 42

Match problem input’s are available in the run script (seeRun file) directly with the id of the problem.

Multiple choice problems

header: The answer to life, the universe and any other things is multiple: true

limit: 2

error_message: "Wrong answer. Don't panic, and read Hitchhiker's Guide to the Galaxy."

success_message: "You're right! But don't forget to always take your towel with you."

choices:

- text: It is, of course, 42!

valid: true

- text: It should be *42*

valid: true - text: 43!

feedback: "43 isn't the answer. Maybe can you try to substract one?"

- text: 41?

feedback: "41 isn't the answer. Maybe can you try to add one?"

Choices are described in the choices section of the YAML. Each choice must have a text field (on the frontend) that will be parsed in restructuredText. Valid choices must have a valid: truefield. The field feedback is a message that will be displayed when the student check the choice.

multipleindicates if the student may (or not) select more than one response.

Choices are chosen randomly in the list. If the limit field is set, the number of choices taken equals to the limit.

There is always a valid answer in the chosen choices.

error_messageand success_message are messages that will be displayed on error/success. They are parsed in RST and are not mandatory.

Multiple choice problem input’s are available in the run script (seeRun file) directly with the id of the problem. The input can be either an array of integer if multiple is true or an integer. Choices are numbered sequentially from 0.

(37)

Run file

When the student have submit his/her code, INGInious starts a new Docker container with the right environment for the task (as given in the .task file). Inside this container is launched a script, called run, that you have to provide in the directory of your task.

Here is a simple example of a run file, compatible with the default environment, that simply returns that the student’s code is OK:

#!/bin/bash

The run script is simply an executable application (a bash script, a python script, or a compiled executable runnable by the container). INGInious’ default containers provides commands (also available as python libraries) to interact with the backend.

By default, the script is run inside the container in the /task directory, by a non-root user. You can modify the container to change this (and everything else).

Feedback commands

feedback (obsolete since v0.4)

The feedback command allows you to set the result of your tests. Every argument is optional.

-r, --result STATUS set the result to STATUS. STATUS can be - success (the student succeeded the task), - failed (there are error in the student answer), - timeout (the tests timed out), - overflow (there was a memory/disk overflow) or - crash (the tests crashed) -e, --escape interprets the space-like escape chars.

-f, --feedback MSG set the feedback message to MSG. It is possible to set different messages for each problems. You can use -i to change the problem to which you assign the message -i, --id PROBLEMID set the problem id to which the message from the -f option is assigned. Unused

if -f is not set.

-g, --grade GRADE the grade. Should be a floating point number between 0(no points) and 100(all points) (bonuses, up to 200, are allowed).

-c, --custom CUSTOM add a value VAL to the “custom” dictionnary, with key KEY, where CUSTOM is KEY:VAL, which is stored in DB. Useful for plugins.

-j, --custom-json CUSTOM same as `--custom` but VAL is a json-encoded dictionnary The feedback command can be called multiple times.

feedback --result success --feedback "You're right, the answer is 42!"

feedback-result

The feedback-result command sets the submission result of a task, or a problem, and uses the following syntax : feedback-result [-i|--id PROBLEM_ID] RESULT

The execution result can be of different types:

• success : the student succeeded the task

(38)

• failed : there are error in the student answer

• timeout : the tests timed out

• overflow :there was a memory/disk overflow

• crash : the tests crashed

For instance, the following command will inform that the student succeeded:

In Python : the equivalent command can be directly obtained with:

from inginious import feedback

feedback.set_global_result("success") # Set global result to success

feedback.set_problem_result("failed", "q1") # Set 'q1' subproblem result to failed

feedback-grade

The feedback-grade command sets the submission grade and uses the following syntax:

feedback-grade GRADE

If no grade is specified, the result score will be binary. This means that a failed submission will give a 0.0% score to the student, while a successful submission will give a 100.0% score to the student. For instance, the following command will give an 87.8% grade to the student:

feedback-grade 87.8

from inginious import feedback

feedback.set_grade(87.8) # Set the grade to 87.8%

feedback-msg

The feedback-msg command sets the feedback message associated to the task or a subproblem. It has several optional parameters:

-a, --append append to current feedback, if not specified, replace the current feedback.

-i, --id PROBLEM_ID problem id to which associate the feedback, leave empty for the whole task.

-e, --escape interprets backslash escapes -m, --message MESSAGE feedback message

If the message is not specified, the feedback message is read from stdin. For instance, the command can be used as follows:

feedback-msg -ae -m "This is the correct answer.\n\nWell done!"