Preliminary and System Testing

Preliminary and system testing is a new contest format introduced at the beginning of the 2015-2016 DMOPC season along with the virtual contest system. Usually, the harder problems (i.e. P4, P5, and P6) will be implementing this format. This page is intended to clarify the format for new users, as well as to take any suggestions to improve the system.

What it is

Instead of running your submission on all test cases as we do for P1, P2, and P3, submissions on the harder problems during the contest will be put through preliminary tests, henceforth referred to as pretests. Pretests will, unless otherwise stated, contain:

Sample data given in the problem statement, identified by point values of $0$ ~0~
One test case or a small batch for each of the subtasks, identified by positive point values that reflect their respective subtask's worth

All subtask scores will be scaled down by a factor of $10$ ~10~, giving you a temporary score out of $10$ ~10~ on your pretest submissions. For example, let's say a particular problem has subtasks $1$ ~1~ and $2$ ~2~, worth $20\%$ ~20\%~ and $80\%$ ~80\%~ respectively, if your submission ACs on only the pretest for subtask $1$ ~1~, you will receive a temporary score of $2$ ~2~. This is intended to reflect your expected final score on the problem. However, note that pretests will typically contain weak test data. If your submission ACs on the pretest cases, it does not necessarily mean it will AC in all cases, All contestants are therefore encouraged to double check the time and space complexities of their algorithms, as well as consider any corner cases.

At the end of the contest, all pretest submissions will be rejudged on the full set of test data, this is known as system testing or systests. You can expect the test data for systests to be more complete and include max cases as well as corner cases. Your submission's result on the systest is what ultimately determines your final score for a problem, as well as the final standings for the contest.

Why

The pretest/systest system was introduced with three main goals in our minds:

1. To prevent cheating. The 9-hour-long window for the virtual contest means that it would be incredibly easy for users to leak problem statements ahead of time and share solutions. The pretest system aims to prevent cheating through creating uncertainty of a solution's correctness.

2. To encourage contestants to consider all possible cases. The pretest system encourages contestants to think harder about the correctness of their algorithm rather than to continuously submit WA solutions. An algorithm that WAs on one case is still an incorrect solution. Missing a corner case on a problem will teach contestants to consider all possible cases next time, leading to more efficient coding in general.

3. To encourage contestants to learn the time and space complexities of algorithms. Contestants will have to judge whether their solution will pass based on the constraints, the time limit, and the memory limit. The pretest system encourages deeper understanding of algorithms, which is ultimately the goal of these contests. On a side note, it may be helpful for contestants to know that the judges can process approximately $10^9$ ~10^9~ operations per second. Of course this varies for different languages, but time and memory limits are always created such that correct algorithms will pass in any language.

If you have any questions or concerns, feel free to leave a comment on the contest page. Specific, constructive feedback to improve the current system is appreciated.