We provide all datasets in the JSON Lines text format containing one example for each line.

Datasets are available at https://github.com/wic-ita/data

Subtask 1: Binary Classification

We provide two datasets for model development:

• The train.jsonl which consists of 2,805 training examples. This dataset should be employed to train the model
• The dev.jsonl which consists of 500 training examples. This dataset should be employed to evaluate the model in the training phase, e.g., tune hyper-parameters

The training dataset (train.jsonl) is highly unbalanced, consisting of about 71.27% of positive and 28.73% of negative examples. At the same time, we provide a balanced development set (development.jsonl) consisting of 50% positive and 50% of negative examples. Further, the dev set includes 250 examples where the target word is out of the vocabulary, i.e., the target word never appears in the training set. For each In-Vocabulary target word of the development set, at least one positive and one negative example are provided in the training set.

Subtask 2: Ranking

We provide four datasets for model development: The train_agr.jsonl which consists of 2,805 training examples for which the two annotators agree The train_dis.jsonl which consists of 1,015 training examples for which the two annotators disagree The train.jsonl which consists of 3,820 training examples. This dataset is the union of the train_agr.jsonl and train_dis.jsonl datasets The dev.jsonl which consists of 500 training examples

Both train_agr.jsonl and dev.jsonl contain the same examples of the training and development set of Subtask 1.