Leaderboard

Submission Instruction

If you have a model and would like to make a submission, you can create a submission here and evaluate your predictions (see the format below). Your submission will be reviewed by our team and added to the leaderboard here.

Submission Format

Please submit a .jsonl file follows the format below, each line is for a question

{"qid": "ACTAAP_2014_7_6", "predictions":["apple", "banana", "car",  "...."]}
{"qid": "ACTAAP_2014_7_7", "predictions":["apple", "banana", "car",  "...."]}
{"qid": "ACTAAP_2014_7_8", "predictions":["apple", "banana", "car",  "...."]}
...

Results

Metric = Hit@50 Acc

Model	Participant	ARC	QASC	OBQA	Overall
BM25	USC-INK yuchen.lin@usc.edu 2021-01-01	56.95	58.50	53.99	56.48
DPR	USC-INK yuchen.lin@usc.edu 2021-01-01	68.67	69.36	62.30	66.78
DrKIT	USC-INK yuchen.lin@usc.edu 2021-01-01	67.63	67.49	61.74	65.62
DrFact	USC-INK yuchen.lin@usc.edu 2021-01-01	71.60	72.01	69.01	70.87
BM25+Reranker	USC-INK yuchen.lin@usc.edu 2021-01-01	76.87	75.75	79.23	77.28
DPR+Reranker	USC-INK yuchen.lin@usc.edu 2021-01-01	76.72	81.66	77.16	78.51
DrKIT+Reranker	USC-INK yuchen.lin@usc.edu 2021-01-01	78.44	84.00	79.25	80.56
DrFact+Reranker	USC-INK yuchen.lin@usc.edu 2021-01-01	84.19	89.87	85.78	86.61

Metric = Recall@50 Acc

Model	Participant	ARC	QASC	OBQA	Overall
BM25	USC-INK yuchen.lin@usc.edu 2021-01-01	21.12	16.33	14.27	17.24
DPR	USC-INK yuchen.lin@usc.edu 2021-01-01	28.93	23.19	18.11	23.41
DrKIT	USC-INK yuchen.lin@usc.edu 2021-01-01	27.57	21.25	18.18	22.33
DrFact	USC-INK yuchen.lin@usc.edu 2021-01-01	31.48	23.29	21.27	25.35
BM25+Reranker	USC-INK yuchen.lin@usc.edu 2021-01-01	39.11	29.03	36.38	34.84
DPR+Reranker	USC-INK yuchen.lin@usc.edu 2021-01-01	43.78	40.72	36.18	40.23
DrKIT+Reranker	USC-INK yuchen.lin@usc.edu 2021-01-01	43.14	39.20	35.12	39.15
DrFact+Reranker	USC-INK yuchen.lin@usc.edu 2021-01-01	47.73	44.30	39.60	43.88