Skip navigation
Full metadata record
DC FieldValueLanguage
dc.contributor.authorWarstadt, Alex-
dc.contributor.authorSingh, Amanpreet-
dc.contributor.authorBowman, Samuel R.-
dc.date.accessioned2019-10-10T16:47:40Z-
dc.date.available2019-10-10T16:47:40Z-
dc.date.issued2019-
dc.identifier.urihttp://hdl.handle.net/2451/60441-
dc.description.abstract[Primary paper:] This paper investigates the ability of artificial neural networks to judge the grammatical acceptability of a sentence, with the goal of testing their linguistic competence. We introduce the Corpus of Linguistic Acceptability (CoLA), a set of 10,657 English sentences labeled as grammatical or ungrammatical from published linguistics literature. As baselines, we train several recurrent neural network models on acceptability classification, and find that our models outperform unsupervised models by Lau et al. (2016) on CoLA. Error-analysis on specific grammatical phenomena reveals that both Lau et al.’s models and ours learn systematic generalizations like subject-verb-object order. However, all models we test perform far below human level on a wide range of grammatical constructions.en
dc.description.sponsorshipThis project has benefited from help and feedback at various stages from Chris Barker, Pablo Gonzalez, Shalom Lappin, Omer Levy, Marie-Catherine de Marneffe, Alex Wang, Alexander Clark, everyone in the Deep Learning in Semantics seminar at NYU, and three anonymous TACL reviewers. This project has benefited from financial support to SB by Google, Tencent Holdings, and Samsung Research. This material is based upon work supported by the National Science Foundation under Grant No. 1850208. Any opinions, findings, and conclusions or recommendations expressed in this material are those of the author(s) and do not necessarily reflect the views of the National Science Foundation.en
dc.language.isoenen
dc.titleCoLA: The Corpus of Linguistic Acceptability (with added annotations)en
dc.title.alternativeNeural Network Acceptability Judgmentsen
dc.title.alternativeLinguistic Analysis of Pretrained Sentence Encoders with Acceptability Judgmentsen
dc.title.alternativeCoLA 1.1en
dc.typeDataseten
Appears in Collections:Machine Learning for Language Lab

Files in This Item:
File Description SizeFormat 
1805.12471.pdfThe paper describing the CoLA corpus.712.7 kBAdobe PDFView/Open
1901.03438.pdfThe paper describing the additional validation set annotations.1.08 MBAdobe PDFView/Open
CoLA_grammatical_annotations_major_features.txtThe annotated validation set with phenomenon-level annotations.85.47 kBTextView/Open
CoLA_grammatical_annotations_minor_features.txtThe annotated validation set with phenomenon-level annotations.183.74 kBTextView/Open
Primary Corpus README.txtAdditional documentation for the CoLA 1.1 corpus. Only the raw version is distributed here.6.42 kBTextView/Open
in_domain_dev.txtThe primary CoLA 1.1 corpus.25.35 kBTextView/Open
in_domain_train.txtThe primary CoLA 1.1 corpus.418.53 kBTextView/Open
out_of_domain_dev.txtThe primary CoLA 1.1 corpus.27.1 kBTextView/Open


Items in FDA are protected by copyright, with all rights reserved, unless otherwise indicated.