Title: | Investigating BERT’s Knowledge of Language: Five Analysis Methods with NPIs |
Authors: | Warstadt, Alex Bowman, Samuel R. et al. |
Keywords: | computational linguistics, natural language processing |
Issue Date: | Nov-2019 |
Publisher: | Association for Computational Lingusitics |
Citation: | Anthology ID: D19-1286 Volume: Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP) Month: November Year: 2019 Address: Hong Kong, China Venues: EMNLP | IJCNLP SIG: SIGDAT Publisher: Association for Computational Linguistics Note: Pages: 2877–2887 URL: https://www.aclweb.org/anthology/D19-1286 DOI: 10.18653/v1/D19-1286 |
Series/Report no.: | ACL Anthology;D19-1286 |
Abstract: | Though state-of-the-art sentence representation models can perform tasks requiring significant knowledge of grammar, it is an open question how best to evaluate their grammatical knowledge. We explore five experimental methods inspired by prior work evaluating pretrained sentence representation models. We use a single linguistic phenomenon, negative polarity item (NPI) licensing, as a case study for our experiments. NPIs like any are grammatical only if they appear in a licensing environment like negation (Sue doesn’t have any cats vs. *Sue has any cats). This phenomenon is challenging because of the variety of NPI licensing environments that exist. We introduce an artificially generated dataset that manipulates key features of NPI licensing for the experiments. We find that BERT has significant knowledge of these features, but its success varies widely across different experimental methods. We conclude that a variety of methods is necessary to reveal all relevant aspects of a model’s grammatical knowledge in a given domain. |
URI: | http://hdl.handle.net/2451/60991 |
DOI: | 10.18653/v1/D19-1286 |
Appears in Collections: | Machine Learning for Language Lab |
Files in This Item:
File | Description | Size | Format | |
---|---|---|---|---|
data_generation_code.zip | Python 3 code for data generation. | 226.92 MB | Unknown | View/Open |
modeling_code.zip | Python 3 modeling code based on the jiant package. See scripts/bert_npi for instructions. | 58.17 MB | Unknown | View/Open |
npi_data_all_environments.tsv | Experimental data. | 70.95 MB | Unknown | View/Open |
paper.pdf | Research paper published at EMNLP 2019. | 1.27 MB | Adobe PDF | View/Open |
paper_appendix.pdf | Appendix/supplement to research paper published at EMNLP 2019. | 1.93 MB | Adobe PDF | View/Open |
Items in FDA are protected by copyright, with all rights reserved, unless otherwise indicated.