Advanced Syllabus Analyzer

Abstract

The purpose of this project is to develop a module to perform TF-IDF text similarity scoring analysis for New York University School of Professional Studies. The module is programmed in R programming language and hosted on the shinyapp.io.server. This module features keyword extraction, online source scrape, cloud computing, and Shiny-based user experience. This module allows users to upload a text file of the syllabus locally, and then it will conduct text similarity analysis against the syllabus and O*Net Occupation Database and return 20 most matched occupations that the students are prepared for after taking the courses. The users then can choose up to 5 occupations, and the module will scrape 15 jobs for each occupation from Indeed.com and conduct term frequency analysis for the syllabus and scraped job descriptions.

The underlying algorithms of this module are as following: extract and clean the text, drop all non-alphabetical characters, eliminate multi-space, and lemmatize the all the words; apply TF-IDF analysis and compute the similarity score against the text and the occupation descriptions; scrape job information and conduct term frequency analysis for the text and job descriptions.

With this new tool, instructors at New York University School of Professional Studies would have an easy-to-use R tool to ascertain the matched occupations with the course and adjust the course content or structure accordingly. The module works for both Windows and Macintosh operating systems. Samples and comprehensive are provided in the GitHub Repository.