program代写程序代写、Java编程设计调试、代做Java

日期：2020-11-20 10:56

Lab 11: Natural Language Processing

This lab is bonus and is not required.

Goals for this lab:

? Use HashMaps to quickly store and access data

? Use HashSets to maintain a duplicate free collection of data

? Determine the term frequency of words in a file

? Prevent stopwords from being included in the term frequency

? Write, compile, and execute Java programs using the given instructions

General Guidelines:

? Use meaningful variable names

? Use appropriate indentation

? Use comments, especially for the header, variables, and blocks of code.

Example Header:

Date:

Instructor:

Description: A small description in your own words

that describe what the program does. Any

additional flags needed to allow the

program to compile should also be placed

here.

*/

1. Term Frequency

The area of natural language processing is typically an intersection between linguistics, computer science, and

artificial intelligence. Algorithms and software are developed in order to allow the computer to infer meaning

and make decisions based on human languages. This could be as complex as teaching a computer to carry on a

reasonable conversation with a human, or as “simple” as trying to determine if people liked a movie based on

their social media posts. In both cases, sentences must be broken down into individual words or phrases so

that the computer can process them. One of the first steps is often term frequency analyses, where the

frequency of words are counted, so that the most important or significant words can be determined.

For this exercise you will be writing a program to count the number of words in chunk of text. The input file

will contain multiple lines, and punctuation has already been removed, however not all of the words are

completely lowercase. You will need to break up each line into individual words, and maintain a count of how

many times each word appears in the overall text chunk using a HashMap. Finally, you will need to output the

list of words and their frequencies to the user

To complete this program:

? In a file named FreqCountV1.java, define a class named FreqCountV1 which will perform the text

analysis. The class should contain:

o A private, static method named readTextFile, which takes in a HashMap and returns

nothing and should:

? Prompt the user for the name of the file containing the text, and repeatedly reprompt

the user if the filename they enter does not exist

? Read in the file one word at a time and for each word

? Convert it to all lowercase letters

? If the word already exists in the HashMap, increase its count by one in the

HashMap

? Otherwise, add it to the HashMap and set its count to one

o A private, static method named outputFreq, which takes in a HashMap and returns nothing

and should:

? Output each word and its associated count from the HashMap

? Because words are different lengths, make sure that you determine the length of the

longest word and use that in your formatting to produce a cleaner output (see

outputV1.txt)

o A main method, which should:

? Contain a HashMap that indexes counts using Strings

? Call readTextFile using the HashMap

? Call outputFreq using the HashMap

Complete the program, making sure to compile and run it to verify that it produces the correct results. Note

that you will submit the FreqCountV1.java file to Canvas.

2. Stopwords

One of the initial challenges facing proper term frequency analysis is the occurrence of stopwords, or words

that provide little to no information regarding the text. These are typically common words such as “the”, “a”,

“and”, “have”, or single letters. Thus, to focus on the more important words, we want to perform stopword

removal (or at least prevent their inclusion) in our analysis.

For this exercise you will be improving upon your previous code from part 1 of this lab. You will perform all of

the same steps from last time, but before you read in the text file, you will need to read in a list of stopwords

and store them in a HashSet. Then, as you are reading in your text file, use the HashSet to prevent the

adding of stopwords to the HashMap containing your term frequencies.

To complete this program:

? In a file named FreqCountV2.java, define a class named FreqCountV2 which will perform the text

analysis. The class should contain:

o A private, static method named readStopWordsFile, which takes in a HashSet and

returns nothing and should:

? Prompt the user for the name of the file containing the list of stopwords, and repeatedly

reprompt the user if the filename they enter does not exist

? Read in the file one word at a time and for each word add it to the HashSet

o A private, static method named readTextFile, which takes in a HashMap, a HashSet,

and returns nothing and should:

? Prompt the user for the name of the file containing the text, and repeatedly reprompt

the user if the filename they enter does not exist

? Read in the file one word at a time and for each word

? Convert it to all lowercase letters

? If the word exists in the HashSet, ignore it and do not add it to the HashMap

? If the word already exists in the HashMap, increase its count by one in the

HashMap

? Otherwise, add it to the HashMap and set its count to one

o A private, static method named outputFreq, which takes in a HashMap and returns nothing

and should:

? Output each word and its associated count from the HashMap

? Because words are different lengths, make sure that you determine the length of the

longest word and use that in your formatting to produce a cleaner output (see

outputV2.txt)

o A main method, which should:

? Contain a HashMap that indexes counts using Strings

? Contain a HashSet that stores stopwords as Strings

? Call readStopWordsFile using the HashSet

? Call readTextFile using the HashMap and HashSet

? Call outputFreq using the HashMap

Complete the program, making sure to compile and run it to verify that it produces the correct results. Note

that you will submit the FreqCountV2.java file to Canvas.

Submission

Once you have completed this lab it is time to turn in your work. Please submit the following files to the Lab

11 assignment in Canvas:

? FreqCountV1.java

? FreqCountV2.java

After you have submitted your files, you are welcome to practice some additional coding.

Rubric

Part 1 60 pts:

? 5 points: Program compiles successfully

? 20 points: Correctly implementing the readTextFile method

o 5 points for obtaining the filename from the user and opening the file

o 5 points for converting all words to all lowercase letters

o 5 points for increasing a words frequency count if it does exist

o 5 points for adding the word and setting its frequency count to one if it does not exist

? 20 points: Correctly implementing the outputFreq method

o 5 points for correctly outputting all of the words in the text file

o 10 points for correctly outputting all of the word frequencies in the text file

o 5 points outputting all of the words and their frequencies with proper formatting (See

outputV1.txt)

? 10 points: Correctly implementing the main method

o 4 points for correctly creating the HashMap

o 3 points for calling the readTextFile method with the appropriate input

o 3 points for calling the outputFreq method with the appropriate input

? 5 points: Appropriate commenting is included

Part 2 40 pts:

? 5 points: Program compiles successfully

? 10 points: Correctly implementing the readStopWordsFile method

o 5 points for obtaining the filename from the user and opening the file

o 5 points for adding all of the stopwords from the file to the HashSet

? 20 points: Correctly implementing the outputFreq method

o 5 points for correctly outputting all of the words in the text file, excluding stopwords

o 10 points for correctly outputting all of the word frequencies in the text file, excluding

stopwords

o 5 points outputting all of the words and their frequencies, excluding stopwords, with proper

formatting (See outputV2.txt)

? 5 points: Appropriate commenting is included