代做program、代写Python程序设计

日期：2024-03-27 09:06

Assignment 1

Objective

The aim is to develop queries on your topic, index the Assignment 1 Document Collection, develop a Gold Standard and hence evaluate your system.

Developing Queries

You need to develop 20 queries, based on your chosen topic, as registered on Moodle and agreed with the lecturer. If you want to confirm what your topic is, you need to contact the lecturer. Each query appears in three forms in the Gold Standard: original_query, keyword_query, and kibana_query (see below). The 20 queries should be broken down into the following types:

TypeExplanation

PersonQueries asking for a person name

AnimalQueries asking for a type of animal

OrganisationQueries asking for the name of an organisation

EventQueries asking for an event

PlaceQueries asking for a place name of location

DateQueries asking for a date or time

ReasonQueries asking for a reason for something / cause of something

OtherQueries asking for something else

Note that the queries are classified based on the type of information you are looking for. They are not classified based on the types of information in the query itself. So, for example a query like ‘When did Dire Straits play a concert in Dublin’ mentions an Organisation (Dire Straits), an Event (concert), and a Place (Dublin) in the query but the type of the query is in fact Date, because that is the type of the information we are looking for.

Try to use at least three types of query in your Gold Standard. Try searches of different types for your chosen topic and see what documents you can match. Based on that, decide which query topics work best for you. In general, try to use several different query types, minimum three.

First, you write down the original_query, which is complete and in English. Next, develop a Kibana Query (kibana_query in the .json) which matches relevant documents as well as it can, using keywords from the original_query but possibly combining them into exact phrases, and using a ‘bool’ of type ‘must’ or ‘should’ etc. See the lab sheets for hints. For finding a named entity such as a person or organisation, a phrase match always works well. Finally, keyword_query will be used in the Python as a simple best_fields search but uses exactly the same keywords as the kibana_query.

You need to experiment with various queries to see whether they will retrieve relevant documents for your project. Once you are happy with your query, you will have a kibana_query based on the keywords, possibly combined into phrases etc., and a keyword_query based on the same keywords. Now, keep these fixed, and start formally searching for matching documents to put in the .json Gold Standard.

When you enter a query and obtain results, you need to look at the top 40 documents returned and, for each document decide whether it contains the answer to the query, and if so, why. If a document contains the answer, we say it is ‘relevant’. If no document in the top 40 is relevant, you must change your query in order to meet this condition. So you need to spend some time with Elasticsearch/Kibana in order to develop your queries.

Queries beyond the first 40, we never look at. So, we never know how many relevant documents there really are for a query. We just estimate this, using the first 40.

You are allowed to reverse-engineer this. So, you can do some searches on your topic, and, based on the documents returned, see what questions are going to have relevant documents.

Developing the Gold Standard

Your Gold Standard will be a JSON containing the queries. For each query there will be one or more relevant documents, each specified by their DocID (shown as “_id” in the document collection. For each relevant document, there will be a list of one or more sentences, taken from the document, which contain the answer to the query (we say that the sentences support the answer). Sometimes, two sentences are needed, because one sentence alone does not demonstrate relevance, but in most cases, one sentence will demonstrate the relevance on its own.

The format of the Gold Standard JSON is shown in the file gold_standard.json It shows a complete example of a query and its matching documents. Also see worked_example_gold_standard.txt which shows exactly how the data was derived. Finally, rgs_read_gold_standard.py is a useful program which can read in your .json and print out information about it. If that program crashes, there is a problem with your .json. The error message gives the line number in the .json which is useful for tracking down mistakes.

The JSON you hand in must be syntactically correct (i.e. we must be able parse it after submission, and convert to a Python dictionary). If rgs_read_gold_standard.py can read your .json then it is OK syntactically.

The Document Collection

The document collection is called the ‘Assignment 1 Document Collection’

Evaluating the System

We will evaluate the standard Elasticsearch system, using both the keyword_query and the kibana_query. For keyword_query we will use a simple  “best_fields” search, incorporating the keywords as the query. For kibana_query, we will submit the exact kibana query you provide in the Gold Standard, using the Python interface to Elasticsearch.

We can evaluate the system automatically, at n = 5 and n = 10. Some code will be provided to do this, but you will need to combine the parts together.

Concerning Recall, we will assume that all relevant documents are returned in the first 40. This assumption is unlikely to be true, of course, but it is needed to make the project practical.

Assignment 2

Objective

The aim is to extend the Information Retrieval system you developed in Assignment 1, to make it into a Question Answering system. Almost all the code is provided, you just need to write some small parts and carry out some analysis.

Stages of Question Answering

As discussed in detail in the lectures, the stages of question answering are:

1. Start with a query (NB use original_query form of your queries):

Who played keyboards in the band Dire Straits

2. Extract keywords for search:

play keyboard band dire straits

3. Classify query by expected answer type:

person (in our .json)

PERSON (in SpaCy)

4. Search with elastic to find matching documents

5. From those documents find supporting sentences

6. Search for NEs in the supporting sentences using SpaCy NE recognition

7. Look for sentences which contain at least one NE of correct type (i.e. PERSON in our example)

8. In sentences, identify NEs and also identify relevant keywords (see keywords we extracted above)

9. Score all NEs based on their distance from relevant keywords

10. Select the NE with the best score

11. Return it as the answer.

You will be supplied with a basic program which carries out these stages. When you run it, it will try to answer the questions. However, you must complete the stages below before it can work properly.

The program is called aqu_answer_question_v2.py and you can find it on Moodle. You also need one_query_to_show_qa.json.

Task 1: Extract Keywords for Search

See aqu_extract_keywords_from_original_query( original_query )

This takes original_query as a string and returns a list of keywords. There is basic code (commented out) to tokenise the query and convert words to lemma (root) form. You need to:

?Study your queries

?Decide what words to count as stopwords

?Change the code in the above function to remove those stopwords

Note: At present this function returns a fixed list of words, irrespective of the input! You need to change this function before doing anything else.

Task 2: Classify Query

The next task is to develop the keyword query classifier so that it correctly classifies your 20 queries from Assignment 1.

A basic function is provided as a starting point: aqu_keyword_classify_query( query )

You can do this by using more complicated boolean expressions for each query type, such as:

if ('this' in tokens and 'that' in tokens) or 'other' in tokens:

   return( 'person' )

The function must be able to recognise all .json query types. At present the function is only a draft, and does not.

You are welcome to use a more sophisticated approach.

For testing, write a function called aqu_evaluate_keyword_classify_query() which goes through the queries and classifies them with aqu_keyword_classify_query( query ). It is already in the python file with detailed instructions, you just need to add the code.

Task 3: Query Type Mapping

You need to complete the function

aqu_convert_json_type_to_spacy_type( json_type )

You need to map all .json query types (from Assignment 1) to suitable SpaCy NE types. Where there are several possible SpaCy types, choose what you consider to be the most likely one.

Remember, the SpaCy type you choose will the one that is searched for after NE recognition.

In your video, you will need to justify (very briefly) the mapping you chose.

Task 4: Find Required Number of Docs

You need to look at the documents returned by Elastic when it is queried by the question answering system.

You can do this by finding the exact Elastic query that is used in aqu_answer_the_questions() (it gets printed out) and submitting that in Kibana.

Then, look for answers in the documents returned and find the first document which contains an answer to the query. At present, aqu_answer_the_questions() only looks in the first sentence of each document. Do this for all 20 queries, and hence find the minimum number of documents you need to return in order to be able to find at least one answer to every query. Call that number n.  

Now, you can set this number in aqu_answer_the_questions() to n:

               size = 10, # Max number of hits to return. Default is 10.

When you have found one or more exact answers then add them to your Gold Standard as a new attribute: exact_answers, which is a list of strings (see one_query_to_show_qa.json).

You should include in the list any answers to the question which you find in the first n documents. An answer is an exact substring of the text. Here is an example we saw before:

At the time of their initial breakup in September 1988, the band featured Mark Knopfler, Illsley, rhythm guitarist and backing vocalist Jack Sonni, drummer Terry Williams and keyboardists Alan Clark and Guy Fletcher.

Suppose we are looking for the keyboardists. There are two answers, Alan Clark and Guy Fletcher. We use exact substrings of the text. The previous .json for this query was like this:

     "answer_type":"person",

     "matches":

Between these two lines we have added the exact_answers, in one_query_to_show_qa.json, like this:

     "answer_type":"person",

     "exact_answers": [ "Alan Clark", "Guy Fletcher" ],

     "matches":

Hint: In Kibana left hand window you can put many different queries. When you log off and return, those queries are still there. To run any query, place your cursor in it and click the right arrow. You can save queries you want to show, ready to demonstrate them in the video.

Task 5: Study NE Matching in Documents Returned

aqu_answer_the_questions() gives detailed output. First of all, you should make a close study of all the output produced for each of your queries and make sure you understand it.

Now look at the recognition of NEs, the KEYWORDs used and the scoring of answers. Be ready to explain how this scoring works in your video.

Is it working properly or not? If not, why not? Go through each of your queries in turn and analyse the results at the end of each query. Use text like this:

Analysis:

1. Best scoring answer: Alan Clark

2. Correct/Incorrect: Correct

3. Reason Correct/Reason Incorrect: There is only one NE candidate in this case.

For (1) write down the answer which has the best (lowest) score.

For (2) state whether it is correct or incorrect.

For (3) try to find the reason. If correct, mostly it is because the KEYWORDs are closest to the answer. Sometimes there is only one NE (as above). If incorrect, is the correct NE actually in the sentence? If it is there, why did another one match better? Are the KEYWORDs suitable for the query? If not, why not? Is it a limitation of the scoring function?

Place your transcript for all the queries, as well as this analysis for each query, in the file assignment_2_analysis.docx.

You will need to submit a transcript of the program running on all your queries, and insert comments into it at the end of each query section, as shown at the end of this document. A file assignment_2_analysis.docx is provided for this.

Task 6: Overall Performance Results

Fill in the table in assignment_2_analysis.docx.