{
"cells": [
{
"cell_type": "markdown",
"metadata": {},
"source": [
"# Assignment 3: Hello Vectors\n",
"\n",
"Welcome to this week's programming assignment on exploring word vectors.\n",
"In natural language processing, we represent each word as a vector consisting of numbers.\n",
"The vector encodes the meaning of the word. These numbers (or weights) for each word are learned using various machine\n",
"learning models, which we will explore in more detail later in this specialization. Rather than make you code the\n",
"machine learning models from scratch, we will show you how to use them. In the real world, you can always load the\n",
"trained word vectors, and you will almost never have to train them from scratch. In this assignment, you will:\n",
"\n",
"- Predict analogies between words.\n",
"- Use PCA to reduce the dimensionality of the word embeddings and plot them in two dimensions.\n",
"- Compare word embeddings by using a similarity measure (the cosine similarity).\n",
"- Understand how these vector space models work.\n",
"\n",
"\n",
"\n",
"## 1.0 Predict the Countries from Capitals\n",
"\n",
"In the lectures, we have illustrated the word analogies\n",
"by finding the capital of a country from the country. \n",
"We have changed the problem a bit in this part of the assignment. You are asked to predict the **countries** \n",
"that corresponds to some **capitals**.\n",
"You are playing trivia against some second grader who just took their geography test and knows all the capitals by heart.\n",
"Thanks to NLP, you will be able to answer the questions properly. In other words, you will write a program that can give\n",
"you the country by its capital. That way you are pretty sure you will win the trivia game. We will start by exploring the data set.\n",
"\n",
"\n",
"\n",
"### 1.1 Importing the data\n",
"\n",
"As usual, you start by importing some essential Python libraries and then load the dataset.\n",
"The dataset will be loaded as a [Pandas DataFrame](https://pandas.pydata.org/pandas-docs/stable/getting_started/dsintro.html),\n",
"which is very a common method in data science.\n",
"This may take a few minutes because of the large size of the data."
]
},
{
"cell_type": "code",
"execution_count": 1,
"metadata": {},
"outputs": [],
"source": [
"# Run this cell to import packages.\n",
"import pickle\n",
"import numpy as np\n",
"import pandas as pd\n",
"import matplotlib.pyplot as plt\n",
"\n",
"from utils import get_vectors"
]
},
{
"cell_type": "code",
"execution_count": 2,
"metadata": {},
"outputs": [
{
"data": {
"text/html": [
"
\n",
"\n",
"
\n",
" \n",
"
\n",
"
\n",
"
city1
\n",
"
country1
\n",
"
city2
\n",
"
country2
\n",
"
\n",
" \n",
" \n",
"
\n",
"
0
\n",
"
Athens
\n",
"
Greece
\n",
"
Bangkok
\n",
"
Thailand
\n",
"
\n",
"
\n",
"
1
\n",
"
Athens
\n",
"
Greece
\n",
"
Beijing
\n",
"
China
\n",
"
\n",
"
\n",
"
2
\n",
"
Athens
\n",
"
Greece
\n",
"
Berlin
\n",
"
Germany
\n",
"
\n",
"
\n",
"
3
\n",
"
Athens
\n",
"
Greece
\n",
"
Bern
\n",
"
Switzerland
\n",
"
\n",
"
\n",
"
4
\n",
"
Athens
\n",
"
Greece
\n",
"
Cairo
\n",
"
Egypt
\n",
"
\n",
" \n",
"
\n",
"
"
],
"text/plain": [
" city1 country1 city2 country2\n",
"0 Athens Greece Bangkok Thailand\n",
"1 Athens Greece Beijing China\n",
"2 Athens Greece Berlin Germany\n",
"3 Athens Greece Bern Switzerland\n",
"4 Athens Greece Cairo Egypt"
]
},
"execution_count": 2,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"data = pd.read_csv('capitals.txt', delimiter=' ')\n",
"data.columns = ['city1', 'country1', 'city2', 'country2']\n",
"\n",
"# print first five elements in the DataFrame\n",
"data.head(5)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"***\n",
"\n",
"### To Run This Code On Your Own Machine:\n",
"Note that because the original google news word embedding dataset is about 3.64 gigabytes,\n",
"the workspace is not able to handle the full file set. So we've downloaded the full dataset,\n",
"extracted a sample of the words that we're going to analyze in this assignment, and saved\n",
"it in a pickle file called `word_embeddings_capitals.p`\n",
"\n",
"If you want to download the full dataset on your own and choose your own set of word embeddings,\n",
"please see the instructions and some helper code.\n",
"\n",
"- Download the dataset from this [page](https://code.google.com/archive/p/word2vec/).\n",
"- Search in the page for 'GoogleNews-vectors-negative300.bin.gz' and click the link to download."
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Copy-paste the code below and run it on your local machine after downloading\n",
"the dataset to the same directory as the notebook.\n",
"\n",
"```python\n",
"import nltk\n",
"from gensim.models import KeyedVectors\n",
"\n",
"\n",
"embeddings = KeyedVectors.load_word2vec_format('./GoogleNews-vectors-negative300.bin', binary = True)\n",
"f = open('capitals.txt', 'r').read()\n",
"set_words = set(nltk.word_tokenize(f))\n",
"select_words = words = ['king', 'queen', 'oil', 'gas', 'happy', 'sad', 'city', 'town', 'village', 'country', 'continent', 'petroleum', 'joyful']\n",
"for w in select_words:\n",
" set_words.add(w)\n",
"\n",
"def get_word_embeddings(embeddings):\n",
"\n",
" word_embeddings = {}\n",
" for word in embeddings.vocab:\n",
" if word in set_words:\n",
" word_embeddings[word] = embeddings[word]\n",
" return word_embeddings\n",
"\n",
"\n",
"# Testing your function\n",
"word_embeddings = get_word_embeddings(embeddings)\n",
"print(len(word_embeddings))\n",
"pickle.dump( word_embeddings, open( \"word_embeddings_subset.p\", \"wb\" ) )\n",
"```\n",
"\n",
"***"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Now we will load the word embeddings as a [Python dictionary](https://docs.python.org/3/tutorial/datastructures.html#dictionaries).\n",
"As stated, these have already been obtained through a machine learning algorithm. "
]
},
{
"cell_type": "code",
"execution_count": 3,
"metadata": {},
"outputs": [
{
"data": {
"text/plain": [
"243"
]
},
"execution_count": 3,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"word_embeddings = pickle.load(open(\"word_embeddings_subset.p\", \"rb\"))\n",
"len(word_embeddings) # there should be 243 words that will be used in this assignment"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Each of the word embedding is a 300-dimensional vector."
]
},
{
"cell_type": "code",
"execution_count": 4,
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"dimension: 300\n"
]
}
],
"source": [
"print(\"dimension: {}\".format(word_embeddings['Spain'].shape[0]))"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"### Predict relationships among words\n",
"\n",
"Now you will write a function that will use the word embeddings to predict relationships among words.\n",
"* The function will take as input three words.\n",
"* The first two are related to each other.\n",
"* It will predict a 4th word which is related to the third word in a similar manner as the two first words are related to each other.\n",
"* As an example, \"Athens is to Greece as Bangkok is to ______\"?\n",
"* You will write a program that is capable of finding the fourth word.\n",
"* We will give you a hint to show you how to compute this.\n",
"\n",
"A similar analogy would be the following:\n",
"\n",
"\n",
"\n",
"You will implement a function that can tell you the capital of a country.\n",
"You should use the same methodology shown in the figure above. To do this,\n",
"compute you'll first compute cosine similarity metric or the Euclidean distance."
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"### 1.2 Cosine Similarity\n",
"\n",
"The cosine similarity function is:\n",
"\n",
"$$\\cos (\\theta)=\\frac{\\mathbf{A} \\cdot \\mathbf{B}}{\\|\\mathbf{A}\\|\\|\\mathbf{B}\\|}=\\frac{\\sum_{i=1}^{n} A_{i} B_{i}}{\\sqrt{\\sum_{i=1}^{n} A_{i}^{2}} \\sqrt{\\sum_{i=1}^{n} B_{i}^{2}}}\\tag{1}$$\n",
"\n",
"$A$ and $B$ represent the word vectors and $A_i$ or $B_i$ represent index i of that vector.\n",
"& Note that if A and B are identical, you will get $cos(\\theta) = 1$.\n",
"* Otherwise, if they are the total opposite, meaning, $A= -B$, then you would get $cos(\\theta) = -1$.\n",
"* If you get $cos(\\theta) =0$, that means that they are orthogonal (or perpendicular).\n",
"* Numbers between 0 and 1 indicate a similarity score.\n",
"* Numbers between -1-0 indicate a dissimilarity score.\n",
"\n",
"**Instructions**: Implement a function that takes in two word vectors and computes the cosine distance."
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"\n",
"\n",
" Hints\n",
"\n",
"
\n",
"
\n",
"
Python's NumPy library adds support for linear algebra operations (e.g., dot product, vector norm ...).
\n",
""
]
},
{
"cell_type": "code",
"execution_count": 5,
"metadata": {},
"outputs": [],
"source": [
"# UNQ_C1 (UNIQUE CELL IDENTIFIER, DO NOT EDIT)\n",
"def cosine_similarity(A, B):\n",
" '''\n",
" Input:\n",
" A: a numpy array which corresponds to a word vector\n",
" B: A numpy array which corresponds to a word vector\n",
" Output:\n",
" cos: numerical number representing the cosine similarity between A and B.\n",
" '''\n",
"\n",
" ### START CODE HERE (REPLACE INSTANCES OF 'None' with your code) ###\n",
" \n",
" dot = np.dot(A, B)\n",
" norma = np.linalg.norm(A)\n",
" normb = np.linalg.norm(B)\n",
" cos = dot/(norma * normb)\n",
"\n",
" ### END CODE HERE ###\n",
" return cos"
]
},
{
"cell_type": "code",
"execution_count": 6,
"metadata": {},
"outputs": [
{
"data": {
"text/plain": [
"0.6510956"
]
},
"execution_count": 6,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"# feel free to try different words\n",
"king = word_embeddings['king']\n",
"queen = word_embeddings['queen']\n",
"\n",
"cosine_similarity(king, queen)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"**Expected Output**:\n",
"\n",
"$\\approx$ 0.6510956"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"### 1.3 Euclidean distance\n",
"\n",
"You will now implement a function that computes the similarity between two vectors using the Euclidean distance.\n",
"Euclidean distance is defined as:\n",
"\n",
"$$ \\begin{aligned} d(\\mathbf{A}, \\mathbf{B})=d(\\mathbf{B}, \\mathbf{A}) &=\\sqrt{\\left(A_{1}-B_{1}\\right)^{2}+\\left(A_{2}-B_{2}\\right)^{2}+\\cdots+\\left(A_{n}-B_{n}\\right)^{2}} \\\\ &=\\sqrt{\\sum_{i=1}^{n}\\left(A_{i}-B_{i}\\right)^{2}} \\end{aligned}$$\n",
"\n",
"* $n$ is the number of elements in the vector\n",
"* $A$ and $B$ are the corresponding word vectors. \n",
"* The more similar the words, the more likely the Euclidean distance will be close to 0. \n",
"\n",
"**Instructions**: Write a function that computes the Euclidean distance between two vectors."
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
" \n",
"\n",
" Hints\n",
"\n",
"
\n",
""
]
},
{
"cell_type": "code",
"execution_count": 7,
"metadata": {},
"outputs": [],
"source": [
"# UNQ_C2 (UNIQUE CELL IDENTIFIER, DO NOT EDIT)\n",
"def euclidean(A, B):\n",
" \"\"\"\n",
" Input:\n",
" A: a numpy array which corresponds to a word vector\n",
" B: A numpy array which corresponds to a word vector\n",
" Output:\n",
" d: numerical number representing the Euclidean distance between A and B.\n",
" \"\"\"\n",
"\n",
" ### START CODE HERE (REPLACE INSTANCES OF 'None' with your code) ###\n",
"\n",
" # euclidean distance\n",
"\n",
" d = np.sqrt(np.sum(np.power(A - B, 2)))\n",
"\n",
" ### END CODE HERE ###\n",
"\n",
" return d\n"
]
},
{
"cell_type": "code",
"execution_count": 8,
"metadata": {},
"outputs": [
{
"data": {
"text/plain": [
"2.4796925"
]
},
"execution_count": 8,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"# Test your function\n",
"euclidean(king, queen)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"**Expected Output:**\n",
"\n",
"2.4796925"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"### 1.4 Finding the country of each capital\n",
"\n",
"Now, you will use the previous functions to compute similarities between vectors,\n",
"and use these to find the capital cities of countries. You will write a function that\n",
"takes in three words, and the embeddings dictionary. Your task is to find the\n",
"capital cities. For example, given the following words: \n",
"\n",
"- 1: Athens 2: Greece 3: Baghdad,\n",
"\n",
"your task is to predict the country 4: Iraq.\n",
"\n",
"**Instructions**: \n",
"\n",
"1. To predict the capital you might want to look at the *King - Man + Woman = Queen* example above, and implement that scheme into a mathematical function, using the word embeddings and a similarity function.\n",
"\n",
"2. Iterate over the embeddings dictionary and compute the cosine similarity score between your vector and the current word embedding.\n",
"\n",
"3. You should add a check to make sure that the word you return is not any of the words that you fed into your function. Return the one with the highest score."
]
},
{
"cell_type": "code",
"execution_count": 9,
"metadata": {},
"outputs": [],
"source": [
"# UNQ_C3 (UNIQUE CELL IDENTIFIER, DO NOT EDIT)\n",
"def get_country(city1, country1, city2, embeddings):\n",
" \"\"\"\n",
" Input:\n",
" city1: a string (the capital city of country1)\n",
" country1: a string (the country of capital1)\n",
" city2: a string (the capital city of country2)\n",
" embeddings: a dictionary where the keys are words and values are their embeddings\n",
" Output:\n",
" countries: a dictionary with the most likely country and its similarity score\n",
" \"\"\"\n",
" ### START CODE HERE (REPLACE INSTANCES OF 'None' with your code) ###\n",
"\n",
" # store the city1, country 1, and city 2 in a set called group\n",
" group = set((city1, country1, city2))\n",
"\n",
" # get embeddings of city 1\n",
" city1_emb = embeddings[city1]\n",
"\n",
" # get embedding of country 1\n",
" country1_emb = embeddings[country1]\n",
"\n",
" # get embedding of city 2\n",
" city2_emb = embeddings[city2]\n",
" # get embedding of country 2 (it's a combination of the embeddings of country 1, city 1 and city 2)\n",
" # Remember: King - Man + Woman = Queen\n",
" vec = country1_emb - city1_emb + city2_emb\n",
"\n",
" # Initialize the similarity to -1 (it will be replaced by a similarities that are closer to +1)\n",
" similarity = -1\n",
"\n",
" # initialize country to an empty string\n",
" country = ''\n",
"\n",
" # loop through all words in the embeddings dictionary\n",
" for word in embeddings.keys():\n",
"\n",
" # first check that the word is not already in the 'group'\n",
" if word not in group:\n",
"\n",
" # get the word embedding\n",
" word_emb = embeddings[word]\n",
"\n",
" # calculate cosine similarity between embedding of country 2 and the word in the embeddings dictionary\n",
" cur_similarity = cosine_similarity(word_emb, vec)\n",
"\n",
" # if the cosine similarity is more similar than the previously best similarity...\n",
" if cur_similarity > similarity:\n",
"\n",
" # update the similarity to the new, better similarity\n",
" similarity = cur_similarity\n",
"\n",
" # store the country as a tuple, which contains the word and the similarity\n",
" country = (word, similarity)\n",
"\n",
" ### END CODE HERE ###\n",
"\n",
" return country"
]
},
{
"cell_type": "code",
"execution_count": 10,
"metadata": {},
"outputs": [
{
"data": {
"text/plain": [
"('Egypt', 0.7626821)"
]
},
"execution_count": 10,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"# Testing your function, note to make it more robust you can return the 5 most similar words.\n",
"get_country('Athens', 'Greece', 'Cairo', word_embeddings)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"**Expected Output:**\n",
"\n",
"('Egypt', 0.7626821)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"### 1.5 Model Accuracy\n",
"\n",
"Now you will test your new function on the dataset and check the accuracy of the model:\n",
"\n",
"$$\\text{Accuracy}=\\frac{\\text{Correct # of predictions}}{\\text{Total # of predictions}}$$\n",
"\n",
"**Instructions**: Write a program that can compute the accuracy on the dataset provided for you. You have to iterate over every row to get the corresponding words and feed them into you `get_country` function above. "
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"\n",
"\n",
" Hints\n",
"\n",
"
\n",
""
]
},
{
"cell_type": "code",
"execution_count": 11,
"metadata": {},
"outputs": [],
"source": [
"# UNQ_C4 (UNIQUE CELL IDENTIFIER, DO NOT EDIT)\n",
"def get_accuracy(word_embeddings, data):\n",
" '''\n",
" Input:\n",
" word_embeddings: a dictionary where the key is a word and the value is its embedding\n",
" data: a pandas dataframe containing all the country and capital city pairs\n",
" \n",
" Output:\n",
" accuracy: the accuracy of the model\n",
" '''\n",
"\n",
" ### START CODE HERE (REPLACE INSTANCES OF 'None' with your code) ###\n",
" # initialize num correct to zero\n",
" num_correct = 0\n",
"\n",
" # loop through the rows of the dataframe\n",
" predicted_data = data.apply(lambda x : get_country(x['city1'], x['country1'], x['city2'], word_embeddings)[0], axis=1)\n",
" accuracy = (np.sum(predicted_data == data.country2) / len(data))\n",
"# for i, row in data.iterrows():\n",
"\n",
"# # get city1\n",
"# city1 = None\n",
"\n",
"# # get country1\n",
"# country1 = None\n",
"\n",
"# # get city2\n",
"# city2 = None\n",
"\n",
"# # get country2\n",
"# country2 = None\n",
" \n",
" \n",
"\n",
"# # use get_country to find the predicted country2\n",
"# predicted_country2, _ = None\n",
"\n",
"# # if the predicted country2 is the same as the actual country2...\n",
"# if predicted_country2 == country2:\n",
"# # increment the number of correct by 1\n",
"# num_correct += None\n",
"\n",
" # get the number of rows in the data dataframe (length of dataframe)\n",
"# m = len(data)\n",
"\n",
" # calculate the accuracy by dividing the number correct by m\n",
"# accuracy = None\n",
"\n",
" ### END CODE HERE ###\n",
" return accuracy\n"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"**NOTE: The cell below takes about 30 SECONDS to run.**"
]
},
{
"cell_type": "code",
"execution_count": 12,
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"Accuracy is 0.92\n"
]
}
],
"source": [
"accuracy = get_accuracy(word_embeddings, data)\n",
"print(f\"Accuracy is {accuracy:.2f}\")"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"**Expected Output:**\n",
"\n",
"$\\approx$ 0.92"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"# 3.0 Plotting the vectors using PCA\n",
"\n",
"Now you will explore the distance between word vectors after reducing their dimension.\n",
"The technique we will employ is known as\n",
"[*principal component analysis* (PCA)](https://en.wikipedia.org/wiki/Principal_component_analysis).\n",
"As we saw, we are working in a 300-dimensional space in this case.\n",
"Although from a computational perspective we were able to perform a good job,\n",
"it is impossible to visualize results in such high dimensional spaces.\n",
"\n",
"You can think of PCA as a method that projects our vectors in a space of reduced\n",
"dimension, while keeping the maximum information about the original vectors in\n",
"their reduced counterparts. In this case, by *maximum infomation* we mean that the\n",
"Euclidean distance between the original vectors and their projected siblings is\n",
"minimal. Hence vectors that were originally close in the embeddings dictionary,\n",
"will produce lower dimensional vectors that are still close to each other.\n",
"\n",
"You will see that when you map out the words, similar words will be clustered\n",
"next to each other. For example, the words 'sad', 'happy', 'joyful' all describe\n",
"emotion and are supposed to be near each other when plotted.\n",
"The words: 'oil', 'gas', and 'petroleum' all describe natural resources.\n",
"Words like 'city', 'village', 'town' could be seen as synonyms and describe a\n",
"similar thing.\n",
"\n",
"Before plotting the words, you need to first be able to reduce each word vector\n",
"with PCA into 2 dimensions and then plot it. The steps to compute PCA are as follows:\n",
"\n",
"1. Mean normalize the data\n",
"2. Compute the covariance matrix of your data ($\\Sigma$). \n",
"3. Compute the eigenvectors and the eigenvalues of your covariance matrix\n",
"4. Multiply the first K eigenvectors by your normalized data. The transformation should look something as follows:\n",
"\n",
""
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"**Instructions**: \n",
"\n",
"You will write a program that takes in a data set where each row corresponds to a word vector. \n",
"* The word vectors are of dimension 300. \n",
"* Use PCA to change the 300 dimensions to `n_components` dimensions. \n",
"* The new matrix should be of dimension `m, n_componentns`. \n",
"\n",
"* First de-mean the data\n",
"* Get the eigenvalues using `linalg.eigh`. Use `eigh` rather than `eig` since R is symmetric. The performance gain when using `eigh` instead of `eig` is substantial.\n",
"* Sort the eigenvectors and eigenvalues by decreasing order of the eigenvalues.\n",
"* Get a subset of the eigenvectors (choose how many principle components you want to use using `n_components`).\n",
"* Return the new transformation of the data by multiplying the eigenvectors with the original data."
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"\n",
"\n",
" Hints\n",
"\n",
"
\n",
"
\n",
"
Use numpy.mean(a,axis=None) : If you set axis = 0, you take the mean for each column. If you set axis = 1, you take the mean for each row. Remember that each row is a word vector, and the number of columns are the number of dimensions in a word vector.
\n",
"
Use numpy.cov(m, rowvar=True) . This calculates the covariance matrix. By default rowvar is True. From the documentation: \"If rowvar is True (default), then each row represents a variable, with observations in the columns.\" In our case, each row is a word vector observation, and each column is a feature (variable).
Use numpy.argsort sorts the values in an array from smallest to largest, then returns the indices from this sort.
\n",
"
In order to reverse the order of a list, you can use: x[::-1].
\n",
"
To apply the sorted indices to eigenvalues, you can use this format x[indices_sorted].
\n",
"
When applying the sorted indices to eigen vectors, note that each column represents an eigenvector. In order to preserve the rows but sort on the columns, you can use this format x[:,indices_sorted]
\n",
"
To transform the data using a subset of the most relevant principle components, take the matrix multiplication of the eigenvectors with the original data.
\n",
"
The data is of shape (n_observations, n_features).
\n",
"
The subset of eigenvectors are in a matrix of shape (n_features, n_components).
\n",
"
To multiply these together, take the transposes of both the eigenvectors (n_components, n_features) and the data (n_features, n_observations).
\n",
"
The product of these two has dimensions (n_components,n_observations). Take its transpose to get the shape (n_observations, n_components).
\n",
"
\n",
""
]
},
{
"cell_type": "code",
"execution_count": 56,
"metadata": {},
"outputs": [],
"source": [
"# UNQ_C5 (UNIQUE CELL IDENTIFIER, DO NOT EDIT)\n",
"def compute_pca(X, n_components=2):\n",
" \"\"\"\n",
" Input:\n",
" X: of dimension (m,n) where each row corresponds to a word vector\n",
" n_components: Number of components you want to keep.\n",
" Output:\n",
" X_reduced: data transformed in 2 dims/columns + regenerated original data\n",
" \"\"\"\n",
"\n",
" ### START CODE HERE (REPLACE INSTANCES OF 'None' with your code) ###\n",
" # mean center the data\n",
" X_demeaned = X - np.mean(X, axis=0)\n",
" # calculate the covariance matrix\n",
" covariance_matrix = np.cov(X_demeaned, rowvar=False)\n",
" \n",
" # calculate eigenvectors & eigenvalues of the covariance matrix\n",
" eigen_vals, eigen_vecs = np.linalg.eigh(covariance_matrix, UPLO='L')\n",
"\n",
" # sort eigenvalue in increasing order (get the indices from the sort)\n",
" idx_sorted = np.argsort(eigen_vals)\n",
" \n",
" # reverse the order so that it's from highest to lowest.\n",
" idx_sorted_decreasing = idx_sorted[::-1]\n",
" \n",
" # sort the eigen values by idx_sorted_decreasing\n",
" eigen_vals_sorted = eigen_vals[idx_sorted_decreasing]\n",
"\n",
" # sort eigenvectors using the idx_sorted_decreasing indices\n",
" eigen_vecs_sorted = eigen_vecs[:, idx_sorted_decreasing]\n",
"\n",
" # select the first n eigenvectors (n is desired dimension\n",
" # of rescaled data array, or dims_rescaled_data)\n",
" eigen_vecs_subset = eigen_vecs_sorted[:,:n_components]\n",
"\n",
" # transform the data by multiplying the transpose of the eigenvectors \n",
" # with the transpose of the de-meaned data\n",
" # Then take the transpose of that product.\n",
" X_reduced = np.dot(eigen_vecs_subset.T ,X_demeaned.T).T\n",
"\n",
" ### END CODE HERE ###\n",
"\n",
" return X_reduced\n"
]
},
{
"cell_type": "code",
"execution_count": 57,
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"Your original matrix was (3, 10) and it became:\n",
"[[ 0.43437323 0.49820384]\n",
" [ 0.42077249 -0.50351448]\n",
" [-0.85514571 0.00531064]]\n"
]
}
],
"source": [
"# Testing your function\n",
"np.random.seed(1)\n",
"X = np.random.rand(3, 10)\n",
"X_reduced = compute_pca(X, n_components=2)\n",
"print(\"Your original matrix was \" + str(X.shape) + \" and it became:\")\n",
"print(X_reduced)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"**Expected Output:**\n",
"\n",
"Your original matrix was: (3,10) and it became:\n",
"\n",
"
\n",
"
\n",
"
\n",
" 0.43437323\n",
"
\n",
"
\n",
" 0.49820384\n",
"
\n",
"
\n",
"
\n",
"
\n",
" 0.42077249\n",
"
\n",
"
\n",
" -0.50351448\n",
"
\n",
"
\n",
"
\n",
"
\n",
" -0.85514571\n",
"
\n",
"
\n",
" 0.00531064\n",
"
\n",
"
\n",
"
\n",
"\n",
"Now you will use your pca function to plot a few words we have chosen for you.\n",
"You will see that similar words tend to be clustered near each other.\n",
"Sometimes, even antonyms tend to be clustered near each other. Antonyms\n",
"describe the same thing but just tend to be on the other end of the scale\n",
"They are usually found in the same location of a sentence,\n",
"have the same parts of speech, and thus when\n",
"learning the word vectors, you end up getting similar weights. In the next week\n",
"we will go over how you learn them, but for now let's just enjoy using them.\n",
"\n",
"**Instructions:** Run the cell below."
]
},
{
"cell_type": "code",
"execution_count": 59,
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"You have 11 words each of 300 dimensions thus X.shape is: (11, 300)\n"
]
}
],
"source": [
"words = ['oil', 'gas', 'happy', 'sad', 'city', 'town',\n",
" 'village', 'country', 'continent', 'petroleum', 'joyful']\n",
"\n",
"# given a list of words and the embeddings, it returns a matrix with all the embeddings\n",
"X = get_vectors(word_embeddings, words)\n",
"\n",
"print('You have 11 words each of 300 dimensions thus X.shape is:', X.shape)"
]
},
{
"cell_type": "code",
"execution_count": 60,
"metadata": {},
"outputs": [
{
"data": {
"image/png": "iVBORw0KGgoAAAANSUhEUgAAAZsAAAD8CAYAAAChHgmuAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDMuMC4zLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvnQurowAAIABJREFUeJzt3Xt0VdW59/HvQ0CNQkENIolIvCCVkJBABORuERCLAooWRBqkiniqvj0dZhQPbUXFo6+0VXi9UK2Cl1RBhIhXvCBHEGxJJMitVKRRIQgBDAckVAzP+0c2EWKu7L2zd5LfZ4w9WJe51nz2BPJkzTXXXObuiIiIhFOTSAcgIiINn5KNiIiEnZKNiIiEnZKNiIiEnZKNiIiEnZKNiIiEnZKNiIiEnZINYGaJZrYu0nGIiDRUSjYiIhJ2Fs0zCMTFxXliYmKNy5eUlLBlyxYOHTqEu9O2bVsOHjzI3r17OXz4MM2bN+fss8/GzPjmm2/4/PPPadKkCc2bN2fv3r0kJSWF78uIiNSB3NzcXe7eOtJxlNc00gFUJTExkZycnBqXf/nll3nrrbd48sknAdi7dy8lJSWcdtppAIwbN45rr72WK664gpSUFN5880369+9PZmYmb775Zq3qEhGJRmb2eaRjqEiD6kZLTk7m3Xff5Te/+Q3Lli2jZcuWvP/++/To0YPk5GSWLFnC+vXr2bt3L0VFRfTv3x8oTUIiIhI+UX1lU1sXXHABubm5vPHGG9x5550MHjyYRx99lJycHNq1a8fUqVM5ePAg7o6ZRTpcEZFGo0Fd2RQUFHDyySdz/fXXc8cdd/Dxxx8DEBcXx/79+5k/fz4ArVq1omXLlixfvhyArKysiMUsItIYNKgrm7Vr15KZmUmTJk1o1qwZjz/+ONnZ2SQnJ5OYmMhFF11UVnb27NlMmDCBk08+mSFDhkQwahGRhi+qR6Olp6e7btqLiNScmeW6e3qk4yivQXWjiYhIdGpQ3WhHZK/exvTFmygoKia+VSyZQzoyIi0h0mGJiDRaDS7ZZK/exp0L1lJ8qASAbUXF3LlgLYASjohIhDS4brTpizeVJZojig+VMH3xpghFJCIiDS7ZFBQV12q7iIiEX4NLNvGtYmu1XUREwq/BJZvMIR2JbRZzzLbYZjFkDukYoYhERKTBDRA4MghAo9FERKJHg0s2UJpwlFxERKJHg+tGExGR6KNkIyIiYadkIyIiYddok02vXr2O67gxY8aQkpLCQw89VGmZpUuXMmzYsOMNTUSkwWmQAwRqYsWKFbU+5quvvmLFihV8/nlUvnVVRCRqNdorm+bNm+PuZGZm0rlzZ5KTk5k7dy5Q+proV155pazs2LFjWbRoEYMHD2bnzp2kpqaybNkyBgwYwJFXIOzatYvExMRIfBURkajXaK9sABYsWEBeXh5r1qxh165dXHTRRfTr148bb7yRhx56iOHDh7N3715WrFjBM888Q0pKCsOGDSMvLy/SoYuI1CshubIxs6fNbKeZratk/wAz22tmeYHP70NRb7CWL1/OmDFjiImJoU2bNvTv359Vq1bRv39/Nm/ezM6dO3nhhRe4+uqradq0UedlEZGghOon6BzgEeDZKsosc/eoumte1VtKx40bR1ZWFi+++CJPP/10hWWaNm3K4cOHATh48GBYYhQRaQhCcmXj7h8Ae0JxrrrUr18/5s6dS0lJCYWFhXzwwQd0794dgPHjx/Pwww8DkJSUVOHxiYmJ5ObmAjB//vy6CVpEpB6qywECF5vZGjN708wq/ukNmNlEM8sxs5zCwsKwBWNmjBw5kpSUFDp16kS7du148MEHOfPMMwFo06YNF154ITfccEOl57jjjjt4/PHH6dWrF7t27QpbrCIi9Z1V1ZVUqxOZJQKvuXvnCvb9CDjs7vvN7HJghrt3qO6c6enpfmS0Vyjt3r2brl27lg1hzs/PZ9iwYaxb9/0tpwMHDpCcnMzHH39My5YtQx6DiEg4mFmuu6dHOo7y6uTKxt3/1933B5bfAJqZWVxd1F1eQUEBF198MXfccccx20tKSrjppptISkqiW7dudOzYkfT0dC699FK6dOnC1VdfzYEDB4DSLrZJkybRt29fLrjgAl577TUA5syZw/Dhw7nsssvo2LEjd999NwC/+93vmDFjRlldU6ZMYebMmXX0jUVEooC7h+QDJALrKtl3Jt9fRXUHvjiyXtWnW7duHk4LP97qve5/zxMmPeU0aeJ/emGxu7tfc801/txzz/muXbvKyk6ZMsVnzpzp7u4ZGRk+ZMgQLykp8X/+85+ekJDgxcXFPnv2bD/zzDN9165dfuDAAU9KSvJVq1b5v/71L09LS3N395KSEj/33HOPObeISKgAOR6in+uh/IRkNJqZvQAMAOLMbCtwF9AskMxmAaOAW8zsO6AYGB1olIjJXr2NOxespfhQCQBNW7bhifWHOWf1Nrp160Z+fj7r1q3jt7/9LUVFRezfv58hQ4aUHX/ttdfSpEkTOnTowLnnnss//vEPAAYNGsTpp58OwFVXXcXy5cv51a9+xemnn87q1avZsWMHaWlpZWVERBqDkCQbdx9Tzf5HKB0aHTWmL95UlmgALKYZxYdKmL54EyObxlBcXMz48ePJzs6mS5cuzJkzh6VLl35f3uyY8x1Zr2z7jTfeyJw5c/jqq6+YMGFCmL6ViEh0arTT1RQUFVe7fd++fbRt25ZDhw6RlZV1TLmXXnqJw4cP89lnn7FlyxY6dix97fQ777zDnj17KC4uJjs7m969ewMwcuRI3nrrLVatWnXMFZKISGPQaB+Lj28Vy7YKEk58q9iy5XvvvZcePXrQvn17kpOT2bdvX9m+jh070r9/f3bs2MGsWbM46aSTAOjTpw/jxo1j8+bNXHfddaSnlw4KOeGEE7jkkkto1aoVMTExYf52IiLRpdEmm8whHcvu2TRt2Yb4XzxGbLMYMod0ZETaT8rK3XLLLRUe37t37wpfM3DGGWfwyCM/7DE8fPgwH330ES+99FLovoSISD3RaLvRRqQlcP9VySS0isWAhFax3H9VMiPSEkJe14YNGzj//PMZOHAgHTpU+3iRiEiDE7KHOsMhXA91iog0VNH6UGej7UYrL3v1NqYv3kRBUTHxrWID3Wmhv8oREWmMlGz44TM324qKuXPBWgAlHBGREGi092yOVv6ZG6DsmRsREQmekg01e+ZGRESOn5INxz5bU5PtIiJSO0o2lD5zE9vs2ActjzxzIyIiwdMAAb4fBKDRaCIi4aFkEzAiLUHJRUQkTNSNJiIiYadkIyIiYadkIyIiYadkIyIiYadkIyIiYadkIyIiYadkIyIiYReSZGNmT5vZTjNbV8l+M7OZZrbZzD4xs66hqFdEROqHUF3ZzAEuq2L/UKBD4DMReDxE9YqISD0QkmTj7h8Ae6ooMhx41kt9BLQys7ahqFtERKJfXd2zSQC+PGp9a2DbD5jZRDPLMbOcwsLCOglORETCq66SjVWwzSsq6O5PuHu6u6e3bt06zGGJiEhdqKtksxVod9T6WUBBHdUtIiIRVlfJZhHw88CotJ7AXnffXkd1i4hIhIXkFQNm9gIwAIgzs63AXUAzAHefBbwBXA5sBg4AN4SiXhERqR9CkmzcfUw1+x34ZSjqEhGR+kczCIiISNgp2YiISNgp2YiISNgp2YiISNgp2YiISNgp2YiISNgp2YiISNgp2YiISNgp2YiISNgp2YiISNgp2YiISNgp2YiISNgp2YiISNgp2YiISNgp2YiISNgp2YiISNgp2YiISNgp2YiISNgp2YiISNgp2YiISNiFJNmY2WVmtsnMNpvZ5Ar2DzCzvWaWF/j8PhT1iohI/dA02BOYWQzwKDAI2AqsMrNF7r6hXNFl7j4s2PpERKT+CcWVTXdgs7tvcfdvgReB4SE4r4iIVGDOnDkUFBTU+jgzm2Nmo8IQUrVCkWwSgC+PWt8a2FbexWa2xszeNLOkyk5mZhPNLMfMcgoLC0MQnohIw3K8ySaSQpFsrIJtXm79Y6C9u3cB/h+QXdnJ3P0Jd0939/TWrVuHIDwRkeiWn5/Pj3/8YzIyMkhJSWHUqFEcOHCA3Nxc+vfvT7du3RgyZAjbt29n/vz55OTkMHbsWFJTUykuLiYxMZF77rmHPn36AJxqZqlm9pGZfWJmC83s1PJ1mlk3M/sfM8s1s8Vm1jawfamZpQeW48wsP7A83syyzexVM/uXmd1qZr82s9WBuk6r6juGItlsBdodtX4WcEzKdff/dff9geU3gGZmFheCukVEGoRNmzYxceJEPvnkE370ox/x6KOPcttttzF//nxyc3OZMGECU6ZMYdSoUaSnp5OVlUVeXh6xsbEAnHTSSSxfvhzga+BZ4DfungKsBe46ui4za0bpL/6j3L0b8DRwXw3C7AxcR+ntk/uAA+6eBqwEfl7VgUEPEABWAR3M7BxgGzA6EEwZMzsT2OHubmbdKU1yu0NQt4hIVOjVqxcrVqwgPz+fYcOGsW7dulod365dO3r37g3A9ddfz3//93+zbt06Bg0aBEBJSQlt27at9Pif/exnRxZjgFbu/j+B9WeAl8oV70hp4njHzI4cs70GYb7v7vuAfWa2F3g1sH0tkFLVgUEnG3f/zsxuBRZTGvDT7r7ezCYF9s8CRgG3mNl3QDEw2t3Ld7WJiNRbK1asCOr4wA/9Mi1atCApKYmVK1fW6PhTTjmlVtUB69394gr2fcf3vV4nldv376OWDx+1fphq8klInrNx9zfc/QJ3P8/d7wtsmxVINLj7I+6e5O5d3L2nuwf3tyIiEkF/+tOf6Ny5M507d+bhhx8GoHnz5kGd84svvihLLC+88AI9e/aksLCwbNuhQ4dYv349UJqI9u3bV9mpSoCvzaxvYH0c8D/lymwCWpvZxVDarXbUwK18oFtgOWQj10LRjSYi0mjk5uYye/Zs/va3v+Hu9OjRg/79+wd93gsvvJBnnnmGm2++mQ4dOnDbbbcxZMgQbr/9dvbu3ct3333Hr371K5KSkhg/fjyTJk0iNja2siufDGCWmZ0MbAFuOHqnu38bGAI908xaUpoLHgbWA38A5pnZOGBJ0F8swKK5Nys9Pd1zcnIiHYaISJkZM2awe/du7rnnHgB+97vf0bp1a/7rv/6L/fv3H9c9m+O9z1MRM8t19/SgTxRimhtNRKQWovkX9GimZCMiUgv9+vUjOzubAwcO8M0337Bw4UL69u1b/YFVSExMZNpzi+n9wBLOmfw6vR9YQvbqbSGKODrono2ISC107dqV8ePH0717dwBuvPFG0tLSgjpn9upt3LlgLcWHSgDYVlTMnQvWAjAiraIJWeof3bMREYmw3g8sYVtR8Q+2J7SK5cPJP6nVuXTPRkREKlRQQaKpant9pG40EZHjkL16G9MXb6KgqJj4VrFkDul43F1e8a1iK7yyiW8VG2yYUUNXNiIitXTkHsu2omKc7++xHO9N/cwhHYltFnPMtthmMWQO6RiCaKODko2ISC1NX7yp7Gb+EcWHSpi+eNNxnW9EWgL3X5VMQqtYjNJ7NfdfldxgBgeAutFEpBG49957ycrKol27dsTFxdGtWzdatmzJE088wbfffsv555/Pc889x8knn8xLL73E3XffTUxMDC1btuSDDz74wfnCcY9lRFpCg0ou5enKRkQatJycHF5++WVWr17NggULODLC9aqrrmLVqlWsWbOGCy+8kKeeegqAe+65h8WLF7NmzRoWLVpU4Tkru5fSkO6xhJqSjYg0aMuXL2f48OHExsbSokULrrjiCgDWrVtH3759SU5OJisrq2ySy969ezN+/HiefPJJSkpKKjxnY7jHEmpKNiLSoFX2LOH48eN55JFHWLt2LXfddRcHDx4EYNasWUybNo0vv/yS1NRUdu/+4au3GsM9llBTshGRBq1Pnz68+uqrHDx4kP379/P6668DsG/fPtq2bcuhQ4fIysoqK//ZZ5/Ro0cP7rnnHuLi4vjyyy8rPO+ItAQ+nPwT/vXAT/lw8k+UaKqhAQIi0qBddNFFXHnllXTp0oX27duTnp5Oy5Ytuffee+nRowft27cnOTm57P0wmZmZfPrpp7g7AwcOpEuXLhH+Bg2DpqsRkQZv//79NG/enAMHDtCvXz+eeOIJunbtGumwwiJap6vRlY2INHgTJ05kw4YNHDx4kIyMjAabaKKZrmxEpFEI5fQy0UxXNiIiEdIYpvCPdiEZjWZml5nZJjPbbGaTK9hvZjYzsP8TM9M1rIjUmVBPLyO1F3SyMbMY4FFgKNAJGGNmncoVGwp0CHwmAo8HW6+ISE01hin8o10ormy6A5vdfYu7fwu8CAwvV2Y48KyX+ghoZWZtQ1C3iEi1NL1M5IUi2SQARz/1tDWwrbZlADCziWaWY2Y5hYWFIQhPRBo7TS8TeaFINlbBtvJD3GpSpnSj+xPunu7u6a1btw46OBERTS8TeaEYjbYVaHfU+llAwXGUEREJm4Y+hX+0C8WVzSqgg5mdY2YnAKOB8vNyLwJ+HhiV1hPY6+7bQ1C3iIjUA0Ff2bj7d2Z2K7AYiAGedvf1ZjYpsH8W8AZwObAZOADcEGy9IiJSf4TkoU53f4PShHL0tllHLTvwy1DUJSIi9Y9eMSAiImGnZCMiImGnZCMiImGnZCMiImGnZCMiImGnZCMh9/DDD3PgwIFIhyEiUUTJRkKuqmRTUlJS4XYRadiUbBqpZ599lpSUFLp06cK4ceP4/PPPGThwICkpKQwcOJAvvvgCgPHjxzN//vyy45o3bw7A0qVLGTBgAKNGjeLHP/4xY8eOxd2ZOXMmBQUFXHLJJVxyySVlx/z+97+nR48eTJs2jZEjR5ad75133uGqq66qw28uIhHh7lH76datm0vorVu3zi+44AIvLCx0d/fdu3f7sGHDfM6cOe7u/tRTT/nw4cPd3T0jI8NfeumlsmNPOeUUd3d///33/Uc/+pF/+eWXXlJS4j179vRly5a5u3v79u3Lzu3uDvjcuXPd3f3w4cPesWNH37lzp7u7jxkzxhctWhTmbyzSeAA5HgU/v8t/dGXTCC1ZsoRRo0YRFxcHwGmnncbKlSu57rrrABg3bhzLly+v9jzdu3fnrLPOokmTJqSmppKfn19huZiYGK6++moAzIxx48bx/PPPU1RUxMqVKxk6dGhovpiIRK2QTFcj9Yu7Y1bRWx++d2R/06ZNOXz4cNlx3377bVmZE088sWw5JiaG7777rsJznXTSScTEfP8ukRtuuIErrriCk046iWuuuYamTfXPUKSh05VNIzRw4EDmzZvH7t27AdizZw+9evXixRdfBCArK4s+ffoAkJiYSG5uLgCvvPIKhw4dqvb8LVq0YN++fZXuj4+PJz4+nmnTpjF+/Pggv42I1Af6lbIRSkpKYsqUKfTv35+YmBjS0tKYOXMmEyZMYPr06bRu3ZrZs2cDcNNNNzF8+HC6d+/OwIEDOeWUU6o9/8SJExk6dCht27bl/fffr7DM2LFjKSwspFOnTiH9biISnaz0flJ0Sk9P95ycnEiHIWFw6623kpaWxi9+8YtIhyLSoJhZrrunRzqO8tSNJmGTn5/PX//617L1nJwcbr/9drp168Ynn3zC9ddfH5J6srOz2bBhQ0jOJSLhoW40CZsjyebIKLf09HTS00P/C1d2djbDhg1Tl5xIFNOVTSOWvXobvR9YwjmTX6f3A0vIXr3tmP21efDz9ttvp1evXpx77rllD4FOnjyZZcuWkZqaykMPPcTSpUsZNmwYAFOnTmXChAkMGDCAc889l5kzZ5bV+/zzz9O9e3dSU1O5+eaby2YdaN68OVOmTKFLly707NmTHTt2sGLFChYtWkRmZiapqal89tlnddF0IlJLSjaNVPbqbdy5YC3biopxYFtRMXcuWFuWcNavX899993HkiVLWLNmDTNmzODWW2/l5z//OZ988gljx47l9ttvLzvf9u3bWb58Oa+99hqTJ08G4IEHHqBv377k5eXxn//5nz+I4R//+AeLFy/m73//O3fffTeHDh1i48aNzJ07lw8//JC8vDxiYmLIysoC4JtvvqFnz56sWbOGfv368eSTT9KrVy+uvPJKpk+fTl5eHuedd174G09Eak3JppGavngTxYeOnaes+FAJ0xdvAmr/4OeIESNo0qQJnTp1YseOHTWK4ac//SknnngicXFxnHHGGezYsYP33nuP3NxcLrroIlJTU3nvvffYsmULACeccELZlVG3bt0qfYhURKKP7tk0UgVFxVVur82Dn3DsA541HeFY0UOh7k5GRgb333//D8o3a9asrM6qHiIVkegT1JWNmZ1mZu+Y2aeBP0+tpFy+ma01szwz01jmKBDfKrbK7bV58LMy1T3cWZGBAwcyf/58du7cWVbv559/HvJ6RKRuBduNNhl4z907AO8F1itzibunRuP478Yoc0hHYpvFHLMttlkMmUM6Asc++NmlSxd+/etfM3PmTGbPnk1KSgrPPfccM2bMqLKOlJQUmjZtSpcuXXjooYdqFFenTp2YNm0agwcPJiUlhUGDBrF9+/Yqjxk9ejTTp08nLS1NAwREolRQD3Wa2SZggLtvN7O2wFJ371hBuXwg3d131eb8eqgzvLJXb2P64k0UFBUT3yqWzCEdGZGWEOmwRCQI0fpQZ7DJpsjdWx21/rW7/6Arzcz+BXwNOPBnd3+iinNOBCYCnH322d2q60IREZHvRWuyqXaAgJm9C5xZwa4ptaint7sXmNkZwDtm9g93/6CigoFE9ASUXtnUog4REYlS1SYbd7+0sn1mtsPM2h7VjbazknMUBP7caWYLge5AhclGoou62kQkFIIdILAIyAgsZwCvlC9gZqeYWYsjy8BgYF2Q9UodqO7BTxGRmgo22TwADDKzT4FBgXXMLN7M3giUaQMsN7M1wN+B1939rSDrlTpQ3YOfIiI1FdRDne6+GxhYwfYC4PLA8hagSzD1SGRU9+CniEhNaboaqVR1D36KiNSUko1UqroHP0VEakpzo0mljow602g0EQmWko1UaURagpKLiARN3WgiIhJ2SjYiIhJ2SjYiIhJ2SjYiIhJ2SjYiIhJ2SjYiIhJ2SjYiIhJ2SjYiIhJ2SjYiIhJ2SjYiIhJ2SjYiIhJ2Sjb1xKxZs3j22WcBmDNnDgUFBRGOSESk5jQRZz0xadKksuU5c+bQuXNn4uPjIxiRiEjNKdlEqWeffZY//OEPmBkpKSmcd955NG/enMTERHJychg7diyxsbHcd999/OUvf2HhwoUAvPPOOzz++OMsWLAgwt9AROR76kaLQuvXr+e+++5jyZIlrFmzhhkzZpTtGzVqFOnp6WRlZZGXl8fll1/Oxo0bKSwsBGD27NnccMMNkQpdRKRCSjZRaMmSJYwaNYq4uDgATjvttErLmhnjxo3j+eefp6ioiJUrVzJ06NC6ClVEpEaCSjZmdo2ZrTezw2aWXkW5y8xsk5ltNrPJwdTZGLg7Zlbj8jfccAPPP/88L7zwAtdccw1Nm6p3VESiS7BXNuuAq4APKitgZjHAo8BQoBMwxsw6BVlvgzZw4EDmzZvH7t27AdizZ88x+1u0aMG+ffvK1uPj44mPj2fatGmMHz++LkMVEamRoH4FdveNQHW/hXcHNrv7lkDZF4HhwIZg6m7IkpKSmDJlCv379ycmJoa0tDQSExPL9o8fP55JkyYRGxvLypUriY2NZezYsRQWFtKpk/K4iESfuuhvSQC+PGp9K9CjDuqt1zIyMsjIyKhw39VXX83VV199zLbly5dz00031UVoIiK1Vm2yMbN3gTMr2DXF3V+pQR0VXfZ4FfVNBCYCnH322TU4vXTr1o1TTjmFP/7xj5EORUSkQtUmG3e/NMg6tgLtjlo/C6j08Xd3fwJ4AiA9Pb3SpCTfy83NjXQIIiJVqotutFVABzM7B9gGjAauq4N667Xs1duYvngTBUXFxLeKJXNIR0akJUQ6LBGR4xLs0OeRZrYVuBh43cwWB7bHm9kbAO7+HXArsBjYCMxz9/XBhR15RUVFPPbYY2E5d/bqbdy5YC3biopxYFtRMXcuWEv26m1hqU9EJNyCSjbuvtDdz3L3E929jbsPCWwvcPfLjyr3hrtf4O7nuft9wQYdDcKZbKYv3kTxoZJjthUfKmH64k1hqU9EJNw0g8Bxmjx5Mp999hmpqalkZmaSmZlJ586dSU5OZu7cuQD8x3/8B4sWLQJg5MiRTJgwAYCnnnqK3/72t+Tn53PhhRdy0003kZSUxODBgykuLqagqLjCOivbLiIS7ZRsjtMDDzzAeeedR15eHj179iQvL481a9bw7rvvkpmZyfbt2+nXrx/Lli0DYNu2bWzYUPpo0fLly+nbty8An376Kb/85S9Zv349rVq14uWXXya+VWyFdVa2XUQk2inZhMDy5csZM2YMMTExtGnThv79+7Nq1Sr69u3LsmXL2LBhA506daJNmzZs376dlStX0qtXLwDOOeccUlNTgdIhzPn5+WQO6Uhss5hj6ohtFkPmkI51/t1EREJBk2iFgHvFI7QTEhL4+uuveeutt+jXrx979uxh3rx5NG/enBYtWrB7925OPPHEsvIxMTEUFxeXjTrTaDQRaSiUbI7T0fOT9evXjz//+c9kZGSwZ88ePvjgA6ZPnw7AxRdfzMMPP8ySJUvYvXs3o0aNYtSoUdWef0RagpKLiDQYSjbH6fTTT6d379507tyZoUOHkpKSQpcuXTAzHnzwQc48s3TShb59+/L2229z/vnn0759e/bs2VN2v0ZEpLGwyrqAokF6errn5OREOgwRkXrDzHLdvdJXvkSKBgiIiEjYKdmIiEjY6Z5NkDSHmYhI9ZRsgnBkDrMjU8scmcMMUMIRETmKutGCoDnMRERqRskmCJrDTESkZpRsgqA5zEREakbJJgiaw0xEpGY0QCAImsNMRKRmlGyCpDnMRESqp260ahQUFJRNnLl06VKGDRsGwJw5c7j11lsjGZqISL2hZFON+Ph45s+fH+kwRETqNSWbo/zmN7/hscceK1ufOnUqf/zjH+ncuXOVx7366qv06NGDtLQ0Lr30Unbs2AFAYWEhgwYNomvLGJQPAAAHSUlEQVTXrtx88820b9+eXbt2AfD888/TvXt3UlNTufnmmykpKamqChGRei2oZGNm15jZejM7bGaVzjJqZvlmttbM8swsaqdxHj16NHPnzi1bnzdvHhdddFG1x/Xp04ePPvqI1atXM3r0aB588EEA7r77bn7yk5/w8ccfM3LkSL744gsANm7cyNy5c/nwww/Jy8sjJiaGrKys8HwpEZEoEOwAgXXAVcCfa1D2EnffFWR9YZWWlsbOnTspKCigsLCQU089lbPPPrva47Zu3crPfvYztm/fzrfffss555wDlL4ueuHChQBcdtllnHrqqQC899575ObmliWy4uJizjjjjDB9KxGRyAsq2bj7RgAzC000UWDUqFHMnz+fr776itGjR9fomNtuu41f//rXXHnllSxdupSpU6cClb8u2t3JyMjg/vvvD1XYIiJRra7u2TjwtpnlmtnEOqrzuIwePZoXX3yR+fPn1+j1zQB79+4lIaF0+PMzzzxTtr1Pnz7MmzcPgLfffpuvv/4agIEDBzJ//nx27twJwJ49e/j8889D+TVERKJKtcnGzN41s3UVfIbXop7e7t4VGAr80sz6VVHfRDPLMbOcwsLCWlQRGklJSezbt4+EhATatm1bo2OmTp3KNddcQ9++fYmLiyvbftddd/H222/TtWtX3nzzTdq2bUuLFi3o1KkT06ZNY/DgwaSkpDBo0CC2b98erq8kIhJxIXkttJktBe5w92pv/pvZVGC/u/+hurL1/bXQ//73v4mJiaFp06asXLmSW265hby8vEiHJSINWLS+FjrsMwiY2SlAE3ffF1geDNwT7nqjwRdffMG1117L4cOHOeGEE3jyyScjHZKISEQElWzMbCTw/4DWwOtmlufuQ8wsHviLu18OtAEWBgYRNAX+6u5vBRl3WIXq7ZsdOnRg9erVYYhQRKR+CXY02kJgYQXbC4DLA8tbgC7B1FOX9PZNEZHQ0wwC5ejtmyIioadkU47evikiEnpKNuXo7ZsiIqGnZFOO3r4pIhJ6enlaOXr7pohI6CnZVEBv3xQRCS11o4mISNgp2YiISNgp2YiISNgp2YiISNgp2YiISNgp2YiISNiF5H024WJmhUCkXmEZB+yKUN21VZ9ihfoVr2INn/oUb32KtT0wxd2fiHQgR4vqZBNJZpYTjS8gqkh9ihXqV7yKNXzqU7z1KVaIznjVjSYiImGnZCMiImGnZFO5qOrvrEZ9ihXqV7yKNXzqU7z1KVaIwnh1z0ZERMJOVzYiIhJ2SjYBZnaNma03s8NmVukoDjO7zMw2mdlmM5tclzEeFcNpZvaOmX0a+PPUSsrlm9laM8szs5w6jrHKdrJSMwP7PzGzrnUZXwXxVBfvADPbG2jLPDP7fSTiDMTytJntNLN1leyPmratQazR1K7tzOx9M9sY+FnwfyooExVtW8NYo6ZtAXB3fUq7Ei8EOgJLgfRKysQAnwHnAicAa4BOEYj1QWByYHky8H8rKZcPxEUgvmrbCbgceBMwoCfwtwj+3dck3gHAa5GKsVws/YCuwLpK9kdT21YXazS1a1uga2C5BfDPaP13W8NYo6Zt3V1XNke4+0Z331RNse7AZnff4u7fAi8Cw8Mf3Q8MB54JLD8DjIhADFWpSTsNB571Uh8BrcysbV0HGhAtf6814u4fAHuqKBI1bVuDWKOGu293948Dy/uAjUD5F1tFRdvWMNaoomRTOwnAl0etbyUyf8Ft3H07lP6jA86opJwDb5tZrplNrLPoatZO0dKWtYnlYjNbY2ZvmllS3YR2XKKpbWsi6trVzBKBNOBv5XZFXdtWEStEUds2qjd1mtm7wJkV7Jri7q/U5BQVbAvLcL6qYq3FaXq7e4GZnQG8Y2b/CPymGW41aac6a8saqEksHwPt3X2/mV0OZAMdwh7Z8Ymmtq1O1LWrmTUHXgZ+5e7/W353BYdErG2riTWq2rZRJRt3vzTIU2wF2h21fhZQEOQ5K1RVrGa2w8zauvv2wCX8zkrOURD4c6eZLaS0u6gukk1N2qnO2rIGqo3l6P/I7v6GmT1mZnHuHo3zZUVT21Yp2trVzJpR+sM7y90XVFAkatq2ulijrW3VjVY7q4AOZnaOmZ0AjAYWRSCORUBGYDkD+MFVmZmdYmYtjiwDg4EKRwSFQU3aaRHw88Donp7A3iNdgxFQbbxmdqaZWWC5O6X/d3bXeaQ1E01tW6VoatdAHE8BG939T5UUi4q2rUms0dS2gEajHfkAIyn9reXfwA5gcWB7PPDGUeUup3Tkx2eUdr9FItbTgfeATwN/nlY+VkpHVq0JfNbXdawVtRMwCZgUWDbg0cD+tVQyAjCK4r010I5rgI+AXhGM9QVgO3Ao8G/2F9HatjWINZratQ+lXWKfAHmBz+XR2LY1jDVq2tbdNYOAiIiEn7rRREQk7JRsREQk7JRsREQk7JRsREQk7JRsREQk7JRsREQk7JRsREQk7JRsREQk7P4/VCFI60iJ6C8AAAAASUVORK5CYII=\n",
"text/plain": [
""
]
},
"metadata": {
"needs_background": "light"
},
"output_type": "display_data"
}
],
"source": [
"# We have done the plotting for you. Just run this cell.\n",
"result = compute_pca(X, 2)\n",
"plt.scatter(result[:, 0], result[:, 1])\n",
"for i, word in enumerate(words):\n",
" plt.annotate(word, xy=(result[i, 0] - 0.05, result[i, 1] + 0.1))\n",
"\n",
"plt.show()"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"**What do you notice?**\n",
"\n",
"The word vectors for 'gas', 'oil' and 'petroleum' appear related to each other,\n",
"because their vectors are close to each other. Similarly, 'sad', 'joyful'\n",
"and 'happy' all express emotions, and are also near each other."
]
}
],
"metadata": {
"anaconda-cloud": {},
"coursera": {
"schema_names": [
"NLPC1-3"
]
},
"kernelspec": {
"display_name": "Python 3",
"language": "python",
"name": "python3"
},
"language_info": {
"codemirror_mode": {
"name": "ipython",
"version": 3
},
"file_extension": ".py",
"mimetype": "text/x-python",
"name": "python",
"nbconvert_exporter": "python",
"pygments_lexer": "ipython3",
"version": "3.7.1"
}
},
"nbformat": 4,
"nbformat_minor": 1
}
\n",
"\n",
"### 1.1 Importing the data\n",
"\n",
"As usual, you start by importing some essential Python libraries and then load the dataset.\n",
"The dataset will be loaded as a [Pandas DataFrame](https://pandas.pydata.org/pandas-docs/stable/getting_started/dsintro.html),\n",
"which is very a common method in data science.\n",
"This may take a few minutes because of the large size of the data."
]
},
{
"cell_type": "code",
"execution_count": 1,
"metadata": {},
"outputs": [],
"source": [
"# Run this cell to import packages.\n",
"import pickle\n",
"import numpy as np\n",
"import pandas as pd\n",
"import matplotlib.pyplot as plt\n",
"\n",
"from utils import get_vectors"
]
},
{
"cell_type": "code",
"execution_count": 2,
"metadata": {},
"outputs": [
{
"data": {
"text/html": [
"
\n",
"\n",
"You will implement a function that can tell you the capital of a country.\n",
"You should use the same methodology shown in the figure above. To do this,\n",
"compute you'll first compute cosine similarity metric or the Euclidean distance."
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"### 1.2 Cosine Similarity\n",
"\n",
"The cosine similarity function is:\n",
"\n",
"$$\\cos (\\theta)=\\frac{\\mathbf{A} \\cdot \\mathbf{B}}{\\|\\mathbf{A}\\|\\|\\mathbf{B}\\|}=\\frac{\\sum_{i=1}^{n} A_{i} B_{i}}{\\sqrt{\\sum_{i=1}^{n} A_{i}^{2}} \\sqrt{\\sum_{i=1}^{n} B_{i}^{2}}}\\tag{1}$$\n",
"\n",
"$A$ and $B$ represent the word vectors and $A_i$ or $B_i$ represent index i of that vector.\n",
"& Note that if A and B are identical, you will get $cos(\\theta) = 1$.\n",
"* Otherwise, if they are the total opposite, meaning, $A= -B$, then you would get $cos(\\theta) = -1$.\n",
"* If you get $cos(\\theta) =0$, that means that they are orthogonal (or perpendicular).\n",
"* Numbers between 0 and 1 indicate a similarity score.\n",
"* Numbers between -1-0 indicate a dissimilarity score.\n",
"\n",
"**Instructions**: Implement a function that takes in two word vectors and computes the cosine distance."
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"
"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"**Instructions**: \n",
"\n",
"You will write a program that takes in a data set where each row corresponds to a word vector. \n",
"* The word vectors are of dimension 300. \n",
"* Use PCA to change the 300 dimensions to `n_components` dimensions. \n",
"* The new matrix should be of dimension `m, n_componentns`. \n",
"\n",
"* First de-mean the data\n",
"* Get the eigenvalues using `linalg.eigh`. Use `eigh` rather than `eig` since R is symmetric. The performance gain when using `eigh` instead of `eig` is substantial.\n",
"* Sort the eigenvectors and eigenvalues by decreasing order of the eigenvalues.\n",
"* Get a subset of the eigenvectors (choose how many principle components you want to use using `n_components`).\n",
"* Return the new transformation of the data by multiplying the eigenvectors with the original data."
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"