In general, when you collect, clean, and format data, you are working with independent files, but most of the time your analytic work needs to combine files from different sources to answer the questions you define in your research design. It is rarely the case that your research question depends on one particular data set or on several files from one source. Integration is the process of combining your data to produce the file you need. The material on storage focuses on how to code for data retrieval, updating, and sharing; topics that go beyond simply saving your files for later use.

Integrating Data

Preparing the Integration: Cleaning the Keys

To be able to integrate different data sets, you need one column that serves as a key to merge the data. Sometimes, you may even need two columns. That key column is the common column in every data set to be integrated. The headers do not have to be the same (no need for that), but the contents do. For example, if you have two data files of cities from different countries, the key could be the city name. If one file has more cities than the other, it is not a problem, as long as the cities that appear on both files are written using the same spelling. For this example, I will combine some known data sets from different sources to show the issues you face when integrating data.

1. • Human Development Index (HDI)

2. • Freedom of the World Index (WORLD)

3. • Economic Freedom Index (ECO)

4. • Freedom of the Press (PRESS)

5. • A table with the country names and their ISO and FIPS (CODES) from America's Open Geocode (http://www.opengeocode.org/) database

The unit of analysis is the country; the column with the country names can be used as the key, but I prefer some kind of country code, so the ISO or FIPS codes are the best option to ease the merging. Notice that I will use the official link to the data this time (the ones provided in the previous footnotes) to avoid scrapping (and mistakes in Wikipedia information). Let's start by getting the ISO and FIPS codes.