Mapping in D3

Tue, Oct 31, 2017

To make a map in D3, we must first start with a file that has cartographic information. We can then convert this for use in D3 or other software tools. Some of the most common formats of cartographic data files:

  • ESRI Shapefile — This comes as a .zip file. When you extract it, it will contains multiple files required for calculating the cartographic coordinates, projection, and other metadata. These files are mostly used with a GIS software program, like ArcGIS or it’s free open source alternative, QGIS. This is the most common data format of maps in the GIS world, and generally you will start with this type of file. We will need to covert it to a format more friendly to JavaScript and web software.
  • KML/KMZ or XML — These are formats that use XML (a generic HTML) to store the data. The KML format (and its identical, but compressed version KMZ) is mostly used by Google Earth and Google Maps for storing cartographic data.
  • GeoJSON — This is a JSON format that is a specific standard for storing cartographic features. The GeoJSON website outlines the specification requirements for displaying shapes, lines, and other features.
  • TopoJSON — Similar to the above GeoJSON, but it’s much more compact and efficient. GeoJSON files can often become very large and very process intensive. Designed by Mike Bostock, TopoJSON was aimed for efficiency without losing much quality. It’s designed to be used for any type of shape topology, not just cartographic purposes.

Projections

Before we begin, it’s also important to understand how map projections work. Projections is the cartographic science of taking a 3D shape, like a sphere, and representing it as a 2D plane surface.

Map projection

D3 offers several choices for how to display map projections.

Process Overview

The first steps in the process is to convert our cartographic data as a topojson. This is the format most ideal with D3 and will give us the best results. The process will vary depending on what you start with. In most cases, using the tools shown in this page will work with different starting files.

The goal is to convert the file to an EPSG:4326 coordinate system, or verify that it is already set to this. This will use standard latitude and longitude coordinates for all the points in your shape.

Step 1: Starting with a Shapefile

In this example, we will start with a shapefile of election precincts in Contra Costa County, California. Most shapefiles are available for download on the websites of local government agencies. This one was provided by the elections office of Contra Costa County.

Download Election Precincts for Contra Costa County (.zip)

Optional: Viewing the file in Mapshaper.org

We can view the file in MapShaper. This process is just informational, so we can see what it looks like. Clicking the “i” icon and hovering over the various features of the map helps us understand the type of data associated with each feature.

Viewing a shapre file in mapshaper

Note: Mapshaper has the ability to export files as GeoJSON and TopoJSON. Sometimes this would be OK to perform at this time. But in our example, we won’t use MapShaper to do this right now, because our shapefile is in a different coordinate system other than EPSG:4326. We need to first convert it before exporting as TopoJSON!

Step 2: Checking the Embedded Coordinate System

Shapefiles have a coordinate system embedded in them. You can think of this like an Cartesian X and Y grid, but where the values have different meaning depending on the type of map and where you got your shapefile.

D3 likes EPSG:4326, because it uses a common latitude/longitude coordinate system. Some shapefiles might already be setup with EPSG:4326, so this step can be optional.

To verify the coordinate system of your shape file, upload the .prj file to Prj2epsg.org. The .prj file is inside your shapefile .zip. You’ll need to unzip the shapefile temporarily so we can document which coordinate system it’s using. Make sure you keep the files intact and don’t throw away your .zip file just yet, we’ll still need it!

upload your .prj file

Prj results

We can see from the results, our shape file is embedded with EPSG:2227 coordinates. We need to convert it from EPSG:2227 to EPSG:4326!

Step 3: Converting to EPSG:4326

Next, we’ll visit the Ogre to Ogre (ogr2ogr) web client to convert our file to the correct EPSG:4326 coordinate system.

Ogr2ogr is a free command line program that can be installed though a package called GDAL, or through Homebrew.

However, there is an easier way using a free online web client and it doesn’t require installing anything. The web client should work for most applications. The drawback is that there are limited options for conversion, and there is a file size limit. Should either of these be an issue, then you’ll need to install ogr2ogr manually and run the following command from the Terminal program (You’ll need to do this from the folder where your shapefile is located.):

ogr2ogr -f GeoJSON output_file.geojson -t_srs EPSG:4326 your_shapefile.shp

For everyone else using the web client version, just visit: ogre.adc4gis.com. You’ll need to upload the entire shapefile .zip file.

Ogr2Ogr convert

With some browsers like Chrome, it may output the GeoJSON file as text in your window. Simply press Command (Ctrl on PC) + S to save the file with a .json file extension. The name of the file will be used later, so make sure the file name has no spaces and is well understood.

Step 4: Converting to TopoJSON

Next, we’ll upload our GeoJSON file to Mapshaper, and convert it to TopoJSON. Visit the Mapshaper website, and upload your GeoJSON file that you saved from ogr2ogr (not the original .zip shapefile!).

Export to TopoJSON with mapshaper

Advanced Users Note: If you already know the metadata you want to associate with each feature, you can add id-field= to the command line options during export and it will become the “id” property of each feature. In this current Contra Costa County Precinct, for example, we can put id-field=SPCTNM so that the SPCTNM field, which is the precinct name, will be associated with each feature’s id. Without doing this, all metadata will still be preserved.

Step 5: Building a D3 Map

Let’s start with a very basic starter template:

<!DOCTYPE html>
<html>
<head>
    <meta charset="utf-8">
    <title>D3 Map Example</title>
</head>
<body>


<script src="https://d3js.org/d3.v4.min.js"></script>
<script src="https://d3js.org/topojson.v2.min.js"></script>
<script>


</script>
</body>
</html>

Notice we’re loading in two D3 libraries; the main library, and one topojson sub-library. In the <script> tag, we’ll setup our basic SVG, and setup a path function which will draw our map.


var svg = d3.select("body")
    .append("svg")
    .attr("width", 960)
    .attr("height", 600);

var projection = d3.geoMercator()
    //center of your map
    .center([-121.979141, 37.940119])
    .scale(60000);//zoom factor

var path = d3.geoPath()
    .projection(projection);

d3.json("ccc_precinct_topo.json", function(error, mapData){
    if (error) throw error;

    console.log(mapData);

    svg.selectAll("path")
        .data(topojson.feature(mapData, mapData.objects.ccc_voters_geo).features)
        .enter()
        .append("path")
        .attr("d", path)
        .attr("stroke", "#000000")
        .attr("fill", "#ffffff");

});

The above code assumes that you named your TopoJSON file ccc_precinct_topo.json. Also, we need to figure out what the features object is in your data. If you followed this tutorial, Mapshaper would have made the object property name the same as the file name of your GeoJSON file (the one you exported from ogr2ogr). You can also find this out by looking at your console data, which we exported using console.log(mapData).

Looking at console to find object property for features

Note that in this example our property name is ccc_voters_geo (yours may be different). We need to use this in our data() function:

.data(topojson.feature(mapData, mapData.objects.ccc_voters_geo).features)

Change your mapData.object.[name of property] to match your console.

You should see a map appear:

Map so far

Step 6: Loading Other Data to Color the Map Features

Next, let’s load a .csv (spreadsheet data) so that we can color-code these precincts. We’ll use the latest election results from the 2016 primary elections.

2016 Primary Election Results (Excel)

Let’s make a map to find out who voted for Hillary Clinton vs Donald Trump. Open the election results in Excel, then we’ll create a new Spreadsheet document, and copy over just the columns we want to display on our map.

Sort just the data you need with precincts in one of the columns, and save as a .csv file.

Step 7: Loading in multiple data files using d3.queue()

Sometimes we need to load in multiple data files simultaneously. D3 has a separate utility called d3.queue() for doing just this. We first create a d3.queue() object, and then load in each file. We then specify a function, which will only be called once all of the previous files are loaded. This is important to use, because sometimes one file might take longer than another, and you want to ensure both are loaded before running the function.

var queue = d3.queue()
  .defer(d3.json, "ccc_precinct_topo.json") 
  .defer(d3.csv, "CCC_Primary_results.csv")
  .await(ready); //name of function that will be called

Later, we put in the function that will be called once each of these is loaded:

function ready(error, data1, data2){
  if (error) throw error; //will give us info if something goes wrong.

  //data1 and data2 have the info for each file we loaded in.

}

Step 8: Understanding d3.map()

In order to pair the data from our spreadsheet to the shapefiles, we’ll need a special object variable. D3 has something called d3.map() built just for this purpose. It’s a really simple function that allows you to create an object, and set the keyword for this object. In addition, there are some function for retrieving the data based on the keyword you set. Let’s look at an arbitrary example:

//just an example, don't use this in your map

var my_map = d3.map(); //we create a d3.map() object

//using the set function, we can associate data with keywords
my_map.set("dog", 32894);
my_map.set("cat", 54334);
my_map.set("mouse", 32452);

//later on, if we want to get the data, we can retrieve it by keyword
my_map.get("dog"); //returns 32894
my_map.get("cat"); //returns 54334
my_map.get("mouse"); //returns 32452

So, now that we generally understand how maps work, let’s use it in our example. We can use a special feature of d3.queue(), which will allow us to run a temporary intermediate function to do something to the data before our ready function is called.


var election_map = d3.map(); //create the map object

var queue = d3.queue()
  .defer(d3.json, "ccc_precinct_topo.json") 
  .defer(d3.csv, "CCC_Primary_results.csv", function(d){ 
      election_map.set(d["Precinct"], {"trump" : d["Trump"], {"clinton": d["Clinton"]}); 
  })
  .await(ready); 

What this does is allows us to recall either candidate’s results based on the precinct name.

election_map.get("Alhambra101") // will return object {trump: 11, clinton: 18}

Since some the shapefile data has precinct names associated with each shape, when we make our D3 map, we can reference this election_map variable to extract the voter data.

Step 9: Integrating the map