Canada map update: working with large CanVec data sets

In a recent post I mentioned I’m working on a reference map of Canada. Well, even though I’m still chugging along at it I submitted it to the map gallery for the NACIS 2017 conference in Montreal. Now I have to finish it!

Dealing with the CanVec land cover data is challenging. Merging, deleting duplicate geometries, simplifying, etc. the  large shapefiles really slowed down my laptop. And since I can’t stop working on client projects to work on my own hobby map I decided to use my old htpc to do the grunt work.

I installed qGIS on my htpc running Linux and installed software so I can access the desktop remotely from my laptop. That way I can keep an eye on the status of the geoprocessing in qGIS and if (when) it locks up, it doesn’t affect my other work. I wish I’d thought of this months ago!

First I ran the “merge vector layers” tool on multiple 250k wooded area shapefiles from CanVec (one for each province and territory); I did this for a few at a time. For each merged shapefile I ran the “delete duplicate geometries” tool. This worked for a while but when I got to the last large merged shapefile qGIS choked on deleting the duplicate geometries. So I tried running the “dissolve” tool with the final merged shapefile but that just froze. It finally worked with OGR dissolve.

But the resulting file is 650MB, and I get an error when I try to load the layer in Illustrator – apparently it thinks the file is corrupted but I can open it in qGIS just fine.

I am now trying GRASS tools: importing the merged and dissolved shapefile into GRASS then running the following series of v.clean commands: break, rmline,  rmdangle,  rmbridge, rmdupl then prune. If that works I will create a step-by-step guide with screenshots to show how I made this large dataset a more manageable size. I also tried using the v.clean advanced tool to run the above commands as a batch, but again my computer choked.

Perhaps I should have run all these GRASS tools on the original 250K CanVec shapefiles before merging and dissolving. I could have run it as a batch job – perhaps I’ll try that next.

In the meantime I am running the v.generalize GRASS tool with different options, and then I will load a raster landcover image to manually digitize three areas of missing data in northern Quebec and the Northwest Territories

missing land use data in Quebec and NWT

I downloaded a DEM from GeoGratis and used Pyramid Shader to generate relief shading, but I don’t think it’s in high-enough resolution for my purposes. I will continue to search for other elevation data. I haven’t looked yet, but there may be CanVec contour shapefiles which I can use in qGIS to generate a DEM, then use Pyramid Shader again.

relief derived from Canada opendata dem

Below is a screenshot without the relief, but showing a close-up of my preliminary city label placement, the rivers layer (not yet labelled), and the unstyled roads layer (from Natural Earth large-scale data) for which I have not yet applied a hierarchy.

Canada roads layer

As you can see, I still have a loooooong way to go. At last count I have over 60 layers in Illustrator but many are similar layers from different sources and I haven’t yet decided which ones to use.

first half of layer list

second half of layer list

 

 

 

 

 

 

 

 

 

 

 

 

 

I want to have this done in mid-September so I can print out my first draft. Hopefully I will have a decent map for the gallery when this is finished. I’ve been working on this sporadically for the past few months but now I have only 3 months to finish it – an unrealistic deadline but I’m hoping not impossible.

One thought on “Canada map update: working with large CanVec data sets

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s