↑ OpenStreetmap Hacker's guide ↑↑ Net & Web  

The canonical map server toolchain

This page gives an overview of how geospatial information from OpenStreetmap is turned into map tiles for online viewing. This is the most common usage of OpenStreetmap data, well-maintained tools exist for it and can serve as a basis for customisation.

This diagram shows the successive data repositories and processing steps:

Starting point: packed OpenStreetmap geospatial database

Source OpenStreetmap, Geofabrik or other sources
Tools Non-interactive downloaders like wget or curl helpful
Result After download, file on your local disk

These can be huge files that are continually being updated. You can (possibly should, depending on your use case) download a partial database of only one country or region.

They contain most things you can see on a map, such as rivers, roads, houses, towns, etc. They do not contain sea shore lines or elevation data. Shore line data is generally needed in addition to the OSM database (unless perhaps you are rendering only a landlocked region). The three archives from this directory are needed, as well as borders and inhabited regions from naturalearthdata.com. This shell script from this git repository can be used to download most of them.

Import into a live database

Tool osm2pgsql
Documentation Wiki page; more doc links
Result Live geospatial database

This step copies the information from the OpenStreetmap database packages into a live database to allow querying and selecting. If you downloaded only regional data, the rest of the world will contain no features, but this will not prevent it from being rendered as blank tiles later.

This step is very demanding in terms of memory usage and disk access.

Live geospatial database

Tools PostgreSQL and PostGIS
Documentation PostgreSQL manual; plenty of manual pages installed with PostgreSQL; PostGIS manual
Result Default storage in /var/lib/postgres/data/, default port 5432

PostgreSQL is a freely-licenced database. PostGIS adds data types and queries for geometrical objects and relations between them, such as intersections, distance, area etc.

Rendering bitmap images with Mapnik

Tool Mapnik library
Documentation Rendering style configuration documentation, wiki main page, outdated Python API documentation
Style examples Old OpenStreetmap style, Style share wiki page

Mapnik is a very versatile and powerful map rendering toolkit. It is written in C++, but is almost undocumented. The old versions 2.x came with Python bindings that were documented; with version 3.0.0 they have been moved to a separate repository and there is no longer any online documentation. Obviously a library on its own does not do anything, but Mapnik is so frequently mentioned that it warrants a section of its own. Multiple renderers using the Mapnik library exist.

The appearance of the map it produces is determined by a configuration file in XML format. For each map scaling, it contains layers to be rendered in order from bottom (occluded by higher layers) to top. For each layer it can be selected which features are shown, to the point of complicated SQL queries. Such style files can be very complex. The old OpenStreetmap style which is written in XML is 6469 text lines. To handle this complexity better, several style generators with simpler input languages have been written.

The Mapnik library can read information from various sources, not just from a live database. (There is even an input plugin that reads packed OSM data directly, but it seems to be considered obsolete.) In the canonical toolchain, the archives of continent outlines, borders and city limits we downloaded above are read by Mapnik in addition to data from the live database.

Rendering on demand

Tool renderd (part of distribution of Apach module mod_tile)
Documentation Manual page and supplementary documentation
Configuration /etc/renderd.conf (or config file specified with -c option)
Control Default socket /var/run/renderd/renderd.sock
Result Metatiles; default output directory /var/lib/mod_tile

renderd is a background daemon designed to render tiles on demand from the Apache module mod_tile or other programs. By default it generates "metatiles", mini-archives of 8 by 8 tiles. Rendering larger metatiles helps to render efficiently because there has to be some overlap between neighbouring tiles, and storing metatiles helps keep the total number of files down and reduces the load on the file system driver.

The individual tiles are in PNG format. The world is made up of 4z tiles in zoom level z because each successive level zooms in by a factor 2. To avoid having directories with too many subdirectories or files, the metatiles are stored in a deep directory structure corresponding to bit fields in the tile coordinates:

<base dir>/<style>/<zoom>/<16*A+a>/<16*B+b>/<16*C+c>/<16*D+d>/<8*(16*E+e)>.meta
x = AAAABBBBCCCCDDDDEMMM
y = aaaabbbbccccddddemmm

The index of the tile within the metatile file is 8*M+m. The <style> directory allows generating different map appearances on the same server.

As a curiosity, the tiles served by openstreetmap.org are actually in JPEG format instead of PNG. Everything else is the same, even the file extension of the tile URLs is .png, but the images are JPEGs. Apparently this is a non-standard conversion performed after tile generation to save network bandwidth.

Serving tiles

Tool Apache with module mod_tile
Documentation Apache V2.4 documentation home page; wiki page on mod_tile
Configuration /etc/httpd/conf/, /etc/httpd/conf/extra/mod_tile.conf
Result Tiles served at path http://server.foo/osm_tiles/<zoom>/<x>/<y>.png

The Apache web server together with the mod_tile module deliver the tiles to the client (which is usually running a JavaScript "slippy map" that chooses which tiles to download and puts them together to form the map). mod_tile connects to a running renderd to render metatiles on demand when they do not yet exist or are out of date. The decision which tiles are out of date seem to be taken purely based on their last rendering time, no check for an updated database seems to be done.

Displaying the map

Tools Emerillon; various browser plugins

This step is strictly out of the scope of this page, but of course the tiles are pretty useless without a viewer. I don't have much to say on this, as I have mostly used OSM data offline.

You can also search the web or the OpenStreetmap Wiki for other viewers, but be wary, there are many obsolete ones which do not compile any more or crash.

References

This workflow is fairly standard, so there actually exist tutorials about recreating it. They take the form of "wetware programs", lists of steps readers are supposed to follow blindly, with no explanation. Both tutorials assume Ubuntu Linux, apparently what the OSM project itself uses.


Licensed under the Creative Commons Attribution-Share Alike 3.0 Germany License

TOS / Impressum