), which is about 7%
of the world data in size, and which resulted in a database of 33 GB on disk.
| ↑ OpenStreetmap Hacker's guide | ↑↑ Net & Web |
Installation -- Database setup -- Rendering -- Viewing -- Link
This page describes how to render tiles without setting up a web server with a module that triggers rendering on demand. This allows to render tiles on a powerful desktop computer for later viewing on a mobile device that need not waste energy on rendering. This possibility is one of the strengths of OpenStreetmap, which can be legally downloaded and distributed, over commercial services.
This tutorial will stay as closely as possible to the tile server processing chain as possible, as it is comparatively well documented. Though most of this page is linux distribution independent, some commands will be valid verbatim only for Arch Linux. I will however describe what is going on, so you can substitute your distribution's package manager and similar. Existing tile server tutorials for Ubuntu Linux may also be helpful if you are using that distribution.
You need to install the packages postgresql, mapnik, protobuf-c (dependency for osm2pgsql-git), boost (dependency for mod_tile-git), and the AUR (i.e. source) packages osm2pgsql-git and mod_tile-git. I give the dependencies for AUR packages explicitly in case you use an AUR installer that does not handle dependencies. The necessary dependencies of binary packages, postgis (for mapnik) and protobuf (for protobuf-c) will be pulled in automatically.
You should also clone at least the first of the following two style repositories:
git clone https://github.com/openstreetmap/mapnik-stylesheets git clone https://github.com/gravitystorm/openstreetmap-carto.git
The first is the style we are going to use. The second contains a download script that will make it easier to retrieve auxiliary data. I have mirrored the script here so you don't have to clone the whole repository (or in case it vanishes overnight).
This section will show how to set up the database that makes the raw data from OpenStreetmap accessible. To do that, I will reference two user names which may differ between systems: postgresql and mapper. The first is the (Linux) user that runs the PostgreSQL server and is usually fixed by your Linux distribution. The second is the database user that owns the geospatial database we set up. This you can choose yourself, and different tutorials use different names. But later Mapnik has to be told the right user name to access the database with. The one used in the old OSM style (mapnik-stylesheets repository retrieved above) is mapper, so we will set it up that way.
Creating and initialising the database
Most of the work will be done using the postgresql user, but you have to log in as root first, because postgresql typically has no valid password set. Then execute the following commands as root:
su -l postgres -c "initdb --locale en_IE.UTF-8 -E UTF8 -D /path/to/lots/of/space" systemctl start postgresql
The first command sets some initialisation data for the PostgreSQL server, among others the path where to store the database. This is usually chosen as /var/lib/postgres/data, which is usually on the system disk; if you have a large data disk, it makes more sense to put it there, because you will need the space. If you have already set up a database and would like to change the directory, you can just move the data and replace the directory by a symbolic link.
The second command performed as root starts the PostgreSQL server.
Now log in as the postgresql user by running su -l postgres as root, and execute the following commands:
createuser mapper createdb -E UTF8 -O mapper gis psql -d gis -f /usr/share/postgresql/contrib/postgis-2.1/postgis.sql psql -d gis -c "ALTER TABLE spatial_ref_sys OWNER TO mapper;" psql -d gis -U mapper -f /usr/share/postgresql/contrib/postgis-2.1/spatial_ref_sys.sql
They create our database user and a database called gis where the OpenStreetmap data will be stored. The three psql commands operate on that database (-d gis). The first executes a lengthy SQL script that comes with the postgis package. For practical purpose this script makes our database into a geospatial database by defining the relevant types and functions. The second psql command changes the owner of the table of coordinate systems, to allow the third command to run as it is. The third executes an SQL script importing a number of coordinate systems. The location of the script may depend on your postgis version and Linux distribution. (It used to be enough to run /usr/share/osm2pgsql/900913.sql from the osm2pgsql package, which adds the pseudo-mercator system only, but recent versions of postgis require at least the latitude/longiture coordinate system to be in the table too.)
Importing OpenStreetmap data
After downloading the compressed OpenStreetmap data, this step is performed with the osm2pgsql program. Its most important command-line options are the following. (complete option documentation here)
An example osm2pgsql command line is:
osm2pgsql -s -C 2000 --number-processes 3 -U mapper myregion.osm.bz2
When I did this, I had expected -C to set the total RAM usage, so I received the error messages:
WARNING: Failed to fork helper process 1: Cannot allocate memory. Trying to recover. WARNING: Failed to fork helper process 2: Cannot allocate memory. Trying to recover.
The import still proceeded successfully. Probably osm2pgsql simply fell back to using only one process. A more serious caveat is to have some swap space available. The linux kernel sometimes promises processes more RAM than it can deliver, and kills some of them when they try to access it. (This is called memory overcommit and can be disabled, but that will prevent many applications from running, because many allocate more RAM than they end up using.) If this issue catches you, osm2pgsql will be killed with a message referring to memory. The only time when I experienced this, I had no swap space at all, but when you see this kind of message you may try to increase swap.
It should be said that this step is the bottleneck in the chain of processing
steps necessary for rendering. The actual rendering that is performed later
will take more time with a slow computer, but will complete eventually. The
database import and the resulting database require a certain amount of
resources. With an oldish Core2Duo box with 3 GB of RAM, I have imported data
for the Alps region (available on
Geofabrik), which is about 7%
of the world data in size, and which resulted in a database of 33 GB on disk.
The default OpenStreetmap style for Mapnik has inputs other than the OpenStreetmap database, notably the outlines of continents, countries and cities. The shell script get-shapefiles.sh from the openstreetmap-carto repository (mirrored here) downloads most of them:
./get-shapefiles.sh
The downloaded data are partly postprocessed and end up in the data subdirectory. I had to rename the post-processed files ne_10m_populated_places_fixed... to the equivalent names without the _fixed suffix, because the default style expects them to have the plain names. In addition you need to download the following two files manually:
wget https://planet.openstreetmap.org/historical-shapefiles/shoreline_300.tar.bz2 wget https://planet.openstreetmap.org/historical-shapefiles/processed_p.tar.bz2
This section will describe how rendering tiles works. It will basically describe how to render the whole world but omit details you should absolutely be aware of if you have limited computing power and do not want to render the whole world. Map rendering without a high-end server is possible, see below.
We will generate tiles by running the daemon renderd, which is part of the mod_tile package. Normally mod_tile triggers tile generation on demand as part of the Apache web server. But renderd can be easily made to generate tiles by any program, see below.
First, renderd has to be configured for the database setup by editing /etc/renderd.conf. While renderd actually has a manual page, its configuration file is entirely undocumented. It is organised in sections denoted by header names in square brackets. Section names starting with "renderd" refer to renderd itself or remote instances of it. (Apparently distributed rendering is possible.) The "mapnik" section sets some paths for the Mapnik rendering library. The other sections, notably "default", refer to different map styles to be generated on the same server and set their Mapnik styles and other options. I have changed the following entries from the default:
[renderd] section:
[mapnik] section:
[default] style section:
Configuring Mapnik consists of defining the appearance of the map at all zoom levels. Because creating such a style description from scratch is a major project, we will use the default OpenStreetmap style we downloaded above. The main style file is osm.xml in the top-level directory of the repository. It includes several auxiliary files in the inc subdirectory.
A few parameters relating to the database have to be set in the include file describing the main data source, inc/datasource-settings.xml.inc, based on inc/datasource-settings.xml.inc.template. After copying from the template, set the following parameters:
<Parameter name="host">127.0.0.1</Parameter> <Parameter name="port">5432</Parameter> <Parameter name="dbname">gis</Parameter> <Parameter name="user">mapper</Parameter> <Parameter name="password"></Parameter> <Parameter name="estimate_extent">true</Parameter>
They give the host running the PostgreSQL server, its port number, the database name, database user and non-existing password. (The "extent" is the region over which a map projection is valid, and having Mapnik estimate it is simplest.)
Some additional parameters have to be set in inc/settings.xml.inc, also copied from the corresponding .template. The comments in the file provide strong hints on how to set them. There is a curiosity regarding the syntax: Placeholders are denoted by %(name)s, apparently with the "s" part of the placeholder, so it has to be removed when substituting a value. The default values are:
<!ENTITY symbols "symbols"> <!ENTITY osm2pgsql_projection "&srs900913;"> <!ENTITY dwithin_node_way "&dwithin_900913;"> <!ENTITY prefix "planet_osm"> <!ENTITY world_boundaries "/data/openstreetmap/world_boundaries">
The world_boundaries parameter contains the directory where you stored the downloaded shapefile and supplementary data (see above).
Now you are set up to render.
Tile rendering
Basically, one starts renderd and runs a program like render_list which comes with mod_tile and renderd to trigger rendering of all or some tiles in a given zoom level.
Before starting with the details, one remark related to systemd that may or may not be Arch Linux specific. The logging daemon that is part of systemd, systemd-journald, is configured by default to log everything including debug messages. Because renderd outputs a lot, this causes the logs to grow huge. Edit /etc/systemd/journald.conf (as root) and set MaxLevelStore and MaxLevelSyslog both to info instead of debug.
To start rendering, one first has to start renderd. There is no need to run it as root. I have used the http user which also runs the web server when renderd is run for on-demand rendering. Because http has no login shell, you still have to start the program from a root shell via sudo or su. Before that, you have to create a subdirectory of /var/run to which http can write. Annoyingly, /var/run is a RAM disk, so one has to recreate this directory time and again after every reboot. The complete procedure (as root) is:
mkdir /var/run/renderd chown http.http /var/run/renderd sudo -u http renderd
If you want to view error messages about renderd.conf (sensible on a first run), add the -f option to dump renderd output to the shell. Syntax error messages about comment lines starting with ";" can be ignored; apparently even commented-out lines are parsed (and complained about), but it seemed to do no harm.
Once renderd is runnning, render_list can be used to submit requests to renderd. I have put its usage message up here. It can request specific tiles specified by its input, but that is not usually of interest when trying to render a map of any size. It can also make renderd generate a range of tiles. Its most important options are:
It is important to know that a zoom level range cannot be combined with a coordinate range, and in particular the rendered tile range is not restricted to the region the database holds information for. So when rendering a range of zoom levels, the whole world will always be rendered, creating many empty tiles at great computational expense! That is why one should probably not use render_list as-is.
If render_list gives you the following message:
rendering failed with command 4, pausing.
this is its way of saying an error occurred; "command" in this context seems to mean "status". This particular message meant I had forgotten to start the PostgreSQL daemon, which of course also has to be running. Other error reports can be expected to be equally cryptic.
With anything less than a high-end server, one can only create a database of part of the world and as a consequence can only render that part of the world at most. For time reasons, one might even want to restrict rendering to a smaller region than what the database covers, such as a travel destination.
As render_list does not allow combining a geographic region with a range of zoom levels, I have written a Perl script that runs it several times, once for each zoom level. The first step in using it is finding out the tile indices of the region one wants to render. This is easily done with the OpenStreetmap online map. Go to the left upper corner of the region (at any zoom level) and use the "open image" or similar option in the context menu of your browser. Read off the tile coordinates and zoom from the tile image URL, which has the following form:
http://*.tile.openstreetmap.org/<zref>/<x0>/<y0>.png
<zref> is the zoom level in which the coordinates we read off are defined, and <x0> and <y0> are the coordinates of the left upper corner of the region. Then go to the right lower corner of the region at the same zoom level, and read off <x1> and <y1>, the coordinates of this corner. Then, if renderd is running and render_list is available, you can run the Perl script render_list-rect.pl as follows:
render_list-rect.pl <zref> <x0> <x1> <y0> <y1> <z0> <z1> [ ... ]
The first five arguments are the region coordinates we just obtained. <z0> and <z1> are the range of zoom levels to render. (<zref> serves only as a reference for interpreting the region coordinates.) Optional following arguments are passed through to render_list to allow for example -n or -t. The Perl script simply scales the region coordinates by the appropriate power of 2 for each zoom level to render and runs render_list once for each.
The generated tiles with their usual directory structure can be copied to the
device where you want to view them. There is nothing architecture-specific
about them. Viewing them however requires some more work, as all viewers seem
to be made exclusively for the webserver setup. Dedicated offline tile viewers
do not seem to exist, though
one wiki
page speculates about them.
As a consequence, you have to install the web server apache and the tile server module mod_tile on the computer where you want to view the tiles. By default, mod_tile seems to be set up for operation on a virtual server, i.e. a machine that serves requests to multiple servers with different names. I have disabled that and configured the tile cache directory by editing /etc/httpd/conf/extra/mod_tile.conf as follows:
#<VirtualHost *:80>
# ServerName tile.openstreetmap.org
# ServerAlias a.tile.openstreetmap.org b.tile.openstreetmap.org c.tile.openstreetmap.org d.tile.openstreetmap.org
# DocumentRoot /var/www/html
DocumentRoot /data/openstreetmap/www
...
# ModTileTileDir /var/lib/mod_tile
ModTileTileDir /data/openstreetmap/mod_tile
...
#</VirtualHost>
The VirtualHost tags and the entries ServerName and ServerAlias have been commented out to disable the virtual server features. I am not sure if DocumentRoot needs to be set, but I set it to an existing empty directory to be on the safe side. ModTileTileDir is the directory you copied the tiles to (not including the "default" style directory). In addition, I had to comment out a comment line lacking the #.
Start the webserver (or set it up to be started automatically) but do not start renderd, as it cannot generate anything without a database anyway. If your package automatically starts renderd, this is probably harmless.
When you have set up the local tile server, you still need a viewer you can
redirect at the local web server. Because using a bulky browser to run a
simple program is technologically daft, I chose to adapt the stand-alone viewer
Emerillon (Emerillon + libchamplain:
2.3 MB; Firefox: 77 MB).
Emerillon uses the library
libchamplain to display the map.
Though the library
allows many kinds of
tile sources, Emerillon does not make use of this diversity but simply uses
the hard-coded default tile sources, with OSM tiles the default. The simplest
way to redirect it to local tiles is to make a modified copy of libchamplain in
which you replace the string
"http://tile.openstreetmap.org/#Z#/#X#/#Y#.png" by
"http://127.0.0.1/osm_tiles/#Z#/#X#/#Y#.png" with a hexadecimal editor
(fill the surplus space with zeros). Then you can run Emerillon with the
local tile server like this:
LD_PRELOAD=/path/to/modified-libchamplain.so emerillon
Licensed under the Creative Commons Attribution-Share Alike 3.0 Germany License 