You can automate a lot of tasks in QGIS using Python scripting (PyQGIS) and the Processing Framework. Most of the time, you would run these scripts manually while QGIS is open. While that is helpful, many times you need a way to run this jobs via the command-line and without needing to open QGIS. Fortunately, you can write standalone python scripts that use QGIS libraries and can be run via the command-line. In this tutorial, we will learn how to write and schedule a job that uses the QGIS Processing framework.
Let’s say we are working on some analysis using shapefiles of a region. The shapefiles are updated on a daily basis and we always need the latest file. But before we can use these files, we need to cleanup the data. We can setup a QGIS job that automates this process and runs it daily so you have the latest cleaned up shapefiles for your work. We will write a standalone Python script that downloads a shapefile and run topological cleaning operations on a daily basis.
Geofabrik provides daily updated shapefiles of OpenStreetMap datasets.
We will use shapefiles for Fiji for this exercise. Download the fiji-latest.shp.zip and unzip it to a folder on your disk.
Data Source [GEOFABRIK]
roads.shp
file and click Open.roads
layer as
Input layer. We will use EPSG:3460 Fiji 1986 / Fiji Map Grid
CRS as the Target CRS. Click Run.EPSG:3460 Fiji
1986 / Fiji Map Grid
as the CRS. This will ensure that our original and
reprojected layers will line up correctly.v.clean
algorithm. Search for this algorithm in the Processing Toolbox
and double-click it to launch the dialog.snap
tool to remove
duplicate vertices that are within 1 meter of each other. Select
Reprojected layer
as the Layer to clean. Choose snap
as
the Cleaning tool. Enter 1.00
as the
Threshold. Leave the other fields blank and click
Run.Cleaned vector layer
is the layer with topological errors corrected.
You will also have a Errors layer
which will highlight the features
which were repaired. You can use the errors layer as a guide and zoom in to
see vertices that were removed..bat
file. This file will
first set the correct configuration options and then call the python
script. Create a new file named launch.bat
and enter the following
text. Change the values according to your QGIS configuration. Don’t forget
to replace the username with your own username in the path to the python
script. The paths in this file will be the same on your system if you
installed QGIS via the OSGeo4W Installer
. Save the file on your
Desktop.Note
Linux and Mac users will need to create a shell script to set the paths and environment variables.
REM Change OSGEO4W_ROOT to point to the base install folder
SET OSGEO4W_ROOT=C:\OSGeo4W64
SET QGISNAME=qgis
SET QGIS=%OSGEO4W_ROOT%\apps\%QGISNAME%
set QGIS_PREFIX_PATH=%QGIS%
REM Gdal Setup
set GDAL_DATA=%OSGEO4W_ROOT%\share\gdal\
REM Python Setup
set PATH=%OSGEO4W_ROOT%\bin;%QGIS%\bin;%PATH%
SET PYTHONHOME=%OSGEO4W_ROOT%\apps\Python27
set PYTHONPATH=%QGIS%\python;%PYTHONPATH%
REM Launch python job
python c:\Users\Ujaval\Desktop\download_and_clean.py
pause
download_and_clean.py
and save it on your Desktop.from qgis.core import *
print 'Hello QGIS!'
launch.bat
icon. Double-click it
to launch a new command window and run the script. If you see Hello
QGIS!
printed in the command window, your configuration and setup worked
fine. If you see errors or do not see the text, check your launch.bat
file and make sure all the paths match the locations on your system.download_and_clean.py
script to
add the following code. This is the bootstrap code to initialize QGIS.
These are unnecessary if you are running the script within QGIS. But since
we are running it outside QGIS, we need to add these at the beginning. Make
sure you replace the username with your username. After making these
changes, save the file and run launch.bat
again. If you see Hello
QGIS!
printed, you are all set to do add the processing logic to the
script.import sys
from qgis.core import *
# Initialize QGIS Application
QgsApplication.setPrefixPath("C:\\OSGeo4W64\\apps\\qgis", True)
app = QgsApplication([], True)
QgsApplication.initQgis()
# Add the path to Processing framework
sys.path.append('c:\\Users\\Ujaval\\.qgis2\\python\\plugins')
# Import and initialize Processing framework
from processing.core.Processing import Processing
Processing.initialize()
import processing
print 'Hello QGIS!'
ret
value and printing the path to the reprojected layer.roads_shp_path = "C:\\Users\\Ujaval\\Downloads\\fiji-latest.shp\\roads.shp"
ret = processing.runalg('qgis:reprojectlayer', roads_shp_path, 'EPSG:3460',
None)
output = ret['OUTPUT']
print output
launch.bat
and you will see the path to the newly
created reprojected layer.grass.v.clean
algorithm. If you left these blank, the output will be
created in a temporary directory.processing.runalg("grass:v.clean",
output,
1,
1,
None,
-1,
0.0001,
'C:\\Users\\Ujaval\\Desktop\\clean.shp',
'C:\Users\\Ujaval\\Desktop\\errors.shp')
import os
import urllib
import zipfile
import tempfile
temp_dir = tempfile.mkdtemp()
download_url = 'http://download.geofabrik.de/australia-oceania/fiji-latest.shp.zip'
print 'Downloading file'
zip, headers = urllib.urlretrieve(download_url)
with zipfile.ZipFile(zip) as zf:
files = zf.namelist()
for filename in files:
if 'roads' in filename:
file_path = os.path.join(temp_dir, filename)
f = open(file_path, 'wb')
f.write(zf.read(filename))
f.close()
if filename == 'roads.shp':
roads_shp_path = file_path
Task
Scheduler
in Windows. Launch the Task Scheduler and click
Create Basic Task.Note
Linux and Mac users can use cron jobs to schedule tasks.
Daily Download and Cleanup
and click Next.Daily
as the Trigger and click NextStart a program
as the Action and click
Next.launch.bat
script. Click
Next.Below is the full download_and_clean.py
script for your reference.
import sys
from qgis.core import *
import os
import urllib
import zipfile
import tempfile
# Initialize QGIS Application
QgsApplication.setPrefixPath("C:\\OSGeo4W64\\apps\\qgis", True)
app = QgsApplication([], True)
QgsApplication.initQgis()
# Add the path to Processing framework
sys.path.append('c:\\Users\\Ujaval\\.qgis2\\python\\plugins')
# Import and initialize Processing framework
from processing.core.Processing import Processing
Processing.initialize()
import processing
# Download and unzip the latest shapefile
temp_dir = tempfile.mkdtemp()
download_url = 'http://download.geofabrik.de/australia-oceania/fiji-latest.shp.zip'
print 'Downloading file'
zip, headers = urllib.urlretrieve(download_url)
with zipfile.ZipFile(zip) as zf:
files = zf.namelist()
for filename in files:
if 'roads' in filename:
file_path = os.path.join(temp_dir, filename)
f = open(file_path, 'wb')
f.write(zf.read(filename))
f.close()
if filename == 'roads.shp':
roads_shp_path = file_path
print 'Downloaded file to %s' % roads_shp_path
# Reproject the Roads layer
print 'Reprojecting the roads layer'
ret = processing.runalg('qgis:reprojectlayer', roads_shp_path, 'EPSG:3460', None)
output = ret['OUTPUT']
# Clean the Roads layer
print 'Cleaning the roads layer'
processing.runalg("grass:v.clean",
output,
1,
1,
None,
-1,
0.0001,
'C:\\Users\\Ujaval\\Desktop\\clean.shp',
'C:\Users\\Ujaval\\Desktop\\errors.shp')
print 'Success'
This work is licensed under a Creative Commons Attribution 4.0 International License