Geospatial Software and Programming Languages¶

0. Closed and Open Source¶

Before talking about particular software and languages, it’s worth understanding the difference between closed and open source software.

Closed Source / Proprietary Software¶

When you buy software like ArcGIS Pro, or Microsoft Word, have you ever read the license agreement, or did you just click Accept?

Software like this is proprietary software. It’s created by a company like ESRI or Microsoft, and it’s protected by copyright and other laws. To use the software, you have to pay for it, and what you’re actually paying for is a license to use the software. This license usually imposes strict limits on use - for example allowing installation on only one device (or a small number of devices), banning sharing, and prohibiting users from trying to reverse engineer the source code.

The last point there is a crucial one. With the source code closed in this manner, only the company who writes the software can generally see or edit the code. The reason for this is pretty obvious - if you’re a company who writes and publishes a software package, that’s your product, your company’s income. If you let anyone see the code, then people could just copy it, and then why would anyone pay for your product? So you wouldn’t make any money off it, and that’s your company out of business.

Free and Open Source Source (FOSS) Software¶

However, there’s another category of software, and that’s open source software, which is usually free and open source (FOSS). Open Source software has the code for the software available to anyone who wants to see it. Most open source software is also free - and in software, there’s two specific meanings of the word “free”. It can be “free as in speech”, meaning anyone is free to take that source code and do whatever they want with it, including editing it and distributing their versions. It can also be “free as in beer”, meaning there’s no payment required. Most open source software is both, but there are notable exceptions.

The alternative to ArcGIS Pro is QGIS, which is free both as in speech, and as in beer. You don’t have to pay anything for it, and not only can you view the source code, but you can do what you want with it.

FOSS software like this is generally written by anyone and everyone who wants to contribute to it. Usually, there will be one or a small number of project maintainers in charge of the official version, who review all the suggested additions or changes to the code, and publish regular updates. People contribute because they find it useful themselves but want a new feature, because they want the challenge, or because they want to contribute something which will be useful to other people - everyone has different reasons. There’s a huge open source community, and it’s really pretty amazing.

Open source software often has a reputation of lagging behind propietary software. After all, since it’s free, there’s not a company with loads of full-time staff who can be assigned to development of the software. I’m not sure how many staff ESRI has working on updating and adding new features to ArcGIS Pro, but it’s probably a lot.

Some open source software does have full-time staff though, funded by donations and sponsorship. QGIS, for example, has full time staff funded not only by donations from individual users, but also by companies and organisations including other GIS companies which you might think are competitors, like Felt - which is actually the biggest individual sponsor of QGIS - and organisations which use the software, including the Irish Government Office of Public Works (the list is huge). QGIS probably has fewer people working full time on it than ESRI has on ArcGIS Pro, but

Other open source software is more commonly now pursuing other income generation models, for example being free for personal or educational use but charging for commercial use, or allowing free use but offering subscriptions for support services.

The Python language is completely free and open source, maintained by the Python Software Foundation. It does have some full-time staff, funded again by donations, which includes core developers, but also anyone is able to contribute. Probably the vast majority of software written in Python is open source projects, which again anyone can contribute to.

Open Source licenses¶

Open source software isn’t completely uncontrolled - most will still have a license explaining what you can and can’t do with the code. There’s a number of different licenses, with different levels of permissions. One f the most commonm is the MIT license:

It’s a bit shorter than the license agreements for software like Word that nobody ever fully reads, right? It also allow anyone to do anything - including selling adapted versions, with the only condition that the copyright notice be preserved.

Another common one is the BSD 3-clause license:

More restrictive is the GPL - the Gnu Public License - which is too long to paste here. It requires that any modified versions must use the same license, and publish the source code.

It’s not important to memorise the license types, but do bear it in mind for a couple of reasons.

First, simply so that you understand what you’ll be using in this course. I will be advocating using open source software here - in fact, you already are. The Notebooks in which I have written the content for this module use the Python programming language, which is open source and published under a custom open source license. The Notebooks themselves are Jupyter Notebooks, published by Project Jupyter under the BSD 3-clause license, and you may be running them in Google Colab, which is simply a modified version of Jupyter Notebook hosted online - in other words, Google made use of the BSD license and modified the software to run their own version. QGIS is published under a custom GPL-compatible open source license. Both QGIS and ArcGIS Pro also use Python and other open source elements in their software, including the option to write Python code for specific tasks inside the software; and we’ll also be using a few open source Python projects even just in these Notebooks.

Second, so that you understand what you’re allowed to do with certain software and code after the course - including if you want to write any code or contribute to any projects yourself. You might! It’s not as difficult as you might think, and it’s a little bit addictive!

1. Python¶

As I said, these Notebooks use the Python programming language. The Python language itself is open source, and many open source projects are written in Python - including many extremely useful geospatial projects.

I find that using Python is often the best way to teach geospatial data analysis. In part, that’s because I can write Notebooks like these, giving code examples which you can run - and of course there’s no Notebooks for QGIS or ArcGIS. I’ll record some walkthroughs for those, but using the Notebooks means you don’t have to do as much at first, versus if I went straight into QGIS where you’d have to follow written instructions and try to find the right menus to click, type in the right text etc.

But it’s also because using Python, you can do things a little more step by step, breaking things down a bit more. In that sense, it’s similar to how it’s better to teach maths by doing calculations and drawing graphs by hand before jumping into using calculators or software like Excel. So, hopefully using Python like this will let you understand what you’re doing, rather than just knowing which button to click to do what you want.

To use Python, essentially you just type out the commands for what you want to do. You can do this either in an interactive shell, where you type in a command and it runs immediately; or a Notebook like this one, where you type out bits of code and run them when you’re ready; or by writing a script, where you write out a full set of code for what you want to do, and run the entire script to do it.

I won’t say much more about Python here - instead, I have a couple more Notebooks:

The notebook The Python Language explains the fundamental basics of how the Python language works. You’ll find this very useful just to understand better what’s in these Notebooks, even if you don’t continue using Python afterwards.

The notebook Getting Started Seriously with Python will be useful if you’d like to start using Python for geospatial data analysis (or anything else, really) yourself, beyond these Notebooks. It explains some very useful tools and good practices which will make things significantly easier if you want to use Python even semi-regularly, including how to install Python to use it yourself.

2. QGIS¶

QGIS is free and open source desktop GIS software. You can download QGIS from qgis.org and install it on any Windows, Mac, or Linux desktop or laptop computer you want.

With software like Word, we’re used to opening a file, editing it, and saving it within that software. QGIS works a little differently.

When you open QGIS, the first step is to create a new project. The project file is what QGIS will be editing and saving. Into that project you can add geospatial data as different layers. These layers are completely separate from the project file. The QGIS project file saves which layers are part of the project, where on your computer to find the data for those layers, and how to display them. In some cases, QGIS can edit these layers, and you can create new layers within QGIS, but even when you’re doing that, one of the key steps is to identify where and how that layer should be saved. You can think of the QGIS project as a container for your geospatial data, simply controlling how it’s displayed.

QGIS also isn’t really just one piece of software. You can better think of it as a collection of multiple different open source geospatial tools, handily collected all in one place. It uses GDAL - the Geospatial Data Abstraction Library - for many functions, GRASS GIS and SAGA GIS for others, as well as smaller tools. It can be extended using plugins, additional tools for specific purposes written by particular contributors. Plugins aren’t part of the base QGIS package because they might be useful to only a small subset of users, so it’s preferable not to over-complicate things by installing them for everyone by default.

QGIS will also install a version of Python, because it uses a lot of Python behind the scenes. When you click on a toolbar icon, or on a menu item, it will run a specific command in the background - and many of those are in Python.

3. ArcGIS Pro¶

ArcGIS Pro is proprietary desktop GIS software published by ESRI. If you want to use ArcGIS Pro as part of this module, we will add you to the UL Geography ArcGIS license, giving you access to ArcGIS Pro and a range of additional tools provided by ESRI.

ArcGIS Pro works similarly to QGIS, in that the software itself directly saves mainly a project file which contains which layers are part of the project, where to find those on your computer, and how to display them. Like QGIS, it can also be thought of as a container for several different geospatial tools, including some open source tools.

Why might you consider using ArcGIS Pro instead of QGIS? One reason is that ArcGIS Pro is widely regarded as the industry standard, and so if you are working for a company which uses ArcGIS Pro, or aim to work for one in the future, best to stick with the tools you’ll need to know.

ArcGIS Pro also does have more features than QGIS. Many of these are features you’d only need for very specialised applications - but you might have one of those. More significantly, ESRI also provides a huge range of services and adaptions of ArcGIS, including online storage, tools like ArcGIS Online, Survey123, and StoryMaps, and these are all integrated with ArcGIS Pro. If you have any interest in using tools or services in this wider ArcGIS ecosystem, using ArcGIS Pro would be much simpler.

I don’t personally use ArcGIS Pro for much - I make no secret that I am an advocate for open source software, and QGIS is more than enough for the GIS work I don’t use Python for. So ArcGIS Pro content and support for this course will be provided by my colleagues.

4. Felt¶

Felt is a relatively new online GIS platform, which I would describe as the Google Docs equivalent for GIS. Not only does it allow you to work with geospatial data in a browser - such as Chrome, Firefox, or Edge - but multiple people can work on the same map, even at the same time.

Felt is free for limited personal use, and for education use, but does require a subscription for commercial or higher-volume use. However, the ability for multiple people to simultaneously work on the same map, and the online nature, is a game-changing toolset which might just be the perfect option for many of you - or at least an extremely useful additional tool.

5. Binder¶

Binder is an online platform which allows you to run Jupyter Notebooks - like this one - from a Github repository. So, if there’s a repository with Python (or Julia, or R) code in a Jupyter Notebook, you can run - and even edit - that code in Binder. Some of you may be running these Notebooks in Binder.

Binder will set up the environment based on information in the Github repository - so it will use whatever version of Python you want, and whichever Python packages you need for the project (see the Getting Started Seriously With Python for more about Github and environments).

You wouldn’t want to use Binder as your main tool for analysing geospatial data - but it’s certainly an incredibly useful tool for sharing, and for doing smaller tasks.

6. Google Colab¶

Colab is, like Binder, an online platform which allows you to run Python code in Jupyter Notebooks. It has one big limitation in comparison to Binder - the Python version and environment for Colab is fixed for all users, so you have to work with whatever versions of Python and specific Python packages Google has chosen, or which are compatible with what Google has chosen.

However, does have some significant advantages, including integration with not just Github but also Google Drive. It also offers significantly more processing power than Binder, including free access to GPUs. This means you can use Colab for heavy processing tasks, including machine learning, making it a viable option as your main tool for analysing geospatial data.

7. Google Earth Engine¶

Don’t confuse this with Google Earth - which allows you to look at selected satellite imagery for anywhere on the planet.

Google Earth Engine is a game-changing resource which offers access to virtually any publicly available geospatial data, including satellite imagery from the European Space Agency and NASA, as well as thousands of datasets derived from satellite data or created from other sources. Over 70 million gigabytes of data, going back decades.

But it’s not just the access to everything in one place that’s the game changer - it’s the processing. Earth Engine allows you to analyse the data - and it does the analysis on Google’s servers, not on your computer.

This means that high-end data analysis is at the tip of your fingers, regardless of what device you’re using. You could run a detailed analysis of global satellite imagery going back decades while on your phone sitting on a bus.

Earth Engine requires registration, it’s not open to everyone. It is free for academic, non-commercial, and non-profit use, but commercial use requires a subscription through Google Cloud.

8. Others¶

There are other programming languages and tools available. Microsoft’s Planetary Computer is an alternative to Google Earth Engine. There’s other GIS desktop software, including Global Mapper, and apps fo Android and iOS such as SW Maps. A lot of geospatial analysis has also been done using the R programming language, and there’s growing use of the languages Julia and Rust, and the new Mojo, as well as long-standing use of C (which is a much more complicated language to learn and use - you can’t run C in a Notebook, for example, only in complete compiled programs).

In general, the best tool to use is simply one that does what you need and which you’re comfortable with. So if you want to use something else - go for it. In terms of this course, though, there are limits to what support you might be able to get using something else, simply because we don’t use them ourselves. That doesn’t mean they’re bad or that we don’t like them - it’s just that what we use does what we need, and we’re comfortable with it. I do have colleagues in the Departmemt of Mathematics and Statistics and the Department of Computer Science and Information Systems who would strongly encourage using R and Julia, respectively, and certainly Rust and Mojo are growing rapidly and might be well worth learning.