GRASS GIS 7.8.2 released with updated PROJ 6 and GDAL 3 support
What’s new in a nutshell
As a follow-up to the recent GRASS GIS 7.8.1 we have pusblished the new stable release GRASS GIS 7.8.2.
Besides other improvements, the release contains important PROJ 4/5/6 related datum handling fixes, wxGUI fixes and a fix for the vector import from PostGIS databases.
The Geographic Resources Analysis Support System (https://grass.osgeo.org/), commonly referred to as GRASS GIS, is an Open Source Geographic Information System providing powerful raster, vector and geospatial processing capabilities in a single integrated software suite. GRASS GIS includes tools for spatial modeling, visualization of raster and vector data, management and analysis of geospatial data, and the processing of satellite and aerial imagery. It also provides the capability to produce sophisticated presentation graphics and hardcopy maps. GRASS GIS has been translated into about twenty languages and supports a huge array of data formats. It can be used either as a stand-alone application or as backend for other software packages such as QGIS and R geostatistics. It is distributed freely under the terms of the GNU General Public License (GPL). GRASS GIS is a founding member of the Open Source Geospatial Foundation (OSGeo).
GRASS GIS is an open source geoinformation system which is developed by a globally distributed team of developers. Besides the source code developers also message translators, people who write documentation, those who report bugs and wishes and more are involved.
Early days… from pre-Internet to CVS and SVN
While GRASS GIS is under development since 1982 (no typo!) it has been put into a centralized source code management system in December 1999. Why so late? Because the World Wide Web (WWW) became available in the 1990s along with tools like browsers and such, followed by the development of distributed source code management tools. We moved on 29th Dec 1999 (think Y2K bug) the entire code into our instance of CVS (Concurrent Versioning System). With OSGeo being founded in 2006, we migrated the CVS repository to SVN (Subversion for the source code management) and trac (bug and wish tracker) on 8 Dec 2007. See here for historic details on our various bug trackers.
Time to move on: git
Now, after more than 10 years using SVN/trac time had come to move on and join the large group of projects managing their source code in git (see also our related Wiki page on migration). Git comes with numerous advantages, yet we needed to decide which hosting platform to use. Options where github.com, gitlab.com, gitlab or gitea on OSGeo infrastructure, or other platforms. Through a survey we found out that the preference among contributors is GitHub. While not being open source itself it offers several advantages: it is widely known (good to get new developers interested and involved), numerous OSGeo projects are hosted there under the GitHub “OSGeo organization“.
If all fails (say, one day GitHub no longer being a reasonable choice) the import of our project from GitHub to GitLab is always possible. Indeed, we meanwhile mirror our code on OSGeo’s gitea server.
file timestamps (which I tried to preserve for decades :-) have been used to reconstruct the source code history (e.g., releasebranch_3_2)
junk files removed (plenty of leftover old binary files, files consisting of a special char only etc)
having this grass-legacy repo available in parallel to the main grass repo which contains the recent source code we have a continuous source code coverage from 1987 to today in git.
size is about 250MB
What’s missing
the 4.3 source code doesn’t have distinct timestamps. Someone must have once packaged without mtime preservation… a pity. Perhaps a volunteer may fix that by carrying over the timestamps from GRASS GIS 4.2 in case the md5sum of a file is identical (or so).
Trac issue migration
A series of links had to be updated. Martin Landa invested days and days on that (thanks!!). He used the related GDAL efforts as a basis (Even Rouault: thanks!). As the date for the trac migration we selected 2007-12-09 (r25479) as it was the first SVN commit (after the years in CVS). The migration of trac bugs to github (i.e. transfer of trac ticket content) required several steps:
Link updates in the ticket texts:
links to other tickets (now to be pointed to full trac URL). Note that there were many styles of referring in the commit log message which had to be parsed accordingly
links to trac wiki (now to be pointed to full trac URL)
links source code in SVN (now to be pointed to full trac URL)
images and attachments (now to be pointed to full trac URL)
Transferring:
“operating system” trac label into the github issue text itself (following the new issue reporting template)
converting milestones/tickets/comments/labels
converting trac usernames to Github usernames
setting assignees if possible, set new “grass-svn2git” an assignee otherwise
slowing down transfer to match the 60 requests per second API limit rate at github
Fun with user name mapping
Given GRASS GIS’ history of 35+ years we had to invest major effort in identifying and mapping user names throughout the decades (see also bug tracker history). The following circumstances could be identified:
user present in CVS but not in SVN
user present in SVN but not in CVS
user present in both with identical name
user present in both with different name (well, in our initial CVS days in 1999 we often naivly picked our surnames like “martin”, “helena”, “markus”, “michael” … cute yet no scaling very much over the years!) as some were changed in the CVS to SVN migration in 2007, leading to
colliding user names
some users already having a github account (with mostly different name again)
We came up with several lookup tables, aiming at catching all variants. Just a “few” hours to dig in old source code files and in emails for finding all the missing email addresses…
Labels for issues
We cleaned up the trac component of the bug reports, coming up with the following categories which have to be visually grouped by color since the label list is just sorted alphabetically in github/gitlab:
Issue category:
bug
enhancement
Issue solution (other than fixing and closing it normally):
In order to avoid users being flooded by emails due to the parsing of user contributions which normally triggers an email from github) we reached out to GitHub support in order to temporarily disable these notifications until all source code and selected issues were migrated.
The issue conversion rate was 4 min per trac bug to be converted and uploaded to github. Fairly slow but likely due to the API rate limit imposed and the fact that the migration script above generates a lot of API requests rather than combined ones..
Note to future projects to be migrated: use the new gihub import API (unfortunately we got to know about its existence too late in our migration process).
Here out timings which occurred during the GRASS GIS project migration from SVN to github:
grass repo: XX hours (all GRASS GIS 7.x code)
grass-legacy repo: XX hours (all GRASS GIS 3.x-6.x code)
NNN issues: XX hours – forthcoming.
New issue reporting template
In order to guide the user when reporting new issues, we will develop a small template – forthcoming.
Email notifications: issues to grass-dev and commits to grass-commit
We changed the settings from SVN post-hook to Github commit notifications and they flow in smoothly into the grass-commit mailing list. Join it to follow the development.
Overall, after now several months of using our new workflow we can state that things work fine.
https://neteler.org/wp-content/uploads/2024/01/wg_neteler_logo.png00Markushttps://neteler.org/wp-content/uploads/2024/01/wg_neteler_logo.pngMarkus2019-12-01 12:03:542023-10-22 18:58:58Remarks on SVN-trac to GitHub migration
su
# Fedora 23+24:
# install this extra repo
dnf copr enable neteler/GDAL
# A) in case of update, simply
dnf update
# B) in case of new installation (gdal-devel is optional)
dnf install gdal gdal-python gdal-devel
https://neteler.org/wp-content/uploads/2024/01/wg_neteler_logo.png00Markushttps://neteler.org/wp-content/uploads/2024/01/wg_neteler_logo.pngMarkus2016-11-27 00:31:292024-09-25 10:38:38GDAL 2.1 packaged for Fedora 23 and 24
The new version 1.11.0 of GDAL/OGR https://gdal.org/ which offers major new features has been released. GDAL/OGR is a C++ geospatial data access library for raster and vector file formats, databases and web services. It includes bindings for several languages, and a variety of command line tools.
https://neteler.org/wp-content/uploads/2024/01/wg_neteler_logo.png00Markushttps://neteler.org/wp-content/uploads/2024/01/wg_neteler_logo.pngMarkus2014-05-02 12:31:252023-11-24 23:37:31GDAL/OGR 1.11.0 released
We are pleased to announce that the 50th ICA-OSGeo Lab has been established at the GIS and Remote Sensing Unit (Piattaforma GIS & Remote Sensing, PGIS), Research and Innovation Centre (CRI), Fondazione Edmund Mach (FEM), Italy. CRI is a multifaceted research organization established in 2008 under the umbrella of FEM, a private research foundation funded by the government of Autonomous Province of Trento. CRI focuses on studies and innovations in the fields of agriculture, nutrition, and environment, with the aim to generate new sharing knowledge and to contribute to economic growth, social development and the overall improvement of quality of life.
The mission of the PGIS unit is to develop and provide multi-scale approaches for the description of 2-, 3- and 4-dimensional biological systems and processes. Core activities of the unit include acquisition, processing and validation of geo-physical, ecological and spatial datasets collected within various research projects and monitoring activities, along with advanced scientific analysis and data management. These studies involve multi-decadal change analysis of various ecological and physical parameters from continental to landscape level using satellite imagery and other climatic layers. The lab focuses on the geostatistical analysis of such information layers, the creation and processing of indicators, and the production of ecological, landscape genetics, eco-epidemiological and physiological models. The team pursues actively the development of innovative methods and their implementation in a GIS framework including the time series analysis of proximal and remote sensing data.
The GIS and Remote Sensing Unit (PGIS) members strongly support the peer reviewed approach of Free and Open Source software development which is perfectly in line with academic research. PGIS contributes extensively to the open source software development in geospatial (main contributors to GRASS GIS), often collaborating with various other developers and researchers around the globe. In the new ICA-OSGeo lab at FEM international PhD students, university students and trainees are present.
PGIS is focused on knowledge dissemination of open source tools through a series of courses designed for specific user requirement (schools, universities, research institutes), blogs, workshops and conferences. Their recent publication in Trends in Ecology and Evolution underlines the need on using Free and Open Source Software (FOSS) for completely open science. Dr. Markus Neteler, who is leading the group since its formation, has two decades of experience in developing and promoting open source GIS software. Being founding member of the Open Source Geospatial Foundation (OSGeo.org, USA), he served on its board of directors from 2006-2011. Luca Delucchi, focal point and responsible person for the new ICA-OSGeo Lab is member of the board of directors of the Associazione Italiana per l’Informazione Geografica Libera (GFOSS.it, the Italian Local Chapter of OSGeo). He contributes to several Free and Open Source software and open data projects as developer and trainer.
Open Source Geospatial Foundation (OSGeo) is a not-for-profit organisation founded in 2006 whose mission is to support and promote the collaborative development of open source geospatial technologies and data.
International Cartographic Association (ICA) is the world authoritative body for cartography and GIScience. See also the new ICA-OSGeo Labs website.
https://neteler.org/wp-content/uploads/2024/01/wg_neteler_logo.png00Markushttps://neteler.org/wp-content/uploads/2024/01/wg_neteler_logo.pngMarkus2013-10-23 16:06:302023-11-20 10:17:3950th ICA-OSGeo Lab established at Fondazione Edmund Mach (FEM)
The Landsat 8 mission is a collaboration between the U.S. Geological Survey (USGS) and National Aeronautics and Space Administration (NASA) which continues the acquisition of high-quality data for observing land use and land cover change.
The Landsat 8 spacecraft which was launched in 2013 carries they following key instruments:
OLI: the Operational Land Imager which collects data in the visible, near infrared, and shortwave infrared wavelength regions as well as a panchromatic band. With respect to Landsat 7 two new spectral bands have been added: a deep-blue band for coastal water and aerosol studies (band 1), and a band for cirrus cloud detection (band 9). Furthermore, a Quality Assurance band (BQA) is also included to indicate the presence of terrain shadowing, data artifacts, and clouds.
TIRS: The Thermal Infrared Sensor continues thermal imaging and is also intended to support emerging applications such as modeling evapotranspiration for monitoring water use consumption over irrigated lands.
The data from Landsat 8 are available for download at no charge and with no user restrictions.
Using the “Download options”, you can download the data set (requires login). Select the choice:
[x] Level 1 GeoTIFF Data Product (842.4 MB)
You will receive the file “LC80160352013134LGN03.tar.gz”.
Unpacking the downloaded Landsat 8 dataset
To unpack the data, run (or use a graphical tool at your choice):
tar xvfz LC80160352013134LGN03.tar.gz
A series of GeoTIFF files will be extracted: LC80160352013134LGN03_B1.TIF, LC80160352013134LGN03_B2.TIF, LC80160352013134LGN03_B3.TIF, LC80160352013134LGN03_B4.TIF, LC80160352013134LGN03_B5.TIF, LC80160352013134LGN03_B6.TIF, LC80160352013134LGN03_B7.TIF, LC80160352013134LGN03_B8.TIF, LC80160352013134LGN03_B9.TIF, LC80160352013134LGN03_B10.TIF, LC80160352013134LGN03_B11.TIF, LC80160352013134LGN03_BQA.TIF
Note: While this Landsat 8 scene covers the area of the North Carolina (NC) sample dataset, it is delivered in UTM rather than the NC’s state plane metric projection. Hence we preprocess the data first in its original UTM projection prior to the reprojection to NC SPM.
Using the Location Wizard, we can import the dataset easily into a new location (in case you don’t have UTM17N not already created earlier):
grass70 -gui
Now start GRASS GIS 7 and you will find the first band already imported (the others will follow shortly!).
For the lazy folks among us, we can also create a new GRASS GIS Location right away from the dataset on command line:
Landsat 8 Operational Land Imager (OLI) and Thermal Infrared Sensor (TIRS) images consist of nine spectral bands with a spatial resolution of 30 meters for Bands 1 to 7 and 9. New band 1 (ultra-blue) is useful for coastal and aerosol studies. New band 9 is useful for cirrus cloud detection. The resolution for Band 8 (panchromatic) is 15 meters. Thermal bands 10 and 11 are useful in providing more accurate surface temperatures and are collected at 100 meters. Approximate scene size is 170 km north-south by 183 km east-west (106 mi by 114 mi).
Landsat 7
Wavelength (micrometers)
Resolution (meters)
Landsat 8
Wavelength (micrometers)
Resolution (meters)
Band 1 – Coastal aerosol
0.43 – 0.45
30
Band 1 – Blue
0.45 – 0.52
30
Band 2 – Blue
0.45 – 0.51
30
Band 2 – Green
0.52 – 0.60
30
Band 3 – Green
0.53 – 0.59
30
Band 3 – Red
0.63 – 0.69
30
Band 4 – Red
0.64 – 0.67
30
Band 4 (NIR)
0.77 – 0.90
30
Band 5 – Near Infrared (NIR)
0.85 – 0.88
30
Band 5 (SWIR 1)
1.55 – 1.75
30
Band 6 – SWIR 1
1.57 – 1.65
30
Band 7 (SWIR 2)
2.09 – 2.35
30
Band 7 – SWIR 2
2.11 – 2.29
30
Band 8 – Panchromatic
0.52 – 0.90
15
Band 8 – Panchromatic
0.50 – 0.68
15
Band 9 – Cirrus
1.36- 1.38
30
Band 6 – Thermal Infrared (TIR)
10.40 -12.50
60* (30)
Band 10 – Thermal Infrared (TIRS) 1
10.60 – 11.19
100* (30)
Band 11 – Thermal Infrared (TIRS) 2
11.50- 12.51
100* (30)
* ETM+ Band 6 is acquired at 60-meter resolution. Products processed after February 25, 2010 are resampled to 30-meter pixels.
* TIRS bands are acquired at 100 meter resolution, but are resampled to 30 meter in delivered data product.
Natural color view (RGB composite)
Due to the introduction of a new “Cirrus” band (#1), the BGR bands are now 2, 3, and 4, respectively. See also “Common band combinations in RGB” for Landsat 7 or Landsat 5, and Landsat 8.
From Digital Numer (DN) to reflectance:
Before creating an RGB composite, it is important to convert the digital number data (DN) to reflectance (or optionally radiance). Otherwise the colors of a “natural” RGB composite do not look convincing but rather hazy (see background in the next screenshot). This conversion is easily done using the metadata file which is included in the data set with i.landsat.toar:
Now we are ready to create a nice RGB composite (hint 2015: i.landsat.rgb has been renamed to i.colors.enhance):
Select the bands to be visually combined:
… and voilà !
Applying the Landsat 8 Quality Assessment (QA) Band
One of the bands of a Landsat 8 scene is named “BQA” which contains for each pixel a decimal value representing a bit-packed combination of surface, atmosphere, and sensor conditions found during the overpass. It can be used to judge the overall usefulness of a given pixel.
We can use this information to easily eliminate e.g. cloud contaminated pixels. In short, the QA concept is (cited here from the USGS page):
For the single bits (0, 1, 2, and 3):
0 = No, this condition does not exist
1 = Yes, this condition exists.
The double bits (4-5, 6-7, 8-9, 10-11, 12-13, and 14-15) represent levels of confidence that a condition exists:
00 = Algorithm did not determine the status of this condition
01 = Algorithm has low confidence that this condition exists (0-33 percent confidence)
10 = Algorithm has medium confidence that this condition exists (34-66 percent confidence)
11 = Algorithm has high confidence that this condition exists (67-100 percent confidence).
Detailed bit patterns (d: double bits; s: single bits):
d – Bit 15 = 0 = cloudy
d – Bit 14 = 0 = cloudy
d – Bit 13 = 0 = not a cirrus cloud
d – Bit 12 = 0 = not a cirrus cloud
d – Bit 11 = 0 = not snow/ice
d – Bit 10 = 0 = not snow/ice
d – Bit 9 = 0 = not populated
d – Bit 8 = 0 = not populated
d – Bit 7 = 0 = not populated
d – Bit 6 = 0 = not populated
d – Bit 5 = 0 = not water
d – Bit 4 = 0 = not water
s – Bit 3 = 0 = not populated
s – Bit 2 = 0 = not terrain occluded
s – Bit 1 = 0 = not a dropped frame
s – Bit 0 = 0 = not fill
Usage example 1: Creating a mask from a bitpattern
We can create a cloud mask (bit 15+14 are set) from this pattern:
cloud: 1100000000000000
Using the Python shell tab, we can easily convert this into the corresponding decimal number for r.mapcalc:
Welcome to wxGUI Interactive Python Shell 0.9.8
Type "help(grass)" for more GRASS scripting related information.
Type "AddLayer()" to add raster or vector to the layer tree.
Python 2.7.5 (default, Aug 22 2013, 09:31:58)
[GCC 4.8.1 20130603 (Red Hat 4.8.1-1)] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> int(0b1100000000000000)
49152
Using this decimal value of 49152, we can create a cloud mask:
# set NULL for cloudy pixels, 1 elsewhere:
r.mapcalc "cloudmask = if(LC80160352013134LGN03_BQA == 49152, null(), 1 )"
# apply this mask
r.mask cloudmask
In our sample scene, there are only tiny clouds in the north-east, so no much to be seen. Some spurious cloud pixels are scattered over the scene, too, which could be eliminated (in case of false positives) or kept.
Usage example 2: Querying the Landsat 8 BQA map and retrieve the bitpattern
Perhaps you prefer to query the BQA map itself (overlay the previously created RGB composite and query the BSA map by selecting it in the Layer Manager). In our example, we query the BQA value of the cloud:
Using again the Python shell tab, we can easily convert the decimal number (used for r.mapcalc) into the corresponding binary representation to verify with the table values above.
Hence, bits 15,14,13, and 12 are set: cloudy and not a cirrus cloud. Looking at the RGB composite we tend to agree :-) Time to mask out the cloud!
wxGUI menu >> Raster >> Mask [r.mask]
Or use the command line, as shown already above:
# remove existing mask (if active)
r.mask -r
# set NULL for cloudy pixels, 1 elsewhere:
r.mapcalc "cloudmask = if(LC80160352013134LGN03_BQA == 61440, null(), 1 )" --o
# apply the new mask
r.mask cloudmask
The visual effect in the RGB composite is minimal since the cloud is white anyway (as NULL cells, too). However, it is relevant for real calculations such as NDVI (vegetation index) or thermal maps.
We observe dark pixels around the cloud originating from thin clouds. In a subsequent identification/mask step we may eliminate also those pixels with a subsequent filter.
https://neteler.org/wp-content/uploads/2024/01/wg_neteler_logo.png00Markushttps://neteler.org/wp-content/uploads/2024/01/wg_neteler_logo.pngMarkus2013-09-29 23:35:012023-11-20 16:46:07Processing Landsat 8 data in GRASS GIS 7: Import and visualization
https://neteler.org/wp-content/uploads/2024/01/wg_neteler_logo.png00Markushttps://neteler.org/wp-content/uploads/2024/01/wg_neteler_logo.pngMarkus2013-06-17 17:12:552024-01-22 22:30:21Scaling up globally: 30 years of FOSS4G development
https://neteler.org/wp-content/uploads/2024/01/wg_neteler_logo.png00Markushttps://neteler.org/wp-content/uploads/2024/01/wg_neteler_logo.pngMarkus2009-03-11 15:32:002023-10-22 18:44:00Four days of the Toronto Code Sprint 2009
The new winGRASS 6.4.0RC2 is now available! It has been packaged into the OSGeo4W installer which also provides QGIS, Mapserver, GDAL and other goodies.
OSGeo4W has two modes – express and advanced:
“Express” gives you a short list of suggested packages to select from which have been widely tested.
“Advanced” gives a long detailed list of packages and subpackages including development versions and so forth. For the moment GRASS is available in the “advanced” install. After a period of testing it may be moved to the “express” section.
The GDAL/OGR project team is pleased to announce the release of GDAL/OGR 1.6.0. This release represents 11 months of new feature development since the GDAL/OGR 1.5.0 release, as well as many bug fixes. The release source code is available as described at:
https://neteler.org/wp-content/uploads/2024/01/wg_neteler_logo.png00Markushttps://neteler.org/wp-content/uploads/2024/01/wg_neteler_logo.pngMarkus2008-12-10 22:46:002023-10-22 18:44:11GDAL/OGR 1.6.0 Released