Migration to the Open Data Platform

2023-06-28

Source: vignettes/migration_odp.Rmd

About

From VicmapR version 0.2.0, the default platform data is retrieved from is different.

The Enterprise Spatial Services (ESS) Transformation Program from the Department of Energy, Environment and Climate Action (DEECA) has determined that DELWP’s WMS and WFS Open Data services will be hosted in AWS Cloud. This will cause a change of URLs when accessing these services.

This migration of services essentially requires the VicmapR package to restructure where and how it obtains data. We have updated package code to by default use the new Amazon AWS cloud platform as the default source for data moving forward. The legacy platforms and services will be turned off by the 27 March 2023. Meaning that users will no longer be able to access data from these sources from this date.

We have made an effort to update VicmapR in such a way to make existing code work by translating layer names from the legacy names to the new names. However, not all data may be available; or even if data is available there may be changes to formats and structures. However, the new platform is expected to be more scalable (with likely addition of more data moving forward).

Some key changes to the data structure are:

  • Layer names now have a prefix of ‘open-data-platform:’ instead of ‘datavic:’
  • Layer names are now lowercase
  • Attributes (columns) are now in lowercase
  • There are no different layers for subsets of data (e.g. there is only one layer for 2005 modelled EVCs: nv2005_evcbcs, whereas previously there were lots of layer subsets for the different EVC groups)
  • The default chunk limit is 5000 (previously varied between 1500 and 70000 depending on the dataset)

In order to view how old layer names relate to new layer names users can view the exported data.frame (name_conversions).

Comparison of old and new code

Below we show how you might have queried a a layer in the legacy system and in the new system

library(VicmapR)
#> 
#> Attaching package: 'VicmapR'
#> The following object is masked from 'package:stats':
#> 
#>     filter
library(kableExtra)

# Old code for obtaining freshwater wetlands in glenelg plain
evc_18_1_old <- vicmap_query("datavic:FLORAFAUNA1_NV2005_EVCBCS_18_1") %>% 
  filter(BIOREGION == "Glenelg Plain" & X_EVCNAME %in% c("Deep Freshwater Marsh", "Aquatic Herbland")) %>% 
  select(X_EVCNAME) %>%
  head(10) 
#> You are using old layer names. We converted your layer to nv2005_evcbcs with a CQL filter of evc_subgp=18.1. To suppress this message, update your code to use the new layer names (see VicmapR::name_conversions for more details)

collect(evc_18_1_old) %>%
  kbl() %>%
  kable_styling()
id x_evcname geometry
nv2005_evcbcs.5882 Aquatic Herbland MULTIPOLYGON (((140.9806 -3…
nv2005_evcbcs.14290 Deep Freshwater Marsh MULTIPOLYGON (((141.5681 -3…
nv2005_evcbcs.14559 Aquatic Herbland MULTIPOLYGON (((141.0497 -3…
nv2005_evcbcs.18331 Deep Freshwater Marsh MULTIPOLYGON (((141.5998 -3…
nv2005_evcbcs.19701 Deep Freshwater Marsh MULTIPOLYGON (((141.5607 -3…
nv2005_evcbcs.44964 Aquatic Herbland MULTIPOLYGON (((141.1398 -3…
nv2005_evcbcs.45154 Deep Freshwater Marsh MULTIPOLYGON (((141.1711 -3…
nv2005_evcbcs.46267 Aquatic Herbland MULTIPOLYGON (((141 -37.363…
nv2005_evcbcs.46747 Deep Freshwater Marsh MULTIPOLYGON (((141.1332 -3…
nv2005_evcbcs.55241 Aquatic Herbland MULTIPOLYGON (((141.1057 -3… 
# New code for obtaining freshwater wetlands in glenelg plain
evc_18_1_new <- vicmap_query("open-data-platform:nv2005_evcbcs") %>% 
  filter(x_subgroupname == "Freshwater" & 
           bioregion == "Glenelg Plain" & 
           x_evcname %in% c("Deep Freshwater Marsh", "Aquatic Herbland")) %>% 
  select(x_evcname) %>%
  head(10)

collect(evc_18_1_new) %>%
  kbl() %>%
  kable_styling()
id x_evcname geometry
nv2005_evcbcs.5882 Aquatic Herbland MULTIPOLYGON (((140.9806 -3…
nv2005_evcbcs.14290 Deep Freshwater Marsh MULTIPOLYGON (((141.5681 -3…
nv2005_evcbcs.14559 Aquatic Herbland MULTIPOLYGON (((141.0497 -3…
nv2005_evcbcs.18331 Deep Freshwater Marsh MULTIPOLYGON (((141.5998 -3…
nv2005_evcbcs.19701 Deep Freshwater Marsh MULTIPOLYGON (((141.5607 -3…
nv2005_evcbcs.44964 Aquatic Herbland MULTIPOLYGON (((141.1398 -3…
nv2005_evcbcs.45154 Deep Freshwater Marsh MULTIPOLYGON (((141.1711 -3…
nv2005_evcbcs.46267 Aquatic Herbland MULTIPOLYGON (((141 -37.363…
nv2005_evcbcs.46747 Deep Freshwater Marsh MULTIPOLYGON (((141.1332 -3…
nv2005_evcbcs.55241 Aquatic Herbland MULTIPOLYGON (((141.1057 -3…

As we can see, both code chucks now produce the same data due to VicmapR’s in-built translations as part of the update. To check the queries are the same we can compare them:

cat("Old query (translated):\n")
#> Old query (translated):
show_query(evc_18_1_old)
#> <base url>
#> https://opendata.maps.vic.gov.au/geoserver/ows
#> 
#> <body>
#> service: wfs 
#> version: 2.0.0 
#> request: GetFeature 
#> typeNames: nv2005_evcbcs 
#> outputFormat: application/json 
#> count: 10 
#> srsName: EPSG:4283 
#> CQL_FILTER: (evc_subgp=18.1 AND ("bioregion" = 'Glenelg Plain' AND "x_evcname" IN ('Deep Freshwater Marsh', 'Aquatic Herbland'))) 
#> propertyName: geom,x_evcname 
#> 
#> <full query url>
#> https://opendata.maps.vic.gov.au/geoserver/ows?service=wfs&version=2.0.0&request=GetFeature&typeNames=nv2005_evcbcs&outputFormat=application%2Fjson&count=10&srsName=EPSG%3A4283&CQL_FILTER=%28evc_subgp%3D18.1%20AND%20%28%22bioregion%22%20%3D%20%27Glenelg%20Plain%27%20AND%20%22x_evcname%22%20IN%20%28%27Deep%20Freshwater%20Marsh%27%2C%20%27Aquatic%20Herbland%27%29%29%29&propertyName=geom%2Cx_evcname
cat("New query:\n")
#> New query:
show_query(evc_18_1_new)
#> <base url>
#> https://opendata.maps.vic.gov.au/geoserver/ows
#> 
#> <body>
#> service: wfs 
#> version: 2.0.0 
#> request: GetFeature 
#> typeNames: open-data-platform:nv2005_evcbcs 
#> outputFormat: application/json 
#> count: 10 
#> srsName: EPSG:4283 
#> CQL_FILTER: ("x_subgroupname" = 'Freshwater' AND "bioregion" = 'Glenelg Plain' AND "x_evcname" IN ('Deep Freshwater Marsh', 'Aquatic Herbland')) 
#> propertyName: geom,x_evcname 
#> 
#> <full query url>
#> https://opendata.maps.vic.gov.au/geoserver/ows?service=wfs&version=2.0.0&request=GetFeature&typeNames=open-data-platform%3Anv2005_evcbcs&outputFormat=application%2Fjson&count=10&srsName=EPSG%3A4283&CQL_FILTER=%28%22x_subgroupname%22%20%3D%20%27Freshwater%27%20AND%20%22bioregion%22%20%3D%20%27Glenelg%20Plain%27%20AND%20%22x_evcname%22%20IN%20%28%27Deep%20Freshwater%20Marsh%27%2C%20%27Aquatic%20Herbland%27%29%29&propertyName=geom%2Cx_evcname

Old data that is no longer available

Some data layers are not available on the new platform (at least not yet). At time of writing, the swooping birds data is not available. If you try and retrieve a data layer that is no longer available, you will receive the following error:

try({
vicmap_query(layer = "datavic:FLORAFUANA1_SWOOPING_BIRD") 
})
#> Error in convert_layer_name(url) : 
#>   No matching data found on new geoserver platform. Please search for a new layer with listLayers()