On Well Known Text and the Web
Over the past week I have made a start on looking back at the problem of transferring spatial data out of databases into front end applications. This issue stemed from a project I worked on UCL called Lacuna. Lacuna was a 3D, WebGL based Geographic Information System that I worked on for my masters dissertation. I have since deprecated Lacuna because, in all honesty I dived straight in the deep end and at the time I had only a superficial knowledge of JavaScript and PHP. As such it accrued a lot of technical debt from trying to finish my dissertation in a reasonable time frame.
With the knowledge I have accrued since then I have decided to give tackling CRUD operations on spatial databases via web services. One decision I made was it would require something more performant for the backend. For the language I knew PHP5 wasn't going to cut it this time around. I weighed up the pros and cons of various languages such as Node and PHP7 but decided that Go was a solid choice. However for the backend I would keep using PostGIS due to it and PostgreSQL relative maturity.
Go is an interesting language, and I'm equal parts enjoying and bashing my head on the desk playing with it for this latest project. The backend of Lacuna was responsible predominately for getting the 3D geometries stored in PostGIS and bringing them to the front end to be rendered by Three.js.
This shouldn't be an overly complex task, but this time around, I realised that to do it properly is actually quite a significant pain. The major reason for this is Well Known Text. For those of you that aren't familiar Well Known Text is a markup language for geometries. Specifically it is the text based, human readable version of Well Known Binary (WKB), which is the standard for storing geometries in spatial databases. Let me show you an example of what Well Known Text looks like:
POLYHEDRALSURFACE Z (
((0 0 0, 0 1 0, 1 1 0, 1 0 0, 0 0 0)),
((0 0 0, 0 1 0, 0 1 1, 0 0 1, 0 0 0)),
((0 0 0, 1 0 0, 1 0 1, 0 0 1, 0 0 0)),
((1 1 1, 1 0 1, 0 0 1, 0 1 1, 1 1 1)),
((1 1 1, 1 0 1, 1 0 0, 1 1 0, 1 1 1)),
((1 1 1, 1 1 0, 0 1 0, 0 1 1, 1 1 1))
)
You can find a full specification here. As it stands there 18 different Geometry types and 4 variations on each (2D, Z, M, ZM). My job as the developer is to convert these Well Known Text fragments so that I can use clientside in a 3D geometry viewer. This on the surface seems like a plausible task; I mean, firstly someone must have hit this problem before, there's got to be a library out there that does this right? It turns out there are a few and they're great, but inevitably they have their limitations. One recurring issue is the 3D support is varying (either they lack Z or ZM geometries, PolyhedralSurfaces, TIN Z or a combination). Generally this is okay if you are doing things with Leaflet, OpenLayers, Google or Esri APIs and working in a 2D environment. However, as the GI industry progresses we are incrementally seeing more 3D geographic information, with demand increasing for both data and systems that can handle 3D.
In reality most web systems will mostly likely require that the data at some point make it's way into JSON for consumption by the ubiquity that is JavaScript. As an example Terraformer (an open source library from Esri) tackles this by converting to GeoJSON. Again this is great stuff, but unfortunately GeoJSON is a 2D specification (as it stands).
OK you say, well stop being lazy and write your own parser and come up with your own JSON schema. So I am, and I'm about 40% of the way there, but I felt compelled to point out some issues I see with WKT. One of the major ones is the way the text itself is delimited. WKT doesn't appear (to my knowledge) to leverage any obvious standard for structuring data (i.e. JSON, XML or YAML). As such you more-or-less have to write the parser from scratch. Lets take the example above of the POLYHEDRALSURFACE Z: the X Y Z coordinates are delimited by spaces which, amongst other things makes converting them into something more usable like JSON quite a complex task. Delimitation between coordinates in the same part of the geometry, different parts of the same geometry and separate geometries in their entirety are all delimited by commas. This in turn makes it tough to parse effectively. What characters should you split on? Is that even a good methodology? How can I ensure the integrity of the regex if I go down that path? These are of course rhetorical questions but I hope you get my point.
The approach I took in the end was using a combination of string manipulation and regex, but it feels fragile (read: hacky), and by all means this may not the most optimal. Perhaps a better approach would be to break down the parentheses but it's hard to tell which ones are superficial and which are integral to the structure of the text. Unfortunately the specification is quite dense for developers to digest in comparison to something like GeoJSON (MapBox have poked a little fun at this on the wellknown GitHub README).
Again though this poses another problem, now I've parsed the WKT into JSON, but how should the JSON schema look? Maybe like GeoJSON? But really there's no standard so it feels somewhat arbitrary. The real point of this post is to open up a dialogue on how to handle WKT going forward. It seems like a standard that has been adopted by spatial database and as such is not going anywhere fast. Storing 3D data in databases makes sense, but in honesty WKT is a tough thing to transition to something meaningful for clientside apps (unlike JSON, XML, YAML). Should we be working towards better support for 3D in projects like wellknown from MapBox, Terraformer from Esri and GeoScript? And should we be looking at ways to produce a standard for 3D geo data in the web i.e. by extending GeoJSON? I am really keen to hear what approaches people have taken to handling WKT in development.
As an after thought, how about something like this for modelling WKT as JSON?:
{
type : "POYLHEDRALSURFACE"
dimensions : "Z"
geometries : [
[[ {x: 0, y: 0, z: 0}, {x: 0, y: 1, z: 0}, {x: 1, y: 1, z: 0}, {x: 1, y: 0, z: 0}, {x: 0, y: 0, z: 0}]],
[[ {x: 0, y: 0, z: 0}, {x: 0, y: 1, z: 0}, {x: 0, y: 1, z: 1}, {x: 0, y: 0, z: 1}, {x: 0, y: 0, z: 0}]],
[[ {x: 0, y: 0, z: 0}, {x: 1, y: 0, z: 0}, {x: 1, y: 0, z: 1}, {x: 0, y: 0, z: 1}, {x: 0, y: 0, z: 0}]],
[[ {x: 1, y: 1, z: 1}, {x: 1, y: 0, z: 1}, {x: 0, y: 0, z: 1}, {x: 0, y: 1, z: 1}, {x: 1, y: 1, z: 1}]],
[[ {x: 1, y: 1, z: 1}, {x: 1, y: 0, z: 1}, {x: 1, y: 0, z: 0}, {x: 1, y: 1, z: 0}, {x: 1, y: 1, z: 1}]],
[[ {x: 1, y: 1, z: 1}, {x: 1, y: 1, z: 0}, {x: 0, y: 1, z: 0}, {x: 0, y: 1, z: 1}, {x: 1, y: 0, z: 1}]]
]
}
I know what you're thinking; this is longer than the original! That is true, however after gzipping and transfer over the wire the difference will be very negligible because it's a repeating pattern (x, y, z).
Published