Wednesday, March 8, 2017

Leveller for Open Simulator (part 5)

Merging Documents in Leveller

This tutorial looks at merging two terrains using Leveller (a powerful 3D terrain editor for windows from Daylon Graphics) for later use in OpenSimulator (Opensim). This tutorial assumes that you've read Part 4. If you haven't, please do that before reading on.

We'll be using terrain.party to generate 16bit PNG terrain files for us. This online application allows the user to select a portion of the map of the earth and generate a series of PNG files that you can import into Leveller. The reasons for using this online application are:
  • It's free to use;
  • PNG format can be viewed as an image;
  • Files can be generated from anywhere on the Earth ; and
  • It's simple to use.
Now terrain.party has a couple of quirks as well. All water.sea area is set to zero height, so there's no height information beyond the coastline. Also the selection square used can only range between 8km to 60km. That means that there's a limit to how fine the detail is, and if you want to model anything bigger, you have to create several downloads and patch the heightfields together. The terrain.party window looks like:


You can search by name for a location, zoom in or out, and scroll around the map of the Earth by dragging the mouse. You can also decide which which overlay to use, though the two USGS options only work in the USA and territories. You can also decide how large the selection area is and place that where you want, though it doesn't move unless either you manually move it, or your text search moves the map to a location.

Tenerife Island

Tenerife Island was used an an example of creating contours with shapes in Part 3. Here's a map of it...

By Oona Räisänen (Mysid) - Self-made in Inkscape.Topographic data from the Shuttle Radar Topography Mission (SRTM) as modified by CGIAR-CSI.Bathymetric data from SRTM30_PLUS.Relief shading was derived from SRTM data in GIMP.Roads and place names manually retraced from the OpenStreetMap, originally by OpenStreetMap contributors., CC BY-SA 3.0, https://commons.wikimedia.org/w/index.php?curid=11100572
...and here's an view of it from space:



The main difference between this and the previous two examples in Part 4, is that the island is too large to fit into the largest selection size of terrain.party which is 60km. Now one solution to this issue would to find a different source of map data for the island. For this example however, we'll do a work around, of patching together the heightfields from two overlapping PNGs.

To do this, first we'll expand the selection area to 60km and download from first the South West of Tenerife, and then from the North East. There should be an overlap as seen below:

The two selections above are (as taken from the Readme files of each downloaded zip:
Tenerife 60km SW
http://terrain.party/api/export?name=Tenerife+60km+SW&box=-16.355525,28.468774,-16.968654,27.929788

Elevation range: 0 through 3696 meters 
Tenerife 60km NE
http://terrain.party/api/export?name=Tenerife+60km+NE&box=-16.025059,28.768499,-16.639940,28.229514
Elevation range: 0 through 3616 meters

The maximum heightfields vary because one of the maps has height terrain than the other. The scale for each is 1081 / 60000 or 0.0180166666666667 or 2%! We can scale the maximum heightfields down proportionally to 73.92 for Tenerife 60km SW, and 72.32 for Tenerife 60km NE.

Going through the process that we did for both St. Helena and Rottnest, after inspection well use the Merged versions of the PNGs for each: Tenerife 60km SW Height Map (Merged).png and Tenerife 60km NE Height Map (Merged).png. We'll import each with correct elevation ranges, accept the large map, and select ALL of it. We then have two maps:


Each of these documents is 1081×1081 in size, and their heightfields should be correctly scaled so that when we paste them together, there should be a smooth join. The guidelines on each have been changed to show only 1m and 256m, and the difference in what shows depends on what zoom you have. Before we do anything else, we'll save these documents with their default names as Import of Tenerife 60km SW Height Map (Merged).ter and Import of Tenerife 60km NE Height Map (Merged).ter!

 Merging maps

There are two ways to merge these two documents together. The first is to enlarge one of the documents by using Edit > Pad and then copy and paste the terrain of the other document into it. You could also create a new document using File > New and make sure it has enough width and breadth, and then copy and paste both documents into it. In either case, In the first case I'd recommend saving a copy of the padded document with a new name via File > Save As, so that if anything goes wrong, you still have an original version of that document if anything goes wrong.

We'll do the padding option in this example, and use Import of Tenerife 60km SW Height Map (Merged).ter as the document to be padded. Doing Edit > Pad (or Shift + E) with that document selected brings up...



We should select the bottom left box because that's roughly where that document's terrain will sit in the final merge. This will mean that no pixels will be added to the left or bottom of the document. The next question is how much to expand the document? We could just play save and double the Width and Breadth, and crop the document later. Looking at the maps above, there should be about an extra 512px to both dimensions. If we add 600px that gives a margin or error. Either way we may need to crop or pad the document afterwards, to make a document width dimensions evenly divisible by 256m, and hence suitable for export to Opensim. The help page on this dialog says to ...check Also enlarge the coordinate system's extents if you want the ground extents to increase to take into account the new areas. This will preserve gridpost spacing. Otherwise, gridpost spacing will decrease since more heightfield pixels are using the extents. Normally you'll want to enlarge the extents as this will preserve the spatial measures of heightfield formations. Seems like a good idea. So we'll try this:



...and this is the result:


So far so goo. Now would be a good time to Files > Save As this to something like Tenerife (Merged Terrain).ter. Now we swap to Import of Tenerife 60km NE Height Map (Merged).ter and do a Selection > Select Elevations. If we enter -11 to 0 in the following dialog...



...and the select Replace and OK the that on the following dialogue (if it appears), we get the following selection:


Everything but the land is selected. If we use Selection > Invert (or Ctrl + I) we invert the selection and get this...


...which is the selection we want. Now we use Edit > Copy (or Ctrl + C) on this selection and then swap to the padded document. After doing Edit > Paste (or Ctrl + V) we should see this:


The selection's been pasted to the padded document, and the Select tool is now active. This is currently a Floating Selection, so you can move it about first, and when happy with the position, Defoat it to be part of the terrain below. Until you defloat this, you can still press Delete and erase the selection if you want to (maybe you realised you've copied the wrong selection). Using the scroll wheel on your mouse will zoom in and out on the map, which is handy for matching up terrain.


This looks about right, so we'll use Selection > Defloat > Replace, which will replace the terrain underneath with the floating selection. The result is...


...which is not quite right! What's with the dark lines? Looking at the scene, we see:


Oops, the terrain looks slightly off, and the lines are showing the edge of the paste where the heights don't match. We have several options. We could do an Edit > Regress To to restore the map to the previous state, and return the floating selection. e.g.:



Instead of using the mouse to move the floating section, you can just use the arrow keys to move it one pixel at a time (though you need to time this correctly or you'll overshoot). It may just mean you need to move the floating selection a few pixels horizontally or vertically. If the next attempt doesn't work you can regress and try again. It could be that our scaling is slightly out as well, and you could try importing the PNG again with a slightly lower maximum height.


You can also choose other options than Replace when defloating the selection. Use Edit > Regress To as shown above, but when you Defloat the selection you might try Average instead. This will be fine when defloating over similar parts of the map, but defloating over zero or blank areas, not so much. Another option is to play around with the selection before you copy it. Try doing Selection > Shrink four times and then Selection > Feather with 4px. Or you can Selection > Blur it as well. Then when pasting a slightly smoother edge might result.

Alternatively you could Regress as above, swap to the other map and use Filter > Elevate  with Lower and 1, and then copy the altered selection. If the paste is 1px out in elevation, this might be an easier way to change that.

Or it might not. In the end you might just need to use the Blur tool and Smudge tool to soften the edges manually. Of course that's going to reduce accuracy in relation to the real world terrain, but if you can't see the creases it's all good. Here's the result of some blurring and smudging. There's a hint of line, so some more work needs to be done.


Assuming that's done, we still have some cleaning up to do. Here's what the whole map looks like, with the Contours toggled off:


The padded area has a height of -18, and the original no data (ocean is zero). You need to make this standard. Easy to do using the Magic wand tool. Select some part of the padded area, and then select some of the zeroed are while pressing Shift. That selects everything that isn't the island:


You could also do a Selection > Select Elevations and use .1 and 400 as values. That will select everything over zero. Then press Ctrl + I to invert the selection and we get the same result. Either way, if we press Delete then all the selected area becomes -18.


Finally we'll use Filter > Elevate using Lower and 2 to drop everything by 2m, with the ocean floor being at -20 (the lowest you want for most documents getting exported to Opensim. It now looks like:


Looking better, and a lot better than my result in Part 3!

Monday, March 6, 2017

Leveller for OpenSimulator (part 4)

Importing real life geography into Leveller

This tutorial looks at importing real geography into Leveller (a powerful 3D terrain editor for windows from Daylon Graphics) for later use in OpenSimulator (Opensim). Unlike the previous tutorials, which focus on specific tools or processes, this is more about making decisions on how to import and how to modify the results of that.
Elevation and Image formats

Leveller imports a variety of elevation formats. For a complete list of this see Supported Elevation Formats on the Leveller site. Leveller can also import the image formats as heightfields, see Supported Image Formats. If you examine both pages, you'll notice that there's an overlap between elevation and image formats. These formats were used as a convenient way to encode height values into images which could be edited later. We'll be using terrain.party to generate 16bit PNG files for us, which can be used as both a graphic and a heightmap. This online application allows the user to select a portion of the map of the earth and generate a series of PNG files that you can import into Leveller. The reasons for using this online application are:
  • It's free to use;
  • PNG format can be viewed as an image;
  • Files can be generated from anywhere on the Earth ; and
  • It's simple to use.
Now terrain.party has a couple of quirks as well. All water.sea area is set to zero height, so there's no height information beyond the coastline. Also the selection square used can only range between 8km to 60km. That means that there's a limit to how fine the detail is, and if you want to model anything bigger, you have to create several downloads and patch the heightfields together. The terrain.party window looks like:



You can search by name for a location, zoom in or out, and scroll around the map of the Earth by dragging the mouse. You can also decide which which overlay to use, though the two USGS options only work in the USA and territories. You can also decide how large the selection area is and place that where you want, though it doesn't move unless either you manually move it, or your text search moves the map to a location.

Examples used

We'll be using two examples for this tutorial: Saint Helena, and Rottnest Island. All examples are islands because their coastlines give a clearly defined area to work with.

St Helena

This island lies in the South Atlantic Ocean and is famous for being where Napoleon was exiled after the end of the Napoleonic Wars. It was settled by the Portuguese and British, and is currently a British Dependent Territory.

By Oona Räisänen (Mysid) - Self-made in Inkscape. Based on several sources:CGIAR-CSI SRTM 90m DEM Digital Elevation databaseSRTM30_PLUS bathymetry by Scripps Institution of OceanographyTopographic map of Saint Helena by Barry Weaver and Jean-Pierre LangerSaint Helena Island, a photograph by NASA Earth Observatory, CC BY-SA 3.0, https://commons.wikimedia.org/w/index.php?curid=11135951

In terrain party, it looks like this:


Now the smallest selection size that will cover the island is 20km. Below we see a comparison between that, and the largest possible selection of 60km:


We'll make two downloads, one at 60km, and another at 20km, to show the difference. You download by pressing Download button on the map, choosing a filename and saving to an appropriate location. All downloads are ZIP files so you need to open that and inspect the contents:


The image above shows the contents of saint helena 20km terrain.zip. Opening the saint helena 20km README.txt file, we can read where the zip came from, a URL (e.g. http://terrain.party/api/export?name=saint+helena+20km&box=-5.612559,-15.871623,-5.799511,-16.051286) we can use want if we want to export this again, and information about the heightmaps (PNG files) included in the zip. The Height Maps section lists details about the ASTER, SRTM3 v4.1, and SRTM30 Plus PNGs. These are all different sources for the data with different resolutions and accuracy. The saint helena 20km Height Map (Merged).png file is a merging of all of these, and in all the examples we'll use the Merged versions of the PNGs. The Elevation Adjustment section of the readme file gives us information about the range of elevations in each heightmap (in this case 0 through 1250 meters), and advice on transforming these values for "the game" (which is cities:skylines). Lastly there's the Missing Data section, which informs you if there's missing data on any of the maps. You'll find missing data on all the PNGs in all the examples, because no data is included for ocean areas (hence the zero height value).

If we open saint helena 60km Height Map terrain.zip and read the readme file, it says...
No height map data was available. That's a bummer.

If you're way at the poles, that's probably why. Orbital measurements only go
so far away from the equator.

If this area is filled with water, try a box that has less water. That's
probably a good idea for gameplay reasons anyway.

...which is annoying, but at least it tells you why it failed. In any case, our available PNG file looks like:


The bitmap image is 1081x1081 in size and 8bit. To import this into Leveller, go to File > Import, and select the PNG option. Then load the saint helena 20km Height Map (Merged).png file, and you should see something like this:



As you can see, Leveller wants to know the elevation range. Terrain party reported this as being 0m to 1250m, but that will need to be adjusted. There's no need to change the zero, because that's seal level where the "missing data" lives. Now 10m might be a an OK guess, but we need to consider the scale factor. The 1250m maximum was for a 20×20km - or 20000×20000m - area. Assuming that 1px = 1m in Leveller (that can be changed) then that means the image is 1081m wide, so the map has been scaled down to 1081/20000 or to 5.405% If we multiply the maximum height by this, we get 1250 times 0.05405 or 67.5625, which we'll round up to 68m. Enter that as the second span value, select OK, and we get:



This is a standard confirmation screen, which appears when you're importing a terrain over a certain size (starting at 1024, but this can be changed in preferences). We'll press OK, and this takes us to the next dialog:



Because the terrain file's above the limit, this dialog needs you to select which area to import. You can do this manually with the Edit button, or dynamically with the selection marquee. You can also resize the terrain during import by ticking Downsample and selecting a % to downsize to from a drop-down menu. We'll skip both and instead dynamically select all using the marquee. It should now look like this:



Select OK, and the resulting map looks like this:


What's that bite in the South East? If we look back to the PNG we can see that it was on the image. When looking at the scene pane, we see:


Clearly there are errors in the merged version of the heightmap (red arrows). After a search on the net to find an aerial photo of St Helena...

From https://goo.gl/images/0h78ve


...we can see that the spikes on the West and North coasts are aberrations (probably in the original data of one of the sources), the bite in the South East is missing data, but close to the actual coastline. However, the cliffs in the South West are actually there!

There's several options now. You could:
  • Try one of the other PNGs;
  • Try another download using terrain party;
  • Fix the aberrant and missing terrains manually, using the photo as a guide; or
  • Find another source of data that is more accurate and complete.
We'll try another PNG first, and pick saint helena 20km Height Map (SRTM3 v4.1).png because the readme text describes that as...
...provided by the Shuttle Radar Topography Mission but SRTM3 v4.1 is the result of considerable effort from CGIAR-CSI. It has a ~30m resolution in the US and a ~90m resolution elsewhere.

...so it might be the more accurate of the PNGs. Viewing the PNG itself...


..we can see that there's no spikes (white patches), and that the South East corner is intact. We'll follow the same routine as listed above (with a range of 0 to 68 elevations), and the result is:


This is looking better. The fringe around the island is possibly where the radar gave an indeterminate result. We can easily get rid of it however. First we move the cursor around the map to check the height of those areas:


It varies but never goes above 4m, so we'll select Filters > Elevate, and Lower by 4. As we have nothing selected, this will lower the entire heightfield by that much. The new map now looks like:


The small islands around the main one, might actually be tidal in nature, or it could just be the radar going wacky. You can always edit them out manually. The blue sounding area now shows because the points that were zero are now -4, below the water line. Here's what the scene now looks like:


You may need to fiddle with this further. For example, is the height scale correct? You can change that using Filter > Span. However it's a bit hard to fully appreciate the scale here. Perhaps you want to export it to Opensim first to see how it looks in relation to an actual avatar? In that case we need to fix the size of the document.With a 1081×1081 size, that's just above 1024×1024. That could be: one large 1024m² region; 2×2 512m² regions, or 4×4 256m² regions. The first step would be to crop the document to 102×1024. To use Edit > Crop, you need to have a rectangular area selected. You can either do that manually by using the Select tool, or by using Selection > Select Area option. We'll do the latter for precision.



Select OK, and this dialog appears:



We want to Replace as shown. Select OK and you now have the following selection:


Now select Edit > Crop (or press Ctrl + R) and we get (after I adjusted the guidelines):


Of course you could also shrink or enlarge the document as well. The upper practical limits on document size, according to the official FAQ, is about 6000×6000 for the 32 bit version, and 12000×12000 for the 64bit version. The amount of memory you have on your PC will affect performance as well.In any case, suppose you only want a single 256m² using this map. You can resize a whole document by using Edit > Resample. This brings up a drop down list of choices:



The size we want - 256×256 - is on the list so we can select that. The result is now:


Before anything else, I would do a File > Save As and save the re-sampled version as a copy of the original. If you don't, and later want to resize it again, you will have lost data in the sampling down of the document, and would probably have to import the PNG again. Or, you can do an Edit > Regress To. This brings up the following dialog:



In the example above, you could just select Select none of heightfield, or Initial state, and Leveller will recover the original version of the document. Often this is a lot easier, and quicker, than doing a Ctrl + Z (Undo). So, we have our original size restored. What if we wanted to resize to a value not on that list? In that case we'd select Custom on the resampling list, or just Shift + S at any time. This brings up this dialog:



As you can see, we can use any value for width (the horizontal size of the map) and breadth (the vertical size of the map). If these new figures aren't proportional to the originals (e.g. 1024, and 512) then that will change the proportions of the terrain. You may need to use the calculator here to get values for non-square documents. Pressing F1 (as it will for most dialogs) will bring up a HTML help page in your browser, including an explanation of the differences between Plain, Bilinear, and Bicubic resampling. We'll change the size to 4096×4096, leave the rest as is, and get:


The resize worked, but the elevation values stayed the same, so the result looks very flat. We'll Undo to the previous state, resample to 4096×4096 again, but this time tick the Keep elevations proportional box:


This is better than before, but the flat bits around the coast have also been raised and raised above the water level (still zero). An inspect of those shows that they never get above 12m, so if we do a Filter > Elevate with Lower and 12, we get...


Much better! As you can see, the the coastline's been restored.

Rottnest Island

Rottnest Island (known as Wadjemup to the local Noongar people, and otherwise colloquially known as Rotto) is an island 18km off the coast of Western Australia, and due west of Fremantle. A sandy, low-lying island formed on a base of limestone, Rottnest covers 19km². We'll be using Rotto as our second example in importing real life terrains.

By Moondyne (openstreetmap.org) [CC BY-SA 2.0
(http://creativecommons.org/licenses/by-sa/2.0)],
via Wikimedia Commons


Here's what Rottnest Island looks from the air:

By Created by the uploader.
Any IP involved in the creation of this image is released into
the public domain. [Public domain], via Wikimedia Commons

Rottnest varies in from Saint Helena in several ways: it's nor round, squarish shape, it's a generally flat area, and it has several lakes and reservoirs.We'll address these issues, as otherwise importing the terrain follows much the same steps as St Helena. Using terrain.party, we can see that the whole island can be covered using a 12km selection area:



We could use a selection of 11km, but that cuts very close to the coastline of the island. Downloading that selection we open Rottnest Island 12km terrain.zip and find the ASTER 30m, SRTM3 v4.1, SRTM30 Plus, Merged, and Readme files. The Readme file tells us that the files were made from the http://terrain.party/api/export?name=Rottnest+Island+12km&box=115.568385,-31.952871,115.441188,-32.060669 selection, and that the height ranges are between 0m (coastline) and 82m. After inspection, the Merged version of the PNG looks like the best one to use.



Once again the image is 1081×1081px (a standard size with terrain.party) but instead of a 20km selection as we used with St Helena, this selection is 12km. That gives us a ratio of 1081/12000 or 0.0900833333333333 which we can round to 9%. Multiplying 9% by 82, we get 7.38m for the highest elevation. Our import dialog should look like:



We select OK, and also OK the Proceed with Import dialog that appears (because the size is larger than 1024 a side) after this . This then brings up the following dialog:



Rather than accept the default, or select all of the imported terrain, we'll select a custom are, as below:



The map of this (with 1px and 256pm guidelines) looks like:


As you can see, you could export this using File > Export tiles as either a 4×2 tileset of 256m², or a 2×1 tileset of 512m². The scene looks like:


We still have the same issue that St Helena had. The ocean areas here are really zero elevation, and the coastline looks like there's a shape drop. This means that if exported this into Opensim, the ocean would be either all at 20m, or 0m! Moving the cursor around the coastline of the map, we can see that the heights are between 2.1 and 2.9. If we do a Filter > Elevate with Lower and 3, we get:


Clearly a value of 3 is too much. Next we'll Undo, and try the Filter > Elevate with 2.5:


This is better! Of course we could instead raising or lowering the elevation of the entire heightmap, change the Water Level. You can do this by using Edit > Water level > Elevation and entering a new value, or by using the Water Tool and moving the mouse. We didn't use either of these, because as we're exporting later to Opensim, we want the water level to stay at zero. See below for fixing the ocean height levels properly. Rottnest lies on a coastal shelf, but that's more than 2.5m deep!

But, what about the ocean?!


There's one thing remaining before we can export these documents to Opensim - the ocean. In both cases the heightmap imported via the PNG file has "missing data". There were no values for any of the water areas, so these were "zeroed out" and shown as 0m. Over the course of fiddling with the documents both heightfields were lowered using Filter > Elevate, so that what was zero is now a negative value. While that might look OK in the Scene pane, it's going to flat and either very shallow or very deep. Either way that's not very natural looking.

So, we need to modify those areas, but at the same time not modify the land area that we've taken care to import intact. The easiest way is to use shapes as contours to generate terrain in those areas, and to also use a selection to make sure that it's only those areas affected. Before you continue, go read Part 3 about how to use shapes as contours in Leveller. Pause. Now that's done, consider what we need for Opensim. Usually in Opensim the water level is set at 20m. and the bottom of a region is 0m. You can go lower than that by manually lowering terrain using the Edit Land tools, but anything lower than zero tends to be a dark murky unreliable space. The following image shows the Master Builder avatar standing at -9m in a test region. Not pleasant!


In any case, if you export a .ter file into OpenSim, anything lower than zero will be set to zero, so with the default setting anything lower than -20m in Leveller will be lost. I prefer to use -18m as the bottom height for a "default region" in Leveller, as it gives me some leeway in using the Perlin filter to make it bumpier. Things are different if you are using a water level higher than 20 in Opensim. If you have that set to 40, but are still using zero as the water level in Leveller (good practise), then you'd need to enter 40 for height adjustment in the File > Export tileset dialog instead of the default of 20. However, this also means that you can go down to -40m (or -38 to play safe) in Leveller when creating the sea floor.

Well assume that both examples are for default water height regions in Opensimulator. The first thing we can do is create a set of contour shapes around the land terrain. Here's one version of this for St Helena:


I haven't been too fussed about accuracy here with heights. The map for St Helena does show some undersea gradients, but according to the legend they go down to -3000m. Even adjusting for scaling, that would still be -162.15m, far too low for our purposes! Instead, this should give use a receding ocean floor. The next step is selecting only the ocean.We do this by using Selection > Select Elevations and using 0 and 400 for the elevation values. This selects everything above the water level. Then we use Selection > Invert Selection (or Ctrl + I). Everything that isn't the island is now selected:


Use Shapes > Heightfield from Points and we get the following:


As you can see only the selected area's been affected. The land remains untouched. Maybe we don't want an perfectly smooth ocean floor however. We use Filter > M-R > Perlin Noise and enter the following in the dialog box that pops up:



The How to Draw setting of Add means that the noise generated (in + and - height values) will be added to the existing height value. You could also try Average for a gentler result. Select OK, and we get this:


Here's a closeup of that:


Looks good. Save that document! We'll use the same process for Rottnest as well. Here's the contours:


And here's the finished result:


The Finished Terrains

You can now export the St Helena and Rottnest documents using File > Export tilesets as shown in Part 2, in whatever combination of tiles you want. Here's how St. Helena looks exported to a grid of 1×1 of 1024m², on the map, from a distance, and compared to an avatar:






And here's how Rottnest looks being exported to a 4×2 grid of 256m², from a distance and with an avatar:






Next Time...

In Part 5 we import real life geography for Tenerife!