Klokan Technologies GmbH

WebGL Earth: Open Source 3D Globe for Web Browser

2011-01-25T21:51:00.023+01:00

WebGL Earth is an open source software enabling to explore, zoom and “play” with the 3D globe directly in a web browser on any platform including mobile devices - without a plugin. The project lives from support and cooperation of developer community.

Live demo:
http://www.webglearth.com/.
Project page with source code:
http://www.webglearth.org/.

It is written in JavaScript using HTML5 Canvas tag with WebGL extension. The code uses Closure Library and Compiler - the same toolset which is behind GMail, Docs, Maps and other Google products.

The project displays detailed street-level data (via OpenStreetMap), detailed aerial imagery (via Bing Maps) and any other custom maps which are available in popular Mercator tiles (prepared by MapTiler, GDAL2Tiles, TileCache, GeoWebCache, etc).

Browser supporting WebGL, like Mozilla Firefox 4 Beta, Google Chrome Beta, WebKit nightly or other is necessary to try this project out.

If you don’t have one, you can at least check the video:

WebGL Earth is available to embed in your own websites via a simple JavaScript API, which we hope to extend soon - based on requests of community. Check the first set of examples demonstrating the API:

Developers can also modify the code completely and use the rendering core as a component for other projects. The best place to start with development is the Quickstart document.

Users are welcome to submit feedback. Developers can join mailing-list, join our community, create a new amazing applications from the source code and make contribution to WebGL Earth or help with reporting and fixing of bugs.

We are looking forward to hear from you. The future of this open-source project is in your hands!

Closure Compiler for OpenLayers 3.x

2010-12-02T15:34:00.001+01:00

You probably heard about the JavaScript Google Closure Tools which were released as open-source (Apache License 2.0) by Google more then an year ago. It is a set of tools which are in use for development of Gmail, Google Maps, Reader, Docs and other popular Google products.

In this blog post I would like to summarize what cool features the Closure Tools offer to developers of large open-source JavaScript projects, and I would like to suggest (re)writing OpenLayers 3.0 the way, that it is fully usable with Closure Tools. We can all profit from what is already out there - and write smaller and faster web and mobile applications easier with the future versions of OpenLayers, if we decide to go this way.

The most important tool in the Closure set is the Compiler. All other tools are optional and are build around the features of the compiler. The compiler is usable from command line or as a web application. You can also HTTP POST your source code and get back the compiled code. The compiler can be utilized by the existing building scripts of OpenLayers easily.

The compiler has several cool features:

It compiles readable JavaScript into compressed machine-readable JavaScript.
Documentation of the code with JSDoc Tags is important: the compiler reads it and you get warnings during compilation for typos in documentation, wrong use of a @constructor, wrong type of a variable, misuse of a field annotated with @private and @protected, etc.
If you write a reusable JavaScript library, such as OpenLayers, you formally export your public API - and the compiler optimizes your internal code.
The end applications can be compiled together with the library - and then the unused parts of the library are removed from the produced code. Dependencies are solved automatically by the compiler.
Compiler accepts constants to remove unwanted functionality - this allows compilation only for particular browser such as Mobile WebKit, for only one of Quirks mode or Strict mode, compilation without support of IE6, etc.
Debugging with FireBug is possible even for the compiled version of the source code.
Compiler supports generation of dynamically loadable modules, which can significantly speed up loading of the end application, because the code for advanced functionality can be loaded only when it is required.

Let's go through these points one by one:

1. It compiles readable JavaScript into compressed machine-readable JavaScript.

The compiler loads the whole source code of the supplied JavaScript application into memory, makes detailed analyze of the code, prepares internal graph of dependency and while knowing the syntax and partly also semantic of JavaScript language, it makes the compilation.

If you use the default "SIMPLE_OPTIMIZATION" then Closure Compiler behaves just as another JavaScript minimizer: whitespace and comment removal, renaming local variables and function parameters to shorter names (only symbols that are local to functions). It can be directly used with the OpenLayers Build Profiles as they are available now - and you get around 20% smaller OpenLayers.js file on the output, compared to the default jsmin-based minimization of the code. Check also latest #2871 and #2873 if you want to have smaller results for OL. BTW official releases of jQuery are now minimized with Closure compiler and SIMPLE_OPTIMIZATION.

The truly powerful is the compiler when it is used with the "ADVANCED_OPTIMIZATION". Then it uses several tricks to shorten the code further, including the dead code removal and function inlining. But the JavaScript code must be ready for the compilation and it can't contain buggy code patterns. It means that the programmers are pushed to write their JavaScript code in a more readable and maintainable way. Typical problem is with using of "this" outside of constructors and prototype methods. Programmers can't use some of the techniques which the JavaScript language offers, but this restrictions also help to keep the code readable and reusable by other people later on. In general, programmers should follow the rules described in the JavaScript Style Guide. There is also a tool for checking and automatic corrections of the JavaScript code called Linter which can help. OpenLayers 2.x is not following these rules right now, but I think for OpenLayers 3.0 it would make sense to write the code with these rules in mind.

2. Documentation of the code with JSDoc Tags is important: the compiler reads it

OpenLayers 2.x, same as other big JavaScript projects, are using JSDoc comments for inline documentation of the source code. Google Closure compiler is parsing such JSDoc tags, and then using it to enhance the code optimization process and to print warnings during compilation for potentially buggy code patterns or mistakes, if used with "--warning_level=VERBOSE".

In fact through JSDoc comments the JavaScript language is enhanced with complete type system for variables and with visibility of objects.

The JSDoc set of annotations and type expressions which Google Closure compiler understands is described at http://code.google.com/closure/compiler/docs/js-for-compiler.html.

The introduced type system is powerful. OpenLayers can profit from possibility to define your own object types, for example the "OpenLayers.Location" object can be a new type which always must contain ".x" and ".y" with type float. Compiler can enforce this: whenever a function expects "OpenLayers.Location" as a parameter you receive warning during compilation if you use an object which does not comply.

3. Public API is formalized - the compiler needs it

It is important to have a formal definition of the public API which you are exporting from the library such as OpenLayers - compilation of a publicly usable JavaScript library is not possible without that. It means you are pushed to keep the API clean - as well as documented correctly with JSDoc.

4. The end applications can be compiled together with the library

The end-user applications can be programmed traditionally: So the OpenLayers library is used as a standalone OpenLayers.js script, which is included in the website in the HTML header. The application itself has a separate code base, and uses the public OpenLayers API. This approach is common with OpenLayers 2.x.

Alternative approach is that the code of the application is merged with the code of the library and then the result is compiled together - this way the unused parts of the library are removed automatically from the final code. Dependencies in the code are solved by the compiler.
Only the JavaScript functionality, which is really in use by the end application is then part of the final compiled .js file, which is used for deployment and production use.

5. Constants defined during compilation to remove unwanted functionality

Variables annotated with the @define JSDoc tag can be redefined during compilation, and then the compiler can recognize blocks of code conditioned with such variables as unreachable. Because the dead code is removed automatically - such source code is completely stripped from the final result .js file.

Typical use case for this is a block of code executed only if some DEBUG variable is set to true, but there are also other use cases:

A source code of a JavaScript library can contain plenty of functions to handle differences between Quirks and Strict mode of a web browser, determined by the DOCTYPE tag in the beginning of the HTML page. The general JavaScript library must support both modes, but if you are a developer of a web application you control in which rendering mode is your application used. This means the handling of Quirks mode can be easily stripped from library which you are using.

Similarly - if you are developing a mobile web application for iPhone or Android platform, you don't really need in your application code which is specific for Internet Explorer or for older versions of Firefox or other browsers.

The Closure Library, the standard JavaScript library coming with Closure Tools, supports already this kind of conditional compilation for different browsers (via goog.userAgent.ASSUME_MOBILE_WEBKIT) and rendering modes (goog.dom.ASSUME_STANDARDS_MODE).

I saw at FOSS4G that for OpenLayers 3.x is planned closer interoperability with existing general libraries such as Prototype, jQuery or Closure Library for accessing DOM, operation on strings or other basic functionality - to not reinvent wheel inside of OpenLayers and eliminate duplicate code for the same functionality in final applications, which are already using one of these libraries anyway.
The conditional compilation with constants supported by the Closure compiler can help in this case.

6. Debugging with FireBug

The minimized JavaScript code is normally very hard to debug, because the code is obfuscated and there is no reference to the original formatting and variable names.

Closure Compiler comes with two handy approaches for debugging the code compiled in the ADVANCED_OPTIMIZATION mode:

First is the parameter "--debug=true", which causes that the renamed variables which are normally shortened to one or two letters keep meaningful names - e.g. "OpenLayers.Location.prototype.setValue" becomes $$OpenLayers$Location$$$$$setValue$$, instead of "aa" for example. This parameter is very often used with "--formatting=PRETTY_PRINT".

The second option is usage of the final compiled code together with the Inspector extension for Firefox. Compiler then generate a mapping file between original source code and the compiled code and the extension simplifies debugging with Firebug, once such mapping file is loaded.

7. Dynamically loadable modules

Google Maps API V3 as well as other Google products are compiled to load only a small core (bootstrap) of the necessary functionality and later load extensions with more code.
This approach can significantly speed up the first appearance of a web application, which is important especially on mobile devices.

The Closure compiler of course supports this form of compilation, with parameters on the command line.

The whole project can be compiled with a simple Makefile or custom merge and build scripts, but there are also several tools available for simplifying the compilation, module dependencies and handling of the code which is spread over several files - these tools are useful especially if you are using Closure Library or you need compilation into modules: Plovr, ClosureBuilder, or Closure Modules. With Closure library comes also closurebuilder.py and depswriter.py.

I am very keen to discuss with OpenLayers developers and the community the application of Closure compiler and it's advantages and disadvantages.

We have used the compiler and Closure library already at some projects which are now in production use, so we have practical experience with these tools - for example from the development of the web interface for our MapRank Search product.

If there is an interest I can publish another blog post with a code of a simple project and step-by-step guide demonstrating the use of Closure Compiler and Library.

The Closure tools are definitively worth attention for any web developer. Please write into the comments what do you think about the subject of Closure Compiler and OpenLayers V3.

Custom style for Google Maps

2010-05-21T11:24:00.000+02:00

I am impressed by the new functionality in Google Maps API V3: StyledMaps

You can create your own dynamic style for Google Maps tiles - and change the look and feel of the base maps which you want to include in your website.

It is quite a natural step, something what open-source projects like Cascadenik and products of other companies such as CloudMade made possible already some time ago. The adoption of this functionality by Google brings this possibility to masses.

If you want to define your own styles for the map you can use the online Style Map Wizard tool. More info and the official announcement is available here.

FOSS4G 2010: Vote for OldMapsOnline.org!

2010-05-12T09:36:00.002+02:00

I submitted a proposal for the presentation in the FOSS4G conference in Barcelona and I would like to ask the OSGeo community and other people who plan to visit the FOSS4G conference to vote for the presentation: "OldMapsOnline.org: Open Source & Online Tools for Old Maps".

In the OldMaspOnline.org project we are developing open-source software and designing online tools for collaborative georeferencing, annotation, 3D visualisation, accuracy analysis and geometadata specification for old maps (or in general any raster images) from the web browser.

One of the very interesting and practical results is an online service for georeferencing of the scanned maps or any other online images. You can directly from the webbrowser georeference any online image published as JPEG, any image already published on Wikimedia or Flickr, collection of online tiles (Zoomify, DeepZoom, ...) or imagery published on one of the supported image servers (IIPImage, Lizardtech Express MrSID, DigiTool, ...). In this moment the service is under active development and I would like to announce the results at FOSS4G 2010 in Barcelona.

In OldMapsOnline.org we have also contributed to several open-source projects. You can find our code in OpenLayers (Zoomify support in 2.9), GDAL (GDAL2Tiles), GeoTools, IIPImage and other FOSS projects.

We have produced two new open-source projects:

- MapTiler: user-friendly tile map publishing a la Google Maps: http://www.maptiler.org/

- IIPImage JPEG2000: open-source server software for fast delivery of the ultra high resolution raster imagery directly from JPEG2000 or TIFF files. MooViewer, OpenZoom, Zoomify, DeepZoom or OpenLayers provides the attractive user-experience on the client side (usually in the web browser). http://help.oldmapsonline.org/jpeg2000/

If you are interested to know more about our project, feel free the explore our websites: http://help.oldmapsonline.org/.

To vote for the presentation please visit: http://2010.foss4g.org/review/ and follow instructions there - the deadline is this Friday (May 14th)! I am looking forward to meet you in Barcelona ;-)

OziExplorer OZF format specification + open-source decoder!

2010-01-15T10:55:00.049+01:00

Yesterday I was testing the OZEX project, which intends to be an open-source replacement for the popular OziExplorer software. The most interesting on OZEX is that it is able to decode and display the Ozf2 and Ozfx3 binary files on Linux and other platforms and that the decoder is completely open-source!
I know about other nice and open-source projects targeted to OziExplorer users with an advanced GUI and interesting features, look at the QLandKarte GT screenshots for example.

This is for the first time I see open-source implementation of the OZF2 and OZFx3 binary format!
It would be excellent to create a decoder also in the GDAL library (as a driver) because it would bring the OZF reading/decoding functionality into several open-source projects.

I have submitted to the gdal svn a documentation of the format derived from the source codes, some sample files as well as links to the OZEX GPL code. More sample files can be generated with the img2ozf utility (runs well under Wine).
Unfortunately I am now busy on another projects but I hope that some of the GDAL developers finds a bit of time to do the coding of the OZF driver...

GDAL already has a preliminary support for the OziExplorer's .map files (textual metadata, think of advanced ESRI World File with included info about the map projection), but support for the binary formats from the OZF family (version 2 and version 3) would move the compatibility to a different level.

OziExplorer is a very popular in the GPS and GeoCaching community. Support of the maps generated or georeferenced with this software in the OSGEO open-source tools would be great! I hope to see it in the near future in MapTiler, GRASS, QGis, MapServer, GeoServer and all the other FOSS GIS tools! Anybody interested in the coding for GDAL?

IIPImage JPEG2000: Free Software for Zoomable High Resolution Online Images

2010-01-13T15:33:00.006+01:00

As a technical manager of the OldMapsOnline.org project I am very pleased to post to this blog a note about our results:

Moravian Library and the OldMapsOnline.org project are proud to announce the release of a new version of the open-source IIPImage server software (http://help.oldmapsonline.org/jpeg2000/).

The freely available IIPImage software can be used for stunning online presentations of scanned documents, paintings, maps, books, newspapers, photographs or other high-resolution images on the web directly from JPEG2000 or TIFF files.

The new version allows direct publishing from JPEG2000 images to a wide variety of different client technologies based on AJAX, Adobe Flash or Silverlight. These include popular pan&zoom viewers based on Zoomify or Seadragon technology (including the Seadragon AJAX viewer and the Seadragon iPhone application) as well as it's own AJAX enabled IIPMooViewer. The documents provided by IIPImage can be displayed in any web browser and on a number of platforms - Windows, Mac, Linux or iPhone.

The software is primarily targeted at institutions who operate their own server connected to the Internet and who want to publish large collections of digital images directly from JPEG2000 or
TIFF files.

Institutions who does not have the necessary infrastructure can follow our alternative tutorial at http://help.oldmapsonline.org/publish/ on how to achieve the same using standard web hosting and free software.

IIPImage is a light-weight client-server system for fast and efficient online viewing and zooming of ultra high-resolution images. It is designed to be bandwidth and memory efficient and usable over a slow Internet connection even on gigapixel sized images.
It is available for free, under an open source license (GNU GPL). We recommend installing
the software on a Linux (or other UNIX) server. We have prepared an easy to install binary package for Debian and Ubuntu with step-by-step instructions for installation.

JPEG2000 support has been implemented using the Kakadu library, which provides one of the fastest implementations of the JPEG2000 ISO standard and is redistributable for non-commercial use.

The enhancement of IIPImage was developed by the Moravian Library and the OldMapsOnline.org project with the support of grants from the Ministry of Culture of the Czech Republic.

The Moravian Library (http://www.mzk.cz/), based in Brno, Czech Republic, is a research institution and a legal deposit library. Project OldMapsOnline.org (http://www.oldmapsonline.org/) is a research project of the Moravian Library that aims to develop software to assist in the management, manipulation and visualisation of historical map collections on the web. The project team is designing online tools for publishing, collaborative georeferencing, annotation, 3D visualisation, accuracy analysis and geometadata specification for old maps.

For more information and for the IIPImage JPEG2000 software, see http://help.oldmapsonline.org/jpeg2000/.

Raster map reprojection (warping) with JavaScript and HTML5 Canvas

2009-10-24T17:36:00.014+02:00

I wanted to create something nice during the FOSS4G Code Sprint, and I ended up programing a prototype of the client-side raster reprojection implemented with the HTML5 Canvas, JavaScript and Proj4js. It is kind of port of gdalwarp into the web-browser environment.

To try it you need a web browser which supports HTML5, it means the latest version of one of the Firefox, Chrome, Safari or Opera.

Have a look at the demo:

Raster map reprojection (warping) with JavaScript in a web-browser with HTML5 Canvas

Link: http://examples.maptiler.org/map-reprojection-html5-canvas/

It is just a prove of concept in this moment, but I am sure you will come up with plenty of ideas for practical application... post a comment! ;-)

Garmin Custom Maps KMZ from GeoTIFF via MapTiler / GDAL2Tiles

2009-10-10T19:51:00.084+02:00

Garmin rolled out beta version of firmware for their latest generation handheld GPS receivers allowing you to display scanned paper maps and another raster data in their navigation devices. They have chosen to do that with a simplified variant of Google Earth's KMZ format, so you can prepare maps for your Garmin even in Google Earth.

On the Internet you can find plenty of free maps in the standardized GIS formats, which can be now transfered into your GPS device. Look at USGS topographic maps, USFS maps, or NOAA Nautical Charts, etc.
These geodata has already high quality georeference embedded, but they are using different coordinate systems and map projections then Google Earth and this complicates the conversion into the form supported by latest Garmin handhelds.

I am the author of MapTiler - a graphical application for Windows, Linux and Mac OS X, which allows extremely easy and user-friendly transformation of raster geodata into the web presentation (mashups) and tile overlay for Google Maps, Bing, Yahoo Maps, OpenStreetMap or Google Earth. You can download and use this software for free. And it is open-source!

The application now supports direct export into the Garmin Custom Maps KMZ. If you have one of the supported Garmin devices you can try an example map: the USGS map of Grand Canyon encoded for Garmin GPS KMZ (9 MB) or the alternative from USDA Forest Service. (10 MB)

Because I don't have a Garmin device with support of the KMZ, I would like to kindly ask those who have it and are going to use the Garmin rendering functionality to donate a few dollars via PayPal, so I can buy one of these devices and also cover the time I spent on the development of this new feature and on the new release of MapTiler.

PLEASE WHEN YOU WILL ANNOUNCE SOMEWHERE THIS NEW GARMIN FUNCTIONALITY OF MAPTILER PUT THERE A NOTE ABOUT THE DONATION AS WELL.

You can also include the donation gadget into your own web / blog post. Just copy and paste the HTML code.

Donation is as easy as a click on the "Give" button in the PayPal giving widget in this blog post - you can use either PayPal or your credit card.

I am keen to see if it is possible to raise enough funds to cover the costs for development this way.

Thank you, if you send a donation...

Klokan Petr Pridal

P.S.
Steps for Garmin map rendering:

Download and install latest version of MapTiler application. When you start it: In the first step choose the Google Earth tiles rendering, then open your geodata file and continue till the Tile Details page. Choose from the tile formats one of the Garmin Custom KMZ maps. Probably best option is the 512 version. By default it renders the maximal quality where the map has less then 100 tiles (limit of the Garmin devices). If you are fine with lower image quality just choose smaller number for the maximal zoom level. In most cases the default should be fine. Then continue in the GUI to the final rendering step.
The result is the map.kmz file, which can be directly used in your Garmin GPS.

If you would like to use the command line for rendering instead of the GUI then it is done by calling:

python gdal2tiles.py -p garmin map.tif

You need the latest development version of GDAL to do that - and the new version of gdal2tiles which I will commit to SVN during the FOSS4G Code Sprint on this Saturday here in Sydney.

It would be cool to buy the Garmin Oregon 400t (1099 CHF ~ 1065 USD) to test and be able to really use this MapTiler functionality, once I am back in Switzerland. ;-)

I welcome patches improving the user interface or the functionality.

Tiles à la Google Maps: Coordinates, Tile Bounds and Projection...

2008-11-27T08:41:00.000+01:00

I prepared an online tool I was missing when I started to study how the tiling using Spherical Mercator runs in Google Maps, Virtual Earth, Yahoo Maps, OpenStreeMap and others... This tool can help you to understand how are the tiles referenced and how they are stitched together in the browser.

I think this mashup could be an excellent starting point for people who are interested in this subject, because it contains all the math what you need for overlaying your own geodata from external sources: it can be tiles pre-generated by Maptiler/GDAL2Tiles following the OSGeo TMS recommendations, it can be raster data from WMS servers probably cached by TileCache, or tiles from MSR MapCruncher etc.

The tool is excellent for fast debugging of tiling for particular area, because you can use the search for display the tiles of a given place.

A transparent tile with correct coordinates is displayed for every original map tile by calling Google Chart API service.
Whereever you click the longitude/latitude boundaries of the underlaying tile are displayed in WGS84 datum. Boundaries are also displayed in projected Mercator coordinates (EPSG:900913, EPSG:3785) and in pixels for active zoom level. This information is all you need for generating custom map tiles.

The site contains also a Python script doing the coordinates and bounds calculation offline.

So enjoy this mashup:

Tiles à la Google Maps: Coordinates, Tile Bounds and Projection

BTW This demo is part of the documentation of MapTiler application.

PNG palette with variable alpha: small and great for webbrowser

2008-11-05T09:01:00.006+01:00

I am working on the MapTiler/GDAL2Tiles open-source project for user-friendly static tile publishing of maps in the style of Google Maps. This project started as my Google Summer of Code project for OSGEO and results are published in coming GDAL 1.6.

Recently I was testing several optimization of PNG for minimizing the tile size with preserving reasonable image quality as this is crucial for my application. Such optimization speeds up the download and also saves a lot of space on the hard disk. There is plenty of tools available for optimizing the size of PNG: advpng, pngrewrite, pngcrush, pngquant, scolorq, pngnq... I wanted to choose one of them for postprocessing of tiles (or probably later on integrate such optimization directly into GDAL).

The best optimization of the PNG filesize is always based on reduction of the amount of used colors: when you switch from RGB(A) model to palette. What I didn't know before is that PNG supports palette with variable alpha. It means that part of the definition of every color in the palette is also its transparency. It is possible to use functionality usually available only with separate alpha channels but with palette!

After switching to PNG with palette you need just half or even less of the space for your image raster data. Of course conversion from RGBA to palette is lossy operation for images with a lot of colors... but with well chosen algorithm for quantization it can be almost not recognizable for humans. Especially for small tiles like in MapTiler case.

The PNG with the alpha palette is correctly displayed in all latest web-browsers like Firefox 2+, Safari, Explorer 7+ and also in Google Earth.
But wait that is not all! Internet Explorer 5.5 & 6 displays such PNG files with (limited) transparency without HTML hacking! That is just great because it allows you to easily display map overlays in the old browsers by viewers like OpenLayers or Google Maps and still be able to dynamically change opacity for map overlay containing transparent parts!

Well, if it is so great, how to produce such file? There is very nice open-source application for converting standard RGBA PNGs into this PNG8 with alpha palette:

PNGNQ: http://pngnq.sourceforge.net/ (by Stuart Coyle and Greg Roelofs)

It is using The NeuQuant Neural-Net image quantization algorithm (by Anthony Dekker) which was documented in this PDF article. With slightly modified version of this algorithm it converts standard PNG24 with RGBA into PNG8 with alpha palette. It is quite fast.
The result PNG file has usually half of the size of the original.
If you are crazy about the produced png size you can try to compress the file with pngcruch postprocessing step. Sometimes it decrese the size of the result even a bit more.

The documentation at the PNGNQ project page recommends to run:

pngnq -n 256 image.png && pngcrush image-nq8.png smallimage.png

This PNGNQ tool is excellent for post-processing of PNG tiles generated from MapTiler/GDAL2Tiles...

And it seems that the source code of the utility itself as well as of dependencies like NeuQuant algorithm is compatible with BSD license so they have a chance to find a way into GDAL or other open-source tools which are processing maps...