With my new (old) project, I am once again revisiting AIS. AIS, short for Automatic Identification System, is an automatic tracking system that uses transponders on ships and is used by vessel traffic services (VTS). Since I reside close to the port of the city of Hamburg, I came up with the idea of recording and processing this data. This is interesting real-time data, especially in a well-frequented port like Hamburg. It is also a motivation to explore technologies like TypeScript and MongoDB in more detail.
It is well known how AIS information, which is transmitted alternately over two channels in the VHF radio range, is processed. A USB-TV stick with a chipset that is suitable for use as an SDR1 receiver is usually sufficient. Integrated Solutions which solve the SDR aspect of receiving AIS signals as well as decoding received signals to into AIVDM/AIVDO Protocol data stream without much configuration effort can easily found.2,3 In practice it has proven useful to use a Raspberry Pi as the base system. A Raspberry Pi enables AIS data to be received, stored and dispatched continuously without having to keep the desktop computer running or the laptop becoming stationary. Some providers even offer solutions for this. Professional AIS service providers rely partly on crowdsourcing to build up their database and, in return, offer access to their API where the users can complete their own database.
A new (old) Project
When I first tried to develop a system for processing and displaying AIS data, I used a Redis in-memory data store. Redis is an in-memory key/value database and also offers a simple message broker. With multiple receivers and adding data from an AIS provider, my first solution will soon reach its limit. In order to detect duplicates of AIS messages and check their plausibility, a database is needed that offers better support when searching data with the possibility of processing geographic information. Conveniently, this database also offers a publish/subscribe system to send new data to a frontend in real time. MongoDB offers this functionality and was chosen in the new development of AIS Tools. To take a look at a new technology as a JavaScript developer, TypeScript was also chosen. It promises a library that will work in both the backend and the frontend. AIS Tools prove that this is possible.
AIS Tools - One library for the Backend, Frontend and Dispatcher
The AIS Tools (Github) are basically a model system for processing and storing AIVDM/AIVDO reports which are distributed via AIS. The following tasks are involved:
Decode AIVDM/AIVDO Protocol Reports
Once a AIS Reports is received via VHF, processed and published on an UDP data stream, we get a string like this: !AIVDM,1,1,,A,139gmv0P00Pea<lN`pOf4?wb20Rj,0*56
In this case, we got a position report as an AIVDM package where the information is encoded as bitfields in a sentence. There is a comprehensive documentation available on gitlab where all aspects of the AIVDM/AIVDO Protocol is described.4
All types of AIVDM/AIVDO Reports contain information about the number of fragments and the number of the fragments in a set. Some reports transport data in multiple sentences. Reports are transmitted over two channels A and B which is also a default parameter in this string, followed by the payload of this report, and a checksum.
The AIS Tools decode these messages for Position Reports (Types 1,2,3) as well as Static and Voyage Related Data (Type 5) All other Data received is stored as default data for later processing. For the purpose of displaying positions and the shape of a vessel on a Map, the two reports are sufficient.
A Position Report contains information related about the current status of a vessel. Except for the obligatory Gauss Kruger coordinate, current speed, course, heading, if moored or under way, accuracy of the given location etc. as well as the UTC second, when the report is generated.5
First of all, Static and Voyage Related Data include the name of the vessel. Furthermore the dimensions of the vessel as well as the position of the AIS Transceiver can be found is this report. On top of this, the type of the vessel and destination and estimated arrival time, among others.6
Store AIS reports and Publish/Subscribe with MongoDB
If multiple receivers store data in parallel as well as delayed, duplicate reports will occur inevitably. Moreover, a vessel will send its position within an interval even if it is not changed (for example if it is moored or as a ferry wait for passengers on the pier). It is advantageous to filter and aggregate these positions.
MongoDB offers a mechanism where geo information can be processed and queried. This helps to reduce the messages sent to a frontend and reduces data stored in the database. Geo information data is basically stored, indexed and queried as GeoJSON.7
As soon as a new report has been identified, it should be published so that clients can be informed about it in real time. For this purpose, the report is written in a special collection, which is configured as a capped collection. Capped collections are fixed-size collections that support high-throughput operations that insert and retrieve documents based on insertion order.8
By default, MongoDB will automatically close a cursor when the client has exhausted all results in the cursor. However, for capped collections a Tailable Cursor remains open after the client exhausts the results in the initial cursor.9 A collection with a CursorFlag tailable added returns a EventEmitter instead of a collection of data. Once a document is written in the collection, a data event is fired and a subscriber gets informed in real time about new positions and vessel information arriving or moving in an observed area.
Convert ship data and positions to GeoJSON objects
One aspect addressed in AIS Tools is to convert Position and Static Data Reports into GeoJSON objects. These operations help to display the shape of a vessel to its latest position on a Map and display the track of past positions of its voyage. Doing this processing in the model system offered from AIS Tools reduces the complexity of a possible front end and can be reused for further processing and analysing in a backend scenario.
An impression of COSCO HIMALAYAS, a Cargo Ship and three Tugs in Hamburg Harbour.
Screenshot taken from AIS PLTR, a frontend which uses AIS Tools.
Ready-made services
The provision of a model system and the structured storage of AIVDM/AIVDO Protocol Reports in a database was the task of AIS Tools and is therefore fulfilled. The task of collecting reports from different sources and passing them on to a frontend via a backend is the output of further projects. These projects will be presented in future articles.
AIS-UDP
AIS data received via SDR is commonly delivered via a UDP data stream containing AIVDM/AIVDO Protocol Reports. A project that can be configured for your own needs can be found in a separate Github Repository AIS-UDP. A seperate article goes into details of this project10.
AIS-HTTP
If AIS Reports are transmitted to AIShub, AIShub offers access to their Database. The focus of this project is not to download the entire database, but to enrich the data for a geographical area within the reach of the own Receivers.
AIShub allows polling of the service for no more than once a minute. The developer currently operates three Raspberry Pis receivers at different locations and transfers the data to AIShub. For this reason, polling reports from the AIShub API can be carried out every 20 seconds. The timing for this operation is also part of this project, which can also be found at Github AIS-HTTP. Details of this project can be found in a seperate article11.
Using AIS Tools in a Backend/Frontend scenario
Finally, the data collected should be displayed in a front end.12 A backend is also required to provide the front end with data via a REST service and a web socket.13 Since the database operations are isolated in a separate class, the AIS Tools model system can be reused in the frontend, provided that a database class prepared for the frontend is used. In this case, it is not read directly from the database, but via a REST service as a proxy, which is provided by the backend. The publish/subscribe function using capped collections and tailable cursor is simulated using a web socket service.
Introducing the backend and frontend in more detail would go beyond the scope of this article and is presented in follow up article14.
Who like to take a glimpse on the visualization of activities taken by vessels in the Hamburg harbor can have a look at AIS PLTR. This is an AIS Chartplotter, implemented using AIS Tools, where movements and voyage history from vessels can be studied in real time.