In a previous article, I introduced AIS Tools. A library for processing AIS Reports. But how can it be used so collect data from various sources? An explanation of this task is the goal of this article.
As I wrote in my previous article, AIS Tools are a library for processing AIS reports. On the one hand, we receive these reports via a UDP stream, especially if they are to be processed in real time. On the other hand, these reports can also be loaded from an API. In this case, it is historical data.
The AIS tools should help ensure that this data is stored structured in a database, in this case MongoDB, so that it can later be visualized in real time in a front end. The latter is the subject of another article. This article is about collecting data.
Collect reports from a UDP Source
A UDP source, as described, delivers a stream of AIVDM/AIVDO Protocol Messages. This data needs to be decoded and stored in a database. The majority of this work is done by AIS Tools. The remaining task is the setup and configuration of the AIS Tools that can be started on the server as a service.
The above diagram illustrates how the infrastructure is made to receive and dispatch AIS messages with a Raspberry Pi. In the beginning is a AIS broadcast which can be received with a suitable Antenna, tuned to 162Mhz, via a USB SDR Receiver (eg. Nooelec NESDR Nano)
On the side of the Raspberry Pi, rtl_ais is installed as well as AIS Dispatcher. The AIS Dispatcher is an AIS data forwarding utility which allows to stream the UDP Data delivered by rtl_ais to multiple receivers. In particular, the data can be sent to AISHub, which makes this tool available, and in return, enables access to their API. One destination address is a UDP Dispatcher Service, which stores the AIS reports into MongoDB.
AIS Tools do not offer any functionality for collection and proessing UDP data soruces. This task is done with a seperate project AIS-UDP
Installation
The installation is done by clone the project repository and installing the dependencies.1
2
3git clone https://github.com/3epnm/ais-udp
cd ais-udp
npm install
Before the project can be started, the configuration must first be adjusted.
Configuration
The configuration is carried out with node-config-ts. A simple but effective configuration manager for typescript based projects. The easiest way to adapt the configuration is to adapt the file default.json from the config directory of the project folder. In practice, it has proven useful to use a new configuration file depending on the NODE_ENV variable. How this works is explained in the documentation of node-config-ts. Refer to the npm page node-config-ts to learn all features it offers.
The configuration file has the following content, which is explained below.1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24{
"database": {
"url": "mongodb://127.0.0.1:27017",
"options": {
"useNewUrlParser": true,
"useUnifiedTopology": true
},
"dbName": "ais_tracker",
"sender": "***"
},
"dispatcher": {
"udpPort": 10110
},
"logger": {
"level": "info",
"filter": 0,
"filename": ""
},
"ssh": {
"enabled": false,
"forward": "27017:127.0.0.1:27017",
"host": "***"
}
}
Database section
The configuration of the database is as follows:
| Parameter | Description |
|---|---|
| url | The connection url to the MongoDB instance. |
| options |
Options to setup the Database Connection. All options of the native MongoDB Driver are possible. |
| dbName | The name of the database where to store report data. |
| sender |
The name of the sender. Makes it possible to distinguish between those who have created or updated a report in the case of several transmitters. |
AIS Tools initializes the database with the necessary collections and more important creates the necessary indexes. Default documents and position documents expire after one day and position documents are set with a geospatial index. Shipdata documents do not expire and will be updated if a new report is received.
Dispatcher section
The only configuration option is the the port number of the UDP stream with AIVDM/AIVDO Protocol Reports.
The below image illustrates, how detailed a vessels voyage can be visualized in an area, from which a good reception of AIS Signals is possible. In the shown example, every 2s a position report is recorded.
A harbor ferry at the “Landungsbrücken” jetty.
Logger section
Logging is done with winston universal logging library.
| Parameter | Description |
|---|---|
| level |
The logging levels are named after npm logging levels and allows to configure how verbose the messages are written to a logfile or to stdout. |
| filter |
The filter is a MMSI Number - an identifier every AIS Report has and which is unique to the vessel. Used to log the processing of Reports for a specific vessel. |
| filename |
The filename where the log file is written to. If empty, the messages are written to stdout. |
The logger uses the winston-daily-rotate-file to archive log files if used in a debug level for a longer period of time. In addition, the logger can also be controlled via the NODE_ENV environment variable. If set to “debug”, the log messages also written to stdout, regardless of whether the filename parameter is set or not.
SSH section
The following configuration enables an SSH tunnel to be created if required, which is started as a child process.
| Parameter | Description |
|---|---|
| enabled | Whether the function is used or not |
| forward | Which port from the source is forwarded to a port at the destination |
| host | The host of the source |
Start the Service
Once the configuration is done, the service can be started with npm start
Poll reports from a HTTP Source
As mentioned in the previous paragraph, the reports are also sent to AISHub. This gives the user access to an API where AIS reports can be queried once a minute for a specific area.
It is not necessary to install the HTTP service on a Raspberry Pi. Since the reports are sent from Raspberry Pi to AISHub, it is most practical to query the API from the Raspberry Pi.
An additional project was created to load the AIShub reports and save them in the shared database. AIS HTTP. Instalaton and configuration is similar to AIS-UDP.
Installation
The installation is done by clone the project repository and installing the dependencies.1
2
3git clone https://github.com/3epnm/ais-http
cd ais-udp
npm install
Before the project can be started, again the configuration must first be adjusted.
Configuration
The configuration file has the following content, which is explained below.1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32{
"database": {
"url": "mongodb://127.0.0.1:27017",
"options": {
"useNewUrlParser": true,
"useUnifiedTopology": true
},
"dbName": "ais-tracker",
"sender": "***"
},
"dispatcher": {
"second": 0,
"timeout": 61,
"rest": {
"username": "***",
"latmin": 53.36837696691308,
"latmax": 53.60574780621288,
"lonmin": 9.652862548828127,
"lonmax": 10.244407653808596
}
},
"logger": {
"level": "info",
"filter": 0,
"filename": ""
},
"ssh": {
"enabled": false,
"forward": "27017:127.0.0.1:27017",
"host": "***"
}
}
The configuration options for the database, logger and ssh section is the same as done for the AIS-UDP above.
Dispatcher section
In the dispatcher section, the configuration the AIShub service is done:
| Parameter | Description |
|---|---|
| second |
The Second of the Minute, when the API is called. Useful, if more then one UDP stream is offered to AISHub and the API can be called more often than once per minute. |
| timeout |
The timeout between two API calls. In the example above, 61s is configured which is suitable for AIShub |
| rest |
In this subsection, the username for the api request as well as the desired geographic boundery box is configured for which vessels and positions are requested. Refer API Documentation page at AIShub. for more details about this service |
Start the Service
Once the configuration is done, the service can be started with npm start
The author currently has three AIS received Raspberry Pis in different locations and also receives additional AIS reports from AISHub every 20 seconds so that all movements in the port of Hamburg can be recorded.
The following picture shows how the API queries complement areas at least every 20 seconds, which are in the interest of the author but outside the reception range of the Raspberry Pis.
KAJA JOSEPHINE is leaving Hamburg for the North Sea.
Conclusion
The author prefers Supervisor as a Process Control System because it runs well on a Raspberry Pi and his service provider offers Supervisor for installing and registering services on the main server. To control a controlled teardown if a service is stopped, a little afford is taken to setup the process behavior of the program.
Since three Raspberry Pis are used in production which are from a different iteration (2,3,4) the most powerful is chosen as the host which polls the AISHub API for all three accounts. This reduces the load on the other two with less memory and cpu performance.