The Big Data Pipeline
Multi-gigabyte AIS CSV files streamed row-by-row via Python generators, cleaned, chunked, and processed in parallel across CPU workers — without ever loading the full file into memory
📡
Streaming Process
  • Reads header once → O(1) column lookup
  • Yields 7 fields from ~50 columns per row
  • Chains 2 CSV files into one continuous stream
  • Assigns global chunk_id for ordered merging
  • 10 GB file uses same RAM as one chunk
click to explore
raw chunks
dispatched
⚙️
Parallel Workers
  • Pool.imap_unordered — N cores in parallel
  • parser.py cleans inside each worker: Class A only, 9-digit MMSI, valid coords
  • Groups by MMSI, sorts by timestamp
  • Downsamples to 5-min for anomalies A & C
  • Detects A, B, C, D1, D2 per chunk
click to explore
chunk results
out-of-order
🔗
Merge Process
  • Buffers results, reassembles by chunk_id
  • Compares boundary: last of chunk N vs first of N+1
  • Catches anomalies spanning chunk boundaries
  • A & C use sampled records; D uses full-res
  • D1 and D2 tracked separately
click to explore
global events
all vessels
🏆
Scoring & Output
  • Aggregates all events per vessel
  • Computes DFSI score for every MMSI
  • Ranks vessels by suspicion descending
  • Writes dfsi_results.csv + map CSVs
  • MemoryMonitor logs RAM throughout
click to explore
🚢
Big Data
2
CSV input files
7 / ~50
columns kept per row
5-min
downsample interval A&C
60 kn
teleportation threshold
4h / 1km
going dark threshold
DFSI
final vessel ranking
💻 Hardware Specifications
CPU
Intel Core i7-12700H 14 cores  ·  24 MB Cache  ·  up to 4.70 GHz
RAM
16 GB DDR5 4800 MHz
SSD
1 TB PCIe® 4.0 NVMe™ M.2
Anomaly Detection Engine
Four suspicious behaviours detected in rules.py — every consecutive AIS ping pair checked against all rules simultaneously
🌑
ANOMALY A
Going Dark
Vessel disables AIS transponder while continuing to move — conceals route, port calls, and cargo activity from authorities and tracking systems.
gap > 4 hours distance > 1 km haversine dist
  • Detected by detect_going_dark() in rules.py
  • Stationary vessels excluded — movement required
  • Runs on 5-min downsampled records
  • DFSI contribution: max_gap_hours / 2
444 events
detected
in dataset
🤝
ANOMALY B
Loitering & Transfer
Two vessels linger together at sea at very low speed — indicates covert ship-to-ship transfer of cargo, fuel, or sanctioned goods outside port oversight.
proximity < 0.5 km SOG < 1 kn duration ≥ 2h
  • Detected via detect_loitering_transfers()
  • Spatial grid bucketing for fast proximity search
  • Outside port zones only — port visits excluded
  • Runs post-merge on globally merged sampled records
0 events
detected
not triggered
⚓
ANOMALY C
Draft Change
Ship draught changes significantly during an AIS blackout while at sea — strongly suggests covert loading or unloading of cargo away from port inspection.
gap > 2 hours draught Δ > 5% at sea only
  • Detected via detect_draft_change()
  • Vessel must be outside all port zone polygons
  • Hardest anomaly to explain innocently
  • DFSI contribution: count × 15 — highest weight
0 events
detected
not triggered
👥
ANOMALY D
Teleportation / Cloning
Same MMSI broadcasts from two physically impossible locations — reveals identity cloning where a shadow vessel spoofs a legitimate ship's AIS identity.
speed > 60 kn D1: gap ≤ 30 min D2: 30 min–24h
  • D1 — near-simultaneous: two ships, one MMSI at same time
  • D2 — impossible relocation after a longer AIS gap
  • Rejects (0,0) coordinates and gaps < 30 seconds
  • D2 valid only if both points at sea (land mask filter)
D1 50   D2 0
events detected

Suspicion Scoring

Dark Fleet Suspicion Index

Every vessel receives a DFSI score combining four anomaly signals. Higher = more suspicious. Ranked in dfsi_results.csv
DFSI =
  max_gap_h / 2
  + draft_changes × 15
  + D1_episodes × 20
  + valid_D2_nm / 10
Top 5 DFSI MMSI
MMSI DFSI Score
246782000 140
245223000 140
210399000 80
210388000 80
111265586 73
D1 episodes: consecutive D1 events within 2h merged into one spoofing episode  ·  D2 filter: coarse land mask removes land-crossing jumps  ·  Output: dfsi_results.csv

🌑 Anomaly A — Going Dark

Blue = last known position  ·  Red = reappearance  ·  Orange line = AIS blackout gap

Gap > 4h Distance > 1km

👥 Anomaly D1 — Near-Simultaneous Identity Cloning

Blue = Vessel A (southern cluster)  ·  Orange = Vessel B (northern cluster)  ·  Dashed = same MMSI, same time, two locations

Gap ≤ 30 min Speed > 60 kn

🚀 Anomaly D2 — Impossible Relocation

Cyan = last known position  ·  Orange = reappearance  ·  Purple line = impossible jump (30 min to 24h gap)

30 min – 24h Speed > 60 kn

📈 Memory & Runtime Benchmarks

Heatmap overview, speedup scaling, runtime behaviour, and Amdahl's Law fit across chunk sizes and worker counts

w4 benchmark · 250K & 100K chunks
Peak Memory and Runtime Heatmap
Peak process memory (MB) and runtime (s) across all configurations
Speedup vs Number of Workers
Speedup vs number of workers across chunk sizes
Runtime vs Chunk Size
Runtime vs chunk size for 1–4 workers
Amdahl's Law Analysis 250K
Amdahl's Law fit — chunk size 250K (P = 67.8% parallel)

💻 Worker Memory Profiles

Peak memory scaling, per-worker footprints, and system RSS breakdown at 250K and 100K chunk sizes

w4 benchmark · 250K & 100K chunks
Peak Worker Memory vs Chunk Size
Peak worker memory vs chunk size (1–4 workers)
Per-Worker Memory Distribution w4 c250k
Per-worker memory distribution — 4 workers, 250K chunk
System Memory Timeline w4 c250k
System-wide RSS timeline — 4 workers, 250K chunk
Per-Worker Memory Distribution w4 c100k
Per-worker memory distribution — 4 workers, 100K chunk
What We Found
pipeline design · results · performance
🚨 Detection & Design
Pipeline overview and anomaly results
The vessel anomaly detection pipeline successfully identifies anomalous behavior, as confirmed by the visualizations. For the given datasets, only two anomaly types were detected: A – going dark and D1 – cloning episodes.
This implementation is designed as a single-node, streaming, incrementally merged pipeline that leverages multiprocessing within one machine. It focuses on efficient memory usage and controlled parallelism rather than distributed execution. All intermediate data is processed in memory without writing partial results to disk. This design minimizes I/O overhead and simplifies the pipeline, but also contributes to the observed memory bandwidth bottleneck at higher levels of parallelism.
📈 Performance & Conclusions
Parallelism gains, optimal config, hardware note
The performance results show that our vessel anomaly workflow benefits strongly from parallelism up to two workers, after which performance stops improving because the system becomes limited by memory throughput rather than CPU power.
Memory consumption increases predictably with larger chunks, and workers share the load evenly throughout execution. Since only about two thirds of the workload can run in parallel while the remaining portion must be merged sequentially, the speedup naturally levels off. This means that two workers and a 250 K chunk size represent the practical optimum for the vessel anomaly calculation pipeline.
It is important to note that these performance characteristics are specific to the hardware configuration used in this study, and different systems may exhibit different optimal settings.
1 / 2