Flights during a storm
This vizualization took way more time than I expected. The idea came up on 29th June, when I was sitting in Microsoft’s office right next to the airport in Warsaw. The storm during that afternoon was huge. I immediately checked flightradar24 to see, what the airtraffic looks like. I wondered what it would look like if plotted with the clouds and maps. Fastforward 3(!) months and here it is:
You will find the detailed explaination below.
1. Flight data
I have conducted quite an extensive research and decided that Flightradar’s data is the most reliable, so I took out my credit card and started the trial. From the website I picked the flights and downloaded csv with columns: Timestamp, UTC, Callsign, Position, Altitude ,Speed, Direction. All the files that I used can be found here.
The main issue with the data was the fact that the time between observations(rows) was not constant. To give you an example: during the take off or landing you would have around 10 observations per minute, while on the cruising attitude 1, maybe 2 per minute. I had to interpolate the “missing” observations’ coords basing on the condition that the change between two observations was linear.
The code that I used to prepare the data was following:
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 |
import re from numpy import genfromtxt import numpy as np import os.path from distribution import uniform_distribution #the function I have written #Get list of all the CSV path = '.' r = re.compile(".*csv") csv_list = list(filter(r.match,os.listdir(path))) csv_count = len(csv_list) def data_extract(file1): #Rename files and write them to a separate folder file2 = file1.replace('Flight_','')[:5] + '.csv' fin = open(file1) data = fin.read().splitlines(True) fout = open('source files/'+ file2,"w+") for line in data[1:]: line = line.replace('"','') fout.write(line) fin.close() fout.close() #Create arrays from the data and filter it by time data = genfromtxt('source files/'+ file2, delimiter=",", dtype = None, names="timestamp, utc, callsigb, pos1, pos2, att, speed, direction") data = data[::-1] data = data[data['timestamp']>1498731036] #Interpolate the data coords = uniform_distribution (data) return coords |
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 |
#Interpolate the coords from numpy import genfromtxt import numpy as np import math import bisect def uniform_distribution (data): #Get min and max time from the data max_t = max(data['timestamp']) min_t = min(data['timestamp']) #Divide it by 12 seconds, because this is my aim diff = (max_t-min_t)//12 rounded = math.ceil(diff) #Create new array with time distributed by 12 seconds new_data = np.arange(min_t, max_t, 12) pos1 = np.empty(len(new_data)) pos2 = np.empty(len(new_data)) for i,timestamp in enumerate(new_data): #Find next timestamp bigger than current index = bisect.bisect(data['timestamp'], timestamp) #Smaller smaller = data['timestamp'][index-1] smaller_pos1 = data['pos1'][index-1] smaller_pos2 = data['pos2'][index-1] #Bigger bigger = data['timestamp'][index] bigger_pos1 = data['pos1'][index] bigger_pos2 = data['pos2'][index] #Linear interpolation pos1[i]=smaller_pos1 + ((bigger_pos1-smaller_pos1)/(bigger - smaller))*(timestamp-smaller) pos2[i]=smaller_pos2 + ((bigger_pos2-smaller_pos2)/(bigger - smaller))*(timestamp-smaller) #Merge columns and return them new_coords = np.column_stack((pos1, pos2)) return new_coords |
2. Map and clouds
I received the cloud data from radar-opadow and its team got it from IMGW – a polish institution. The files are available here. These picutres need to be reprojected using proj4. The process is explained here. This was the slowest part and I ended up taking screenshots of the htmls I created (not proud of that). That left me with dozen pictures, which would be later used as the backgrounds of the plot. The example html with clouds overlay can be found here (click save as).
3. Plotting
At the beginning I was aiming to create an animated plot and write it into a gif, but I could not find a way to change the background with the animation. As a solution to the problem, I have decided to create a loop that will produce frames. The code is below:
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 |
import re from numpy import genfromtxt import numpy as np import os.path from distribution import uniform_distribution #Find all the backgrounds path1 = '.' r1 = re.compile(".*png") png_list = list(filter(r1.match,os.listdir(path1))) png_count = len(png_list) path = '.' r = re.compile(".*csv") csv_list = list(filter(r.match,os.listdir(path))) csv_count = len(csv_list) fig = plt.figure(figsize = (50,80)) lines = [plt.plot([], [], 'r-', markersize=10,linewidth=5)[0] for _ in range(csv_count)] extracted_data = [] for i in range(csv_count): extracted_data.append(data_extract(csv_list[i])) if __name__ == '__main__': for line in lines: line.set_data([], []) #Loop all the frames for i in range(0,475): for index, line in enumerate(lines): line.set_data(((extracted_data[index][1+i:10+i,1]-13)*(600/13)-45)*(1326/531), (392-(extracted_data[index][1+i:10+i,0]-49)*(430/6)+40)*(983/392)) l = i//50 img = plt.imread(png_list[l]) plt.imshow(img, zorder = 1,interpolation='nearest') #extent=[14, 27, 49, 55]) plt.axis('off') plt.savefig('final_gif/' + str(i) + "_ready.jpg", bbox_inches='tight', dpi=40) print('Frame {}'.format(i)) |
This code produced 476 frames that later were made into a mp4 video. Initially I wanted to use python to create it, but my experiments left me with files of 500+ MB. This is why I decided to hand those over to my friend, who is a Photoshop pro. He managed to create a much lighter mp4 file.
What do you think? I will be happy to get any kind of feedback! Those are my baby steps in Python, so probably everything could be done better 🙂
Thanks!
