Convert timestamps from data in unix and datetime in python

If you have a large volume of data with timestamps and you are interested to convert the timestamps in unix and datetime timestamps in a very fast way, follow the steps of this tutorial.

Requirements

-Anconda Python Development Environment
-Data in a csv format
-Timestamps included in your data

Setup

To have access to spyder it is recommended to download first Anaconda. After the download has been completed, you can run Anaconda and runch Spyder. If the launch button does not show but an Install button shows instead, make sure to install Spyder and then run it. After you open Spyder, you can create a new file and save the file in the same directory where your data is located. Now we are ready to start coding.

Step 1 Data

Let's assume that we had an eye tracker and we collected a large volume of data from users eyes. The data from the eye tracker has timestamps. The timestamps however are those from the eye tracker. For this example our data was collected from Pupil Labs Eye Tracker VR Add on. Pupil labs eye tracker saves data from pupil and gaze of the user. Each data collected from this eye tracker has timestamps. The eye tracker saves also a json file that is automatically generated along with pupil and gaze data for that specific user. The json file is helpful to convert the eye tracker timestamps into unix and datetime timestamps. Have a look at the json file attached below.
Info Json File
The json file has two important variables to make the timestamp conversion which are: "start_time_synced_s": 787247.659352 and "start_time_system_s": 1657719864.2510924 Keep in mind that those variables are different for each recording that we make from the eye tracker. We will use these two variables in our python script to make the timestamps conversion. For that specific user we have also data from the user gaze. Have a look also to our gaze data that we have collected for this specific user from the eye tracker:
Download Gaze Positions CSV File
This data has a column named gaze_timestamp. This is the column that we will use in our python script to make the timestamps conversion.

Step 2 Convert Eye Tracker Timestamps to Unix and Datetime Timestamps

Now we can write a python script that takes the two values from our json file and convert eye tracker timestamps into unix and datetime timestamps. Have a look at the script below. Copy and paste the script into your own computer and make sure the script and the data are saved into the same location in your computer.
                                        
                                        import json
                                        import pandas as pd

                                        pd.options.display.float_format = '{:}'.format
                                        DATAFRAME_HEAD_COUNT = 3
                                        jsonFile = open('info.player.json')
                                        meta_info = json.load(jsonFile)
                                        start_timestamp_unix = meta_info["start_time_system_s"]
                                        start_timestamp_pupil = meta_info["start_time_synced_s"]
                                        start_timestamp_diff = start_timestamp_unix - start_timestamp_pupil

                                        "gaze"
                                        gaze_positions = pd.read_csv('gaze_positions.csv')
                                        gaze_positions.head(DATAFRAME_HEAD_COUNT)
                                        gaze_positions["gaze_timestamp_unix"] = gaze_positions["gaze_timestamp"] + start_timestamp_diff
                                        gaze_positions.head(DATAFRAME_HEAD_COUNT)
                                        gaze_positions["gaze_timestamp_datetime"] = pd.to_datetime(gaze_positions["gaze_timestamp_unix"], unit="s")
                                        gaze_positions.head(DATAFRAME_HEAD_COUNT)
                                        gaze_positions.to_csv("gaze_positions_unix_datetime.csv")
                                        
                                    
After we run the above script, we now have two more columns in our data named gaze_timestamp_unix and gaze_timestamp_datetime. Have a look at the file below.
Download Gaze Positions in CSV With Converted Timestamps
We can now see that our python script successfully converted the eye tracker timestamps into Unix and datetime timestamps.

Conclusions

In this tutorial we used Spyder application in Anaconda and python code to convert the eye tracker timestamps into Unix and Datetime timestamps utilizing a json file generated from the eye tracker. You can utilise the python script to convert timestamps for any data you have and generate unix and datetime timestamps with or without the json file. If you are interested to filter the data and make some calculations using python check out this tutorial: Filter Large Volumes of data in python based on a criteria