TL; DR

I created a timeline visual from Elite Dangerous in-game events. Why? On the one hand, to further my Python learning and explore AWS DynamoDB. On the other hand, because I ultimately would like to provide some context to me in the game which is not present natively. By focusing on visualising a subset of the available events, I gain a foothold in parsing the game data into a database and putting an API in front of it.

The project page provides an overview while this post covers this part of the story.

The outcome

Here is what the current application looks like:

And an overview video:

Getting to here

Backend - DynamoDB

I elected to use DynamoDB Local (Downloadable Version) as this was my first NoSQL project. The consensus at the time of research was to go with a single table design for almost anything. The current best practice page maintains this:

For DynamoDB, by contrast, you shouldn’t start designing your schema until you know the questions it will need to answer. Understanding the business problems and the application use cases upfront is essential.

Also:

You should maintain as few tables as possible in a DynamoDB application. Most well-designed applications require only one table.

Access patterns

The following questions were the intent of the application to answer:

• Who has interdicted me?
• Who has killed me?

Source data

Both of these questions conveniently match to natively logged data in the game journal files.

Interdiction event:

{ “timestamp”:“2017-11-05T18:18:16Z”, “event”:“Interdicted”, “Submitted”:true, “Interdictor”:“Harald Irongut”, “IsPlayer”:false, “Faction”:“The Ant Hill Mob” }

Killed event (solo and by a group of players):

{ “timestamp”:“2018-04-23T07:30:27Z”, “event”:“Died”, “KillerName”:“Cmdr Flight Assist”, “KillerShip”:“diamondback”, “KillerRank”:“Dangerous” }
{ “timestamp”:“2018-06-30T14:19:02Z”, “event”:“Died”, “Killers”:[ { “Name”:“Cmdr deZpe [ROA]”, “Ship”:“ferdelance”, “Rank”:“Elite” }, { “Name”:“Cmdr 1st.BoneS [RoA]”, “Ship”:“ferdelance”, “Rank”:“Deadly” } ] }

I created a load script to parse through all my ED log files (998, ~300MB) covering November 2017 through to June 2019.

Table and schema

I created a single table with the following schema:

KeySchema=[
        {
            'AttributeName': 'event',
            'KeyType': 'HASH'  #Partition key
        },
        {
            'AttributeName': 'occurred',
            'KeyType': 'RANGE'  #Sort key
        }
    ],
    AttributeDefinitions=[
        {
            'AttributeName': 'event',
            'AttributeType': 'S'
        },
        {
            'AttributeName': 'occurred',
            'AttributeType': 'S'
        },

    ]

Middleware - REST API endpoints

Python was the language of choice for this project, and research led me into Flask vs Django territory for the production of python based REST APIs.

Flask over Django (or vice versa) appeared to be a subject of much discussion online. As someone new to this area, I picked Flask on the back of documentation that I thought was easier to digest.

It seemed natural to map REST endpoints to the expected data questions. Therefore, I created the following API resource paths:

• /interdictions/
• /deaths/

In both cases, I support a parameter isPlayer which influences the resulting table query.

The boto3 library interacts with the DynamoDB backend with the constructed query. The endpoint then parses the results and produces a JSON response for the frontend to manage.

jsonItem = {
                'id' : number,
                'start' : occurred,
                'content' : interdictor,
                'isPlayer' : isPlayer,
                'group' : '0'
            }

Frontend - Single web page with vis.js

A JavaScript call to the above endpoints creates a JSON vis.js dataset. The timeline module provides a visual. Three groups match the expected result:

• Interdiction by
• Killed by solo
• Killed by wing members

I have created a working prototype of the Timeline module with my data. You can see a variety of examples on the vis.js website that illustrate the range of possible functionality.

Conclusion

Getting to this point felt like a milestone moment. There’s lots more I could do - ultimately migrating to a serverless AWS application would be the end goal.

Visually representing the data is the first step to achieving context. Ideally, I’d like to drive the query with in-game interactions for near-time information (you’ve met this player before, prior combat results).

Acknowledgements

• Trek10.com’s post on From relational DB to single DynamoDB table: a step-by-step exploration.