From Flight Sim to Digital Twin: The Evolution of MSFS

I recently gave a talk about past, present, future of MSFS at Nerd Nite Tokyo, titled “From Flight Sim to Digital Twin: The Evolution of MSFS

There’s a lot of things that you could talk about when projecting the future of what MSFS could be, based on the progress made over 40+ years.

However, a lot of these things are already recognised, roadmapped, and already under development eg. Flight Modelling, AI content gen, weather (aerial only generally) , seasons, SDK, AI traffic, ATC, etc etc

I chose to focus on one aspect that currently doesn’t seem to be on the radar and/or roadmap – or discussed much at all except for occasional lonely threads in the MSFS forums.

That focus aspect was water simulation, which I think is a key element in taking MSFS to the next level out to 2030 and beyond.

Realistic water simulation is critical for lots of reasons, two in particular:

  • Modelling the full behavior of natural systems
  • Provide realistic environments for navigation

Here’s one good use case:
|475px;x285px;
ref: Queensland floods: Burketown residents warned of crocodile-infested waters ahead of expected peak | Queensland | The Guardian

What you can currently do in MSFS:
|464px;x261px;

Ditto for the tides.

I gave two other use cases as examples which are even more critical:

  • Sea Level Rise
  • Simulating the global underwater environment

I’ll post the video link if it gets posted, or I’ll post the slides instead.

It would be good to get more of a conversation going about the need for water simulation in MSFS to get some more focus and visibility.

Location Manager for MSFS – Development Overview

Location Manager for MSFS – The Challenge of Enhancing the UI/UX

Following on from my previous post about Aircraft Manager for MSFS – Development Overview , here is a short overview of how the development of Location Manager panned out. To recap what the problem is: MSFS is like an interactive earth simulator, enabled by flying around complex mini-simulations in the form of aircraft across the whole history of flight. The two methods for currently managing locations in MSFS are show below.

Inbuilt MSFS method 1 for location management

Inbuilt MSFS method 1 for location management

Inbuilt MSFS method 2 for location management

Inbuilt MSFS method 2 for location management

But there’s no way to save and/or favorite locations in the sim itself. While you can save locations outside MSFS in all sorts of ways (OneNote, LittleNavMap, Excel, etc) it’s not the most convenient of methods if you want to build up a log of world exploration discoveries for easy access within the simulator. This isn’t the same as loading a saved flight plan either. There are many use cases where you don’t actually want to use a flight plan, but just go to a location and start flying from there. So the intent here was to derive a solution to solve this problem.

Step 1 – Validate the Technical Approach

In development, it’s more efficient if you can leverage past knowledge and skills so you don’t have to reinvent the wheel every time. Having just completed Aircraft Manager for MSFS, I’d climbed the learning curve on both the MSFS UI framework as well as some Javascript solutions to integrate into that. So it seemed logical to ask, how could I apply that to the unique problem of location management? First off, just as in the previous post, I needed to validate some technical issues before diving into the prototyping stage. This was a very different project from Aircraft Manager, as that relied mainly on the inbuilt aircraft database supplied by the sim. In the case of Location Manager for MSFS, one of the big issues here was we needed to develop a robust way to save locations with associated user data via the inbuilt MSFS UI storage system. There were also unanswered questions as to how robust this storage system is in terms of how much data can be safely stored, on both PC and XBox. I have received some advice from one of the Asobo devs on this (thanks!) but still need a little more clarity, so need to keep an eye on this. In summary, some of the biggest technical issues to be validated were:

  • How to capture and save the location information from the MSFS UI
  • Where would be the best place to insert the UI to manage saved locations
  • How to replicate the map zoom functionality in MSFS for saved locations (this was tricky!)
  • How to efficiently store locations for easy retrieval, display, updates, and deletion

So I got to work, solving each of these unknowns in turn, or at least getting me into the ballpark of “this looks doable in a performant manner so the projects not dead and I can keep going”. Like the previous project, there were a number of dead ends and frustration moments which necessitated some creative thinking in order to spark the idea for another way to solve the problem. Some of these solutions come from gaining a deeper understanding of the MSFS UI framework, some from creatively applying workarounds. A workaround is a little different from a hack btw. A hack is typically a short term fix, designed as a purely quick and dirty temporary measure while you search for a better more reliable long term solution to a problem. A workaround is a solution you have confidence in being robust enough for long term use, even though it’s not how you originally envisioned it working. Workarounds are always preferable to hacks, giving you better maintainability over the long term.

Step 2 – Validate the Prototype

After gaining sufficient confidence I had working solutions to the core technical issues, I could start constructing a full prototype incorporating these into a working demo. This lets you test & tweak the full workflow, and weed out other issues/bugs.

Early Prototype of the Map interface

Early Prototype of the Map interface

Early Prototype of the location management interface

Early Prototype of the location management interface

Location Manager for MSFS Free Demo

The next stage from this is to get a demo into peoples hands to get wider feedback, while continuing to tweak workflows and solve issues/bugs as they appear. Similar to AM, I fired up a thread at the Flight Simulator forums to see if anyone is interested in trying it out. Have at it, all feedback is welcome: The Good, The Bad, and The Ugly. You can download the free demo lite version from the Location Manager for MSFS project page. Like the AM demo, it’s fully functional for casual use cases, with no obligation to upgrade. Enjoy!

Location Manager for MSFS Pro Release

Version 1 of LM Pro

Version 1 of LM Pro

Location Manager Pro V1.0 was released on 2023-02-14. It includes a toolbar widget for accessing LM from fly mode, so you can teleport to saved locations as well as add new one. The release announcement on the MSFS Forums is here @ Aircraft Manager Pro + Location Manager Pro now available.

Start Anywhere in MSFS

It was quickly followed up with another solution to a long standing request from users of MSFS: the ability to start cold and dark from anywhere. You are currently limited to being able to start only from airports or in the air.

With V1.0.3 of Location Manager we introduce Start Anywhere, where you can teleport to saved locations in Fly mode and start on land, water, or air to begin your flight experience.

More details in the video below:

Happy Flying!

Aircraft Manager for MSFS – Development Overview

Enhancing the MSFS UI & UX to solve the Aircraft Management Problem.

MSFS (Microsoft Flight simulator) has been a going concern for 40 years, with some ups and downs along the way, and the latest iteration launched in 2020 is revolutionizing the flight simulation space. It’s a complex simulation for sure, not without issues or growing pains, but it’s undeniable that when it works well it is an amazing experience in either flatscreen or VR. I’ve been using it extensively on and off since the launch, and while I like the UI there’s always been a few niggling issues with the main interface that I’ve been itching to scratch. One of the pain points is with the Aircraft Management aspect via the Aircraft Selection Screen, and Aircraft Manager for MSFS is my solution to that. This blog post is a short story of how that came about as a new addon for MSFS called Aircraft Manager.

MSFS Aircraft Selection

MSFS Aircraft Selection Screen

A couple of times in the past I’d taken a look through the User Interface code along with exploring the official SDK, but as there was no documentation on it I steered clear of doing anything with it because it looked too complex. Diving into a complex codebase with no documentation to guide you can be a bit of a scary prospect, and you need to be prepared to persist with it in order to get that series of lightbulb moments where you being to understand it.

Anyway, towards the end of 2022 I was casting around for new side project ideas that could solve a real problem and make an impact. Two areas in MSFS stood out for me as candidates to develop something for:

  • Aircraft Selection and management, made more difficult with the growing number of aircraft now available both paid and free.
  • Managing locations inside the sim. MSFS is like an interactive earth simulator, enabled by flying around complex mini-simulations in the form of aircraft across the whole history of flight. But there’s no way to save and/or favorite locations in the sim itself.

So I decided to look at enhancing the Aircraft Management problem first. More about Location Management in a follow-up post.

The Problem To Solve

Some of the issues with it are:

  • A growing aircraft collection means too many planes, and it’s easy to get overwhelmed deciding what to fly using the current UI
  • Frustration you can’t tag your aircraft the way you like
  • Can’t sort or filter them in meaningful ways to use them in various contexts eg: racing, stol, vfr etc
  • Can’t take easily accessible short notes ingame as reminders before flying
  • [insert your itch here]

First Steps to Developing a Solution to Aircraft Management for MSFS

Getting to grips with the MSFS UI (User Interface) was the first big rock to move, as we say in project management. You can’t cut into a codebase unless you know what you’re doing.

Unfortunately, there is no documenation on the UI at all (some is finally coming in SU12 release, though) so it meant doing it the hard core way: reading and debugging, piecing it together line by line.

Fortunately, this is one of my core skills as a developer. I’m a regular bug magnet, so my debugging and analysis skills are fairly well developed. That said, I knew it wouldn’t be easy, which is why I’d avoided it so far!

So I blocked some time out and I went for that long overdue deep dive into the MSFS UI codebase. Three days later, after some hair pulling, screaming, wtf’s, and aha’s, I surfaced with *some* idea of how it was put together. Shoutout to both Microsoft and Asobo for providing the source code in the first place, and also to the dev(s) that architected and implemented it. Impressive framework effort, and not an easy project to develop even leveraging a UI library.

My pathway to developing this was through iterating rapid prototypes to learn more about how the MSFS UI framework worked, while testing out ideas for how I might enhance it, before launching into the actual development solution. This next phase was even more educational, as I ran across the “gotcha’s” inherent in an undocumented system built on top of  Game UI library Coherent GT.

Not having the ability to implement anything at the Coherent GT level meant understanding fully what might be possible to do, or not, and finding the best solution. I had a short list of technology solutions I wanted to use, based on my previous experience, and prioritised them into a risk reduction approach to prove various technical issues could be solved in order to develop a working enhancement solution.

I worked my way through various ideas I had, and was getting a little dejected when they all ended up on the cutting room floor. This project involved lots of creative pauses, while I had to go for a walk and let it stew for a while in order to spark an idea to find another solution to try. Eventually, I hit paydirt with a robust solution I could build upon that seemed reliable enough to surgically insert into the MSFS UI system and enhance it without disrupting regular use, a pretty critical requirement!

Developing the Prototype

Prototyping a more complete solution was the next stage, where you take the bits you now know work reliably and start to craft a proof of concept that covers the core use cases. The goal of the product development wasn’t to develop an overly complex solution, but a lean mean aircraft management machine, that solves a couple of core use cases very well, as opposed to being a swiss army knife solution.

Once I had a working prototype that was functional enough to prove the concept I reached out to other MSFS users at the MSFS forums to see if anyone was interested to test it and give me some feedback.

Early Prototype of Aircraft Manager for MSFS

Early Prototype of Aircraft Manager for MSFS

Shoutout to the following users for taking a chance on testing and providing feedback for Aircraft Manager for MSFS:

It’s always vital to touch base with other users, as even though I’m scratching my own real itch with the UI and solution other users have different itches, and sometimes they overlap in interesting ways. Stress testing for performance is also critical, as is squashing bugs which are easy to miss even with a QA checklist.

The end result (but not quite the final one, it’s changed already!) you can see in the video walkthrough below:

Update: Does it work to solve real users’ problems?

Very well! Possibly the most useful add-on ever for MSFS, in this great review from one of the top MSFS streamers.

AM Free Demo

You can download the free demo lite version from the Aircraft Manager for MSFS project page. It’s fully functional for casual use cases, with no obligation to upgrade. Enjoy!

AM Pro Release

Version 1 of Aircraft Manager Pro for MSFS

Version 1 of Aircraft Manager Pro for MSFS

I’m currently in the final stages of release for the pro version of Aircraft Manager for MSFS. Making a product is only part of the challenge, there’s a lot of other elements that go into a successful rollout, open source, free, or paid.
This stage always reminds me of the 90/90 rule:

The first 90 percent of the code accounts for the first 90 percent of the development time. The remaining 10 percent of the code accounts for the other 90 percent of the development time.[1][2]

Tom CargillBell Labs

It’s not just code. Website, payments, fulfillment, support, maintenance and the rest all need to be done. That’s a lot, and I found I had to refresh a few of these areas as well, which hung up the deployment of the pro version. The release announcement on the MSFS Forums is here @ Aircraft Manager Pro + Location Manager Pro now available.

Auto Control Preset Switching

The ability to automatically switch device control presets on a per aircraft basis has been a long standing request from the users in MSFS.

Item #17 at 749 votes in the most recent wishlist is “Aircraft Specfic Control Profiles”.

Well, now we have it! With V1.0.2 of Aircraft Manager you are able to save per aircraft control presets and have them switch automatically as a bank of device preset changes whenever you switch aircraft. No more manual switching needed. See the video below for more details.

The Story Continues: Location Manager for MSFS

As I mentioned at the start, the other problem I came up with to solve was:

  • Managing locations inside the sim. MSFS is like an interactive earth simulator, enabled by flying around complex mini-simulations in the form of aircraft across the whole history of flight.

After getting hung up on some of the product deployment issues, I decided to take a look at this problem while waiting on people to get back to me.

Location Manager for MSFS free version is now available from the project page,

I’ll be writing more about this shortly in a follow up post. Stay tuned, and thanks for reading if you manged to stick it out this far!

Update: Part 2 is now available at Location Manager for MSFS – Development Overview.

Immersive Audio and Musical AI

The sound and music are 50% of the entertainment in a movie. George Lucas

As it is in games, I would also say the same for emerging mediums such as VR, AR, and MX.

I’ve been working recently on a couple of projects related to a few different areas, such as:

As part of this, last week I presented an introductory tutorial at the VRHubTokyo Meetup on Spatial Audio in Unity3D.

This is a really interesting area, with a lot of challenges still to be overcome.

This panel video of immersive audio experts is an excellent introduction to a lot of the issues:

We also had another Canvas Tokyo Musicians Meetup where we discussed a few of these topics, among others.

You can read the event report here: Musicians Meetup #2 Event Report – Mon 11th Sept @ Tacchi Studios

Check out the Spatial Music Visualizer which I also used to discuss a few of the issues around spatial music.

Stay tuned for more news soon!

Chatbot development with Azure Bot Service

Chatbot development and deployment

There’s a lot of hype at the moment about AI in general, and chatbot’s being the next big “app market”. Reality is the tech is still very immature and there are not a lot of use cases where the chatbot workflow is a better UX than simple point and click with a well defined menu and information architecture.

FAQ’s are an important corner of the customer support sector and is one area where they may have use so I decided to test out Microsoft’s bot service, which is still in preview mode: https://azure.microsoft.com/en-us/services/bot-service/

I created an FAQ chatbot for one of my own products as a practical test, which you can see running live here: http://www.harpninja.com/wp/faqhelp/
As you can see from the image there’s quite a bit to it, architecturally and technology wise, with a number of different services from NLU (Natural Language Understanding) to the bot development to connecting various user touchpoint services. There are a number of different types of bots you can implement and an FAQ bot is best implemented using another service they have called https://qnamaker.ai

Overall it’s a pretty nice developer workflow considering the different services they have hooked up to make the development and deployment a relatively painless experience. This FAQ chatbot is using their inbuilt web chat preview which made it a snap to integrate into WordPress without having to code up a plugin. The bot has also been deployed to Skype, Slack, and Facebook Messenger.

There’s a lot of different options for developing chatbots, which this article gives a great overview of: https://www.oreilly.com/ideas/bots-what-you-need-to-know , but I would definitely recommend Microsoft (the new Apple!).

Cognitive Technology – a Quickstart Guide Infographic

“I’m sorry Dave, I’m afraid I can’t do that” HAL 9000

Cognitive Technology, Smart Apps, Data Science, Artificial Intelligence…the list of new technology buzz goes on and on.

But what do they actually mean?

Cognitive Technology Infographic

Here’s an infographic to help explain what Cognitive Technology is and how it can be applied for business use.

The use cases are many and varied, depending on your business needs.

  • Expert systems
  • Autonomous vehicle control on Land. Sea, and Air
  • Robotics
  • Sales and inventory forecasting
  • Price modeling
  • Image, video, and handwriting recognition
  • Text and document analysis for classification, concept, and sentiment extraction
  • Customer segmentation and market analysis
  • Social network analysis for brand and product affinity
  • Anomaly detection for fraud and other behavior patterns

There are also many vendors in this space ranging from the big 4

  • IBM
  • Microsoft
  • Google
  • Amazon

to many smaller players focused on providing different levels of service.

Enjoy!
Image Credit: Evolution by Jakob Vogel from the Noun Project

Cognitive Technology Quickstart Infographic

Cognitive Technology Quickstart Infographic

Pin It on Pinterest