Obsolete But Not Gone

This post was born out of a bit of an embarrassing situation, in which I had to obsolete some fields from an AppSource app. Just a quick one to let you know what the official process is, and also that you don’t have to wait for the next major version of BC to fully obsolete a field.

The ‘Situation’

This app has table extensions for the Sales Line table and the posted sales line tables (invoice and credit memo). Not until their first live implementation did we find out that the field numbers in the credit memo line table were not aligned, so at that point BC will yell at you that field types are not compatible. Oops….

For those that don’t understand, let me explain. Field values in the Sales Line table are copied to their posted counterparts by a command called ‘TransferFields’. This command basically copies all the field values from the source table into the target table. Couple things to note: the fields must have the same field numbers, and they must have the same data types. In our case, we had fields 1 and 2 in the Sales Line table, and their corresponding fields in the posted credit memo line had field numbers 2 and 3. The posting process tried to put a number 2 value into field number 1, with the predictable result that there was a data type mismatch.

Why is this a problem? Can’t you just change the field number? Well no, because once an app is published on AppSource, you are not allowed to make any ‘breaking changes’, and renumbering a field is considered a breaking change. Doesn’t matter how I personally feel about that (I don’t agree that field numbers are breaking) and so we did not have any other choice but to use the obsoletion process.

As a side note – yes this exposed a serious issue in our process. Posting a credit memo had clearly not been tested, and matching field numbers is something that a BC developer with my experience should never have a problem with. To my client’s credit: no fingers were pointed, we addressed the issue, and we shared the cost of fixing it.

Obsoleting a Field

Alright, so the official process of obsoleting a field is described in the ‘deprecation guidelines‘. Skip the preprocessing piece if you have not seen that before, and focus on the steps in making code obsolete. There are plenty of blog posts that explain the process itself, so I will not go into any detail.

My main concern was how fast can we get this into AppSource. The process to make a field obsolete has two stages:

  • First, the ObsoleteState of the field is set to ‘Pending’.
    • This means that the field has been marked, but it can still be used in the app
    • The purpose of this stage is to flag the code to any party that has an extension of the code, so that they can take steps to address the change
    • All references to the field will be shown in the VSCode problems window with a warning. There is a reason and a tag property that is used to define which in BC version the change will become permanent
  • Second, the ObsoleteState of the field is set to ‘Removed’.
    • In this state, the field still exists but it cannot be used any longer
    • All references to the field will show up as errors in the problems list

There is a perfectly valid reason why there are two steps. My concern was how long it would take us to get the fields to be removed. The documentation does not address the required AppSource timeline, and I could not find any definite answers in Yammer. The only timeline reference that I could find in any Microsoft documentation was that code must be in ‘Pending’ state for one entire major version. This issue came up in early October, days after 2022 wave 2 was released. If this was true, we would have to wait until 2023 wave 1 (April next year!!) to get the fields obsoleted.

What We Did

Some partners assured me that there is no mandatory wait time for a whole major cycle. The intention of that timeline is not to limit partners but a practice that Microsoft uses. This is to make sure that the partners always have at least one full release cycle to address any compatibility issues due to obsolete code in the base app.

So, with that in mind we went to work. What saved us is that this particular app only had one implementation live, so all we had to do is make sure that they upgrade the app to the ‘Pending’ version as soon as humanly possible. Depending on the number of live implementations you have, this could take longer. I actually don’t even know if there is a dashboard where you can see the live versions that are in use.

I have to say that I was very impressed with how smooth the process of pushing a new version of an app into AppSource is now. After properly testing the change, we pushed the ‘Pending’ app to AppSource, and that was done and ready to publish within a day. The end user then upgraded the app, and we pushed the ‘Removed’ app to AppSource right away. We were able to address the issue within days, and the client lived happily ever after.

Repair Companion Tables

One of my clients has been seeing these weird data issues when migrating databases to the cloud. The most important purpose of this post is to point out a handy data tool that is buried deep in the application, read on to learn what I am talking about.

The problems that my client is experiencing mostly have to do with various ways in which they get data from OnPrem to the cloud. One way is to use the cloud migration tools, another is configuration packages. In addition to using different tools, it feels like there may be some issues with the BC platform’s capability to properly manage data in table extensions. I’ll tell you why I think that.

Missing Data

The first indication was a problem where invoices were migrated into the cloud. The migration process seems to have completed, and the posted invoice list shows a list of invoices. The weird part is that when you try to open an invoice, it shows an empty invoice page. Click ‘View Table’ from the page inspector, and you get nothing, a list of zero records even though the posted invoice list shows the records. Go to the admin center to look at the capacity, it tells us there are like 1154 invoices but when you drill down into the number you get another empty list.

This feels very familiar to me, very much like when a lot of companies tried to migrate straight into SQL Server and we would see null field values. As we all know, BC can’t handle null values. Instead of getting an error saying ‘there are null values!! I don’t know what to do!!’ you get weird behavior like empty lists for tables that you know have plenty of records.

Solving The Problem

With the explosion of moving functionality into separate apps, I had a feeling that the problem had something to do with table extensions (which as you know have added fields in what is called ‘Companion Tables’). I had posted the question on Twitter and there were suggestions to uninstall and re-install apps. This worked sometimes but not all the time.

The original Tweet asking for help on this issue

What it feels like to me is that either there are null values in records, or maybe records are missing in companion tables altogether. We don’t have access to Azure SQL so there is no way for me to actually prove that there is an issue with table extensions. SOMETHING is wrong here though, and for the longest time the only way that I thought we could fix it was to uninstall/re-install apps until the issue was fixed. The reason why this works is that each time you install an app, it will update the schema and make sure that data integrity remains intact. In other words, it will make sure that all records in the main tables have corresponding records in all companion tables.

And then I became aware of a VERY handy little tool. This tool is not available when you are working in local containers, but when you are in a cloud tenant, you will have access to it. I’m talking about the “Repair Companion Table Records” process in the “Cloud Migration Management” page.

The tool will go through all tables that have table extensions, and make sure that each record in extended tables has a corresponding record in each companion table. I still can’t prove my theory but I do know that running this process fixed the problem for my client.

Use SystemId

If you’re like me, and you use the ‘Page Inspector’ a lot, you have undoubtedly scrolled all the way down, and noticed this group of fields at the bottom. One of those fields is called ‘SystemId’. Today’s post describes some of the important things that you need to know about this field.

The Basics

Toward the end of the ‘NAV’ era, you may have noticed a field called ‘id’ in a lot of tables. This field was meant to have a single-field unique attribute for each record in the database. The original purpose was to provide a more industry standard way to get to records from a webservice, and it was implemented as an actual field of each table for which such functionality was provided.

In BC, the ‘id’ field has been obsoleted and replaced by a new system field called “$systemId”, an attribute that is assigned at the system level. The logic that assigns the values is not accessible to us, it all happens behind the scenes. You can read more about the field itself in Docs.

Some important things to remember:

  • Each SystemId value is unique, you will never see any SystemId value repeat across any table. You could cheat the system and manually assign a value to the field, but if you let BC assign its value it will always be unique
  • Its value cannot be renamed. Unlike PK fields in the database, the SystemId is not editable

So How Does It Work?

You will not see the SystemId as a field anywhere. Drill down into a table’s design and it is not there as a field. You can see the value when using the Page Inspector, but none of the pages have the field visible to the user. That would be pointless, because the SystemId does not have a functional purpose. It’s just a random GUID value that holds no meaning. However, EVERY table has a SystemId field, and since its value is always unique, you could theoretically use the field as its primary key.

You can leave the field alone altogether and just continue using the regular PK fields, and BC will automatically take care of the SystemId. If you are just using or developing ‘normal’ functionality, you will probably never even be aware of the presence of the SystemId.

One really handy thing though is that this is a single field unique value! You could capture a tablerelationship in a single field regardless of the target table’s primary key! You could link to the Sales Line’s SystemId field with a single field foreign key, and not have to worry about document type or line number. All you need to do is store the SystemId as a GUID field, with a tablerelationship to the target table’s SystemId field.

To retrieve a record using its SystemId, you use the GetBySystemId method instead of the regular Get method that you are used to.

How Is It Used?

The main purpose of the SystemId is to facilitate the API. The BC API endpoints are formatted in such a way that you specify the systemId as the unique value for the record that you want to retrieve. Yes you can still do a filter on the PK fields, but the standard way to get a particular record from an API endpoint is to provide its SystemId field value. When posting a new record, the response will return the new record’s SystemId.

You can look at a bunch of tables that are part of the API and see that a lot of tables have new fields for these SystemId fields. Take for instance the Customer table. You can see a bunch of new fields like “Currency Id” and “Payment Terms Id” with logic in OnValidate to update the ‘normal’ fields that are all still there. The new SystemId fields do not replace the existing related fields, they live side by side.

Still Confused

Yes it is still confusing, well maybe ‘cumbersome’ is a better word. The mechanism is fairly straightforward , but it has not made our job easier. Not every tablerelationship has been updated with the SystemId, and where you do see those ‘new’ ones, there is a double field relationship that has to be maintained with validation code. Especially when you need to expand standard APIs, you will have to create those additional SystemId fields on top of existing foreign key fields to ensure data integrity.

AppSource Test Drive

Everybody knows about AppSource by now. Everybody is also struggling how to make AppSource work for them, and especially how to provide customers and prospects a trial of their functionality. You could create a sandbox environment and try things out in there, but that doesn’t have test data that is specific for your product. You could install the product into a production tenant, but then you have an app in there that you might not want after all.

One of the lesser known features of AppSource is the Test Drive. This feature provides an ISV partner a completely isolated trial experience  of their product, in an environment that is completely in their control. What’s even better is that the Test Drive can be done in a number of different ways, so you can tailor it exactly to your requirements.

The Test Drive can be a part of a comprehensive marketing strategy, in which you can implement an environment that can showcase even the most complex features of your software, in a way that provides ample opportunity to your customers to learn how to use your product in a non-production environment that is still in the cloud, without having to get a team of consultants onsite.

The way that it works is essentially that the Test Drive is a standalone tenant that has a template company. This template company has your product already installed, and it has proper test data already populated. You can create all the data that you need for your product to run properly. Then, through the SaaSification techniques, you would implement a path into the features of your product, taking the user into your product one step at a time.

If you are interested in providing a Test Drive, please watch this video, in which I go into some more detail about this feature.

To find out more about the test drive, and other information about apps for Business Central, visit http://aka.ms/ReadyToGo

Ready To Go

I’m on my way home from a bucket list kind of trip this past week. This post is more than just a travel log, I’ll get to a good link that you will be able to use to get Ready to Go for Business Central. Bear with me and let me tell the story of my trip 🙂

Through my work I’ve had the opportunity to travel pretty much to the opposite side of the globe to teach the CRS workshop for developing extensions for Business Central with VSCode. The first 2 day workshop was in Hong Kong on Monday and Tuesday, and then Thurday and Friday I hosted the same workshop in Manila. To have some time for sightseeing I arrived in Hong Kong on Saturday early morning. My hotel was on Hong Kong Island, and I spent the day mostly walking around the area near the hotel. It was a very long trip, and I had a looooong night sleep to recover from the journey.

On my flight over, I was a little too vigorous trying to clean the keyboard on my laptop, and it stopped working. As I kind of need a keyboard for the workshop, part of my Sunday sightseeing was to roam the city in search of an external keyboard, which fortunately I found. It’s an unusual souvenir, but I am actually typing this on a keyboard with Chinese characters.

Instead of my original plan to take a ferry across the harbor and find some of the famous food places, I stayed around town on the island. There was an awesome dragon boat race, and I took the trolley up to Victoria Peak, which is where I took the picture at the top of this post. It was fantastic to be in Hong Kong, the people were very hospitable, I really hope I get the chance to go back there some day and spend some serious time exploring the area there.

These workshops are part of a program called ‘Ready to Go’, which was created to help the Microsoft partner channel build the skills that are necessary to succeed in the new ecosystem for Business Central. I’ll post something with more details probably in a few days, and go into a little more detail then, but the link that you want to jot down is http://aka.ms/ReadyToGo. On this page, you will find a ton of good content, and even more links to other places with even more content. This link can be your starting point for any information that you might need to succeed.

The workshops were a success, and I’ve received lots of positive feedback. It was a pleasure visiting these two places, and I hope to go back there one day. These are people that are Ready to Go!

Microsoft 365 Business Central

For about a year now, we have been talking about “Tenerife”. Despite going to great lengths to calm down the partner channel, the name and what it stood for has caused massive misunderstandings and great anxiety. Hopefully that anxiety will come to an end because today the new name has been announced: Microsoft Dynamics 365 Business Central (click here for the announcement and click here for the overview page). A catchy, easy-to-pronounce, 14 syllable name, and it is scheduled to be released on April 2, 2018.

I just wanted to put this out there, and I don’t have a ton of things to say right now, but watch this space for much more stuff to come. This week is Directions Asia in Bangkok, and there will be plenty of information coming out of that event. With the release of the new product there will be some new requirements for partners to get their IP into AppSource, and I will have plenty of things to share about that. Microsoft is working incredibly hard on getting all the information out there, including new material in the learning portal (the link that I had wasn’t working when I wrote this, so I owe you that one) and a ton of new and updated content in the new technical documentation site called ‘docs’.

I am in a very fortunate position to be involved in the very early stages of Business Central, I’ve had the pleasure to be working with the new product for a while now. I have to say I was skeptical of the Web Client and having everything in the cloud, but as I’ve gotten used to how it all works, and how the new ecosystem is forcing to improve our internal processes, I can’t help but be happy about the way that my industry is taking me into a more professional approach to my business. No longer do we get away with flying by the seat of our pants, and do whatever we can get out there in a short term, quick and dirty way. We must adapt and get ALL of our ducks in a row. Our approach to design, architecture, coding, marketing, delivery, automated testing, EVERYTHING must be in top shape in order to be successful in the new age.

This is the time where you have to adapt, or be disrupted. For me personally, it scares the heck out of me, but I also welcome the challenge. I am looking forward to what is to come next, I hope you are too.

Dynamics 365 Primer

Real quick one today, basically a repost of a CustomerSource post today.

The new Dynamics 365 offerings are a bit…. shall we say…. confusing? Ambiguous? The Dynamics 365 page itself seems to have a large number of products on it, they call it all sorts of names, there seem to be a couple of different editions, for different markets. This post is a good attempt at creating some clarity around Dynamics 365.

In a nutshell:

  • Microsoft wants to re-invent business processes. Personally I think that’s BS because the processes themselves won’t really change, it’s just that those processes will be executed by a set of online integrated products. Don’t get me wrong, the technology is moving at lightning speed, and the capabilities are mind-blowing. Microsoft calls it ‘digital transformation’. Now it is REALLY awesome to be able to have a conversation with a prospect, create a quote/order right from your email, have everything posted immediately in your financial system, and see sales figures in a dashboard in real time. The technology to make that happen is now becoming a reality, but the business process itself is really not that different.
  • Clarity about the Dynamics Portfolio. Really important to understand the various products and how they connect and relate. Dynamics 365 Business/Enterprise are based on NAV/AX respectively, GP/SL will technically not have a cloud offering. The consequences for SL/GP partners is that they will have to learn NAV/AX to go into the cloud.
  • Other Microsoft technologies, such as the common data model, PowerBI, Power Apps, Flow, Azure… These will all be offered in a way that allows you to integrate them in a seamless way. You can extract data for your PowerBI dashboard from any other Dynamics offering, and present the relationships in one look. You can use Flow and Power Apps to build your custom workflow, and use data from multiple systems in real time. VERY powerful stuff.

Have a read in this article, there are a ton of links in there for you to follow down the rabbit hole.