Month: June 2022

Repair Companion Tables

One of my clients has been seeing these weird data issues when migrating databases to the cloud. The most important purpose of this post is to point out a handy data tool that is buried deep in the application, read on to learn what I am talking about.

The problems that my client is experiencing mostly have to do with various ways in which they get data from OnPrem to the cloud. One way is to use the cloud migration tools, another is configuration packages. In addition to using different tools, it feels like there may be some issues with the BC platform’s capability to properly manage data in table extensions. I’ll tell you why I think that.

Missing Data

The first indication was a problem where invoices were migrated into the cloud. The migration process seems to have completed, and the posted invoice list shows a list of invoices. The weird part is that when you try to open an invoice, it shows an empty invoice page. Click ‘View Table’ from the page inspector, and you get nothing, a list of zero records even though the posted invoice list shows the records. Go to the admin center to look at the capacity, it tells us there are like 1154 invoices but when you drill down into the number you get another empty list.

This feels very familiar to me, very much like when a lot of companies tried to migrate straight into SQL Server and we would see null field values. As we all know, BC can’t handle null values. Instead of getting an error saying ‘there are null values!! I don’t know what to do!!’ you get weird behavior like empty lists for tables that you know have plenty of records.

Solving The Problem

With the explosion of moving functionality into separate apps, I had a feeling that the problem had something to do with table extensions (which as you know have added fields in what is called ‘Companion Tables’). I had posted the question on Twitter and there were suggestions to uninstall and re-install apps. This worked sometimes but not all the time.

The original Tweet asking for help on this issue

What it feels like to me is that either there are null values in records, or maybe records are missing in companion tables altogether. We don’t have access to Azure SQL so there is no way for me to actually prove that there is an issue with table extensions. SOMETHING is wrong here though, and for the longest time the only way that I thought we could fix it was to uninstall/re-install apps until the issue was fixed. The reason why this works is that each time you install an app, it will update the schema and make sure that data integrity remains intact. In other words, it will make sure that all records in the main tables have corresponding records in all companion tables.

And then I became aware of a VERY handy little tool. This tool is not available when you are working in local containers, but when you are in a cloud tenant, you will have access to it. I’m talking about the “Repair Companion Table Records” process in the “Cloud Migration Management” page.

The tool will go through all tables that have table extensions, and make sure that each record in extended tables has a corresponding record in each companion table. I still can’t prove my theory but I do know that running this process fixed the problem for my client.

Remove BC Bloatware

The number of apps that come with a standard container has exploded over the past handful of releases. My local setup involves doing development in Hyper-V virtual machines, so local system resources are at a premium. It is very easy to trim the fat so to speak, so read on if you want to know how.

What’s with these apps?

From foreign languages to IRS reports. From integration and migration tools to email functionality and even connectors to external systems like the Spotify app. Having these in separate apps is great, because it is very easy to get rid of them.

The downside is that each app comes with its own set of table extensions, which are implemented as companion tables. Each time that a database action is taken against the main table, the system also has to maintain each companion table. Multiply this issue with the number of companies in your system and you can imagine the performance hit this could take.

Funny as it seems if you know what my desk looks like, but I HATE clutter in the extension list, especially if it is functionality that I just don’t ever use. I will NEVER have a need to use the Norwegian language, nor will I EVER need to use the Paypal links in a local container. In other words, I will never need to use the vast majority of these apps. Most importantly though I had started noticing a real slowing down of the performance in my local containers. I had even started wondering if it is time to replace my machine.

Get rid of these apps!

Lucky for us, it is VERY easy to get rid of apps. The BCContainerHelper module has two commands for you. One is UnInstall-BcContainerApp, which we will use in today’s post. The second one is UnPublish-BcContainerApp, which is useful in case you want to completely get rid of the app altogether. If you want to be a real ninja about it, follow the links to see the underlying PowerShell logic that you can use for inspiration. Me, I like to keep it simple, so I’ll use the BCContainerHelper Cmdlets.

If you were thinking ‘never say never’ when you were reading the previous paragraph, you were absolutely right. What if I get a Norwegian customer tomorrow? Let’s use the uninstall Cmdlet. This will leave the app in the system, ready to be installed again at a later date

UnInstall-BcContainerApp `
    -containerName 'MyContainer' `
    -name 'Shopify Connector' `
    -doNotSaveData `
    -doNotSaveSchema `
    -force
  • The name of the app is enough to uninstall the app. You could also specify the publisher and version if you want but for our purposes the name is enough since all apps in the standard container are published by Microsoft
  • The ‘doNotSaveData’ parameter is used to make sure that the data is deleted from the companion tables, important because we are going to get rid of those with the next parameter
  • The ‘doNotSaveSchema’ parameter is used to remove the companion tables from the system. If you do not set this parameter, the schema will remain in the app database

If you are absolutely certain that you will never use the app, you can use the UnPublish Cmdlet instead and REALLY clean up that app list.

Bonus Company Removal

The standard container also comes with a pre-configured company called ‘My Company’ that I personally never use, and we have a command to remove that too:

Remove-CompanyInBcContainer `
    -containerName 'MyContainer' `
    -companyName 'My Company'

Get rid of half the companies, get rid of 50% of the unnecessary companion tables.

Put it All Together

In my personal ‘arsenal’ of goodies, I keep a set of scripts to create new containers. One of those is a script called ‘RemoveBloatware.ps1’ that lists just about every app in the standard container, something like this:

$MyContainerName = 'MyContainer'

Remove-CompanyInBcContainer `
    -containerName $MyContainerName `
    -companyName 'My Company'

UnInstall-BcContainerApp `
    -containerName $MyContainerName `
    -name 'AMC Banking 365 Fundamentals' `
    -doNotSaveData `
    -doNotSaveSchema `
    -force

UnInstall-BcContainerApp `
    -containerName $MyContainerName `
    -name 'Business Central Cloud Migration - Previous Release' `
    -doNotSaveData `
    -doNotSaveSchema `
    -force

UnInstall-BcContainerApp `
    -containerName $MyContainerName `
    -name 'Business Central Cloud Migration - Previous Release (US)' `
    -doNotSaveData `
    -doNotSaveSchema `
    -force

UnInstall-BcContainerApp `
    -containerName $MyContainerName `
    -name 'Shopify Connector' `
    -doNotSaveData `
    -doNotSaveSchema `
    -force

# etcetera, add any app that you don't want

Now you can call this script from your NewContainer script and have a nicely trimmed container. It made a real difference for me, I can really notice the better performance. Very useful when you’re in the thick of coding and you need to deploy code changes frequently.

Free Training

There are SO MANY resources to learn about BC and AL development out there. Some of the best of them are totally For Freeeee!! As I always like to point out: free is in everybody’s budget. Today I noticed a video that popped up in my YouTube timeline. It was a video that I had recorded a few years ago, and I noticed that it had more than 30K views! Just an unreal number, and I want to share the story about these videos.

It Starts…

Back in 2018 I was one of the owners of a well-known company (I’m gonna keep the name out of this post for personal reasons). We were working closely with Microsoft on the cutting edge of all the new technologies. We had developed the material for a number of workshops, and we all traveled to a bunch of different places all over the world to teach NAV people the intricacies of the new technology stack, new processes in the channel, and the philosophy that would become the path that we are now all traveling. Chances are that if you attended any event that was organized or sponsored by Microsoft, you have attended one of our workshops.

At some point, we were stretched quite thin. There were more requests for these workshops than we had staff to hold them. Microsoft then came up with the brilliant idea to record our workshops and create a series of videos that would be made available on PartnerSource.

Creating Content

We created a Very. Long. List. of topics, and the word came in from Microsoft: start creating content! The task of actually creating the videos fell to me, because of my demeaner during in-person classes and my pleasant baritone voice *ahem*. The truth is that most of my co-owners thought this would be a terribly tedious task, and I was the only one that was actually excited to record all of this material. Also, I had a LOT to learn and this was a perfect opportunity to do just that. As far as I’m concerned, this was by far the coolest project I had ever taken on in my professional career. It was explained to me that Microsoft would provide the content, and all that I had to do was record the videos.

It would take a whole series of posts to tell the stories of creating the content, so let me just summarize. Despite assurances from one of my co-owners, nobody at Microsoft knew about the expectation that they would provide ANY sort of content, other than meeting with me to briefly discuss the outline of the content and providing answers to questions. For a number of topics we had some content from short presentations at various events, but none of it was nearly good enough to make it into the videos. As it turned out, I had to create most of the material from scratch. A LOT of willing and eager Microsoft people spent a very limited time with me to first explain the basics and then go over the results of my material before I recorded it.

Let me just make one thing very clear. I enjoyed every single minute of the process of creating and recording the material. It was an absolute joy to work with every single person from the BC team, many of whom had to endure completely ignorant questions from me. I am super grateful to have had the opportunity to work with each and every one of them. I could not have done any of this without their help.

Over the course of 6-7 months, I created more than 20 hours of videos. The topics range from a condensed version of our 2-day AL Development workshop, to videos about how to get your app into AppSource, to automated testing, to source code management. Picked up a bunch of skills that I still benefit from today. It really was one of the best projects of my career.

The Academy That Never Was

The initial idea was that Microsoft would create some sort of ‘academy’ that would be accessible in PartnerSource. Partners would pay a fee for to provide training to their staff, of which our company would receive a percentage. All good with us, because there were BIG plans for the ISV Development Center, so we didn’t think we would have much more time to do in-person workshops anymore anyway.

Soon it became clear that this academy was not going to happen. There were calls from partners that they would not want to pay for this, since they never had to pay to attend any in-person events. At some point about half the videos were done, most of the material for the rest was ready to record, and the question was to continue recording or to stop the project.

There was talk of putting the videos on PartnerSource but not behind a paywall, which just made no sense to me at all. Most developers that I know don’t even have access to PartnerSource, so they would never even see it. Besides, if you are going to provide this content for free, why not just put all of it on YouTube? Just upload it to the public and let anyone that wants to learn all the skills that you need to make it as an AL developer. Once I heard that the content was going to be made available for free, I went all in and talked to anyone that wanted to listen that it should be made available publicly.

To make a long story short, they did end up putting the videos on YouTube (they are all in a single playlist that you can access here).

Unexpected Impact

It’s been almost four years since I created these videos. Still today, every once in a while, videos from this playlist show up in my YouTube timeline, like today. It just struck me that this video had 30K (thirty THOUSAND!!!) views. I was just so surprised about how many people have watched videos that I created. Thinking about all the people that have learned these skills, partly as a result of listening to me explain them. It just makes my head spin.

There are two things about these videos that I take full credit for. First, since I was responsible for the content, I had decided that I wanted to make proper full-length videos. Not condensed summary videos with the high level view of the topic, but deep down detailed videos with ALL the information that you need to execute on that topic.

The second thing is getting the content into YouTube. The project manager told me that my relentless lobbying to every person in the BC team was a key factor in getting them to put these videos on YouTube. I was paid to create the videos but getting them to YouTube was totally done with a community spirit. This is by far the most impactful contribution I have ever made to the BC community, one that I am extremely proud of.