Masterarbeit/QuellCode/CentronERP/docs/operations/build-server-and-automated-builds.md

High level overview

There are a couple of different pieces involved in creating a build.

The most central piece is the Centron.Scripts project in this repository, it does most of the heavy work.
There is also the azure-pipelines.yml that specifies how Azure DevOps works with the Centron.Scripts project, and what to do with the resulting files.
The builds will be placed in our software builds sharepoint.
If the build references one or multiple c-entron Tickets, these Tickets will automatically get the version number of that build, and the ticket will be forwarded to the quality-control department.
There is also a downloader that automatically downloads the latest versions of c-entron.NET and Web-Service from the sharepoint to our own internal network shares. This makes it easier for everyone working at NEXOWARE to access the latest versions.

We have two build servers:
CS-UL-DEVOPS01 (hosted by celos) that has currently 3 build agents, and CS-UL-DEVOPS02 (hosted at celos) that has currently 3 build agents.

All of the users, passwords and other credentials that I talk about in this document are also available in our password server.

Centron.Scripts

The Centron.Scripts project knows how to build the c-entron.NET, the c-entron Web-Service, and also the installers for all of them.

It does a handfull of things:

• It creates the version number
• It builds the projects using dotnet.exe
• It deletes unneeded files from the build output
• It signs the applications and installers
• It creates archives of everything and moves them into the [git repository root]/artifacts directory
• It runs the tests and places the test result in the [git repository root]/artifacts directory

So when you run this project, after a couple of minutes you will end up with perfect installers for c-entron.NET and c-entron Web-Services without any more input required from you. Also, you will get a result file of all the tests.

Code signing

Code signing is optional - you can create a build without having our code signing certificate and password - it is of course still be recommended to do it.

You can activate code signing by setting two environment variables:
CENTRON_BUILD_CODE_SIGNING_CERTIFICATE and
CENTRON_BUILD_CODE_SIGNING_CERTIFICATE_PASSWORD

CENTRON_BUILD_CODE_SIGNING_CERTIFICATE should contain the path to the code signing certificate file. CENTRON_BUILD_CODE_SIGNING_CERTIFICATE_PASSWORD should contain the password for that code signing certificate.

Right now, all of the following files for c-entron.NET are signed:

• c-entron 2.0.exe
• c-entron.NET Installer.msi

And these for the c-entron Web-Service:

• Centron.Host.WindowsService.exe
• c-entron Connection Manager.exe
• Centron.Interfaces.dll
• Centron.WebServices.Core.dll
• Centron.Core.dll
• c-entron Web-Service Installer.msi

If one of these environment variables is empty or not set, the applications and installers will not be signed at all.
You can read in the console output whether the files were signed or not.

Automatic versioning

There is a version.json file in the root directory of the git repository. This version specifies the first 3 parts of the version number, for example 2.0.2601. The last part will be automatically generated by using Nerdbank.GitVersioning.

Nerdbank.GitVersioning is a very feature rich library to automatically generate version numbers for your git repository, but we only use it in a very simplistic simple way.

It basically counts how many commits there have been in the repository, since the version.json file was last changed. That count is the last part of the version number.

For example:
We change the version number in version.txt to 2.0.1911, and then have 2 commits, the version number will be 2.0.1911.3.
Note: It will be 3 and not 2 because it also counts the commit where the version.txt was changed.

This means that the version number is always increasing automatically.

azure-pipelines.yml

The azure-pipelines.yml file contains a couple of tasks, some for building the project, others for deploying it.

Tasks

So the tasks work roughly like this:

• Authenticate with the Azure DevOps NuGet server for c-entron Office
• It downloads our code-signing certificate from Azure DevOps (it's stored in Azure DevOps -> c-entron.NET -> Pipelines -> Library -> Secure files -> c-entron code signing certificate.pfx)
• It updates the Azure DevOps version number to show the version number of the build
• It then runs the Centron.Scripts and also sets the environment variables for code signing to work (by using the certificate file downloaded earlier, and the password that is added as a secure variable to the build pipeline)
• After the build succeeded, it will upload all the created installer archives (c-entron.NET Installer and c-entron Web-Service Installer) to our Software Builds sharepoint documents.
  They will be uploaded to "/c-entron.NET or c-entron Web-Service/vfirst three parts of the version number/vcomplete version number/artifact name".
  So for example:
  /c-entron.NET/v2.0.1908/v2.0.1908.3/c-entron.NET Installer.zip and
  /c-entron Web-Service/v2.0.1908/v2.0.1908.3/c-entron Web-Service Installer.zip

Note: The upload to the Software Builds sharepoint documents is only done for real builds, it is not done for pull request builds.

c-entron Tickets

If a real build was created, then c-entron Tickets that have been fixed or resolved by that real build will get the version number of that build, and also will be forwarded to our quality-control department.

This works by using so called Web Hooks from Azure DevOps. In our case: When a build finishes, a HTTP API will be triggered.
This HTTP API will then figure out which pull request was completed for that build, and whether this pull request references any c-entron Tickets.

To figure out which pull request was completed by the build, it calls the Azure DevOps API using an API key generated by the build@c-entron.de user.

It then searches for ticket numbers in the title and description of the pull request - the following formats are currently supported:

• Ticket 12345
• Ticket#12345
• Ticket #12345
• Ticket# 12345
• Ticket: 12345

When one or more ticket numbers were found, it will call our own c-entron API (https://c-suite.c-entron.de/CentronServiceProductive) to insert the build version into the ticket, and then forward it to the quality-control department (Personal I3D 372).
To call our own c-entron API the sw_centron AD user is used - that matches to the TFS user in our own c-entron database.
This user also has the email address build@c-entron.de to send out ticket forwarding emails.

It is possible to skip the forwarding by adding one of the following tags to the description of the PR:

• [skip-forwarding]
• [skip-fwd]
• [skip-forward]

If this is done it will only insert the build version into the ticket, but it does not forward the ticket. This allows you to link several PRs without the ticket getting send to QS before it is finished.

The HTTP API is currently running on our own web-server CS-UL-ERPWEB01, available at https://erp.c-entron.de/DevOpsCentronTicketBridge - you can enter this URL in the browser and should see a short message that the bridge is running.
It is running as the windows service DevOpsCentronTicketBridge, located at C:\Program Files (x86)\DevOpsCentronTicketBridge.

Basically everything about it can be configured in the appsettings.json file:

• The URL where it is available
• The Personal Access Token used to communicate with the Azure DevOps API
• The regular expressions that are used to find the ticket numbers in the title and description of the pull requests
  These regular expressions should have exactly one group (the round brackets) where the number will be located.
  For example: Ticket ([0-9]{5,6}) or Ticket#([0-9]{5,6})
• Where the c-entron Web-Service is running at, and which user to login with
• To which user the tickets should be forwarded

The whole source-code of this web hook is available on our Azure DevOps server too.

Azure DevOps setup

There are a couple of things configured in Azure DevOps to make this all work:

• The azure-pipelines.yml build pipelines is configured as a branch policy for the master branch and all release-branches (release/vXYZ) with trigger automatic.
  This means the pipeline will be automatically executed for pull requests.
• There is a secure variable configured for the build pipeline called codeSigningCertificatePassword
• There is the code signing certificate added as a secure file called c-entron code signing certificate.pfx
• In Project settings -> Service connections is a connection for Software Builds - Sharepoint - it uses the build@c-entron.de user to authenticate
• In Project settings -> Service hooks is a web hook created to make the c-entron Ticket workflow possible.
  It has a trigger of type Build completed and a couple filters:
  Build pipeline = c-entron.NET CI
  Build Status = Succeeded
  It performs a HTTP POST to https://erp.c-entron.de/DevOpsCentronTicketBridge with:
  Resource details to send = All
  Resource version = 1.0

Azure DevOps Build Agents

We have two build-servers called CS-UL-DEVOPS01 and CS-UL-DEVOPS02.

CS-UL-DEVOPS01 is accessible with its full name CS-UL-DEVOPS01.group.celos.de
I have my own local admin account there LocalAdminDH, but Celos Computer GmbH can add real accounts based on our windows accounts.
Stanislaus Lieb also can access this server.

There are currently 3 build agents installed on that machine:

• E:/AzureDevOpsAgent1 called CS-UL-DEVOPS01.1 and
• E:/AzureDevOpsAgent2 called CS-UL-DEVOPS01.2 and
• E:/AzureDevOpsAgent3 called CS-UL-DEVOPS01.3

CS-UL-DEVOPS02 is accessibly with its full name CS-UL-DEVOPS02.group.celos.de
Login same as CS-UL-DEVOPS01

There are currently 3 build agents installed on that machine:

• E:/AzureDevOpsAgent1 called CS-UL-DEVOPS02.1 and
• E:/AzureDevOpsAgent2 called CS-UL-DEVOPS02.2 and
• E:/AzureDevOpsAgent3 called CS-UL-DEVOPS03.3

These build agents connect to Azure DevOps using API keys that were generated by the build@c-entron.de user.
All of them are configured to run as a windows service using the local user AzureDevOpsAgentUser. This user is also an administrator on the machine (it has to be, otherwise the deploy tasks will fail).

We have a couple of software requirements for the build server:

• Visual Studio 2022 (authenticated with the build@c-entron.de user)
  • Make sure to install all .NET Framework targeting packs, so building c-entron.NET will work correctly
  • Install these workloads:
    • ASP.NET and web development
    • .NET Desktop development
• WIX 3 Toolset build tools (see here)
• WIX 3 Toolset Visual Studio 2022 Extension (also see here)
• .NET Core SDK 8.0.x (the latest one, see global.json) (get it here)
• .NET Core SDK 7.0.3x (for c-entron Office) (get it here)
• .NET Core SDK 6.0.1x (for Outlook Add-In) (get it here)
• An SQL-Server for our end-to-end tests (MSSQL 2017 prefered right now)
  • Make sure to set the environment-variables so the end-to-end tests can actually find the database. You can find the environment-variable names in the Database.cs file.
• As we are using the Azure DevOps task razorspoint.rp-build-release-pnptasks.RP-PnPPowerShell.PnPPowerShell@3 to upload the finished versions to sharepoint, we have to make sure the SharePoint upload works correctly too.
  Try running Find-Module -Name "SharePointPnPPowerShellOnline" -RequiredVersion "3.23.2007.1" on the build server.
  If it works, you should be good to go. If it doesn't we might have to add NuGet as a PowerShell PackageProvider.
  To do that, execute this command Install-PackageProvider -Name NuGet -MinimumVersion 2.8.5.201 -Force

Automatic downloader

We have an automatic downloader that downloads the latest versions of c-entron.NET and c-entron Web-Service every day at around 6 AM.

This downloader is running on the server CS-UL-CENTRON as the windows service SharePointVersionDownloader.
It is running as the user svc_SoftwareCentron@group.celos.de because the process needs write permissions to the network shares where the downloaded files will be placed.
On that server the downloader is located at C:\Program Files (x86)\SharePointVersionDownloader.

The downloader can be configured completely through the appsettings.json file.
If you want to download, for example, the latest 2.0.1908 version of c-entron.NET, then you have to add a job to the appsettings.json for that.
Another job will be required for the latest 2.0.1909 version, or for c-entron Web-Service versions.

For examples on how to configure these jobs, take a look at the current appsettings.json configuration, or even better, the source-code.

The source-code for this downloader is available on our Azure DevOps server.