Masterarbeit/QuellCode/CentronERP/docs/guides/development/end-to-end-testing.md

How to EndToEnd test

General

EndToEnd tests are supposed to test everything from the WebServiceBL layer and downwards (see the diagram here).
In other words, with EndToEnd tests we are testing the WebServiceBL, the BL and the database.

The project can be found at /tests/Centron.Tests.EndToEnd and all the tests are supposed to be in the Tests namespace.

For example, there are tests for the c-entron.NET Receipt Area are at /tests/Centron.Tests.EndToEnd/Tests/Receipts/ReceiptTest.cs.

The tests are executed as part of every Pull-Request.
If one of the tests fails, the PR cannot be merged.

Caution!
Some tests are a bit flaky sometimes. That means they fail even tho they should not. In these cases you can just let the PR get built again, and the tests should run successful.

We should try to make all tests NOT flaky.

Different kinds of tests

We can roughly categorize all tests into 2 categories:

1. General purpose tests
2. Specialized tests

They are only different in what they do, and it's not a hard categorization either.
You still create both kinds of tests the same way, and they will look the same.
They both inherit from EndToEndTest.cs and only differ in what and how much they do.

You can also inherit from CentronTest.cs, if you don't need database access for your test.

And also, there is a PerformanceTest.cs, that measures the performance of some code, and fails if the performance-goal is not met.
We can use this kind of test to make sure, that we don't have too harsh performance-regressions when we change some things.

For example, we have a test that makes sure big receipts can be saved in less than 5 seconds.
If we add something to the SaveReceipt method, or change some other things around, and the receipt suddenly takes 8 seconds to save, then we made a mistake and have to correct it.

General purpose tests

These tests are general purpose, doing every day things that the user does too.
For example creating a new customer, adding addresses and contact-persons to it, renaming contact-persons, removing addresses and so on.

The general purpose tests are supposed to cover really much code.
We want them to fail if we break a workflow that the user is doing every day.

Specialized tests

The specialized tests are way smaller than the general purpose tests and are usually created as a response to a ticket that was fixed, or a new feature that was created.

For example make sure that a setting works as expected, or make sure that a bug was fixed.

These specialized tests are supposed to cover really small amounts of code, just enough to make sure the specific thing works as expected.
We want them to fail if we break a specialized setting that some users might be using.

How does it work?

1. Before each test is executed a database backup (/Infrastructure/DatabaseBackup.zip) is restored
2. The DAOFactory is configured to use that new database
3. Any new scripts (IScriptMethod) are executed
4. The Test.Execute method is executed
5. The database backup is removed

This workflow means that every test always has the same starting configuration.
This fact is extremely useful because it allows us to execute these tests today, or in a month, or in a year, and we will still get the same result in the end.
If you create new entries in a table for example, they will get the same I3D every time.

Actually, that's the most important thing about these tests, and why they even work at all: The tests are repeatable!

If they were not repeatable, they would fail without any real breaking changes, and then they are not useful to find the breaking changes.

How to run the tests on your machine

Because we restore the database backup before each test, and the backup is on your local machine, the SQL Server also needs to be on your local machine.
So if I work on my own machine CS-ULM-DANHAE, and I want to run some tests, I absolutely need a SQL Server installed on my own machine (CS-ULM-DANHAE\SQL2017 in my case).
You cannot use a SQL Server on a different machine.

You need at least an SQL Server 2017.

To configure the connection to the SQL Server you have to change some small things in the /Infrastructure/Database.cs class.

You will see three properties at the top: Server, Username and Password. They all look similar to this:

public static string Server
{
    get
    {
        var server = GetEnvironmentVariable("CENTRON_TESTS_DATABASE_SERVER");

        if (string.IsNullOrWhiteSpace(server) == false)
            return server;

        throw new Exception("You have to configure the SQL server for EndToEndTests");
        return "YOUR-MACHINE\\SQL2017"; //Enter your local SQL server here
    }
}

You can see, it at first tries to get the configured value from an environment variable - that is exactly how the code gets the connection when it's run by the build server.
The build-server will set the environment variables to a SQL Server installed on the build-server itself (so it's locally to where the tests will be executed).

But if the environment variable is not set, it will throw an Exception.

If you want to configure the EndToEnd tests to use your own local SQL Server, you just have to comment out the exception, and change the return value of the property.

public static string Server
{
    get
    {
        var server = GetEnvironmentVariable("CENTRON_TESTS_DATABASE_SERVER");

        if (string.IsNullOrWhiteSpace(server) == false)
            return server;

        //throw new Exception("You have to configure the SQL server for EndToEndTests");
        return "CS-ULM-DANHAE\\SQL2017"; //Enter your local SQL server here
    }
}

If you do the same thing for the Username and Password properties, you are done and can execute the tests locally on your machine.

❗ Make sure to not commit these changes! ❗

How data is validated

When executing the test, you will probably create some data or some objects.
For example, you will create a new customer, a new ticket, or a new receipt.

The test has to validate, that these objects look exactly how expected.

This is done with a class called CentronVerifier. It has a Verify method that takes the data you want to verify, and a name for it.

this.Verifier.Verify("CreatedOffer", offer);

This will create a file called CreatedOffer.actual.txt that is basically the offer json serialized.
It will then compare this file, to a CreatedOffer.expected.txt file.
If they differ, something has changed, and the test fails.
If they are the same, everything still works as expected, and the test continues.

The first time you execute a test, of course there will be no *.expected.txt file, and thus the test will fail.
You're supposed to go ahead, and rename the *.actual.txt one to *.expected.txt.
So when you execute the test again, a new *.actual.txt file will be created, and compared to the *.expected.txt.

This also means, the *.expected.txt should be added to source-control, but the *.actual.txt files should NOT be added to source-control.

You can also verify data from a table by using the VerifyTable method.

this.Verifier.VerifyTable("TheCustomer", "dbo.Kunden", where: "I3D = 12345");

With that you have to provide a name again, the table that you want to verify, and a optional where filter.
A SELECT * FROM [Table] WHERE [Filter] statement will be executed, and the result will be used as data for the Verify method.
So everything with the *.expected.txt and *.actual.txt files works exactly the same as with the Verify method.

Tips and other useful things

Run any sql statement in your test

The base class for all tests EndToEndTest.cs provides a useful method Sql to run any sql statement in your test.
You can use it for example if you easily wanna change a setting in the database before running your test. Or you want to change a configuration property on a customer or something else.

this.Sql("UPDATE Stammdat SET Wert = 1 WHERE I3D = 1234");

Debugging a test

You can also very easily debug a test by setting breakpoints in your code, and then Right-Click -> Debug the test in the Test Explorer.

Diff view for *.expected.txt and *.actual.txt

If a test fails because something has changed between a *.expected.txt and a *.actual.txt file, you can see a diff between those two files.
This allows you to more easily spot the differences.

It is pretty easy to do. If a test fails because of a this.Verifier.Verify call, you just have to debug the test.

Now if the Verify call fails the test will fail as usual, but at the same time a Diff-View will open in Visual Studio with both the *.expected.txt and the *.actual.txt in it.

You can then more easily decide whether the changes are fine, or whether you broke something.

How to create a test

Lets say we want to write a general purpose test for some Helpdesk stuff - creating a new ticket for a customer, setting some values and saving it.

The real general purpose HelpdeskTest is of course supposed to cover even more of the helpdesk code.
For example forwarding it, adding documents and history entries, and closing the ticket.

Step 1

Make sure to configure the database connection so you can run them on your computer.

Step 2

Create a new namespace/folder under the /Tests namespace: WikiHelpdesk.

Step 3

In that namespace, create a new class WikiHelpdeskTests.cs.

Step 4

Make that class inherit from EndToEndTest.

Step 5

Implement the constructor for the class, and override the Execute method.
Your class should look like this now:

using Centron.Tests.EndToEnd.Infrastructure;
using Xunit.Abstractions;

namespace Centron.Tests.EndToEnd.Tests.WikiHelpdesks
{
    public class WikiHelpdeskTests : EndToEndTest
    {
        public WikiHelpdeskTests(ITestOutputHelper testOutputHelper) 
            : base(testOutputHelper)
        {
        }

        public override void Execute()
        {
        }
    }
}

This is the basic setup you always have to do for a test.
Everything that follows now is actually implementing the test and validating data.

Step 6

Create a new method CreateTicket

private HelpdeskDTO CreateTicket()
{
    using (var session = new BLSession())
    {
        //Get a LoggedInUser for the current user using the base class method
        var loggedInUser = this.GetLoggedInUser(11); // Default is 11 if no parameter is provided

        //Create a new ticket
        var ticket = session.GetBL<HelpdeskWebServiceBL>().GetNewTicketFromCustomer(loggedInUser.User, 10022);
        
        //Set some values
        ticket.ShortDescription = "Mein super wichtiges Ticket!";
        ticket.Description = "Mit einer nicht sehr ausführlichen Beschreibung.";

        //Save the helpdesk
        var saveHelpdeskResult = session.GetBL<HelpdeskWebServiceBL>().SaveHelpdesk(ticket, loggedInUser);
        //Verify the saveHelpdeskResult
        this.Verifier.Verify("SaveHelpdeskResult", saveHelpdeskResult);

        //Load the complete helpdesk object
        var savedTicket = session.GetBL<HelpdeskWebServiceBL>().GetHelpdeskByI3D(saveHelpdeskResult.Data, loggedInUser);
        //Verify the savedTicket
        this.Verifier.Verify("SavedTicket", savedTicket);

        return savedTicket.Data;
    }
}

I wanna especially highlight the two lines that use the this.Verifier:

this.Verifier.Verify("SaveHelpdeskResult", saveHelpdeskResult);

and

this.Verifier.Verify("SavedTicket", savedTicket);

These calls make sure that the data is actually validated.

Step 7

Call the new CreateTicket method in the Execute method.

public override void Execute()
{
    var ticket = this.CreateTicket();
}

You created your first test! 🎉

Step 8

As you know from the chapter on how data is actually validated the test will fail right now.
There is no SaveHelpdeskResult.expected.txt and no SavedTicket.expected.txt.

So lets run the test by opening the Test Explorer in Visual Studio and Right-Click -> Run the test.

It will take a little while, but after like 40 seconds you should get a failed test result.

Step 9

If you activate Show All Files in the solution explorer for the Centron.Tests.EndToEnd project, you should see the SaveHelpdeskResult.actual.txt file.

Go ahead and rename it to SaveHelpdeskResult.expected.txt.
It automatically gets picked up by the csproj file too.

Step 10

Run the test again!

...

As expected, it still fails because of the second Verify call.
But you now know what to do.

Step 11

A new file SavedTicket.actual.txt was created.

Feel free to open it, and look at the data that is verified.
Every single line in that JSON document has to stay exactly the same.
Every property in that file is not allowed to change.

Rename it to SavedTicket.expected.txt.

Step 12

Run the test once again!

...

IT WORKS! 🎉

The test is finished!
You can now add even more logic to the Execute method.