Masterarbeit/QuellCode/CentronERP/docs/reference/edi/edi-import-rules.md

EDI-Service Import Process

This document outlines the technical specifications and process flow for how the c-entron.NET EDI-Service downloads EDI documents from suppliers and imports them into the database.

1. System Architecture

1.1 Components

• EdiDownloadService: ASP.NET Core BackgroundService for scheduled downloads
• SupplierEdiWebServiceBL: Business logic layer for EDI web services
• SupplierEdiBL: Core business logic for supplier EDI operations
• EDIConnectBL: Connection handling for FTP/SFTP/FTPS

1.2 Execution Frequency

• Runs every 30 minutes (configurable)
• Initial 1 minute delay after system startup
• Log cleanup for entries older than 185 days occurs between 00:00-02:00

2. Document Processing Flow

2.1 Initialization & Configuration

EdiDownloadService.ExecuteAsync
└── SupplierEdiWebServiceBL.EDIDownloadStartAsync
    └── SupplierEdiBL.DownloadStartAsync
        ├── GetSupplierEdiConfigurations
        └── ProcessIndividualConfigurations

2.2 Download Process

For each supplier configuration:

1. Connection Selection:

   • FTP/FTPS: Uses Ftp_DownloadAsync()
   • SFTP: Uses sFtp_DownloadAsync()

2. File Filtering:

   // Get list of already processed files
   var usedFiles = UsedFiles(config);
   
   // Filter available files
   foreach (FtpListItem file in serverFiles.Data.Where(f => f.Type == FtpObjectType.File))
   {
       // Skip if specific file expected but doesn't match
       if (!string.IsNullOrEmpty(expectedFile) && expectedFile != file.Name) continue;
   
       // Skip blacklisted files (those that failed multiple times before)
       if (badFiles.Any(f => f.Value.Contains(file.Name))) continue;
   
       // Skip already processed files
       if (usedFiles.IndexOf(file.Name) > -1) continue;
   
       // Skip files not matching mask pattern
       if (!FitMask(file.Name, config.Mask)) continue;
   
       // Process file
       var distriFiles = await DownloadFtpFile(config, file.Name);
       if (distriFiles != null)
           if (await ApplyDistriToCentron(distriFiles, config, null)) ++nSaved;
   }
   

2.3 ZIP Handling

Files are processed differently based on extension:

if (Path.GetExtension(fileName).ToLower() == ".zip")
{
    // Add ZIP file to list
    distriFiles.Add(new EDIDistriFile() { DistriName = fileName });
    
    // Extract contents
    ZipExtract(result.Data, distriFiles);
}
else
{
    // Handle non-ZIP file
    MemoryStream dataStream = new MemoryStream();
    result.Data.CopyTo(dataStream);
    distriFiles.Add(new EDIDistriFile() { 
        UnpackName = fileName, 
        DistriName = fileName, 
        XmlDatei = dataStream 
    });
}

2.4 Database Import

Import logic varies by document and supplier type:

switch (config.EdiDataType)
{
    case (int)EdiDataType.OpenTrans21:
        if (config.ObjectKind == (int)EDIConnectionObjectKind.OrderResponse) 
            isOk = await ReadOT21Response(xmlData, config, deal);
        if (config.ObjectKind == (int)EDIConnectionObjectKind.Delivery) 
            isOk = await ReadOT21Delivery(xmlData, config, deal);
        if (config.ObjectKind == (int)EDIConnectionObjectKind.Invoice) 
            isOk = await ReadOT21InvoiceAsync(xmlData, config, deal);
        break;
    
    // Additional formats (Also, AlsoCH, Herweck, etc.)
    ...
}

3. Database Schema

3.1 Primary EDI Tables

Table	Description	Key Columns
[dbo].[EDIInvoiceHead]	Stores EDI invoice headers	I3D, SupplierI3D, OrigFileName
[dbo].[EDIInvoicePositions]	Stores EDI invoice line items	I3D, InvoiceHeadI3D
[dbo].[EDIDeliveryHead]	Stores EDI delivery headers	I3D, SupplierI3D, OrigFileName
[dbo].[EDIDeliveryPositions]	Stores EDI delivery line items	I3D, DeliveryHeadI3D

3.2 File Tracking

The system prevents duplicate imports by checking the OrigFileName column:

// For invoices
if (config.ObjectKind == (int)EDIConnectionObjectKind.Invoice)
{
    var used = this.Session.GetGenericDAO<EDIInvoiceHead>().GetEntityList(f => f.SupplierI3D == config.SupplierI3D);
    return used.Select(f => f.OrigFileName).ToList();
}

// For delivery notes
if (config.ObjectKind == (int)EDIConnectionObjectKind.Delivery)
{
    var used = this.Session.GetGenericDAO<EDIDeliveryHead>().GetEntityList(f => f.SupplierI3D == config.SupplierI3D);
    return used.Select(f => f.OrigFileName).ToList();
}

4. Special Case Handling

4.1 Distributor-Specific Processing

• ITScope: Uses LoadITScopeReceiptAsync() for specialized receipt handling
• EGIS: Uses CheckEgisAsync() for EGIS-specific downloads
• Supplier-Specific Format Handlers:
  • OpenTrans 2.1 (ReadOT21* methods)
  • Also (ReadAlso* methods)
  • AlsoCH (ReadAlsoCH* methods)
  • Herweck (ReadHerweck* methods)
  • Komsa (ReadKomsa* methods)
  • Alltron (ReadAlltron* methods)
  • Zugferd (ReadZugferd* methods)

4.2 Error Handling & File Blacklist

• Failed downloads are tracked in a separate list
• The system can be configured to retry previously failed downloads
• Errors are logged with detailed exception information
• A file blacklist mechanism prevents repeated processing of problematic files

4.2.1 File Blacklist Implementation

The system maintains a blacklist of files that have repeatedly failed processing:

// In DownloadStartAsync method:
badFiles = GetDownloadWithError(config.SupplierI3D, config.ObjectKind);

// Files with more than 3 recorded exceptions are blacklisted
await Ftp_DownloadAsync(config, badFiles.Where(f => f.ID > 3).ToList(), expectedFile);

The blacklist is populated from the EDIManagementLog table using SQL:

private List<IntStringList> GetDownloadWithError(int distributorI3D, int objectKind)
{
    ReceiptLogKind logKind;
    // Map objectKind to appropriate log kind...
    
    string sSql = $@"select COUNT(*) ID, l.FileName Value from EDIManagementLog l
                      Where l.State = {(int)EDILogState.Exception}
                            and l.EDIReceiptLogKind = {(int)logKind}  
                            and l.DistributorI3D = {distributorI3D.ToString()}  
                      Group By l.FileName ";

    return Session.Advanced.RawSqlAccess.ExecuteQuery<IntStringList>(sSql, null).ToList();
}

During download processing, blacklisted files are skipped:

// In both Ftp_DownloadAsync and sFtp_DownloadAsync methods
if (badFiles.Any(f => f.Value.Contains(file.Name))) continue;

This prevents the system from repeatedly trying to process files that have caused multiple exceptions, reducing system load and avoiding potential endless error loops.

5. Logging System

The EDI process uses NLog for comprehensive logging:

// Log start of EDI process
Logger.Info($"EDI Download starts.");

// Log errors with full exception details
Logger.Error(exception, "EDI Download ERROR");

// Detailed operation logs via _eDILogBL
_eDILogBL.WriteEdiDownloadLog(config, EDILogState.DownloadTest, fileName: expectedFile, comment: $"File: {expectedFile} has already been exported.");

6. Testing & Debugging

6.1 Test Mode

A test mode is available (isTest parameter in DownloadStartAsync):

• Files are not deleted from remote server
• More detailed logs are generated
• Can target specific files via expectedFile parameter

6.2 System User

The system uses a designated system user account:

var user = this.Session.GetGenericDAO<AppUser>().GetById(
    new AppSettingsBL(Session).GetSettings(ApplicationSettingID.CentronSystemUser)
    .GetInt(ApplicationSettingID.CentronSystemUser, null));

7. Security Considerations

• Connection credentials are securely stored in supplier EDI configuration
• Supports secure protocols: FTPS (FTP with SSL/TLS) and SFTP
• Files are processed in memory to minimize disk exposure
• System user permissions control database operations