# EDI-Service Import Process

This document outlines the technical specifications and process flow for how the c-entron.NET EDI-Service downloads EDI documents from suppliers and imports them into the database.

## 1. System Architecture

### 1.1 Components

- **EdiDownloadService**: ASP.NET Core BackgroundService for scheduled downloads
- **SupplierEdiWebServiceBL**: Business logic layer for EDI web services
- **SupplierEdiBL**: Core business logic for supplier EDI operations
- **EDIConnectBL**: Connection handling for FTP/SFTP/FTPS

### 1.2 Execution Frequency

- Runs every 30 minutes (configurable)
- Initial 1 minute delay after system startup
- Log cleanup for entries older than 185 days occurs between 00:00-02:00

## 2. Document Processing Flow

### 2.1 Initialization & Configuration

```
EdiDownloadService.ExecuteAsync
└── SupplierEdiWebServiceBL.EDIDownloadStartAsync
    └── SupplierEdiBL.DownloadStartAsync
        ├── GetSupplierEdiConfigurations
        └── ProcessIndividualConfigurations
```

### 2.2 Download Process

For each supplier configuration:

1. **Connection Selection**:
   - FTP/FTPS: Uses `Ftp_DownloadAsync()`
   - SFTP: Uses `sFtp_DownloadAsync()`

2. **File Filtering**:
   ```csharp
   // Get list of already processed files
   var usedFiles = UsedFiles(config);
   
   // Filter available files
   foreach (FtpListItem file in serverFiles.Data.Where(f => f.Type == FtpObjectType.File))
   {
       // Skip if specific file expected but doesn't match
       if (!string.IsNullOrEmpty(expectedFile) && expectedFile != file.Name) continue;
       
       // Skip blacklisted files (those that failed multiple times before)
       if (badFiles.Any(f => f.Value.Contains(file.Name))) continue;
       
       // Skip already processed files
       if (usedFiles.IndexOf(file.Name) > -1) continue;
       
       // Skip files not matching mask pattern
       if (!FitMask(file.Name, config.Mask)) continue;
       
       // Process file
       var distriFiles = await DownloadFtpFile(config, file.Name);
       if (distriFiles != null)
           if (await ApplyDistriToCentron(distriFiles, config, null)) ++nSaved;
   }
   ```

### 2.3 ZIP Handling

Files are processed differently based on extension:

```csharp
if (Path.GetExtension(fileName).ToLower() == ".zip")
{
    // Add ZIP file to list
    distriFiles.Add(new EDIDistriFile() { DistriName = fileName });
    
    // Extract contents
    ZipExtract(result.Data, distriFiles);
}
else
{
    // Handle non-ZIP file
    MemoryStream dataStream = new MemoryStream();
    result.Data.CopyTo(dataStream);
    distriFiles.Add(new EDIDistriFile() { 
        UnpackName = fileName, 
        DistriName = fileName, 
        XmlDatei = dataStream 
    });
}
```

### 2.4 Database Import

Import logic varies by document and supplier type:

```csharp
switch (config.EdiDataType)
{
    case (int)EdiDataType.OpenTrans21:
        if (config.ObjectKind == (int)EDIConnectionObjectKind.OrderResponse) 
            isOk = await ReadOT21Response(xmlData, config, deal);
        if (config.ObjectKind == (int)EDIConnectionObjectKind.Delivery) 
            isOk = await ReadOT21Delivery(xmlData, config, deal);
        if (config.ObjectKind == (int)EDIConnectionObjectKind.Invoice) 
            isOk = await ReadOT21InvoiceAsync(xmlData, config, deal);
        break;
    
    // Additional formats (Also, AlsoCH, Herweck, etc.)
    ...
}
```

## 3. Database Schema

### 3.1 Primary EDI Tables

| Table | Description | Key Columns |
|-------|-------------|------------|
| `[dbo].[EDIInvoiceHead]` | Stores EDI invoice headers | `I3D`, `SupplierI3D`, `OrigFileName` |
| `[dbo].[EDIInvoicePositions]` | Stores EDI invoice line items | `I3D`, `InvoiceHeadI3D` |
| `[dbo].[EDIDeliveryHead]` | Stores EDI delivery headers | `I3D`, `SupplierI3D`, `OrigFileName` |
| `[dbo].[EDIDeliveryPositions]` | Stores EDI delivery line items | `I3D`, `DeliveryHeadI3D` |

### 3.2 File Tracking

The system prevents duplicate imports by checking the `OrigFileName` column:

```csharp
// For invoices
if (config.ObjectKind == (int)EDIConnectionObjectKind.Invoice)
{
    var used = this.Session.GetGenericDAO<EDIInvoiceHead>().GetEntityList(f => f.SupplierI3D == config.SupplierI3D);
    return used.Select(f => f.OrigFileName).ToList();
}

// For delivery notes
if (config.ObjectKind == (int)EDIConnectionObjectKind.Delivery)
{
    var used = this.Session.GetGenericDAO<EDIDeliveryHead>().GetEntityList(f => f.SupplierI3D == config.SupplierI3D);
    return used.Select(f => f.OrigFileName).ToList();
}
```

## 4. Special Case Handling

### 4.1 Distributor-Specific Processing

- **ITScope**: Uses `LoadITScopeReceiptAsync()` for specialized receipt handling
- **EGIS**: Uses `CheckEgisAsync()` for EGIS-specific downloads
- **Supplier-Specific Format Handlers**:
  - OpenTrans 2.1 (`ReadOT21*` methods)
  - Also (`ReadAlso*` methods)
  - AlsoCH (`ReadAlsoCH*` methods)
  - Herweck (`ReadHerweck*` methods)
  - Komsa (`ReadKomsa*` methods)
  - Alltron (`ReadAlltron*` methods)
  - Zugferd (`ReadZugferd*` methods)

### 4.2 Error Handling & File Blacklist

- Failed downloads are tracked in a separate list
- The system can be configured to retry previously failed downloads
- Errors are logged with detailed exception information
- A file blacklist mechanism prevents repeated processing of problematic files

#### 4.2.1 File Blacklist Implementation

The system maintains a blacklist of files that have repeatedly failed processing:

```csharp
// In DownloadStartAsync method:
badFiles = GetDownloadWithError(config.SupplierI3D, config.ObjectKind);

// Files with more than 3 recorded exceptions are blacklisted
await Ftp_DownloadAsync(config, badFiles.Where(f => f.ID > 3).ToList(), expectedFile);
```

The blacklist is populated from the `EDIManagementLog` table using SQL:

```csharp
private List<IntStringList> GetDownloadWithError(int distributorI3D, int objectKind)
{
    ReceiptLogKind logKind;
    // Map objectKind to appropriate log kind...
    
    string sSql = $@"select COUNT(*) ID, l.FileName Value from EDIManagementLog l
                      Where l.State = {(int)EDILogState.Exception}
                            and l.EDIReceiptLogKind = {(int)logKind}  
                            and l.DistributorI3D = {distributorI3D.ToString()}  
                      Group By l.FileName ";

    return Session.Advanced.RawSqlAccess.ExecuteQuery<IntStringList>(sSql, null).ToList();
}
```

During download processing, blacklisted files are skipped:

```csharp
// In both Ftp_DownloadAsync and sFtp_DownloadAsync methods
if (badFiles.Any(f => f.Value.Contains(file.Name))) continue;
```

This prevents the system from repeatedly trying to process files that have caused multiple exceptions, reducing system load and avoiding potential endless error loops.

## 5. Logging System

The EDI process uses NLog for comprehensive logging:

```csharp
// Log start of EDI process
Logger.Info($"EDI Download starts.");

// Log errors with full exception details
Logger.Error(exception, "EDI Download ERROR");

// Detailed operation logs via _eDILogBL
_eDILogBL.WriteEdiDownloadLog(config, EDILogState.DownloadTest, fileName: expectedFile, comment: $"File: {expectedFile} has already been exported.");
```

## 6. Testing & Debugging

### 6.1 Test Mode

A test mode is available (`isTest` parameter in `DownloadStartAsync`):
- Files are not deleted from remote server
- More detailed logs are generated
- Can target specific files via `expectedFile` parameter

### 6.2 System User

The system uses a designated system user account:
```csharp
var user = this.Session.GetGenericDAO<AppUser>().GetById(
    new AppSettingsBL(Session).GetSettings(ApplicationSettingID.CentronSystemUser)
    .GetInt(ApplicationSettingID.CentronSystemUser, null));
```

## 7. Security Considerations

- Connection credentials are securely stored in supplier EDI configuration
- Supports secure protocols: FTPS (FTP with SSL/TLS) and SFTP
- Files are processed in memory to minimize disk exposure
- System user permissions control database operations