Standardizing Multi‑Vendor Firewall Logs with LoongCollector on Alibaba Cloud

Background

In modern, highly connected environments, security threats are diverse, stealthy, and cross‑platform, requiring logs from firewalls, endpoints, and identity systems to build a complete security posture. Enterprises often face multiple vendor products, each emitting non‑standard, fragmented log fields, which raises integration costs and hampers analysis efficiency.

LoongCollector Overview

LoongCollector is a lightweight log‑collection agent that supports mainstream log formats and provides flexible parsing plugins. Its goal is not only to collect logs but also to standardize them for downstream threat detection, compliance auditing, and security analytics.

Prerequisites & Setup

Create a Logstore in Alibaba Cloud SLS and configure the appropriate index for query and analysis.

Install LoongCollector on the collection host.

Create a machine group under the Logstore to associate the collector.

Enabling Syslog Forwarding

Configure the firewall’s syslog settings to forward logs (usually via UDP) to the collector’s listening address. For Longteng WAF, enable the Syslog export option in the system settings; FortiGate and Palo Alto provide similar syslog forwarding configurations.

Longteng WAF Log Integration

Longteng WAF (SafeLine) generates JSON‑compatible logs that can be parsed directly. The following LoongCollector configuration captures and expands the _content_ field into key‑value pairs:

{
  "inputs": [{
    "type": "service_syslog",
    "detail": {
      "Address": "udp://0.0.0.0:5144",
      "ParseProtocol": "rfc5424"
    }
  }],
  "processors": [{
    "type": "processor_json",
    "detail": {
      "SourceKey": "_content_",
      "ExpandConnector": ".",
      "ExpandArray": false,
      "ExpandDepth": 0,
      "IgnoreFirstConnector": true,
      "KeepSource": false,
      "KeepSourceIfParseError": true,
      "NoKeyError": true,
      "Prefix": "",
      "UseSourceKeyAsPrefix": false
    }
  }]
}

After processing, fields such as req_header_raw, risk_level, and action become proper JSON keys, ready for indexing and visualization.

FortiGate Log Integration

FortiGate logs are typically space‑separated key‑value strings. The collector uses a split‑key‑value processor to parse them:

{
  "inputs": [{
    "Type": "service_syslog",
    "Address": "udp://0.0.0.0:9002",
    "ParseProtocol": "rfc5424",
    "IgnoreParseFailure": true
  }],
  "processors": [{
    "Type": "processor_split_key_value",
    "detail": {
      "SourceKey": "_content_",
      "Delimiter": " ",
      "Separator": "=",
      "Quote": "\"",
      "KeepSource": false,
      "UseSourceKeyAsPrefix": false,
      "ErrIfKeyIsEmpty": true,
      "ErrIfSourceKeyNotFound": true,
      "ErrIfSeparatorNotFound": true,
      "DiscardWhenSeparatorNotFound": false
    }
  }]
}

A sample FortiGate log shows fields like srcip, dstip, action, and msg correctly extracted after parsing.

CEF (Common Event Format) Log Integration

For devices that emit CEF logs (e.g., FortiGate in CEF mode), LoongCollector first splits the log on the pipe character, then applies a regex to extract the timestamp, host, CEF version, and the extension string, followed by a key‑value split on the extension:

{
  "inputs": [{
    "Type": "service_syslog",
    "Address": "udp://0.0.0.0:9003",
    "ParseProtocol": "rfc5424",
    "IgnoreParseFailure": true
  }],
  "processors": [
    {
      "Type": "processor_parse_delimiter_native",
      "detail": {
        "SourceKey": "content",
        "Separator": "|",
        "Quote": "\"",
        "Keys": ["time","Vendor","Product","Version","Signature_ID","Name","Severity","Extension"]
      }
    },
    {
      "Type": "processor_split_key_value",
      "detail": {
        "SourceKey": "Extension",
        "Delimiter": " ",
        "Separator": "=",
        "Quote": "\"",
        "KeepSource": false,
        "ErrIfKeyIsEmpty": true,
        "ErrIfSourceKeyNotFound": true,
        "ErrIfSeparatorNotFound": true,
        "DiscardWhenSeparatorNotFound": false
      }
    },
    {
      "Type": "processor_regex",
      "detail": {
        "SourceKey": "time",
        "Regex": "^([A-Z][a-z]{2}\\s+\d{1,2}\\s+\d{2}:\d{2}:\d{2})\\s+(\\S+)\\s+CEF:(\\d+)",
        "Keys": ["Time","Host","CEF_Version"],
        "KeepSource": false,
        "KeepSourceIfParseError": true,
        "NoKeyError": false,
        "NoMatchError": true
      }
    }
  ]
}

The resulting JSON contains structured fields such as Time, Host, Signature_ID, and all extension key‑value pairs.

Palo Alto Networks Log Integration

Palo Alto firewalls can forward logs via syslog or write them to local files. The guide shows how to collect the files with LoongCollector and then parse the CSV‑style log using SPL parse‑csv:

* | parse-csv content as FUTURE_USE, Receive_Time, Serial_Number, Type, Threat_Content_Type, FUTURE_USE_1, Generated_Time, Source_Address, Destination_Address, NAT_Source_IP, NAT_Destination_IP, Rule_Name, Source_User, Destination_User, Application, Virtual_System, Source_Zone, Destination_Zone, Inbound_Interface, Outbound_Interface, Log_Action, FUTURE_USE_2, Session_ID, Repeat_Count, Source_Port, Destination_Port, NAT_Source_Port, NAT_Destination_Port, Flags, IP_Protocol, Action, URL_Filename, Threat_ID, Category, Severity, Direction, Sequence_Number, Action_Flags, Source_Location, Destination_Location, FUTURE_USE_3, Content_Type, PCAP_ID, File_Digest, Cloud, URL_Index, User_Agent, File_Type, X_Forwarded_For, Referer, Sender, Subject, Recipient, Report_ID, Device_Group_Hierarchy_Level_1, Device_Group_Hierarchy_Level_2, Device_Group_Hierarchy_Level_3, Device_Group_Hierarchy_Level_4, Virtual_System_Name, Device_Name, FUTURE_USE_4, Source_VM_UUID, Destination_VM_UUID, HTTP_Method, Tunnel_ID_IMSI, Monitor_Tag_IMEI, Parent_Session_ID, Parent_Start_Time, Tunnel_Type, Threat_Category, Content_Version, FUTURE_USE_5, SCTP_Association_ID, Payload_Protocol_ID, HTTP_Headers, URL_Category_List, Rule_UUID, HTTP_2_Connection, Dynamic_User_Group_Name, XFF_Address, Source_Device_Category, Source_Device_Profile, Source_Device_Model, Source_Device_Vendor, Source_Device_OS_Family, Source_Device_OS_Version, Source_Hostname, Source_MAC_Address, Destination_Device_Category, Destination_Device_Profile, Destination_Device_Model, Destination_Device_Vendor, Destination_Device_OS_Family, Destination_Device_OS_Version, Destination_Hostname, Destination_MAC_Address, Container_ID, POD_Namespace, POD_Name, Source_External_Dynamic_List, Destination_External_Dynamic_List, Host_ID, Serial_Number_2, Domain_EDL, Source_Dynamic_Address_Group, Destination_Dynamic_Address_Group, Partial_Hash, High_Resolution_Timestamp, Reason, Justification, A_Slice_Service_Type, Application_Subcategory, Application_Category, Application_Technology, Application_Risk, Application_Characteristic, Application_Container, Tunneled_Application, Application_SaaS, Application_Sanctioned_State, Cloud_Report_ID, Cluster_Name, Flow_Type | project-away content

After parsing, each column becomes a distinct field in the SLS logstore, enabling precise queries and visualizations.

Result & Benefits

Across all three firewall vendors, LoongCollector transforms raw, heterogeneous syslog streams into well‑structured JSON objects. This uniform format allows security teams to create indexes, run queries, set alerts, and build dashboards without dealing with fragmented log schemas, thereby reducing data‑island problems and improving threat‑analysis efficiency.

Conclusion

Standardizing multi‑source firewall logs with LoongCollector demonstrates that unified log collection is more than a simple pipeline—it requires format compatibility, extensible parsing, and cloud‑native storage. The approach works for current firewall products and can be extended to other security data sources, feeding the Alibaba Cloud SLS intelligence engine for storage, search, analysis, and visualization.