How to Standardize and Collect Enterprise Firewall Logs with LoongCollector

Background

In today’s highly interconnected digital environment, security threats are becoming more diversified, hidden, and cross‑platform. Enterprises must collect logs from firewalls, endpoints, and authentication systems to build a complete security posture, but logs from different vendors are often unstructured and fragmented, leading to high integration costs and low analysis efficiency.

Why LoongCollector?

LoongCollector is a lightweight log‑collection tool that supports mainstream log formats and provides flexible parsing rules. It not only gathers logs but also standardizes them for downstream security analysis, threat detection, and compliance auditing.

Prerequisites

Create a Logstore for log storage and configure related indexes.

Install LoongCollector on the collection machine.

Create a machine group under the Logstore.

Log Collection Workflow

Security vendor firewalls forward logs via the Syslog protocol to a collector machine, where LoongCollector parses and writes them into Alibaba Cloud SLS.

Longting WAF (SafeLine) Integration

Enable Syslog forwarding in the WAF system settings. The WAF logs include protocol, source IP, timestamp, hostname, request method, event ID, attack type, risk level, etc.

{
  "inputs": [{
    "type": "service_syslog",
    "detail": {
      "Address": "udp://0.0.0.0:5144",
      "ParseProtocol": "rfc5424"
    }
  }],
  "processors": [{
    "type": "processor_json",
    "detail": {
      "ExpandArray": false,
      "ExpandConnector": ".",
      "ExpandDepth": 0,
      "IgnoreFirstConnector": true,
      "KeepSource": false,
      "KeepSourceIfParseError": true,
      "NoKeyError": true,
      "Prefix": "",
      "SourceKey": "_content_",
      "UseSourceKeyAsPrefix": false
    }
  }]
}

The above configuration expands the _content_ field of the WAF Syslog message into a normal JSON key‑value structure.

Sample parsed result (truncated):

{
  "scheme": "http",
  "src_ip": "12.123.123.123",
  "src_port": 53008,
  "host": "safeline-ce.chaitin.net",
  "method": "GET",
  "event_id": "32be0ce3ba6c44be9ed7e1235f9eebab",
  "risk_level": "high",
  "action": "deny",
  "req_header_raw": "GET /webshell.php HTTP/1.1
Host: safeline-ce.chaitin.net:8083
User-Agent: PostmanRuntime/7.28.4
..."
}

FortiGate Integration

Configure Syslog forwarding on FortiGate (supports default, CEF, and CSV formats). Example JSON for default format:

{
  "inputs": [{
    "Type": "service_syslog",
    "Address": "udp://0.0.0.0:9002",
    "ParseProtocol": "rfc5424",
    "IgnoreParseFailure": true
  }],
  "processors": [{
    "Type": "processor_split_key_value",
    "SourceKey": "_content_",
    "Delimiter": " ",
    "Separator": "=",
    "Quote": "\"",
    "KeepSource": false,
    "UseSourceKeyAsPrefix": false,
    "ErrIfKeyIsEmpty": true,
    "ErrIfSourceKeyNotFound": true,
    "ErrIfSeparatorNotFound": true,
    "DiscardWhenSeparatorNotFound": false
  }]
}

Sample FortiGate log (default format) and parsed output demonstrate correct key‑value extraction, e.g., the msg field contains "URL belongs to a denied category in policy".

CEF Format Handling

For FortiGate CEF logs, LoongCollector first splits the log by the "|" delimiter, extracts the extension part, and then expands key‑value pairs.

{
  "inputs": [{
    "Type": "service_syslog",
    "Address": "udp://0.0.0.0:9003",
    "ParseProtocol": "rfc5424",
    "IgnoreParseFailure": true
  }],
  "processors": [
    {
      "Type": "processor_parse_delimiter_native",
      "SourceKey": "content",
      "Separator": "|",
      "Quote": "\"",
      "Keys": ["time","Vendor","Product","Version","Signature_ID","Name","Severity","Extension"]
    },
    {
      "Type": "processor_split_key_value",
      "SourceKey": "Extension",
      "Delimiter": " ",
      "Separator": "=",
      "Quote": "\"",
      "KeepSource": false
    },
    {
      "Type": "processor_regex",
      "SourceKey": "time",
      "Regex": "^([A-Z][a-z]{2}\\s+\\d{1,2}\\s+\\d{2}:\\d{2}:\\d{2})\\s+(\\S+)\\s+CEF:(\\d+)",
      "Keys": ["Time","Host","CEF_Version"]
    }
  ]
}

Parsed CEF logs are transformed into a structured JSON format suitable for indexing and analysis.

Palo Alto Networks Integration

Palo Alto NGFW forwards logs via Syslog. After receiving the raw CSV‑style logs, LoongCollector uses SPL parse‑csv to map fields to a structured schema.

* |parse-csv content as FUTURE_USE, Receive_Time, Serial_Number, Type, Threat_Content_Type, FUTURE_USE_1, Generated_Time, Source_Address, Destination_Address, NAT_Source_IP, NAT_Destination_IP, Rule_Name, Source_User, Destination_User, Application, Virtual_System, Source_Zone, Destination_Zone, Inbound_Interface, Outbound_Interface, Log_Action, FUTURE_USE_2, Session_ID, Repeat_Count, Source_Port, Destination_Port, NAT_Source_Port, NAT_Destination_Port, Flags, IP_Protocol, Action, URL_Filename, Threat_ID, Category, Severity, Direction, Sequence_Number, Action_Flags, Source_Location, Destination_Location, FUTURE_USE_3, Content_Type, PCAP_ID, File_Digest, Cloud, URL_Index, User_Agent, File_Type, X_Forwarded_For, Referer, Sender, Subject, Recipient, Report_ID, Device_Group_Hierarchy_Level_1, Device_Group_Hierarchy_Level_2, Device_Group_Hierarchy_Level_3, Device_Group_Hierarchy_Level_4, Virtual_System_Name, Device_Name, FUTURE_USE_4, Source_VM_UUID, Destination_VM_UUID, HTTP_Method, Tunnel_ID_IMSI, Monitor_Tag_IMEI, Parent_Session_ID, Parent_Start_Time, Tunnel_Type, Threat_Category, Content_Version, FUTURE_USE_5, SCTP_Association_ID, Payload_Protocol_ID, HTTP_Headers, URL_Category_List, Rule_UUID, HTTP_2_Connection, Dynamic_User_Group_Name, XFF_Address, Source_Device_Category, Source_Device_Profile, Source_Device_Model, Source_Device_Vendor, Source_Device_OS_Family, Source_Device_OS_Version, Source_Hostname, Source_MAC_Address, Destination_Device_Category, Destination_Device_Profile, Destination_Device_Model, Destination_Device_Vendor, Destination_Device_OS_Family, Destination_Device_OS_Version, Destination_Hostname, Destination_MAC_Address, Container_ID, POD_Namespace, POD_Name, Source_External_Dynamic_List, Destination_External_Dynamic_List, Host_ID, Serial_Number_2, Domain_EDL, Source_Dynamic_Address_Group, Destination_Dynamic_Address_Group, Partial_Hash, High_Resolution_Timestamp, Reason, Justification, A_Slice_Service_Type, Application_Subcategory, Application_Category, Application_Technology, Application_Risk, Application_Characteristic, Application_Container, Tunneled_Application, Application_SaaS, Application_Sanctioned_State, Cloud_Report_ID, Cluster_Name, Flow_Type | project-away content

The resulting structured logs can be indexed and visualized in SLS.

Summary

LoongCollector’s practice shows that unified multi‑source log collection requires format compatibility and extensibility. By standardizing firewall logs from WAF, FortiGate, and Palo Alto, it eliminates data silos, improves threat analysis and compliance reporting, and can be extended to other security scenarios in cloud‑native environments.