Configuring Logback as the Logging Framework for Apache Flink

Flink officially recommends using Logback instead of the default Log4j, and this guide walks through the complete migration process.

First, add Logback dependencies ( logback-core, logback-classic) and log4j-over-slf4j to the project's pom.xml, and exclude Log4j and slf4j-log4j12 from all Flink dependencies to prevent the old logger from being used.

<properties>
  <logback.version>1.2.3</logback.version>
  <log4j-over-slf4j.version>1.7.25</log4j-over-slf4j.version>
</properties>

<dependencies>
  <dependency>
    <groupId>ch.qos.logback</groupId>
    <artifactId>logback-core</artifactId>
    <version>${logback.version}</version>
  </dependency>

  <dependency>
    <groupId>ch.qos.logback</groupId>
    <artifactId>logback-classic</artifactId>
    <version>${logback.version}</version>
  </dependency>

  <dependency>
    <groupId>org.slf4j</groupId>
    <artifactId>log4j-over-slf4j</artifactId>
    <version>${log4j-over-slf4j.version}</version>
  </dependency>
</dependencies>

Next, exclude the transitive Log4j and slf4j-log4j12 artifacts from Flink modules:

<dependency>
  <groupId>org.apache.flink</groupId>
  <artifactId>flink-java</artifactId>
  <version>${flink.version}</version>
  <exclusions>
    <exclusion>
      <groupId>log4j</groupId>
      <artifactId>*</artifactId>
    </exclusion>
    <exclusion>
      <groupId>org.slf4j</groupId>
      <artifactId>slf4j-log4j12</artifactId>
    </exclusion>
  </exclusions>
</dependency>

<dependency>
  <groupId>org.apache.flink</groupId>
  <artifactId>flink-streaming-java_${scala.bin.version}</artifactId>
  <version>${flink.version}</version>
  <exclusions>
    <exclusion>
      <groupId>log4j</groupId>
      <artifactId>*</artifactId>
    </exclusion>
    <exclusion>
      <groupId>org.slf4j</groupId>
      <artifactId>slf4j-log4j12</artifactId>
    </exclusion>
  </exclusions>
</dependency>

Remove the original log4j and slf4j-log4j12 JARs from Flink's lib directory and copy the Logback JARs ( logback-core, logback-classic, log4j-over-slf4j) there.

Update the Logback configuration files ( logback.xml, logback-console.xml, logback-yarn.xml). The example below configures a RollingFileAppender that rolls logs daily, limits each file to 64 MB, and keeps the last 10 days of logs.

<appender name="file" class="ch.qos.logback.core.rolling.RollingFileAppender">
  <file>${log.file}</file>
  <rollingPolicy class="ch.qos.logback.core.rolling.TimeBasedRollingPolicy">
    <fileNamePattern>${log.file}.%d{yyyy-MM-dd}.%i</fileNamePattern>
    <timeBasedFileNamingAndTriggeringPolicy class="ch.qos.logback.core.rolling.SizeAndTimeBasedFNATP">
      <maxFileSize>64MB</maxFileSize>
    </timeBasedFileNamingAndTriggeringPolicy>
    <maxHistory>10</maxHistory>
  </rollingPolicy>
  <encoder>
    <charset>UTF-8</charset>
    <pattern>%d{yyyy-MM-dd HH:mm:ss.SSS} [%thread] %-5level %logger{60} %X{sourceThread} - %msg%n</pattern>
  </encoder>
</appender>

If a dedicated log‑monitoring system (e.g., ELK) is unavailable, you can add an SMTPAppender to email error‑level logs:

<appender name="email" class="ch.qos.logback.classic.net.SMTPAppender">
  <smtpHost>smtp.163.com</smtpHost>
  <smtpPort>25</smtpPort>
  <username>some_username</username>
  <password>some_password</password>
  <to>[email protected]</to>
  <from>[email protected]</from>
  <subject>Flink Error Log</subject>
  <asynchronousSending>false</asynchronousSending>
  <SSL>false</SSL>
  <STARTTLS>true</STARTTLS>
  <layout class="ch.qos.logback.classic.html.HTMLLayout">
    <pattern>%d{yyyy-MM-dd HH:mm:ss.SSS} [%thread] %-5level %logger{60} %X{sourceThread} - %msg%n</pattern>
  </layout>
  <filter class="ch.qos.logback.classic.filter.LevelFilter">
    <level>ERROR</level>
  </filter>
  <cyclicBufferTracker class="ch.qos.logback.core.spi.CyclicBufferTracker">
    <bufferSize>50</bufferSize>
  </cyclicBufferTracker>
</appender>

Don’t forget to place the mail library JAR (e.g., mail.jar) in the lib directory and reference the new appenders in the root logger:

<root level="INFO">
  <appender-ref ref="file"/>
  <appender-ref ref="email"/>
</root>