Automated Mobile App Testing: Transition from Calabash to Appium with BDD Integration

Testing is a critical quality‑assurance activity in mobile app development. For Meituan‑Dianping’s overseas travel apps, manual smoke and regression tests were costly and error‑prone, prompting the team to automate these stable test cases.

Solution selection criteria focused on platform support, stability, maintenance cost, and extensibility. Because maintenance cost was paramount, the team chose Behavior‑Driven Development (BDD) over Test‑Driven Development (TDD) to keep code‑level effort low while writing readable, natural‑language test cases.

Initially the team experimented with Calabash , which satisfied BDD and cross‑platform needs. A sample Calabash scenario looked like:

Scenario: 首页
    Then I press "上海"
    When I press view with id "city"
    Then I see "海外"
    When I press "海外"
    And I press view with id "start_search"
    When I enter "东京" into input field number 1
    Then I press list item number 1
    Then I see "东京"
    When I press "美食"
    ...

However, Calabash suffered from limited community support, Android‑only low‑cost integration, and iOS‑side difficulties (no element IDs, high source‑code integration effort). The team therefore abandoned Calabash.

The next iteration combined Appium with Cucumber (the BDD engine behind Calabash). Using Appium’s Ruby library, the team created a unified project structure where common steps reside in common_steps.rb and platform‑specific configurations are defined in cucumber.yml:

# config/cucumber.yml
---
ios: IDEVICENAME='ios'
android: IDEVICENAME='android'

Platform selection is handled in env.rb:

class AppiumWorld
end
if ENV['IDEVICENAME']=='android'
  caps = Appium.load_appium_txt file: File.expand_path("./../android/appium.txt", __FILE__), verbose: true
elsif ENV['IDEVICENAME']=='ios'
  caps = Appium.load_appium_txt file: File.expand_path("./../ios/appium.txt", __FILE__), verbose: true
else
  caps = Appium.load_appium_txt file: File.expand_path('./', __FILE__), verbose: true
end
Appium::Driver.new(caps)
Appium.promote_appium_methods AppiumWorld::World do
  AppiumWorld.new
end

Dependencies include:

gem 'appium_lib',         '~> 9.4.2'
gem 'rest-client',       '~> 2.0.2'
gem 'rspec',             '~> 3.5.0'
gem 'cucumber',          '~> 2.4.0'
gem 'rspec-expectations','~> 3.5.0'
gem 'spec',              '~> 5.3.4'
gem 'selenium-cucumber','~> 3.1.5'

With this setup, the team could run tests for Android or iOS via cucumber -p android or cucumber -p ios. The new solution brings several advantages:

Improved stability: Appium avoids the ID‑lookup failures that plagued Calabash on certain Samsung devices.

Reduced iOS integration cost: only a re‑signed IPA is needed, no source‑code changes.

Richer element‑location strategies (ID, class, name, XPath), mitigating the lack of IDs on iOS.

Seamless migration of existing Calabash scripts to the new framework while preserving BDD syntax.

Easy Jenkins integration and HTML report generation via Cucumber’s --format html --out reports.html option.

Automation run results on the overseas travel line show that the total execution time for both Meituan and Dianping apps is about 20 minutes, significantly lowering manual effort. Although the new suite’s execution time per run is slightly higher than the original Calabash suite, the relationship between test count and duration is not linear, and overall productivity gains are evident.

Issues and outlook

Two main challenges remain:

Scrolling vs. swiping: Switching from scroll_uiselector to UIAutomation2’s swipe doubled success rates and halved execution time.

Scenario granularity: Determining optimal Scenario split to maximize reuse across test cases.

Future work includes automatic cloud‑test triggering via Jenkins pipelines and fostering a “everyone is a test engineer” culture.

References: Appium documentation, appium/ruby_lib docs, selenium‑cucumber‑ruby canned steps.

Author: Li Cheng, Senior Android Engineer in Meituan‑Dianping’s overseas travel R&D group, with experience in cross‑platform mobile development and automated testing.