How to Remove Duplicate Strings and Objects from Java Lists Efficiently

1. Remove duplicate strings from a List

public List<String> removeStringListDupli(List<String> stringList) {
    Set<String> set = new LinkedHashSet<>();
    set.addAll(stringList);
    stringList.clear();
    stringList.addAll(set);
    return stringList;
}

Java 8 alternative:

List<String> unique = list.stream().distinct().collect(Collectors.toList());

2. Remove duplicate objects

Define a Person class with id and name, and override equals() and hashCode() so that List.contains() can identify duplicates.

public class Person {
    private Long id;
    private String name;
    // constructors, getters, setters omitted for brevity
    @Override
    public boolean equals(Object o) {
        if (this == o) return true;
        if (o == null || getClass() != o.getClass()) return false;
        Person person = (Person) o;
        return id.equals(person.id) && name.equals(person.name);
    }
    @Override
    public int hashCode() {
        int result = id.hashCode();
        result = 31 * result + name.hashCode();
        return result;
    }
}

Example deduplication using a manual contains check:

Person p1 = new Person(1L, "jack");
Person p2 = new Person(3L, "jack chou");
Person p3 = new Person(2L, "tom");
Person p4 = new Person(4L, "hanson");
Person p5 = new Person(5L, "胶布虫");
List<Person> persons = Arrays.asList(p1, p2, p3, p4, p5, p5, p1, p2, p2);
List<Person> personList = new ArrayList<>();
persons.stream().forEach(p -> {
    if (!personList.contains(p)) {
        personList.add(p);
    }
});
System.out.println(personList);

3. Remove duplicates based on a specific field (id)

Using a TreeSet with a comparator that compares the id:

public static List<Person> removeDupliById(List<Person> persons) {
    Set<Person> personSet = new TreeSet<>((o1, o2) -> o1.getId().compareTo(o2.getId()));
    personSet.addAll(persons);
    return new ArrayList<>(personSet);
}

Java 8 stream version:

import static java.util.Comparator.comparingLong;
import static java.util.stream.Collectors.collectingAndThen;
import static java.util.stream.Collectors.toCollection;

List<Person> unique = persons.stream().collect(
    collectingAndThen(
        toCollection(() -> new TreeSet<>(comparingLong(Person::getId))),
        ArrayList::new)
);

Generic predicate for distinct by key:

public static <T> Predicate<T> distinctByKey(Function<? super T, Object> keyExtractor) {
    Map<Object, Boolean> map = new ConcurrentHashMap<>();
    return t -> map.putIfAbsent(keyExtractor.apply(t), Boolean.TRUE) == null;
}

// usage
persons.stream()
       .filter(distinctByKey(p -> p.getId()))
       .forEach(System.out::println);

http://stackoverflow.com/questions/30745048/how-to-remove-duplicate-objects-from-java-arraylist http://blog.csdn.net/growing_tree/article/details/46622579