How to remove duplicate elements from a List in Python
Three ways in which Lists in Python can be deduplicated.
A list data structure is capable of storing elements of different data types and multiple occurrences. In some cases, lists need to be deduplicated, which means we have to remove copies of elements from the data structure.
In this article, we will see how duplicates can be removed from a list using plain python and numpy
1. Using Set¶
Set is made of only unique elements, duplicate insertions are ignored. In this approach, first the list is typecasted to a set and then typecasted back into a list.
duplicate_list = [56, 4, 81, 56, 9, 4] cleaned_list = list(set(duplicate_list)) print(cleaned_list)
[56, 81, 4, 9]
2. Using dict.fromkeys()¶
Using this approach, order can be preserved.
dict in python has a method
fromkeys() which takes in an iterable of hashable objects and creates a dictionary from them, while setting all values as None.
duplicate_list = [56, 4, 81, 56, 9, 4] cleaned_list = list(dict.fromkeys(duplicate_list)) print(cleaned_list)
[56, 4, 81, 9]
3. Using numpy.unique(duplicate_list)¶
Numpy is known for it’s versatility in dealing with array operations. Using
numpy.unique() does not preserve order rather it sorts the array in ascending order.
import numpy as np duplicate_list = [56, 4, 81, 56, 9, 4] cleaned_list = list(np.unique(dupllicate_list)) print(cleaned_list)
[4, 9, 56, 81]
FREE VS Code / PyCharm Extensions I Use
✅ Write cleaner code with Sourcery, instant refactoring suggestions: Link*
PySaaS: The Pure Python SaaS Starter Kit
🚀 Build a software business faster with pure Python: Link*