Get Unique Values from List of Dictionary

We are given a list of dictionaries and our task is to extract the unique values from these dictionaries, this is common when dealing with large datasets or when you need to find unique information from multiple records. For example, if we have the following list of dictionaries: data = [{'name': 'Alice', 'age': 25}, {'name': 'Bob', 'age': 30}, {'name': 'Alice', 'age': 22}] then we want to extract the unique values (like Alice, Bob, 25, 30, 22).

Using a Set

set is a built-in Python data structure that only allows unique elements and we can use it to collect unique values from a list of dictionaries by iterating over the dictionaries and adding the values to the set.

data = [{'name': 'Alice', 'age': 25}, {'name': 'Bob', 'age': 30}, {'name': 'Alice', 'age': 22}]

# Using a set to collect unique values
res = set(val for d in data for val in d.values())

print(res)

{'Alice', 22, 30, 25, 'Bob'}

Explanation:

• This code uses a generator expression val for d in data for val in d.values() to iterate over each dictionary in the list (data), then over each value within the dictionary.
• The values are added to a set which ensures that only unique values are stored automatically removing any duplicates hence the result will be a set of unique values: {'Alice', 25, 30, 22, 'Bob'}.

Using itertools.chain()

itertools.chain() function is a great tool for flattening lists of dictionaries. By chaining the values of all dictionaries together we can then pass them into a set to ensure uniqueness, this method is effective for large datasets because itertools.chain() creates an iterator and doesn't require the entire list to be stored in memory.

from itertools import chain

data = [{'name': 'Alice', 'age': 25}, {'name': 'Bob', 'age': 30}, {'name': 'Alice', 'age': 22}]

# Flattening the list and using a set to get unique values
res = set(chain.from_iterable(d.values() for d in data))

print(res)

{'Alice', 22, 'Bob', 25, 30}

Explanation:

• chain.from_iterable() function is used to flatten the list of dictionaries by iterating over the values of each dictionary as it essentially "chains" all the values into a single iterable then result is then passed into a set to remove any duplicates ensuring that only unique values are retained.

Using Dictionary Keys

A we know that dictionaries automatically enforce uniqueness for keys we can take advantage of this property by converting the values from the list of dictionaries into dictionary keys, this guarantees that only unique values are retained.

data = [{'name': 'Alice', 'age': 25}, {'name': 'Bob', 'age': 30}, {'name': 'Alice', 'age': 22}]

# Using dictionary keys to collect unique values
res = list(dict.fromkeys(val for d in data for val in d.values()))

print(res)

['Alice', 25, 'Bob', 30, 22]

Explanation: 

• dict.fromkeys() method creates a dictionary where each value from the list of dictionaries is used as a key since dictionary keys are unique any duplicates are automatically removed.
• After generating the dictionary we convert it back to a list using list() which gives us the unique values: ['Alice', 25, 30, 22, 'Bob'].

Using a List

Another approach to collect unique values from a list of dictionaries is by using a list and manually checking if a value already exists before appending it while this method works, it can be less efficient than using a set due to the additional membership check for each value.

data = [{'name': 'Alice', 'age': 25}, {'name': 'Bob', 'age': 30}, {'name': 'Alice', 'age': 22}]

# Using a list and checking for uniqueness
res = []
for d in data:
    for val in d.values():
        if val not in res:
            res.append(val)

print(res)

['Alice', 25, 'Bob', 30, 22]

Explanation:

• The code iterates through each dictionary in the list data and then over each value within those dictionaries.
• Before adding a value to res, it checks whether the value is already present in the list. This ensures that only unique values are appended to the list.