Learn With Me: Elixir - Comprehensions (#21)

A comprehension is a type of statement that offers a shortcut for processing collections. If you've ever worked with Python, this will be very familiar, since Python also offers comprehensions as a major language feature.

Comprehensions allow collections of data to be transformed or filtered, similar to Enum.map/2 and Enum.filter/2, but with different syntax. In some cases, the comprehension syntax is easier to read and maintain than the equivalent operation using Enum functions.

Comprehension Syntax

A comprehension starts with a for and is followed by a generator expression. A generator expression shows what should be done with the data in a collection.

Here is a very simple example of a comprehension that iterates through a list but does nothing with that data. The result of each iteration is stored in a list, so the comprehension evaluates to a list

iex> for fruit <- ["apples", "guavas", "pinapples", "bananas"], do: fruit
["apples", "guavas", "pinapples", "bananas"]

In this example, the comprehension iterates over the list of fruit I provided, assigning each element to the variable fruit. Then the code after the do: keyword indicates how I am transforming the data. In this case, each element in the list is left as-is. The results of the comprehension are stored in a new list.

That's an interesting way of iterating, but this particular comprehension was quite useless. Let's do something more interesting.

Simple Data Transformation Example

Let's try a simple data transformation. We'll palindromize the items in the list of fruit by taking each fruit and concatenating it with a reversed version of the fruit.

iex> available_fruit = ["apples", "guavas", "pinapples", "bananas"]
["apples", "guavas", "pinapples", "bananas"]
iex> for fruit <- available_fruit, do: fruit <> String.reverse(fruit)
["applesselppa", "guavassavaug", "pinapplesselppanip", "bananassananab"]		

In this example, we iterate over each fruit, transforming it to make a palindrome. The fruit string is concatenated with a reversed version of the same string, resulting in a list of fruit palindromes.

List comprehension is not the only way to do this. We can also do it with Enum.map/2.

iex> Enum.map(available_fruit, fn fruit -> fruit <> String.reverse(fruit) end)
["applesselppa", "guavassavaug", "pinapplesselppanip", "bananassananab"]

As far as I know of, comprehensions are not any better or worse than the functions in the Enum module: the syntax is just different. You can just use whichever one you prefer.

Data Combination Example

We can create combinations of multiple collections using multiple variables in the generator expression. This is a feature that is quite unique to comprehensions.

iex> letters = ["a", "b", "c", "d"]
["a", "b", "c", "d"]
iex> numbers = [1, 2, 3, 4]
[1, 2, 3, 4]
iex> for letter <- letters, number <- numbers, do: {letter, number}
[
  {"a", 1},
  {"a", 2},
  {"a", 3},
  {"a", 4},
  {"b", 1},
  {"b", 2},
  {"b", 3},
  {"b", 4},
  {"c", 1},
  {"c", 2},
  {"c", 3},
  {"c", 4},
  {"d", 1},
  {"d", 2},
  {"d", 3},
  {"d", 4}
]		

The variable letter is assigned the first letter and there are then multiple iterations where number is assigned every number in the number collection, giving us four tuples where the first element is "a". Then the comprehension repeats with the letter assigned to "b". This continues until the comprehension goes through every possible combination in the collection. This is much like a nested loop construction in languages like C# or Javascript.

This isn't as easy to do with Enum.map/2

iex> Enum.map(letters, fn letter -> Enum.map(numbers, fn number -> {letter, number} end) end)
[
  [{"a", 1}, {"a", 2}, {"a", 3}, {"a", 4}],
  [{"b", 1}, {"b", 2}, {"b", 3}, {"b", 4}],
  [{"c", 1}, {"c", 2}, {"c", 3}, {"c", 4}],
  [{"d", 1}, {"d", 2}, {"d", 3}, {"d", 4}]
]		

We almost did it by nesting a call to Enum.map/2 inside another. However, we now have a list of lists of tuples instead of just a list of tuples. We can fix that by using Enum.flat_map/2 for the final result, which flattens the multiple lists and puts all the elements into a single list.

iex> Enum.flat_map(letters, fn letter -> Enum.map(numbers, fn number -> {letter, number} end) end)
[
  {"a", 1},
  {"a", 2},
  {"a", 3},
  {"a", 4},
  {"b", 1},
  {"b", 2},
  {"b", 3},
  {"b", 4},
  {"c", 1},
  {"c", 2},
  {"c", 3},
  {"c", 4},
  {"d", 1},
  {"d", 2},
  {"d", 3},
  {"d", 4}
]

You can see that the comprehension is much better way to express this sort of operation. It's just easier to read and understand.

There's no limit to how many variables or collections you use in comprehension. You can generate combinations from ten collections if you wanted to, or even use the same collection twice. Let's generate a list of possible coordinates using ranges describing the width and height of a game board.

iex> board_rows = 1..10
1..10
iex> board_columns = 1..6
1..6
iex> for row <- board_rows, column <- board_columns, do: {column, row}
[
  {1, 1},
  {2, 1},
  {3, 1},
  {4, 1},
  {5, 1},
  {6, 1},
  {1, 2},
  {2, 2},
  {3, 2},
  {4, 2},
  {5, 2},
  {6, 2},
  {1, 3},
  {2, 3},
  {3, 3},
  {4, 3},
  {5, 3},
  {6, 3},
  {1, 4},
  {2, 4},
  {3, 4},
  {4, 4},
  {5, 4},
  {6, 4},
  {1, 5},
  {2, 5},
  ...
]

I truncated the output, but 60 sets of coordinates are generated and put in a list.

Filtering Through Comprehensions

Data transformation is not the only thing that comprehensions can do. We can also do filtering in the comprehension. Here's the same combination example from the previous section, but with filtering added. In this example, we only use the odd numbers or any letter that isn't "b".

The filtering expression comes after the first comma, but before do:.

iex> for letter <- letters, number <- numbers, letter != "b" and rem(number, 2) == 1, do: {letter, number}
[{"a", 1}, {"a", 3}, {"c", 1}, {"c", 3}, {"d", 1}, {"d", 3}]

Any combination of variables that doesn't meet the conditions in the filter expression is ignored.

The same operation using the Enum module functions would be lengthier and very difficult to understand.

iex> Enum.flat_map(Enum.filter(letters, &(&1 != "b")), fn letter -> Enum.map(Enum.filter(numbers, &(rem(&1, 2) == 1)), fn number -> {letter, number} end) end)
[{"a", 1}, {"a", 3}, {"c", 1}, {"c", 3}, {"d", 1}, {"d", 3}]

Yikes! That's a lot harder to read!

Binary Comprehensions

There's a different syntax for binary comprehensions. You have to put << >> delimiters around the generator expression. In the following example, I convert bytes in a binary to a list of integers, while filtering out any byte value greater than or equal to 200.

binary_data = <<22, 34, 128, 204>>
iex> byte_list = for << byte <- binary_data >>, byte < 200, do: byte
[22, 34, 128]		

Strings are binaries too, so if we want to we can extract the bytes in a string by using a binary comprehension.

iex> string = "It's string time!"
"It's string time!"
iex> byte_list = for << byte <- string >>, do: byte + 100
[173, 216, 139, 215, 132, 215, 216, 214, 205, 210, 203, 132, 216, 205, 209, 201,
 133]

Comprehension Scoping

Any variables created in a comprehension are limited in scope to that comprehension and will not be available after the comprehension has finished. You will also not affect any variables of the same name in an outer scope.

iex> number = 8
8
iex> for number <- [1, 2, 3], letter <- ["a", "b"], do: {letter, number}
[{"a", 1}, {"b", 1}, {"a", 2}, {"b", 2}, {"a", 3}, {"b", 3}]
iex> number
8
iex> letter
** (CompileError) iex:5: undefined function letter/0

iex>		

Controlling What Comprehensions Return

A comprehension puts the generated values into a list by default, but you can provide a collection it can insert the generated elements into.

iex> for number <- [1, 2, 3], letter <- ["a", "b"], into: Map.new(), do: {letter, number}
%{"a" => 3, "b" => 3}

In this case, I caused the comprehension to insert key-value pairs into an empty map by creating an empty map after the into: keyword. There are only two keys in the final map because keys in a map are unique. The {"a", 1} pair was overwritten with {"a", 2}, which was in turn overwritten with {"a", 3}.

We can also insert generated data into an already-existing list.

iex> for number <- [1, 2, 3], letter <- ["a", "b"], into: ["Bob", :ok, 2.3] , do: {letter, number}
["Bob", :ok, 2.3, {"a", 1}, {"b", 1}, {"a", 2}, {"b", 2}, {"a", 3}, {"b", 3}]

Anything that implements the Collectable protocol can be used with the into: clause. I haven't yet learned about protocols or collectables, but from the context, it sounds like a common interface that can be used to insert data into a data structure, much like how an enumerable appears to implement an inferface that allows iteration over a data structure.

Comprehensions can also make use of pattern matching to extract values from elements in an enumerable. We'll discuss how pattern matching works in the next post.