Learn With Me: Elixir - Elixir 1.8 (#52)

Around the time I wrote this, Elixir 1.8 had recently been released. I was curious to see if anything significant had changed in what I've learned so far. There were some things that had changed, but nothing that was terribly significant to me.

I did upgrade to Elixir 1.8, and I'll be using that from here on forward until a newer version comes out. I plan on silently upgrading to patch version updates, which should mainly be bug fixes.

I'm going to briefly talk about the changes in version 1.8. If you want to see the details, look at the Elixir 1.8 changelog.

Custom Struct Inspections

There's something in Elixir called the Inspect protocol, which allows data to be printed out using the inspect/2 function. By default, this will print all the fields in a struct. You can now customize which fields are printed and which are not. This is a nice feature if your struct contains sensitive data and you don't want your sensitive data to appear in places like log files or error information. You can use the @derive keyword to control which fields are displayed, using the only: option to specify which fields should be displayed or the except: option to specify which fields should be hidden.

Here's an example.

defmodule RegistrationInfo do
	@derive {Inspect, only: [:username}}
	defstruct [:username, :email, :password]
end

I suspect this will mean more to me after I learn about the Inspect protocol and indeed protocols in general, but it's nice to know that this can be done.

Date and Time Changes

I've noticed that it seems that every major version of Elixir since 1.3 has been making improvements to the date and time functionality, and this release is no exception. The DateTime type can now make use of time zone behaviors to become time-zone-aware. I'm unclear about exactly what those behaviors are, but it's good to know that this is being improved. Elixir itself is only aware of the UTC time zone, but the developer can plug in a time zone database and Elixir will use that information. It would not surprise me to see packages in the future that contain time zone information.

Also, a variety of other helpful functions have been added to the Date and Time modules.

Performance Improvements

Elixir has made some performance improvements to both the speed at which it compiles and the speed of the code the compiler produces. None of these improvements are are totally revolutionary in nature, but are incremental improvements over the previous version. That's usually the nature of performance improvements. It's nice to see that the Elixir developers are working on this.

Enum Changes

Enum.into/2 was changed so that passing a non-empty list into it is deprecated. The changelog recommends using the Kernel.++/2 or Keyword.merge/2 (for keyword lists) functions instead. I'm not sure why this was done. The changelog talks about inconsistencies with maps, which I don't understand. In my look at Enum.into/2, I had no problem passing empty or non-empty maps, so I don't know what they are referring to.

Let's check out the changes. I'm going to repeat the same function call with an empty list that I had previously tried in the Enum.into/2 examples.

#Elixir 1.8 does not like me passing in a non-empty list
iex> Enum.into(1..10, ["a", "b", "c"])
warning: the Collectable protocol is deprecated for non-empty lists. The behaviour of things like Enum.into/2 or "for" comprehensions with an :into option is incorrect when collecting into non-empty lists. If you're collecting into a non-empty keyword list, consider using Keyword.merge/2 instead. If you're collecting into a non-empty list, consider concatenating the two lists with the ++ operator.
  (elixir) lib/collectable.ex:83: Collectable.List.into/1
  (elixir) lib/enum.ex:1227: Enum.into_protocol/2
  (stdlib) erl_eval.erl:670: :erl_eval.do_apply/6
  (elixir) src/elixir.erl:258: :elixir.eval_forms/4
  (iex) lib/iex/evaluator.ex:257: IEx.Evaluator.handle_eval/5

["a", "b", "c", 1, 2, 3, 4, 5, 6, 7, 8, 9, 10]
#Trying Kernel.++/2. I wasn't expecting that to work since 1..10 is a range, not a list
iex> 1..10 ++ ["a", "b", "c"]
** (ArgumentError) argument error
    :erlang.++(10, ["a", "b", "c"])
#Convert the range to a list and we can concatenate them, although I would have to reverse
#the order to match the first example.
iex> Enum.to_list(1..10) ++ ["a", "b", "c"]
[1, 2, 3, 4, 5, 6, 7, 8, 9, 10, "a", "b", "c"]
#An empty list is not a problem
iex> Enum.into(1..10, [])
[1, 2, 3, 4, 5, 6, 7, 8, 9, 10]
#Maps are not a problem
iex> Enum.into(%{name: "Bob", age: 32}, %{location: "home"})
%{age: 32, location: "home", name: "Bob"}
iex> Enum.into(%{name: "Bob", age: 32}, %{})
%{age: 32, name: "Bob"}

That change didn't make a lot of sense to me, but I'm sure I'm only seeing part of the picture.

Comprehension Changes

There have been some changes to comprehensions in Elixir 1.8. I'm going to discuss those changes in the following sections.

Changes to Into

You might recall in the section covering comprehensions I talked about how a comprehension can insert the results into a collection using into:. Well, the same change to Enum.into/2 has also been made here. You can no longer pass a non-empty list to the into: option.

I'm going to repeat the example from Lwm 21.

iex> for number <- [1, 2, 3], letter <- ["a", "b"], into: ["Bob", :ok, 2.3] , do: {letter, number}
warning: the Collectable protocol is deprecated for non-empty lists. The behaviour of things like Enum.into/2 or "for" comprehensions with an :into option is incorrect when collecting into non-empty lists. If you're collecting into a non-empty keyword list, consider using Keyword.merge/2 instead. If you're collecting into a non-empty list, consider concatenating the two lists with the ++ operator.
  (elixir) lib/collectable.ex:83: Collectable.List.into/1
  (stdlib) erl_eval.erl:670: :erl_eval.do_apply/6
  (stdlib) erl_eval.erl:438: :erl_eval.expr/5
  (stdlib) erl_eval.erl:122: :erl_eval.exprs/5
  (elixir) src/elixir.erl:258: :elixir.eval_forms/4

["Bob", :ok, 2.3, {"a", 1}, {"b", 1}, {"a", 2}, {"b", 2}, {"a", 3}, {"b", 3}]
iex> for number <- [1, 2, 3], letter <- ["a", "b"], into: [] , do: {letter, number}
[{"a", 1}, {"b", 1}, {"a", 2}, {"b", 2}, {"a", 3}, {"b", 3}]

Elixir 1.8 will still do what I wanted, but it complains about the non-empty list being deprecated. I suspect I would see the same deprecation message for anything that made use of the Collectable protocol. The version with the empty list works just fine.

Addition of a Reduce Operation

Elixir 1.8 has added a reduce: option to do a reduce as part of a comprehension. Here's what it looks like.

iex> for number <- 1..10, reduce: 0 do
...> sum -> sum + number
...> end
55

The value after the reduce: keyword is the initial accumulator value and a reduce function is required after the do keyword. Previously, you would have had to do the comprehension and then make a separate reduce call on the result of the comprehension, but now you can do it all together. It's probably more efficient that way.

The above example assigns each value in the range to the comprehension-scoped variable number and runs the reduce function for each individual value of number. Since that scoped variable with the current value already exists in a comprehension, the reduce function in the comprehension just needs to receive an accumulator value.

It looks like you can do multiple function clauses with pattern matching, since the reduce function should be a standard anonymous function. So I tried it out and it worked.

iex> for number <- 1..100, reduce: 0 do
...> sum when sum > 200 -> sum
...> sum -> sum + number
...> end
210

Let's try doing something fun that comprehensions make easier: creating combinations.

iex> for number1 <- 1..5, number2 <- 1..5, do: {number1, number2}
[
  {1, 1},
  {1, 2},
  {1, 3},
  {1, 4},
  {1, 5},
  {2, 1},
  {2, 2},
  {2, 3},
  {2, 4},
  {2, 5},
  {3, 1},
  {3, 2},
  {3, 3},
  {3, 4},
  {3, 5},
  {4, 1},
  {4, 2},
  {4, 3},
  {4, 4},
  {4, 5},
  {5, 1},
  {5, 2},
  {5, 3},
  {5, 4},
  {5, 5}
]
``

Now let's create those same combinations, but reduce each combination to a sum.

```elixir
iex> sum_list = for number1 <- 1..5, number2 <- 1..5, reduce: [] do
...> acc -> [number1 + number2 | acc]
...> end
[10, 9, 8, 7, 6, 9, 8, 7, 6, 5, 8, 7, 6, 5, 4, 7, 6, 5, 4, 3, 6, 5, 4, 3, 2]
iex> Enum.reverse(sum_list)
[2, 3, 4, 5, 6, 3, 4, 5, 6, 7, 4, 5, 6, 7, 8, 5, 6, 7, 8, 9, 6, 7, 8, 9, 10]

The result is in reverse order, but a call to Enum.reverse/1 takes care of that. Of course, the order may or may not matter depending on what the results are used for.

Comprehension Revelations

It was when playing with the reduce option in the comprehensions did it suddenly occur to me that in a comprehension everything after the do keyword is essentially a function body. This means that you don't have to put the entire expression on one line, but you can do multiple lines like with a function. In the same manner as a function, writing , do: indicates that the expression is going to be on the same line whereas just writing do indicates that more lines are coming and that they will be terminated with an end.

For example, here's a plain combination example using comprehension where each value in the final result is a single expression on the same line.

iex> for number1 <- 1..5, number2 <- 1..5, do: {number1, number2}
[
  {1, 1},
  {1, 2},
  {1, 3},
  {1, 4},
  {1, 5},
  {2, 1},
  {2, 2},
  {2, 3},
  {2, 4},
  {2, 5},
  {3, 1},
  {3, 2},
  {3, 3},
  {3, 4},
  {3, 5},
  {4, 1},
  {4, 2},
  {4, 3},
  {4, 4},
  {4, 5},
  {5, 1},
  {5, 2},
  {5, 3},
  {5, 4},
  {5, 5}
]

Here's an example of a comprehension where I've changed the value expression after the do to be on multiple lines. It creates a tuple containing the combination of two numbers along with the sum of those two numbers.

iex> for number1 <- 1..5, number2 <- 1..5 do
...> numberSum = number1 + number2
...> {number1, number2, numberSum}
...> end
[
  {1, 1, 2},
  {1, 2, 3},
  {1, 3, 4},
  {1, 4, 5},
  {1, 5, 6},
  {2, 1, 3},
  {2, 2, 4},
  {2, 3, 5},
  {2, 4, 6},
  {2, 5, 7},
  {3, 1, 4},
  {3, 2, 5},
  {3, 3, 6},
  {3, 4, 7},
  {3, 5, 8},
  {4, 1, 5},
  {4, 2, 6},
  {4, 3, 7},
  {4, 4, 8},
  {4, 5, 9},
  {5, 1, 6},
  {5, 2, 7},
  {5, 3, 8},
  {5, 4, 9},
  {5, 5, 10}
]

You can't use multiple function clauses with pattern matching in a comprehension unless you use the reduce option though. I tried and I just got an error message.

iex> for number1 <- 1..5, number2 <- 1..5 do
...> (number1, number2) when number1 > number2 -> {number1, number2, :greater}
...> (number1, number2) when number1 == number2 -> {number1, number2, :equal}
...> (number1, number2) -> {number1, number2, :less}
...> end
** (CompileError) iex:15: the do block was written using acc -> expr clauses but the :reduce option was not given

However, I see nothing that prevents you from calling such a function within the comprehension expression, so that's not such a big obstacle. Here's the same thing from above, but with putting the multiple clauses in another function and then calling it.

iex> create_number_tuple = fn
...> (number1, number2) when number1 > number2 -> {number1, number2, :greater}
...> (number1, number2) when number1 == number2 -> {number1, number2, :equal}
...> (number1, number2) -> {number1, number2, :less}
...> end
#Function<12.99386804/2 in :erl_eval.expr/5>
iex> for number1 <- 1..5, number2 <- 1..5 do
...> create_number_tuple.(number1, number2)
...> end
[
  {1, 1, :equal},
  {1, 2, :less},
  {1, 3, :less},
  {1, 4, :less},
  {1, 5, :less},
  {2, 1, :greater},
  {2, 2, :equal},
  {2, 3, :less},
  {2, 4, :less},
  {2, 5, :less},
  {3, 1, :greater},
  {3, 2, :greater},
  {3, 3, :equal},
  {3, 4, :less},
  {3, 5, :less},
  {4, 1, :greater},
  {4, 2, :greater},
  {4, 3, :greater},
  {4, 4, :equal},
  {4, 5, :less},
  {5, 1, :greater},
  {5, 2, :greater},
  {5, 3, :greater},
  {5, 4, :greater},
  {5, 5, :equal}
]

Comprehensions have certainly been like this all along, but I didn't realize it until I was playing around with the reduce option. Nothing I read had mentioned this. So playing around with Elixir is well worth it. It's been helping me to get new insights about how Elixir works.

List Changes

There were a couple new functions added to the List module. The following sections discuss those new functions.

List.myers_difference/3

The new List.myers_difference/3 function is similar to the List.myers_difference/2 function that we went over back in lwm 42. I recommend re-reading that section if you don't remember the details.

The difference between the two myer's difference functions is that List.myers_difference/2 only creates the myer's difference for the top-level list items and that List.myers_difference/3 will create nested myer's differences for lists that contain elements that in themselves can be diffed. The third parameter in List.myers_difference/3 will be the function that will create the nested diff.

Here's an example of a myer's difference using List.myers_difference/2.

iex> List.myers_difference([3, 4, -4, [3, 4, 5], 10], [3, 5, -4, [4, 5, 6], 10])
[
  eq: [3],
  del: [4],
  ins: [5],
  eq: [-4],
  del: [[3, 4, 5]],
  ins: [[4, 5, 6]],
  eq: '\n'
]

There are list elements nested inside the top-level list, but those are treated as whole blocks. We can use List.myers_difference/3 to create a diff for those nested lists as well by passing in List.myers_difference/2 as the function that will determine the nested diffs. Of course, the List.myers_difference/2 function will only work for nested list elements, so I'll have to wrap it in my own anonymous function that calls List.myers_difference/2 when both functions are lists and nil otherwise. Returning a nil indicates that no nested diff is available for these elements.

iex> List.myers_difference([3, 4, -4, [3, 4, 5], 10], [3, 5, -4, [4, 5, 6], 10], fn
...> (elem1, elem2) when is_list(elem1) and is_list(elem2) -> List.myers_difference(elem1, elem2)
...> (_, _) -> nil
...> end)
[
  eq: [3],
  del: [4],
  ins: [5],
  eq: [-4],
  diff: [del: [3], eq: [4, 5], ins: [6]],
  eq: '\n'
]

There you have it. The diff function will call List.myers_difference/2 when two lists are diffed, otherwise it just declines to provide a diff. The nested diff will appear under a ":diff" key in the resulting keyword list.

If I wanted to get really, fancy I could revise the function to use List.myers_difference/3 to create a recursive nested diffing that can diff any level of nesting, but that would require a named function to be able to make the recursive call. It would go something like this.

def diff_func(elem1, elem2) when is_list(elem1) and is_list(elem2), do: List.myers_difference(elem1, elem2, diff_func)
def diff_func(_, _), do: nil

Here's an example of it where I define a module inline in IEx.

iex> defmodule Diff do
...> def diff_func(elem1, elem2) when is_list(elem1) and is_list(elem2), do: List.myers_difference(elem1, elem2, &diff_func/2)
...> def diff_func(_, _), do: nil
...> end
{:module, Diff,
 <<70, 79, 82, 49, 0, 0, 5, 72, 66, 69, 65, 77, 65, 116, 85, 56, 0, 0, 0, 181,
   0, 0, 0, 17, 11, 69, 108, 105, 120, 105, 114, 46, 68, 105, 102, 102, 8, 95,
   95, 105, 110, 102, 111, 95, 95, 7, 99, ...>>, {:diff_func, 2}}
iex> List.myers_difference([3, 4, -4, [3, 4, 5], 10], [3, 5, -4, [4, 5, 6], 10], &Diff.diff_func/2)
[
  eq: [3],
  del: [4],
  ins: [5],
  eq: [-4],
  diff: [del: [3], eq: [4, 5], ins: [6]],
  eq: '\n'
]
iex> List.myers_difference([3, 4, -4, [3, [8, 9, 10], [5, 10, 20]], 10], [3, 5, -4, [4, [8, 9, 10], [6, 10, 20, 30]], 10], &Diff.diff_func/2)
[
  eq: [3],
  del: [4],
  ins: [5],
  eq: [-4],
  diff: [
    del: [3],
    ins: [4],
    eq: ['\b\t\n'],
    diff: [del: [5], ins: [6], eq: [10, 20], ins: [30]]
  ],
  eq: '\n'
]

I tried the previous example again to see if I still got the correct result and then I tried it on lists nested 3 deep. It looks like it worked without a problem. However, I hope never to have to deal with myer's diffs that are this complex.

List.improper?/1

The new List.improper?/1 function just tells us whether a list is an improper list (one whose tail is not a list) or not. I still don't understand why improper lists are even allowed or what the use case for them would be, but at least we can detect them now.

iex> proper_list = [1 | [2, 3]]
[1, 2, 3]
iex> improper_list = [1 | 2]
[1 | 2]
iex> List.improper?(proper_list)
false
iex> List.improper?(improper_list)
true

Other Stuff

Since I have not yet mastered Elixir, there are many changes that mean little to me because they involve aspects of Elixir I have not yet learned. There are also some new functions that I understand, but aren't interesting enough to cover here.

If you're interested in all the little details, please read the Elixir 1.8 changelog. There are lots of things in there I don't fully understand the meaning of, but they give me little hints as to what else is out there that I haven't learned about yet. It turns out that there is quite a lot I haven't learned about yet.