Learn With Me: Elixir - Protocols (#72)
You may remember from my earlier posts that I thought that behaviours and protocols both looked like interfaces, but I wasn't sure what the difference was. It turns out that they are both a kind of an interface. Whereas behaviours were interfaces that modules could implement, protocols are interfaces that data types can implement.
Allowing different implementations of a function for different data types allows for polymorphism in a functional language. This way a function could accept any data type that implemented a particular protocol, and it would be able to work with that data type. It wouldn't have to know anything else about that data type, allowing different data types to put their own spin on a piece of functionality.
A Protocol Usage Example
I think it will take an example to make clear how a protocol can be useful. Let's look at the Kernel.to_string/1 function. That function creates a string representation of a piece of data. Now, it would be a burden for Kernel.to_string/1 to know how to create a string representation for every possible type of data, and it can't possibly know what to do with user-defined data types (structs) that haven't been created yet. So it instead makes a call to a particular interface. That interface call is then directed to an interface implementation for that particular type.
In Elixir, an interface that can be implemented per data type is called a protocol. Like behaviours, protocols are declared separately and define functions that can be called for anything that implements the protocol. Then there are separate protocol implementations, which are applied to one or sometimes more specific types. These protocol implementations contain the code for each function in the protocol.
In the case of Kernel.to_string/1, it makes a call to a protocol called String.Chars. Many of the built-in types in Elixir have an implementation of String.Chars, but not all of them do.
Here's an example of calling Kernel.to_string/1 for various data types
iex> to_string(34)
"34"
iex> to_string(:ok)
"ok"
iex> to_string([1, 2, 3])
<<1, 2, 3>>
iex> to_string({:ok, 3})
** (Protocol.UndefinedError) protocol String.Chars not implemented for {:ok, 3}
(elixir) lib/string/chars.ex:3: String.Chars.impl_for!/1
(elixir) lib/string/chars.ex:22: String.Chars.to_string/1
Integers, atoms, and lists all have implementations of the String.Chars protocol, so Kernel.to_string/1 is able to return a string representation of these data types. Tuples, for some reason unknown to me, do not have an implementation of the String.Chars protocol, so Elixir throws an error indicating that the protocol was not defined for that type.
So not only can we obtain a string representation of Elixir built-in types without having to worry about the implementation details, but any structs defined in the future can provide their own implementation of String.Chars, allowing a string representation to be created when a struct is passed to Kernel.to_string/1. It's quite an elegant solution that allows loose coupling and future extensibility.
This also means that we can create implementations of protocols that are imported from external libraries, allowing us to create data structures that can be plugged into those libraries.
Protocol Declaration
A protocol is declared using the defprotocol keyword. This creates a protocol that can be implemented for a data type.
Here's an example. I created a Describe protocol that can provide a name and description for any data type that implements this protocol.
defprotocol Describe do
def name(item)
def description(item)
end
All functions in a protocol must have at least one parameter, and the first parameter in each function is always a variable representing an instance of the data type that implements the protocol. This is functional programming, not object-oriented programming, so we always need the data passed in as a parameter in order to do something with it. The first parameter in a protocol function would be roughly equivalent to "this" in many object-oriented languages: it contains the data that the function will use to do its work.
Protocol Implementation
A protocol is not implemented in a regular module like behaviors are, but they get their own special implementations, which as specified using the defimpl keyword. This creates a module-like structure (which could be a module under the surface for all I know) that implements the functions from the protocol declaration.
I'm going to give an example of some implementions of the Describe protocol declared above for two types: "Stroopwafel" and "Wrench". Assume that these are structs defined somewhere else.
defimpl Describe, for: Stroopwafel do
def name(_item), do: "stroopwafel"
def description(_item), do: "A sweet treat composed of a waffle with a layer of sweet syrup or caramel in the middle"
end
defimpl Describe, for: Wrench do
def name(_item), do: "wrench"
def description(_item), do: "A useful tool that allows you to turn threaded nuts or bolts"
end
Any function that applies the name/1 or description/1 protocol methods will get different results depending on whether a Stroopwafel or a Wrench struct is passed to it.
Note that these particular implementations didn't do anything with the first parameter because it was just returning static text. In protocol function implementations it's common to access struct properties or call functions in the struct module. If I wanted to, I could have altered the description to include the data contained in the struct.
Each data type can get its own implementation for a protocol and it's always defined outside of the data type module. Indeed, it's entirely possible to define a protocol implementation for a type in a library or even a built-in Elixir type in a completely separate file in a completely separate project. Here's an example of me providing an implementation of Describe for the built-in integer type. There's no need for me to have the Integer module source code: the implementation of the protocol can be provided independently of the type definition.
defimpl Describe, for: Integer do
def name(_integer), do: "integer"
def description(integer), do: "An integer with the value \"#{integer}\""
end
I decided to change the name of the first parameter in the function to make it more closely match what's being passed to it. There are no restrictions on renaming the parameters. The function names and the number of parameters must remain the same as the protocol, but the parameter names are not a integral part of the protocol declaration.
Now what happens if we want to provide the exact same implementation for multiple data types? It would certainly be quite tedious to write the same protocol implementations over and over. Well, Elixir has made this easy for us. You can combine multiple data types into one implementation by specifying a list of types after for: instead of a single data type.
Here's an example of such an implementation
defimpl Describe, for: [Carrot, Lettuce, Celery, Tomato] do
def name(vegetable), do: vegetable.name
def description(vegetable), do: "A healthy #{vegetable.name}, which goes great in a salad"
end
In this example, we retrieve the name of the vegetable from the data structure instead of hardcoding it in the protocol implementation. Which name is returned depends on the data structure that was passed into the function.
Now what happens in this scenario when we need to call a function in the data type module? Since we don't know which type will be passed to the protocol implementation function, we have to use the @for attribute to represent the module associated with that type. This allows us to make the same call to multiple modules that implement the same kind of function.
defimpl Describe, for: [Red, Green, Blue] do
def name(_color), do: @for.name()
def description(_color), do: "A color with the values @for.rgb()"
end
In the example above, I delegate the task of coming up with a name to the data type module, which is represented using @for. At runtime, @for will be replaced with Red, Green, or Blue depending on which data structure is being passed to the function. The description/1 function retrieves the color's RGB values from the corresponding color module.
You can create a protocol implementation for any data type, including built-in data types or your own types created using structs. As long as an implementation of the protocol for that same data type doesn't already exist, it should work.
In fact, it's possible to create an implementation that is applied to all data types using Any after for:. There aren't a lot of things an implementation can do for every possible data type, so you'll have to be careful with this. When using Any, you have to put the following line in the protocol declaration to indicate that an Any implementation is acceptable: @fallback_to_any true. When using Any, Elixir wants you to make sure you know what you are doing, so it makes you jump through some additional hoops.
Here's an example.
defimpl Describe, for: Any do
@fallback_to_any true
def name(_), do: "thing"
def description(_), do: "a thing that does stuff"
end
It seems to me that an "Any" implementation would be too generic to be useful most of the time, but I imagine that it might be useful in a few situations.
It's common for protocol implementations to just call a function in the module associated with that data type. So instead of implementing functionality itself, the function can delegate to a function in the data type module.
For example, in the Elixir source code, the String.Chars protocol implementation for a List and an Integer just calls functions in the List and Integer module.
defimpl String.Chars, for: List do
def to_string(charlist), do: List.to_string(charlist)
end
defimpl String.Chars, for: Integer do
def to_string(term) do
Integer.to_string(term)
end
end
You cannot have two partial defimpl definitions or there will be compile errors. Each defimpl definition must implement all the functions in the protocol. It is possible to have two defimpl protocol definitions for the same type, but they must be complete definitions. The second definition the compiler encounters overrides any previous definitions and the compiler displays a warning.
Note that the defimpl definition acts a lot like a module, making me think that it's probably just a special type of module. It doesn't have to just implement protocol functions: it can contain attributes and non-protocol-related functions. However, since it has no separate module identity, its functions can only be called within the context of a protocol. The only point to having those extra attributes and functions would be if they were used by the implementations of protocol functions.
Using a Protocol
So now that we know how to declare a protocol and define some implementations of that protocol, how do we actually use the protocol? Let's say you know your function receives some data that implements the Describe protocol, and you want to apply a Describe protocol function on that data. You don't care what that data is: you just want to get its name and description. It's actually quite simple: you just need to call the protocol method directly and pass it the necessary parameters.
Here's an example of an "inventory" function that receives an enumerable collection of data structures that implement the Describe protocol. It gathers the names and descriptions from each of the data structures and returns that information in the form of a tuple. That information can then in turn be used for display purposes or something.
def inventory(inventory_items) do
Enum.map(inventory_items, fn item -> {Describe.name(item), Describe.description(item)} end)
end
All we have to do is pass each item to Describe.[function] and Elixir takes care of the magic of calling the correct implementation. That's polymorphism in a functional language. If one of the items did not have an implementation of the Describe protocol, then an error would be thrown.
Protocol Structuring
Sometimes a protocol is included in its own file and sometimes it's included in the same file as another module.
Including a protocol in its own file makes sense if there's no clear place to put it and it's used in many places. For example, the String.Chars protocol in Elixir is defined in its own file (lib/string/chars.ex) along with some protocol implementations for built-in data types. This is because the String.Chars does not have strong associations with any particular type or module.
You can also include a protocol in the same file as the module that primarily uses it. For example, the Enumerable protocol is declared in Enum module source code, since the Enum module is completely oriented around enumerables, which are anything that implement the Enumerable protocol. Since the two are so closely linked, the protocol is contained in the same file as the Enum module.
The Enum module file also contains the implementations for lists, maps, and for some reason, functions (although the function implementation throws an error for almost all of the enumerable function implementations, this implementation must exist for a special reason). If I were to implement the Enumerable protocol for one of my own structs, I'd probably do it in the same file as the struct.
So do what makes the most sense for your situation and for the coding style you use. Most protocols are structured similarly to what I've mentioned here, but there is no well-defined standard that everybody follows.
A Protocol Example
Now it's time for an example project that uses protocols. I created an Elixir project that defines a protocol and then has several implementations of that protocol. You can find this project in the the "lwm 72 - Protocols" folder in the code examples in the Learn With Me: Elixir repository on Github. I'll be showing code from this project as I explain protocols along with command line output showing the final result.
Defining the Protocol and Implementations
Let's start off by taking a look at the protocol definition contained in lib/animal.ex. This is the Animal protocol, which can be implemented by any data structure that represents a particular animal.
defprotocol ProtocolExample.Animal do
@type food :: atom
@spec type(Animal.t()) :: String.t()
def type(animal)
@spec speak(Animal.t()) :: String.t() | nil
def speak(animal)
@spec likes?(Animal.t(), food()) :: boolean
def likes?(animal, food)
@spec name(Animal.t()) :: String.t()
def name(animal)
end
This protocol has four functions:
type/1, which returns a string representing the type of animalspeak/1, which returns a string representing the sound that the animal makes, ornilif the animal does not make a soundlikes?/2, which returns a boolean that indicates if the animal likes a particular food, where the food is represented by an atomname/1, which returns a string containing the name of this particular animal
I have defined three data structures, which represent different types of animals. Along with the module that corresponds to the data structure, each file also contains the Animal protocol implementation for that can be used by a function to generically interact with any type of animal.
Here's the contents of lib/cat.ex, which represents a cat.
#This module defines a Cat struct
defmodule ProtocolExample.Cat do
defstruct name: ""
@sound "meow"
@animal_type "cat"
@foods [:catfood, :chicken, :fish, :mice]
@spec foods() :: list(atom())
def foods(), do: @foods
@spec type() :: String.t()
def type(), do: @animal_type
@spec sound() :: String.t() | nil
def sound(), do: @sound
end
alias ProtocolExample.Animal
alias ProtocolExample.Cat
#This implements the Animal protocol for the Cat struct
defimpl Animal, for: Cat do
def type(_cat), do: Cat.type()
def speak(_cat), do: Cat.sound()
def likes?(_cat, food), do: food in Cat.foods()
def name(cat), do: cat.name
end
You can see that the protocol implementation mostly calls functions in the Cat module to do the work. The Cat module even contains a list of the foods the cat likes, making it easy to implement Animal.likes?/2. Cats have individual names that are assigned when the Cat struct is created.
Here's the code for the Sheep module in lib/sheep.ex.
#This module defines a Sheep struct
defmodule ProtocolExample.Sheep do
defstruct name: "", flock: ""
@sound "baaa"
@animal_type "sheep"
@foods [:grass, :hay, :alfalfa]
@spec foods() :: list(atom())
def foods(), do: @foods
@spec type() :: String.t()
def type(), do: @animal_type
@spec sound() :: String.t() | nil
def sound(), do: @sound
end
alias ProtocolExample.Animal
alias ProtocolExample.Sheep
#This implements the Animal protocol for the Sheep struct
defimpl Animal, for: Sheep do
def type(_sheep), do: Sheep.type()
def speak(_sheep), do: Sheep.sound()
def likes?(_sheep, food), do: food in Sheep.foods()
def name(sheep), do: sheep.name <> " of Flock #{sheep.flock}"
end
The implementation of Animal for Sheep is similar to the implementation for Cat, but the name/1 implementation is a little different. Each individual sheep has a name and a flock, and name/1 creates a string that contains the individual sheep's name along with the flock it belongs to. This makes more sense when you have a lot of sheep to keep track of and brings back memories when I was traveling in Iceland, which has fluffy sheep everywhere.
Here's the code for the Shark module in lib/shark.ex.
#This module defines a Shark struct
defmodule ProtocolExample.Shark do
defstruct tag_number: nil
@sound nil
@animal_type "shark"
@foods [:fish, :seals, :people]
@spec foods() :: list(atom())
def foods(), do: @foods
@spec type() :: String.t()
def type(), do: @animal_type
@spec sound() :: String.t() | nil
def sound(), do: @sound
end
alias ProtocolExample.Animal
alias ProtocolExample.Shark
#This implements the Animal protocol for the Shark struct
defimpl Animal, for: Shark do
def type(_shark), do: Shark.type()
def speak(_shark), do: Shark.sound()
def likes?(_shark, food), do: food in Shark.foods()
def name(shark), do: "##{shark.tag_number}"
end
Sharks don't say much, so speak/1 returns nil for sharks. Sharks don't usually have individual names either, so the implementation of name/1 just returns the shark's tag number. We're going to assume that the sharks we're managing have been individually tagged. Yes, I know that people are not actually a regular part of a shark's diet, and that some types of sharks like people more than others (I'm looking at you, bull and tiger sharks), but it's more fun this way.
Using the Protocol
So now we have a protocol and some implementations of that protocol. Let's do something with them!
The code that creates and displays information about animals can be found in lib/cli.ex. I got a little bit fancy with this example and I experimented with ANSI escape codes to create colored text in ANSI-aware terminals, which is most of them. Why did I do this? Well, I was looking around the IO module documentation and saw that there was an IO.ANSI module that contained the functionality to display colored text. I thought that this would also be a good opportunity to explore that module and see if I could get colored text to display. The documentation was a little vague, but once I figured out that all the IO.ANSI functions did was returns strings of ANSI escape sequences that I needed to append to the text I was displaying, it came together nicely.
I'm not going to go through all the code, since much of it controls how things are displayed on the screen, but I'll go over the more important parts.
The process/1 function receives the command line options, creates a list of animals, and then displays them.
def process(options) do
#Create the animals
animals = create_animals()
#Display the animal information
display_animals_info(animals, options.ansi)
end
There is only one command line option, "--noansi", which turns off ANSI output for those terminals that do not support it. Information about animals will be displayed with or without colors depending on whether ANSI output is enabled.
Here's the create_animals/0 function, which creates the list of animals.
@doc """
Creates a list of animals
These are structs that implement the Animal protocol
"""
@spec create_animals() :: list(animal())
def create_animals() do
[
%Cat{name: "Fluffy"},
%Shark{tag_number: "6433"},
%Cat{name: "Sparky"},
%Sheep{name: "Wooly", flock: "A"},
%Shark{tag_number: "432"},
%Sheep{name: "Tex", flock: "B"},
%Sheep{name: "Splat", flock: "A"}
]
end
A variety of Cat, Sheep, and Shark data structures are created.
Here's the display_animals_info/2 function, which receives a list of animals and displays information about them on the screen.
@doc """
Displays information on the screen describing the list of animals
"""
@spec display_animals_info(list(animal), boolean()) :: :ok
def display_animals_info(animals, use_ansi) do
animals
|> Enum.with_index(1)
|> Enum.each(fn animal -> display_animal_info(animal, use_ansi) end)
end
Pretty much all this does is to package each animal in a tuple with the corresponding index number and pass each tuple to a display_animal_info/1 that displays the information for a single animal. Here's what display_animal_info/2 looks like
@doc """
Displays information on the screen describing a single animal
"""
@spec display_animal_info({animal(), integer()}, boolean()) :: :ok
def display_animal_info({animal, number}, use_ansi) do
IO.puts "-------------"
IO.puts("Animal ##{number} is a #{Animal.type(animal)} named #{Animal.name(animal)}")
animal_sound = animal_sound_to_display_string(Animal.speak(animal))
IO.puts("#{Animal.name(animal)} says #{animal_sound}")
likes_mice = Animal.likes?(animal, :mice)
likes_grass = Animal.likes?(animal, :grass)
likes_seals = Animal.likes?(animal, :seals)
display_like_text(Animal.name(animal), "mice", likes_mice, use_ansi)
display_like_text(Animal.name(animal), "grass", likes_grass, use_ansi)
display_like_text(Animal.name(animal), "seals", likes_seals, use_ansi)
end
This function is, in my opinion, the most interesting one. It indicates which position the animal is in in the list and displays the name of the animal. After that, it outputs the sound the animal makes. I had to pass the sound to another function (animal_sound_to_display_string/1) because nil is a possible result and I want to make sure that nil is converted to a string that makes more sense.
The function then tests if the animal likes mice, grass, or seals, and outputs the result to the screen. If an animal likes the food, some of the text will be green, and if the animal does not like the food, the text will be red.
This code is able to display information about anything that implements the Animal protocol. The code displaying the information has no idea what the data type actually is, but can operate just fine as long as the data being passed to it implements the Animal protocol.
Running the Example
To run the example, build it and create an application package.
> mix escript.build
Compiling 6 files (.ex)
Generated protocol_example app
Generated escript protocol_example with MIX_ENV=dev
Then run the application. The command for MacOS or Linux systems is "./protocol_example" and for Windows systems it is "escript protocol_example". That will give you the following output.
If the output looks all messed up instead of showing colors, you're probably using a terminal that doesn't support ANSI. In that case, run "./protocol_example --noansi" and you'll see plain text without ANSI colors.
Conclusion
I can certainly see why protocols would be useful. I think that they may be even more useful than behaviours.
What I've read indicates that functions with pattern matching are preferred over protocols. So use protocols selectively. You should have a smaller, manageable number of protocols rather than large numbers of them.
Here are some links to some of the resources I used while researching protocols. You may find them useful.