Now that we know how to write basic coroutines and which pitfalls we have to take care of, how about writing a function where remembering the state is required? That is, a function that always gives you the average value of all sent values. This is one of the few cases where it is still relatively safe and useful to combine the coroutine and generator syntax: >>> @coroutine ... def average(): ... count = 1 ... total = yield ... while True:
... total += yield total / count ... count += 1 >>> averager = average() >>> averager.send(20) 20.0 >>> averager.send(10) 15.0 >>> averager.send(15) 15.0 >>> averager.send(-25) 5.0
It still requires some extra logic to work properly though. To make sure we don't divide by zero, we initialize the count to 1. After that, we fetch our first item using yield, but we don't send any data at that point because the first yield is the primer and is executed before we get the value. Once that's all set up, we can easily yield the average value while summing. Not all that bad, but the pure coroutine version is slightly simpler to understand since we don't have to worry about priming:
>>> @coroutine ... def print_(formatstring): ... while True: ... print(formatstring % (yield)) >>> @coroutine ... def average(target): ... count = 0 ... total = 0 ... while True: ... count += 1 ... total += yield ... target.send(total / count) >>> printer = print_('%.1f') >>> averager = average(printer) >>> averager.send(20) 20.0 >>> averager.send(10) 15.0 >>> averager.send(15) 15.0 >>> averager.send(-25) 5.0
As simple as it should be, just keeping the count and the total value and simply send the new average for every new value.
Another nice example is itertools.groupby, also quite simple to do with coroutines. For comparison, we will once again show both the generator coroutine and the pure coroutine version:
>>> @coroutine ... def groupby():
... # Fetch the first key and value and initialize the state ... # variables
... key, value = yield
... old_key, values = key, [] ... while True:
... # Store the previous value so we can store it in the ... # list
... old_value = value ... if key == old_key: ... key, value = yield ... else:
... key, value = yield old_key, values ... old_key, values = key, []
... values.append(old_value) >>> grouper = groupby() >>> grouper.send(('a', 1)) >>> grouper.send(('a', 2)) >>> grouper.send(('a', 3)) >>> grouper.send(('b', 1)) ('a', [1, 2, 3]) >>> grouper.send(('b', 2)) >>> grouper.send(('a', 1)) ('b', [1, 2]) >>> grouper.send(('a', 2)) >>> grouper.send((None, None)) ('a', [1, 2])
As you can see, this function uses a few tricks. We store the previous key and value so that we can detect when the group (key) changes. And that is the second issue; we obviously cannot recognize a group until the group has changed, so only after the group has changed will the results be returned. This means that the last group will be sent only if a different group is sent after it, hence the (None, None). And now, here is the pure coroutine version:
>>> @coroutine ... def print_(formatstring): ... while True: ... print(formatstring % (yield)) >>> @coroutine ... def groupby(target): ... old_key = None ... while True:
... key, value = yield ... if old_key != key:
... # A different key means a new group so send the ... # previous group and restart the cycle.
... if old_key and values:
... target.send((old_key, values)) ... values = []
... old_key = key ... values.append(value)
>>> grouper = groupby(print_('group: %s, values: %s')) >>> grouper.send(('a', 1)) >>> grouper.send(('a', 2)) >>> grouper.send(('a', 3)) >>> grouper.send(('b', 1)) group: a, values: [1, 2, 3] >>> grouper.send(('b', 2)) >>> grouper.send(('a', 1)) group: b, values: [1, 2] >>> grouper.send(('a', 2)) >>> grouper.send((None, None)) group: a, values: [1, 2]
While the functions are fairly similar, the pure coroutine version is, once again, quite a bit simpler. This is because we don't have to think about priming and values that might get lost.
Summary
This chapter showed us how to create generators and both the strengths and weaknesses that they possess. Additionally, it should now be clear how to work around their limitations and the implications of doing so.
While the paragraphs about coroutines should have provided some insights into what they are and how they can be used, not everything has been shown yet. We saw the constructs of both pure coroutines and coroutines that are generators at the same time, but they are still all synchronous. The coroutines allow sending the results to many other coroutines, therefore effectively executing many functions at once, but they can still freeze Python completely if an operation turns out to be blocking. That's where our next chapter will help.
Python 3.5 introduced a few useful features, such as the async and await statements. These make it possible to make coroutines fully asynchronous and non-blocking, whereas this chapter uses the basic coroutine features that have been available since Python 2.5.
The next chapter will expand on the newer features, including the asyncio module. This module makes it almost simple to use coroutines for asynchronous I/O to endpoints such as TCP, UDP, files, and processes.