On the second day of
EnHackathon, I continued to work on the outstanding item
from my last blog post on
After I submitted the pull request for bpo-38314,
I was told by the maintainer that
asyncio is trying to transition from using
unittest.mock based testing to more functional style testing, which
verifies the behaviour as the user would see it.
The only issue is that this kind of functional testing infrastructure doesn’t
yet exist for the
UNIX transport pipes which I was working on: this turned the
task from a very simple method add to a more difficult piece of testing
In order to write a functional test for my pipe method, I would have to actually string together a pipe between a pair of input and output and verify its behaviour. This requires a more in-depth understanding of how pipes work.
UNIX system, most things are represented as “file objects”. Things that
can be read from and written to are obviously analogous to files, but this
metaphor is stretched.
This includes the idea of a “socket”, where a connection to an external service like a network or server can be “written to” or “read from” the file object on disk. This has the effect of sending and receiving data to and from the server.
The “file object” analogy even extends to pipes. A pipe is a file object
representing “connection” between two endpoints; typically one endpoint reads,
and the other writes. Both see the pipe as a “file”, however what is actually
being done is a transfer of information between processes on the system (like
a client and server, or the
stdout of some command and the
stdin of a
To activate a pipe, both the read endpoint and the write endpoint must be opened.
asyncio module provides a wrapper around pipes when they’re attached to
readers on the event loop. When you’ve registered the read (or write) endpoint
of the pipe to the loop, you’re given back an object which represents the
transport object provides methods to pause/resume reading, or in the case
of bpo-38314 a new method
is_reading() to check
it it is actually able to read data across the pipe.
Getting to even that rudimentary understanding took a lot of trial an error.
A few of the choice mistakes I made throughout the day, or particular gotchas of
asyncio event loops:
- Opening a pip file-object in read/write mode blocks until the other endpoint
is opened: this must be done across threads, and can’t be done with
asyncdue to the blocking nature (pipes only really make sense across threads anyway)
- Closing some objects requires information in the buffer to be flushed: if
they’re attached to the event loop this can only be done whilst it is running!
transportobjects won’t happen unless the event loop runs the queued callbacks: this can be done with the internal testing method on
asyncand threading is hard: a lot of the issues I found with reading from my pipe came from the fact that the
asyncevent loop, the write endpoint handle and the read endpoint handle were all on different threads (the main thread and two I opened with
However, the process of contributing a piece of reusable infrastructure off the back of what should have a been a simple two-line method was rewarding.
Hopefully, soon the PR will be ready to be merged in, and alongside my simple
method will be a whole new functional test infrastructure for testing
UNIX pipe objects in