Jekyll2021-10-06T10:40:37+00:00/feed.xmlEnHackathonBlog posts about contributing to Python, written by first-time contributors at the Ensoft office of Cisco.First CPython Core Sprint2020-10-24T00:00:00+00:002020-10-24T00:00:00+00:00/2020/10/24/LewisGaul<p>I was very excited to receive an email from Eric inviting me to join the week-long <a href="https://python-core-sprint-2020.readthedocs.io/">CPython sprint</a>, which took place this week.</p>
<p>This follows on from when I worked with Eric on the subinterpreters CPython project a year ago as part of EnHackathon - see my last EnHackathon <a href="https://enhackathon.github.io/2019/12/11/LewisGaul.html">blog post</a> for context.</p>
<p>Note: This post is cross-posted on my personal blog at <a href="http://lewisgaul.co.uk/blog/coding/2020/10/24/cpython-sprint/">http://lewisgaul.co.uk/blog/coding/2020/10/24/cpython-sprint/</a>.</p>
<h2 id="first-impressions">First Impressions</h2>
<p>We were using Discord as the platform for communication during the sprint, but when I signed in on Monday morning I found nothing was really happening. It turned out that the sprint was largely centered around US time, so I found myself working some unusual hours!</p>
<p>I did manage to get in a call with Tal before the official start of the sprint to talk about the new-contributor experience. Big thanks to him for committing time and effort to improve things in this area.</p>
<p>The sprint officially began with an opening presentation/meeting at 4pm. My favourite part of this was everyone giving a short introduction - there were a lot of familiar names that I wasn’t previously able to put a face to (or even pronounce in some cases!).</p>
<p>After the opening meeting everything really kicked off! There were lots of different discussion channels, which I described at one point as “a more interesting and active version of the python-dev mailing list”.</p>
<p>My first real involvement was in a meeting about how to improve the new-contributor experience, with Tal and a few others. During this meeting I felt incredibly welcomed and that my input was respected despite being a relative outsider to the group.</p>
<p>It was really quite amazing to be a part of such a dedicated group of people, who all do so much for Python in all kinds of different ways (e.g. Steering Council governance, packaging improvements, migrating to GitHub, research into potential major enhancements, adding support for new platforms, …).</p>
<h2 id="my-focus">My Focus</h2>
<p>I was allowed into the sprint as a ‘mentee’, with Eric being kind enough to act as my mentor. My focus was therefore to continue working on the subinterpreters project, with the high-level project aim of landing <a href="https://www.python.org/dev/peps/pep-0554/">PEP 554</a> in upcoming Python 3.10. The pending work before this is expected to be accepted is to make the GIL per-interpreter, which would allow running multiple interpreters in separate threads that are actually able to run in parallel on multiple cores!</p>
<p>I had an open <a href="https://github.com/python/cpython/pull/17575">PR</a> from EnHackathon last year, which was my focus when working on subinterpreters during the sprint. I expected to have time to take on more than this one issue, but I ended up spending a lot of time following/sitting in on other discussions and generally making the most of the event!</p>
<p>The PR required a bit of thought to be sure of the direction we wanted to go in - I had a couple of chats with Eric to make sure he was happy with the proposed solution.</p>
<p>The decision we made was:</p>
<ul>
<li><code class="language-plaintext highlighter-rouge">Py_FinalizeEx()</code> will implicitly clean up any running subinterpreters, but will emit a <code class="language-plaintext highlighter-rouge">ResourceWarning</code> if it has to do so.</li>
<li>An error will be returned if trying to call <code class="language-plaintext highlighter-rouge">Py_FinalizeEx()</code> from a subinterpreter.</li>
</ul>
<p>These two points address the main issue <a href="https://bugs.python.org/issue36225">bpo-36225</a>, but also address related issues <a href="https://bugs.python.org/issue38865">bpo-38865</a> and <a href="https://bugs.python.org/issue37776">bpo-37776</a> relating to the question of calling <code class="language-plaintext highlighter-rouge">Py_FinalizeEx()</code> from a subinterpreter.</p>
<p>In the future there should be no distinction between subinterpreters and the ‘main’ interpreter at the C level, and it should be possible to call <code class="language-plaintext highlighter-rouge">Py_FinalizeEx()</code> from any interpreter, which would implicitly clean up all other interpreters. However, this is not a priority in the short-term.</p>
<h2 id="improvements-for-new-contributors">Improvements for New Contributors</h2>
<p>One of the areas of the sprint I had a strong interest in was the discussion around how to improve the experience for new contributors to CPython. This is an area that has been acknowledged as lacking, and Tal has taken the lead on trying to improve things. I enjoyed getting involved in this as a fairly new contributor myself, meaning I felt I could actually provide some useful input!</p>
<p>Some of the points I had/agreed with were:</p>
<ul>
<li>Having visible metrics could be a good way to encourage newcomers to contribute in ways like trying to revive/close stale issues, e.g. graphs/counters of number of open issues/PRs.</li>
<li>Friendly responses and encouragement to newcomers would be great, even if in the form of a bot message.</li>
<li>We want newcomers to get responses, and to make them feel welcome we want them to feel like they’re on a team with the person responding. This may be easier/less daunting if the people on the ‘first line of response’ are relatively new themselves.</li>
<li>Support guidance/mentoring at all levels. The idea being for everyone to have someone to go to if they don’t have the answer to a question, e.g. perhaps I’d help someone with some basic stuff but then be able to ask someone else if things got out of my depth (which would also be great for learning at lower levels of experience).</li>
<li>Would be nice to have a support chat room intended for new contributors and mentors, hopefully providing quick answers to basic questions and building up more of a new-contibutor community.</li>
<li>Perhaps some public messaging to would-be-contributors could help to clearly acknowledge there’s a problem and talk through how we’re trying to do address it. It could also be worth being clear about the best ways for them to help out, if we decide on what that would be.</li>
</ul>
<p>Another thing that came up is that there are often ‘nitpick’ requests to fix formatting (in C, Python and RST files), which can add hours/days to the PR review process and be quite discouraging to newcomers. The expanded into a discussion about auto-formatting in the entirety of the CPython codebase! I look forward to seeing further discussion on that - I think it could set a great precedent for other large Python projects.</p>
<h2 id="other-interesting-things">Other Interesting Things</h2>
<p>There were lots of interesting discussions going on during the sprint. I couldn’t keep up with all of them, but there were a few that stood out to me as being especially exciting.</p>
<p>Alongside the work on multiple interpreters as a way to improve CPython’s ability to run on multiple cores (i.e. within a single process, rather than requiring multiprocessing), there was discussion around removing the GIL entirely. This follows on from Larry Hastings’s <a href="https://pythoncapi.readthedocs.io/gilectomy.html">Gilectomy project</a>, where it was determined the approach should be to use a tracing garbage collector instead of reference counting. This could in theory make CPython truly multi-core, and would likely require a major version bump to Python 4.0 due to the major changes required to the C API.</p>
<p>There was ongoing discussion around the proposed pattern matching feature, which is now split into three separate PEPs: <a href="https://www.python.org/dev/peps/pep-0634/">PEP 634</a>, <a href="https://www.python.org/dev/peps/pep-0635/">PEP 635</a> and <a href="https://www.python.org/dev/peps/pep-0636/">PEP 636</a>. I think this could be a very interesting new feature and I look forward to trying it out if/when it is collapsed into the main branch.</p>
<p>Mark Shannon sent a proposal promising a 5x speed-up of CPython over the next few years in four phases of 50% increase each. See the <a href="https://mail.python.org/archives/list/python-dev@python.org/message/RDXLCH22T2EZDRCBM6ZYYIUTBWQVVVWH/">email to python-dev</a> for details!</p>
<p>There was some early discussion around native support for ‘exception groups’, perhaps involving the introduction of <code class="language-plaintext highlighter-rouge">try: ... catch: ...</code> in Python - we’ll have to wait and see what comes out from this!</p>
<h2 id="overall-thoughts">Overall Thoughts</h2>
<p>This was the first ever virtual CPython sprint, and the general sentiment seemed to be that people enjoyed it more than expected! One of the clear positives was that it opened things up to people from all over the world, meaning this was the first sprint for quite a few of the participants.</p>
<p>Overall I enjoyed the sprint, and the highlight was just generally being involved in the community and discussions that were going on. I’m excited to see what some of the discussions will lead to, and would like to continue to support the multi-core projects and any new-contributor initiatives.</p>
<p>Once again thanks to Eric for having me along, to Tal for the work he’s doing to help new contributors like me, and to everyone else who contributes to Python as a volunteer!</p>Lewis GaulI was very excited to receive an email from Eric inviting me to join the week-long CPython sprint, which took place this week.Writing Native Apps in Python - Contributing to Beeware2019-12-11T00:00:00+00:002019-12-11T00:00:00+00:00/2019/12/11/CallumWard<p>After having taken a stab at fixing a couple of issues within the main <code class="language-plaintext highlighter-rouge">CPython</code>
implementation of Python, I decided whilst waiting for some of those pull
requests to be merged I might try to contribute to some other open source
projects in the Python ecosystem that are endorsed by the PSF.</p>
<p>Lewis suggested taking a look at <a href="https://beeware.org/">Beeware</a>, which recently
received a grant from the PSF Education group to improve support for Python on
Android, as part of their mission to enable cross-platform Python-based native
apps.</p>
<h2 id="getting-started">Getting Started</h2>
<p>There’s a lot of pieces to Beeware that are in early stages of development and
support, and should be ripe for further contribution. However, the first stages
revolved primarily around familiarising myself with the ecosystem to understand
which pieces would be most suitable.</p>
<p>Beeware has several constituent projects, some of the relevant ones are:</p>
<ul>
<li>
<p><a href="https://github.com/beeware/toga"><code class="language-plaintext highlighter-rouge">toga</code></a>: a cross-platform native widget
toolkit, that provides a high-level API wrapping the concept of an <code class="language-plaintext highlighter-rouge">App</code>,
allowing users to build GUI applications which resolve to using native widgets
on the target platform</p>
</li>
<li>
<p><a href="https://github.com/beeware/colosseum"><code class="language-plaintext highlighter-rouge">colosseum</code></a>: an implementation of the
CSS specification for resolving positions and locations of elements on
a canvas, used by <code class="language-plaintext highlighter-rouge">toga</code> for styling applications</p>
</li>
<li>
<p><a href="https://github.com/beeware/voc"><code class="language-plaintext highlighter-rouge">voc</code></a>: a transpiler which converts Python
bytecode allowing Python to be compiled into Java bytecode and run on the JVM,
enabling Android native apps</p>
</li>
<li>
<p><a href="https://github.com/beeware/batavia"><code class="language-plaintext highlighter-rouge">batavia</code></a>: an implementation of the
Python virtual machine in Javascript, enabling Python bytecode to run on the
browser or over <code class="language-plaintext highlighter-rouge">node.js</code></p>
</li>
</ul>
<p>The one which is most obviously approachable starting from a background of just
Python (as opposed to native widget libraries like Cocoa, GTK+, or Java and
Javascript) is <code class="language-plaintext highlighter-rouge">colosseum</code>, which is a complex problem but well-defined, and
comes with a suite of (currently failing) tests which verify it meets the CSS
specification.</p>
<h2 id="attempting-to-contribute">Attempting to Contribute</h2>
<p>As suggested on the <code class="language-plaintext highlighter-rouge">colosseum</code> <a href="https://colosseum.readthedocs.io/en/latest/how-to/contribute.html">contributing
guide</a>,
I selected a test (at random) from the list of “known failures” and removed it.
Re-running the tests, I took a look at what exactly fails as part of the test.</p>
<p>What I found was not particularly amenable to a contribution: it appears that an
implementation of an object expects a certain class which depending on the type
of the object would contain a certain field. But for the main <code class="language-plaintext highlighter-rouge">layout</code> API, this
object was always being passed in as <code class="language-plaintext highlighter-rouge">None</code>, seemingly because that whole class
of objects hadn’t been implemented:</p>
<div class="language-python highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="k">def</span> <span class="nf">layout</span><span class="p">(</span><span class="n">display</span><span class="p">,</span> <span class="n">node</span><span class="p">,</span> <span class="n">standard</span><span class="o">=</span><span class="n">HTML5</span><span class="p">):</span>
<span class="n">containing_block</span> <span class="o">=</span> <span class="n">Viewport</span><span class="p">(</span><span class="n">display</span><span class="p">,</span> <span class="n">node</span><span class="p">)</span>
<span class="n">font</span> <span class="o">=</span> <span class="bp">None</span> <span class="c1"># FIXME - default font
</span>
<span class="n">node</span><span class="p">.</span><span class="n">layout</span><span class="p">.</span><span class="n">reset</span><span class="p">()</span>
<span class="c1"># 10.1 1
</span> <span class="n">layout_box</span><span class="p">(</span><span class="n">display</span><span class="p">,</span> <span class="n">node</span><span class="p">,</span> <span class="n">containing_block</span><span class="p">,</span> <span class="n">containing_block</span><span class="p">,</span> <span class="n">font</span><span class="p">)</span>
</code></pre></div></div>
<p>Note the <code class="language-plaintext highlighter-rouge">FIXME</code>. It appears that a lot of the current issue with <code class="language-plaintext highlighter-rouge">colosseum</code> is
simply missing functionality: code which has been written in a certain way
(knowing it will be needed to meet the CSS spec) but is being fed by defaults
and stubs.</p>
<p>This represents both a difficulty and an opportunity: once the requirements are
understood, and the problem fleshed out (probably via interacting with the core
devs) there is certainly clear work to be done which just requires skills
already available to our team, designing and coding Python.</p>
<p>After talking to some of the core devs on the
<a href="https://gitter.im/beeware/general"><code class="language-plaintext highlighter-rouge">gitter</code></a>, I’m somewhat hopeful that they’ll
be able to direct us at some work to be done, which might represent good group
projects for further contribution in the New Year.</p>Callum WardAfter having taken a stab at fixing a couple of issues within the main CPython implementation of Python, I decided whilst waiting for some of those pull requests to be merged I might try to contribute to some other open source projects in the Python ecosystem that are endorsed by the PSF.Sh - the baby’s coding2019-12-11T00:00:00+00:002019-12-11T00:00:00+00:00/2019/12/11/EshanSinghal<p>I took my first baby steps in the world of open source today with EnHackathon.
I found an issue quite easily - where shutil.move returns an error related to permissions in a case where the standard unix ‘mv’ does not.
There was a patch attached to the issue but this would break anybody relying on the existing behaviour - so there was a suggestion to add a new parameter in order to
allow for full backwards compatibility. The original issue was raised way back in 2006 and there hadn’t been any activity for 5 years, so my first job was
to see whether this was still an issue today.</p>
<p>After a look at the code, patch and comments on the issue, I felt comfortable understanding the problem at hand and that there hadn’t been significant changes in this
area. The suggestion to add a new parameter throws up an issue, though. Adding a new parameter would require the consumers of the API to amend their calls to use the
new parameter if they so wished. There was something unsatisfactory in special casing shutil.move, the consumer here - but importantly takes in an
optional copy function, so I settled for leaving a comment with my thoughts on taking this issue forward.</p>
<p>Overall, a positive day in that I could take an issue and understand it in whole - but slightly unsatisfactory that I concluded by simply adding a comment in
the hopes of stirring activity rather than making progress towards a pull request.</p>
<p>Eshan</p>Eshan SinghalI took my first baby steps in the world of open source today with EnHackathon. I found an issue quite easily - where shutil.move returns an error related to permissions in a case where the standard unix ‘mv’ does not. There was a patch attached to the issue but this would break anybody relying on the existing behaviour - so there was a suggestion to add a new parameter in order to allow for full backwards compatibility. The original issue was raised way back in 2006 and there hadn’t been any activity for 5 years, so my first job was to see whether this was still an issue today.The End of The Beginning2019-12-11T00:00:00+00:002019-12-11T00:00:00+00:00/2019/12/11/LewisGaul<p>Today was my last day of EnHackathon for 2019. I feel like it went fairly well, with things being a bit clearer given a number of us have now spent a number of days getting familiar with the code and the process. We had a couple of new faces in the group this time, as well as a couple following up on previous contribution efforts.</p>
<p>This time I felt like I had a bit more involvement in a couple of the things other people were working on - I found these issues quite interesting and would recommend reading the respective blog posts:</p>
<ul>
<li>Subinterpreters channel locking (see the <a href="https://github.com/ericsnowcurrently/multi-core-python/issues/19">GitHub issue</a>)</li>
<li><a href="./RebeccaMorgan.html">Subclassing <code class="language-plaintext highlighter-rouge">ctypes</code> classes</a></li>
<li><a href="./EshanSinghal.html">A issue/feature request in <code class="language-plaintext highlighter-rouge">shutil.move</code></a></li>
<li><a href="./CallumWard.html">Looking into contributing to BeeWare</a></li>
</ul>
<h2 id="cleaning-up-subinterpreters-on-shutdown">Cleaning up Subinterpreters on Shutdown</h2>
<p>I spent most of the day looking into the following issue: <a href="https://bugs.python.org/issue36225">https://bugs.python.org/issue36225</a>, following on from my brief look last time. The documentation for the CPython C API says:</p>
<p>“<code class="language-plaintext highlighter-rouge">Py_FinalizeEx()</code> will destroy all sub-interpreters that haven’t been explicitly destroyed at that point.”.</p>
<p>However, this is [no longer?] true - <code class="language-plaintext highlighter-rouge">Py_FinalizeEx()</code> does not clean up subinterpreters, and if there are remaining subinterpreters when calling this function then a <code class="language-plaintext highlighter-rouge">Py_FatalError</code> occurs.</p>
<p>This issue was actually hit by HexChat, a project that presumably embeds CPython and makes use of subinterpreters in some way (they were able to fix the issue from their end, but CPython behaviour should still match what’s documented!).</p>
<p>Thanks to some detailed comments on the BPO issue from Nick Coghlan and Eric Snow I was able to get going on this issue without direct guidance - I first added a testcase to reproduce the issue, and then started working on the fix (test-driven development, woo!).</p>
<p>By the end of the day I was able to get a fix in place and <a href="https://github.com/python/cpython/pull/17575">opened a PR</a> (although I made the mistake of not waiting for the tests to finish running first, and it turns out one of them is failing - I’ll look into that when I get the chance!).</p>
<h3 id="c-api-references">C API References</h3>
<p>A selection of C API functions I had to get somewhat familiar with, all taken from <a href="https://docs.python.org/3/c-api/init.html#high-level-api">https://docs.python.org/3/c-api/init.html#high-level-api</a>.</p>
<p><code class="language-plaintext highlighter-rouge">void Py_EndInterpreter(PyThreadState *tstate)</code><br />
Destroy the (sub-)interpreter represented by the given thread state. The given thread state must be the current thread state. When the call returns, the current thread state is <code class="language-plaintext highlighter-rouge">NULL</code>. The global interpreter lock must be held before calling this function and is still held when it returns. <strong><code class="language-plaintext highlighter-rouge">Py_FinalizeEx()</code> will destroy all sub-interpreters that haven’t been explicitly destroyed at that point.</strong></p>
<p><code class="language-plaintext highlighter-rouge">void Py_Initialize()</code><br />
Initialize the Python interpreter. In an application embedding Python, this should be called before using any other Python/C API functions.</p>
<p><code class="language-plaintext highlighter-rouge">int Py_FinalizeEx()</code><br />
Undo all initializations made by <code class="language-plaintext highlighter-rouge">Py_Initialize()</code> and subsequent use of Python/C API functions, and destroy all sub-interpreters that were created and not yet destroyed since the last call to <code class="language-plaintext highlighter-rouge">Py_Initialize()</code>.</p>
<p><code class="language-plaintext highlighter-rouge">PyThreadState* Py_NewInterpreter()</code><br />
Create a new sub-interpreter. The return value points to the first thread state created in the new sub-interpreter. Note that no actual thread is created.</p>
<p><code class="language-plaintext highlighter-rouge">PyThreadState* PyThreadState_Get()</code><br />
Return the current thread state. The global interpreter lock must be held.</p>
<p><code class="language-plaintext highlighter-rouge">PyThreadState* PyThreadState_Swap(PyThreadState *tstate)</code><br />
Swap the current thread state with the thread state given by the argument tstate, which may be <code class="language-plaintext highlighter-rouge">NULL</code>. The global interpreter lock must be held and is not released.</p>
<p><code class="language-plaintext highlighter-rouge">void PyEval_ReleaseThread(PyThreadState *tstate)</code><br />
Reset the current thread state to <code class="language-plaintext highlighter-rouge">NULL</code> and release the global interpreter lock. The lock must have been created earlier and must be held by the current thread.</p>
<p><code class="language-plaintext highlighter-rouge">void PyEval_RestoreThread(PyThreadState *tstate)</code><br />
Acquire the global interpreter lock (if it has been created) and set the thread state to tstate, which must not be <code class="language-plaintext highlighter-rouge">NULL</code>.</p>
<p><code class="language-plaintext highlighter-rouge">PyGILState_STATE PyGILState_Ensure()</code><br />
Ensure that the current thread is ready to call the Python C API regardless of the current state of Python, or of the global interpreter lock.</p>
<p><code class="language-plaintext highlighter-rouge">void PyGILState_Release(PyGILState_STATE)</code><br />
Release any resources previously acquired.</p>
<h2 id="summary">Summary</h2>
<h3 id="my-contribution-experience">My Contribution Experience</h3>
<p>Overall I’ve had a good - albeit presumably slightly unusual - experience with first-time contribution to CPython, and open source contribution in general. I feel proud of having introduced so many others to open source and CPython, and hope some of them will maintain an interest in contributing and in being involved in the Python community in the future. I aim to continue and increase my involvement in the Python community over the coming years, and will continue to support/encourage others who are interested in getting involved.</p>
<p>I’m planning on writing a standalone blog post soon to talk about my overall experience as a first-time contributor to CPython, intending to join Tal Einat’s collection of <a href="https://github.com/taleinat/python-contribution-feedback">first-time contribution stories</a>. This will probably mostly be bringing together thoughts from previous my blog posts, which you can read at:</p>
<ul>
<li><a href="../../10/27/LewisGaul">Hacktoberfest Event - Getting Started!</a> (27/10/2019)</li>
<li><a href="../../11/04/LewisGaul">Head-first into Python’s Runtime</a> (04/11/2019)</li>
<li><a href="../../11/12/LewisGaul">Visiting the C-side</a> (12/11/2019)</li>
<li><a href="../../11/21/LewisGaul">First Subinterpreters PR</a> (21/11/2019)</li>
</ul>
<h3 id="whats-next-for-enhackathon">What’s Next for EnHackathon?</h3>
<p>Having spoken to other EnHackathon participants, it seems likely that EnHackathon will be going forward in some form in 2020, with some people hoping to continue their contribution early in the year.</p>
<p>I think it’s unlikely I’ll personally be using the days offered by Cisco for open source next year - I’d quite like to spend my time doing something a bit different. This is partly because I know I’m invested in Python anyway and can continue my involvement in my own time! However, if there are more people who would like to get started through EnHackathon next year then I’ll be more than happy to help do some organising and get people going. I’ll also keep trying to think up imaginative ways to encourage more people at work (or out of work) to get involved in Python - if anyone has any suggestions please feel free to <a href="mailto:lewis.gaul@gmail.com">email me</a>! :)</p>Lewis GaulToday was my last day of EnHackathon for 2019. I feel like it went fairly well, with things being a bit clearer given a number of us have now spent a number of days getting familiar with the code and the process. We had a couple of new faces in the group this time, as well as a couple following up on previous contribution efforts.EnHackathon - Finding issues and poking around ctypes2019-12-11T00:00:00+00:002019-12-11T00:00:00+00:00/2019/12/11/RebeccaMorgan<p>I was a little late to the party joining EnHackathon and spent today looking into an issue in ctypes. Although I haven’t written any new code today, I feel that I’m now more comfortable in understanding the bug fix process in Python and have a better idea of how I could contribute more in future.</p>
<h2 id="identifying-issues">Identifying Issues</h2>
<p>As many posts on this blog have discussed, one of the biggest challenges when starting out is identifying appropriate issues on the Python bug tracker, <a href="https://bugs.python.org/">BPO</a>. When looking for issues, I took a couple of different approaches:</p>
<ul>
<li>Investigating the ‘easy’ keyword, which has a helpful shortcut under the ‘Summaries’ section in the sidebar. There are not many unclaimed issues in this category!</li>
<li>Investigating the ‘newcomer-friendly’ keyword (often found alongside ‘easy’), issues which have been identified by maintainers as approachable for first time contributors.</li>
<li>Looking for issues within specific components.</li>
<li>Looking for issues at a particular stage (needs patch).</li>
</ul>
<p>Throughout this process, I focussed on issues more than a few days old, but less than several months old. Once I had found an issue that looked approachable and wasn’t actively being worked on, I started looking into the code.</p>
<h2 id="investigating-ctypes">Investigating ctypes</h2>
<p>Today I’ve been looking into an <a href="https://bugs.python.org/issue38860">issue</a> in ctypes, in which the classmethod <code class="language-plaintext highlighter-rouge">from_buffer_copy</code> did not invoke <code class="language-plaintext highlighter-rouge">__new__</code> and <code class="language-plaintext highlighter-rouge">__init__</code> on a subclass as expected. The person who raised the issue had provided a minimal example, which I explored in gdb.</p>
<p>I started off reading the <a href="https://docs.python.org/3/library/ctypes.html">documentation</a> for the ctypes library and looking around the code. After looking into PyObjects for while, I found the <a href="https://github.com/python/cpython/blob/master/Lib/ctypes/test/test_frombuffer.py">tests</a> for the <code class="language-plaintext highlighter-rouge">from_buffer</code> functionality. Interestingly, the tests explictly check that the
<code class="language-plaintext highlighter-rouge">__init__</code> method is not called for subclasses of structure instantiated by calling <code class="language-plaintext highlighter-rouge">from_buffer_copy</code>. As a newcomer to both contributing and to ctypes, it wasn’t clear how to progress from here - I left a comment on the issue, and am looking forward to getting a response.</p>
<h2 id="summary">Summary</h2>
<p>Contributing to such a large and mature codebase can be intimidating at first, but there are plenty of resources out there to guide you through the process. The <a href="https://devguide.python.org/">Python Developer’s Guide</a> is very thorough and helped me get started quickly. Earlier blog posts on the <a href="/">EnHackathon website</a> give a good overview of what ‘your first issue’ workflow looks like. One of the steps I took while setting up was signing up to the <a href="https://mail.python.org/mailman3/lists/python-dev.python.org/">python-dev</a>, <a href="https://mail.python.org/mailman3/lists/python-ideas.python.org/">python-ideas</a> and <a href="https://mail.python.org/mailman3/lists/core-mentorship.python.org/">core-mentorship</a> mailing lists - it has already been really interesting to see the ongoing discussions about the future of Python.</p>Rebecca MorganI was a little late to the party joining EnHackathon and spent today looking into an issue in ctypes. Although I haven’t written any new code today, I feel that I’m now more comfortable in understanding the bug fix process in Python and have a better idea of how I could contribute more in future. Identifying Issues As many posts on this blog have discussed, one of the biggest challenges when starting out is identifying appropriate issues on the Python bug tracker, BPO. When looking for issues, I took a couple of different approaches: Investigating the ‘easy’ keyword, which has a helpful shortcut under the ‘Summaries’ section in the sidebar. There are not many unclaimed issues in this category! Investigating the ‘newcomer-friendly’ keyword (often found alongside ‘easy’), issues which have been identified by maintainers as approachable for first time contributors. Looking for issues within specific components. Looking for issues at a particular stage (needs patch). Throughout this process, I focussed on issues more than a few days old, but less than several months old. Once I had found an issue that looked approachable and wasn’t actively being worked on, I started looking into the code. Investigating ctypes Today I’ve been looking into an issue in ctypes, in which the classmethod from_buffer_copy did not invoke __new__ and __init__ on a subclass as expected. The person who raised the issue had provided a minimal example, which I explored in gdb. I started off reading the documentation for the ctypes library and looking around the code. After looking into PyObjects for while, I found the tests for the from_buffer functionality. Interestingly, the tests explictly check that the __init__ method is not called for subclasses of structure instantiated by calling from_buffer_copy. As a newcomer to both contributing and to ctypes, it wasn’t clear how to progress from here - I left a comment on the issue, and am looking forward to getting a response. Summary Contributing to such a large and mature codebase can be intimidating at first, but there are plenty of resources out there to guide you through the process. The Python Developer’s Guide is very thorough and helped me get started quickly. Earlier blog posts on the EnHackathon website give a good overview of what ‘your first issue’ workflow looks like. One of the steps I took while setting up was signing up to the python-dev, python-ideas and core-mentorship mailing lists - it has already been really interesting to see the ongoing discussions about the future of Python.EnHackathon - Whipping Subinterpreters into shape2019-11-21T00:00:00+00:002019-11-21T00:00:00+00:00/2019/11/21/BenjaminEdwards<p>Over the past couple of days I have been busy completing the markups to my Pull Request for my changes to <a href="https://bugs.python.org/issue37838">Get Type Hints</a> and learning lots about subinterpreters (including that it’s not subinterpretors). After helping Lewis debug and test his code for <code class="language-plaintext highlighter-rouge">channel_list_interpreters()</code> I set out to find another issue in the module to work on.</p>
<h1 id="threads-vs-interpreters">Threads vs Interpreters</h1>
<p>One of the more interesting things I learnt while talking to Phil is the relationship between threads and interpreters. I had been assuming up to now that there was a kind of hierarchical structure where processes > intepreters > threads. By this I mean you have processes which are what is seen by the outside system. Within processes you have intepreters, which run the python code, and within interpreters you can have different threads running simultaneously. But boy was I wrong.</p>
<p>What is actually going is actually a bit more subtle. Within a process you will have threads which are separate instances of the C code. As there are different instances of C code it means they can run on different cores of the processor at the same time. Then you have interpreters, which are completely orthogonal to threads and are used to run the Python code. So each interpreter runs its own bit of Python code with its own state separate from other interpreters. This means you can have two interpreters on one thread leading to two bits of Python code running on separate states but having to take turns over the processor. Or you can have two threads on one interpreter so there are two bits of Python code running simultaneously but having to share state.</p>
<h1 id="conclusion">Conclusion</h1>
<p>I’m very happy with progress I have made over the past couple days. I have managed to merge down my second Pull Request, which was all down to the quick responses I got when I requested a code review. I am also enjoying learning more about the sub-intepreters project and look forward to contributing further.</p>Benjamin EdwardsOver the past couple of days I have been busy completing the markups to my Pull Request for my changes to Get Type Hints and learning lots about subinterpreters (including that it’s not subinterpretors). After helping Lewis debug and test his code for channel_list_interpreters() I set out to find another issue in the module to work on. Threads vs Interpreters One of the more interesting things I learnt while talking to Phil is the relationship between threads and interpreters. I had been assuming up to now that there was a kind of hierarchical structure where processes > intepreters > threads. By this I mean you have processes which are what is seen by the outside system. Within processes you have intepreters, which run the python code, and within interpreters you can have different threads running simultaneously. But boy was I wrong. What is actually going is actually a bit more subtle. Within a process you will have threads which are separate instances of the C code. As there are different instances of C code it means they can run on different cores of the processor at the same time. Then you have interpreters, which are completely orthogonal to threads and are used to run the Python code. So each interpreter runs its own bit of Python code with its own state separate from other interpreters. This means you can have two interpreters on one thread leading to two bits of Python code running on separate states but having to take turns over the processor. Or you can have two threads on one interpreter so there are two bits of Python code running simultaneously but having to share state. Conclusion I’m very happy with progress I have made over the past couple days. I have managed to merge down my second Pull Request, which was all down to the quick responses I got when I requested a code review. I am also enjoying learning more about the sub-intepreters project and look forward to contributing further.Union Weirdness2019-11-21T00:00:00+00:002019-11-21T00:00:00+00:00/2019/11/21/DanKreso<p>I spent two days this week taking a deeper look at Endian Unions. The two days were productive, with Tom, Anjani, and me realising that there’s more to Endian Unions than meets the eye.</p>
<h3 id="the-1st-day">The 1st Day</h3>
<p>I spent the majority of the first day investigating what testing needed to be added for Endian Unions. The main strategy for this was to look at the existing testing for Endian Structures, and deciding which ones applied to Endian Unions.</p>
<p>For both structures and unions, the user can specify the <a href="https://docs.python.org/3.8/library/ctypes.html#structure-union-alignment-and-byte-order">packing</a> and create objects with <a href="https://docs.python.org/3.8/library/ctypes.html#bit-fields-in-structures-and-unions">bit fields</a>. In deciding how to test these properties for Big and Little Endian Unions I first had to understand how they behaved for Unions in general. I did this by writing quick tests and scratch files.</p>
<p>And I found something fairly unintuitive.</p>
<p>Here’s some code to demonstrate. Hopefully it’s fairly self-explanatory so that if you are not familiar with structures and unions you can understand what they do:</p>
<div class="language-python highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="kn">from</span> <span class="nn">ctypes</span> <span class="kn">import</span> <span class="o">*</span>
<span class="c1"># Create a structure with two c_uint32 fields.
</span><span class="k">class</span> <span class="nc">Struct</span><span class="p">(</span><span class="n">Structure</span><span class="p">):</span>
<span class="n">_fields_</span> <span class="o">=</span> <span class="p">[(</span><span class="s">"a"</span><span class="p">,</span> <span class="n">c_uint32</span><span class="p">),</span> <span class="p">(</span><span class="s">"b"</span><span class="p">,</span> <span class="n">c_uint32</span><span class="p">)]</span>
<span class="c1"># Initialise a null byte array with 8 bytes.
</span><span class="n">buff</span> <span class="o">=</span> <span class="nb">bytearray</span><span class="p">(</span><span class="mi">8</span><span class="p">)</span>
<span class="c1"># Create a structure so that it points to the byte array.
</span><span class="n">simple_struct</span> <span class="o">=</span> <span class="n">Struct</span><span class="p">.</span><span class="n">from_buffer</span><span class="p">(</span><span class="n">buff</span><span class="p">)</span>
<span class="n">simple_struct</span><span class="p">.</span><span class="n">a</span> <span class="o">=</span> <span class="mi">1</span>
<span class="n">simple_struct</span><span class="p">.</span><span class="n">b</span> <span class="o">=</span> <span class="mi">2</span>
<span class="k">print</span><span class="p">(</span><span class="s">"a is {}"</span><span class="p">.</span><span class="nb">format</span><span class="p">(</span><span class="n">simple_struct</span><span class="p">.</span><span class="n">a</span><span class="p">))</span> <span class="c1"># Prints "a is 1"
</span><span class="k">print</span><span class="p">(</span><span class="s">"b is {}"</span><span class="p">.</span><span class="nb">format</span><span class="p">(</span><span class="n">simple_struct</span><span class="p">.</span><span class="n">b</span><span class="p">))</span> <span class="c1"># Prints "b is 2"
# Take a look at how the structure occupies the memory.
</span><span class="k">print</span><span class="p">(</span><span class="s">"Buffer: {}"</span><span class="p">.</span><span class="nb">format</span><span class="p">(</span><span class="n">buff</span><span class="p">))</span> <span class="c1"># Prints "Buffer: b'\x01\x00\x00\x00\x02\x00\x00\x00'"
</span>
<span class="c1"># Create a union with two c_uint32 fields.
</span><span class="k">class</span> <span class="nc">Union_</span><span class="p">(</span><span class="n">Union</span><span class="p">):</span>
<span class="n">_fields_</span> <span class="o">=</span> <span class="p">[(</span><span class="s">"a"</span><span class="p">,</span> <span class="n">c_uint32</span><span class="p">),</span> <span class="p">(</span><span class="s">"b"</span><span class="p">,</span> <span class="n">c_uint32</span><span class="p">)]</span>
<span class="c1"># Initialise a null byte array with 4 bytes. Note that the union requires half
# the memory of the structure as fields "a" and "b" occupy the same bit of memory.
</span><span class="n">buff</span> <span class="o">=</span> <span class="nb">bytearray</span><span class="p">(</span><span class="mi">4</span><span class="p">)</span>
<span class="c1"># Create a union so that it points to the byte array.
</span><span class="n">simple_union</span> <span class="o">=</span> <span class="n">Union_</span><span class="p">.</span><span class="n">from_buffer</span><span class="p">(</span><span class="n">buff</span><span class="p">)</span>
<span class="n">simple_union</span><span class="p">.</span><span class="n">a</span> <span class="o">=</span> <span class="mi">1</span>
<span class="n">simple_union</span><span class="p">.</span><span class="n">b</span> <span class="o">=</span> <span class="mi">2</span>
<span class="k">print</span><span class="p">(</span><span class="s">"a is {}"</span><span class="p">.</span><span class="nb">format</span><span class="p">(</span><span class="n">simple_union</span><span class="p">.</span><span class="n">a</span><span class="p">))</span> <span class="c1"># Prints "a is 2"
</span><span class="k">print</span><span class="p">(</span><span class="s">"b is {}"</span><span class="p">.</span><span class="nb">format</span><span class="p">(</span><span class="n">simple_union</span><span class="p">.</span><span class="n">b</span><span class="p">))</span> <span class="c1"># Prints "b is 2"
</span><span class="k">print</span><span class="p">(</span><span class="s">"Buffer: {}"</span><span class="p">.</span><span class="nb">format</span><span class="p">(</span><span class="n">buff</span><span class="p">))</span> <span class="c1"># Prints "Buffer: b'\x02\x00\x00\x00'"
</span>
<span class="c1"># Create a structure with bitfields. We set "a" to a bitwidth of 24 bits and "b"
# to a bitwidth of 8 bits.
</span><span class="k">class</span> <span class="nc">BitFieldStruct</span><span class="p">(</span><span class="n">Structure</span><span class="p">):</span>
<span class="n">_fields_</span> <span class="o">=</span> <span class="p">[(</span><span class="s">"a"</span><span class="p">,</span> <span class="n">c_uint32</span><span class="p">,</span> <span class="mi">24</span><span class="p">),</span> <span class="p">(</span><span class="s">"b"</span><span class="p">,</span> <span class="n">c_uint32</span><span class="p">,</span> <span class="mi">8</span><span class="p">)]</span>
<span class="c1"># Initialise a null byte array with 4 bytes. Note that the bitfield structure
# requires half the memory of a regular structure.
</span><span class="n">buff</span> <span class="o">=</span> <span class="nb">bytearray</span><span class="p">(</span><span class="mi">4</span><span class="p">)</span>
<span class="n">bitfield_struct</span> <span class="o">=</span> <span class="n">BitFieldStruct</span><span class="p">.</span><span class="n">from_buffer</span><span class="p">(</span><span class="n">buff</span><span class="p">)</span>
<span class="n">bitfield_struct</span><span class="p">.</span><span class="n">a</span> <span class="o">=</span> <span class="mh">0xabcdef</span>
<span class="n">bitfield_struct</span><span class="p">.</span><span class="n">b</span> <span class="o">=</span> <span class="mi">1</span>
<span class="k">print</span><span class="p">(</span><span class="s">"a is {}"</span><span class="p">.</span><span class="nb">format</span><span class="p">(</span><span class="n">bitfield_struct</span><span class="p">.</span><span class="n">a</span><span class="p">))</span> <span class="c1"># Prints "a is 11259375" (0xabcdef in decimal).
</span><span class="k">print</span><span class="p">(</span><span class="s">"b is {}"</span><span class="p">.</span><span class="nb">format</span><span class="p">(</span><span class="n">bitfield_struct</span><span class="p">.</span><span class="n">b</span><span class="p">))</span> <span class="c1"># Prints "b is 1".
</span><span class="k">print</span><span class="p">(</span><span class="s">"Buffer: {}"</span><span class="p">.</span><span class="nb">format</span><span class="p">(</span><span class="n">buff</span><span class="p">))</span> <span class="c1"># Prints "Buffer: b'\xef\xcd\xab\x01'".
# It is easy to see why only a buffer of 4 bytes is required.
</span>
<span class="c1"># Create a union with bitfields, analagously to the bitfield structure.
</span><span class="k">class</span> <span class="nc">BitFieldUnion</span><span class="p">(</span><span class="n">Union</span><span class="p">):</span>
<span class="n">_fields_</span> <span class="o">=</span> <span class="p">[(</span><span class="s">"a"</span><span class="p">,</span> <span class="n">c_uint32</span><span class="p">,</span> <span class="mi">24</span><span class="p">),</span> <span class="p">(</span><span class="s">"b"</span><span class="p">,</span> <span class="n">c_uint32</span><span class="p">,</span> <span class="mi">8</span><span class="p">)]</span>
<span class="n">buff</span> <span class="o">=</span> <span class="nb">bytearray</span><span class="p">(</span><span class="mi">4</span><span class="p">)</span>
<span class="n">bitfield_union</span> <span class="o">=</span> <span class="n">BitFieldUnion</span><span class="p">.</span><span class="n">from_buffer</span><span class="p">(</span><span class="n">buff</span><span class="p">)</span>
<span class="n">bitfield_union</span><span class="p">.</span><span class="n">a</span> <span class="o">=</span> <span class="mh">0xabcdef</span>
<span class="n">bitfield_union</span><span class="p">.</span><span class="n">b</span> <span class="o">=</span> <span class="mi">1</span>
<span class="k">print</span><span class="p">(</span><span class="s">"a is {}"</span><span class="p">.</span><span class="nb">format</span><span class="p">(</span><span class="n">bitfield_union</span><span class="p">.</span><span class="n">a</span><span class="p">))</span> <span class="c1"># Prints "a is 11259375"
# Wait what? I would expect it to print 1 as bitfield_union.b was set last.
</span><span class="k">print</span><span class="p">(</span><span class="s">"b is {}"</span><span class="p">.</span><span class="nb">format</span><span class="p">(</span><span class="n">bitfield_union</span><span class="p">.</span><span class="n">b</span><span class="p">))</span> <span class="c1"># Prints "b is 1".
# I would expect this to be the same as bitfield_union.a.
</span><span class="k">print</span><span class="p">(</span><span class="s">"Buffer: {}"</span><span class="p">.</span><span class="nb">format</span><span class="p">(</span><span class="n">buff</span><span class="p">))</span> <span class="c1"># Prints "Buffer: b'\xef\xcd\xab\x00'".
# When we set bitfield_union.b it is never written to memory!
</span>
<span class="c1"># To check the behaviour, have the two fields be the same bitfield lengths.
</span><span class="k">class</span> <span class="nc">SanityUnion</span><span class="p">(</span><span class="n">Union</span><span class="p">):</span>
<span class="n">_fields_</span> <span class="o">=</span> <span class="p">[(</span><span class="s">"a"</span><span class="p">,</span> <span class="n">c_uint32</span><span class="p">,</span> <span class="mi">32</span><span class="p">),</span> <span class="p">(</span><span class="s">"b"</span><span class="p">,</span> <span class="n">c_uint32</span><span class="p">,</span> <span class="mi">32</span><span class="p">)]</span>
<span class="n">buff</span> <span class="o">=</span> <span class="nb">bytearray</span><span class="p">(</span><span class="mi">4</span><span class="p">)</span>
<span class="n">sanity_union</span><span class="o">=</span> <span class="n">SanityUnion</span><span class="p">.</span><span class="n">from_buffer</span><span class="p">(</span><span class="n">buff</span><span class="p">)</span>
<span class="n">sanity_union</span><span class="p">.</span><span class="n">a</span> <span class="o">=</span> <span class="mi">1</span>
<span class="n">sanity_union</span><span class="p">.</span><span class="n">b</span> <span class="o">=</span> <span class="mi">2</span>
<span class="k">print</span><span class="p">(</span><span class="s">"a is {}"</span><span class="p">.</span><span class="nb">format</span><span class="p">(</span><span class="n">sanity_union</span><span class="p">.</span><span class="n">a</span><span class="p">))</span> <span class="c1"># Prints "a is 2" as we expect.
</span><span class="k">print</span><span class="p">(</span><span class="s">"b is {}"</span><span class="p">.</span><span class="nb">format</span><span class="p">(</span><span class="n">sanity_union</span><span class="p">.</span><span class="n">b</span><span class="p">))</span> <span class="c1"># Prints "b is 2" as we expect.
</span><span class="k">print</span><span class="p">(</span><span class="s">"Buffer: {}"</span><span class="p">.</span><span class="nb">format</span><span class="p">(</span><span class="n">buff</span><span class="p">))</span> <span class="c1"># Prints "Buffer: b'\x02\x00\x00\x00'" as we expect.
</span></code></pre></div></div>
<p>Bitfield unions behave a little strangely when fields are different lengths. The two fields no longer return the same value when you access them, although this doesn’t really matter as you would only ever want to retrieve one at any one time. What we should care about is what the memory looks like. There the weird behaviour is that the 1 doesn’t seem to have overwritten 0xabcdef in the memory, even though it was set after the larger number. There’s no mention of this in the Python docs. If anyone has any explanation for this weirdness contact us!</p>
<h3 id="the-2nd-day">The 2nd Day</h3>
<p>I spent the 2nd day writing test cases to test the Big and Little Endian Union implementation I wrote. Much of the day was spent discussing what exactly constituted a Big/Little Endian Union with Tom and Anjani. Further discussion on this can be read in <a href="./TomasHyland.html">Tom’s post</a> - the concept of “endianness” is subtle for unions with different sized elements.</p>
<p>We concluded that there is still work to do in implementing Endian Unions as currently we do not account for this subtlety.</p>
<h3 id="looking-forward">Looking forward</h3>
<p>Tom, Anjani, and I haven’t figured out whether the remaining work exists in the Python or C code. We also still haven’t fully thought about more complex scenarios, i.e. nested unions with different sizes. This likely means that we will have to continue our work on Endian Unions into the New Year, but we definitely have a better grasp on what is left to do.</p>
<p>I enjoyed these 2 days as it allowed me to ‘get stuck in’ and think more about what is required. Not only have I learnt a fair bit about Python, but also lots about C!</p>Dan KresoI spent two days this week taking a deeper look at Endian Unions. The two days were productive, with Tom, Anjani, and me realising that there’s more to Endian Unions than meets the eye. The 1st Day I spent the majority of the first day investigating what testing needed to be added for Endian Unions. The main strategy for this was to look at the existing testing for Endian Structures, and deciding which ones applied to Endian Unions. For both structures and unions, the user can specify the packing and create objects with bit fields. In deciding how to test these properties for Big and Little Endian Unions I first had to understand how they behaved for Unions in general. I did this by writing quick tests and scratch files. And I found something fairly unintuitive. Here’s some code to demonstrate. Hopefully it’s fairly self-explanatory so that if you are not familiar with structures and unions you can understand what they do: from ctypes import * # Create a structure with two c_uint32 fields. class Struct(Structure): _fields_ = [("a", c_uint32), ("b", c_uint32)] # Initialise a null byte array with 8 bytes. buff = bytearray(8) # Create a structure so that it points to the byte array. simple_struct = Struct.from_buffer(buff) simple_struct.a = 1 simple_struct.b = 2 print("a is {}".format(simple_struct.a)) # Prints "a is 1" print("b is {}".format(simple_struct.b)) # Prints "b is 2" # Take a look at how the structure occupies the memory. print("Buffer: {}".format(buff)) # Prints "Buffer: b'\x01\x00\x00\x00\x02\x00\x00\x00'" # Create a union with two c_uint32 fields. class Union_(Union): _fields_ = [("a", c_uint32), ("b", c_uint32)] # Initialise a null byte array with 4 bytes. Note that the union requires half # the memory of the structure as fields "a" and "b" occupy the same bit of memory. buff = bytearray(4) # Create a union so that it points to the byte array. simple_union = Union_.from_buffer(buff) simple_union.a = 1 simple_union.b = 2 print("a is {}".format(simple_union.a)) # Prints "a is 2" print("b is {}".format(simple_union.b)) # Prints "b is 2" print("Buffer: {}".format(buff)) # Prints "Buffer: b'\x02\x00\x00\x00'" # Create a structure with bitfields. We set "a" to a bitwidth of 24 bits and "b" # to a bitwidth of 8 bits. class BitFieldStruct(Structure): _fields_ = [("a", c_uint32, 24), ("b", c_uint32, 8)] # Initialise a null byte array with 4 bytes. Note that the bitfield structure # requires half the memory of a regular structure. buff = bytearray(4) bitfield_struct = BitFieldStruct.from_buffer(buff) bitfield_struct.a = 0xabcdef bitfield_struct.b = 1 print("a is {}".format(bitfield_struct.a)) # Prints "a is 11259375" (0xabcdef in decimal). print("b is {}".format(bitfield_struct.b)) # Prints "b is 1". print("Buffer: {}".format(buff)) # Prints "Buffer: b'\xef\xcd\xab\x01'". # It is easy to see why only a buffer of 4 bytes is required. # Create a union with bitfields, analagously to the bitfield structure. class BitFieldUnion(Union): _fields_ = [("a", c_uint32, 24), ("b", c_uint32, 8)] buff = bytearray(4) bitfield_union = BitFieldUnion.from_buffer(buff) bitfield_union.a = 0xabcdef bitfield_union.b = 1 print("a is {}".format(bitfield_union.a)) # Prints "a is 11259375" # Wait what? I would expect it to print 1 as bitfield_union.b was set last. print("b is {}".format(bitfield_union.b)) # Prints "b is 1". # I would expect this to be the same as bitfield_union.a. print("Buffer: {}".format(buff)) # Prints "Buffer: b'\xef\xcd\xab\x00'". # When we set bitfield_union.b it is never written to memory! # To check the behaviour, have the two fields be the same bitfield lengths. class SanityUnion(Union): _fields_ = [("a", c_uint32, 32), ("b", c_uint32, 32)] buff = bytearray(4) sanity_union= SanityUnion.from_buffer(buff) sanity_union.a = 1 sanity_union.b = 2 print("a is {}".format(sanity_union.a)) # Prints "a is 2" as we expect. print("b is {}".format(sanity_union.b)) # Prints "b is 2" as we expect. print("Buffer: {}".format(buff)) # Prints "Buffer: b'\x02\x00\x00\x00'" as we expect. Bitfield unions behave a little strangely when fields are different lengths. The two fields no longer return the same value when you access them, although this doesn’t really matter as you would only ever want to retrieve one at any one time. What we should care about is what the memory looks like. There the weird behaviour is that the 1 doesn’t seem to have overwritten 0xabcdef in the memory, even though it was set after the larger number. There’s no mention of this in the Python docs. If anyone has any explanation for this weirdness contact us! The 2nd Day I spent the 2nd day writing test cases to test the Big and Little Endian Union implementation I wrote. Much of the day was spent discussing what exactly constituted a Big/Little Endian Union with Tom and Anjani. Further discussion on this can be read in Tom’s post - the concept of “endianness” is subtle for unions with different sized elements. We concluded that there is still work to do in implementing Endian Unions as currently we do not account for this subtlety. Looking forward Tom, Anjani, and I haven’t figured out whether the remaining work exists in the Python or C code. We also still haven’t fully thought about more complex scenarios, i.e. nested unions with different sizes. This likely means that we will have to continue our work on Endian Unions into the New Year, but we definitely have a better grasp on what is left to do. I enjoyed these 2 days as it allowed me to ‘get stuck in’ and think more about what is required. Not only have I learnt a fair bit about Python, but also lots about C!PRogress2019-11-21T00:00:00+00:002019-11-21T00:00:00+00:00/2019/11/21/JacksonRiley<p>Not one, dear reader, not two, but <em>three</em> pull requests were submitted today. What a rush.</p>
<h2 id="what-i-did">What I did</h2>
<h3 id="collectionsabc-docstrings">Collections.ABC docstrings</h3>
<p><a href="https://bugs.python.org/issue17306">Issue 17306</a> covers adding better docstrings to the Abstract Base Classes in <a href="https://docs.python.org/3/library/collections.abc.html"><code class="language-plaintext highlighter-rouge">collections.abc</code></a>.</p>
<p>I hadn’t had any comments on my patch that I posted to the bpo thread last week, so I went ahead and submitted a pull request. I think this was the right thing to do, and hopefully it’ll get a review before my next day of <code class="language-plaintext highlighter-rouge">EnHackathon</code>.</p>
<h3 id="magicmock-and-divmod">MagicMock and divmod</h3>
<p><a href="https://bugs.python.org/issue34716">Issue 34716</a> covers fixing up an interaction between <a href="https://docs.python.org/3/library/unittest.mock.html#unittest.mock.MagicMock"><code class="language-plaintext highlighter-rouge">unittest.mock.MagicMock</code></a> and the built-in <a href="https://docs.python.org/3/library/functions.html#divmod"><code class="language-plaintext highlighter-rouge">divmod</code></a> function.</p>
<p>My winners of the options from <a href="../12/JacksonRiley.html">last time</a> were</p>
<ul>
<li>
<p>Return a pair of <code class="language-plaintext highlighter-rouge">MagicMock</code> instances. This is very sensible and in-keeping with the rest of the behaviour of <code class="language-plaintext highlighter-rouge">MagicMock</code>.</p>
</li>
<li>
<p>Change the behaviour of <code class="language-plaintext highlighter-rouge">MagicMock</code> more generally such that trying to unpack a <code class="language-plaintext highlighter-rouge">MagicMock</code> instance into two variables, for example, would assign a new <code class="language-plaintext highlighter-rouge">MagicMock</code> instance to each. This would be a bit more far-reaching than this issue but might be worth thinking about. This would fix the issue as a side-effect, as the current result of e.g. <code class="language-plaintext highlighter-rouge">divmod(MagicMock(), 2)</code> is another instance of <code class="language-plaintext highlighter-rouge">MagicMock</code>.</p>
</li>
</ul>
<p>I posted as such on the bpo thread and was informed that the second option was a bit ambitious - there’s no implementation-agnostic way for <code class="language-plaintext highlighter-rouge">MagicMock</code> to “know” how many variables it’s being unpacked into.</p>
<p>I therefore decided to plump for the first option and looked through <code class="language-plaintext highlighter-rouge">unittest.mock</code> to find the right place to add this functionality - I ended up taking a bit of a scenic route but got there in the end. The solution that I went for was to add <code class="language-plaintext highlighter-rouge">__divmod__</code> and <code class="language-plaintext highlighter-rouge">__rdivmod__</code> (<code class="language-plaintext highlighter-rouge">__divmod__</code>’s right-hand cousin) to the list of side-effect methods, which includes methods such as <code class="language-plaintext highlighter-rouge">__eq__</code> and <code class="language-plaintext highlighter-rouge">__iter__</code> which have a default behaviour within <code class="language-plaintext highlighter-rouge">MagicMock</code>.</p>
<p>For instance, <code class="language-plaintext highlighter-rouge">m.__eq__(other)</code> checks by default for object identity: if <code class="language-plaintext highlighter-rouge">m</code> and <code class="language-plaintext highlighter-rouge">other</code> are names for the same object. Adding <code class="language-plaintext highlighter-rouge">__divmod__</code> and <code class="language-plaintext highlighter-rouge">__rdivmod__</code> was simply a matter of defining some new default behaviour - return a pair of <code class="language-plaintext highlighter-rouge">MagicMock</code> instances. This now means that <code class="language-plaintext highlighter-rouge">MagicMock</code> behaves as</p>
<div class="language-python highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="o">>>></span> <span class="kn">from</span> <span class="nn">unittest.mock</span> <span class="kn">import</span> <span class="o">*</span>
<span class="o">>>></span> <span class="n">floordiv</span><span class="p">,</span> <span class="n">mod</span> <span class="o">=</span> <span class="nb">divmod</span><span class="p">(</span><span class="n">MagicMock</span><span class="p">(),</span> <span class="mi">2</span><span class="p">)</span>
<span class="o">>>></span> <span class="k">print</span><span class="p">(</span><span class="n">floordiv</span><span class="p">,</span> <span class="n">mod</span><span class="p">)</span>
<span class="o"><</span><span class="n">MagicMock</span> <span class="nb">id</span><span class="o">=</span><span class="s">'139748283461136'</span><span class="o">></span> <span class="o"><</span><span class="n">MagicMock</span> <span class="nb">id</span><span class="o">=</span><span class="s">'139748283411024'</span><span class="o">></span>
<span class="o">>>></span> <span class="n">floordiv</span><span class="p">,</span> <span class="n">mod</span> <span class="o">=</span> <span class="nb">divmod</span><span class="p">(</span><span class="mi">2</span><span class="p">,</span> <span class="n">MagicMock</span><span class="p">())</span>
<span class="o">>>></span> <span class="k">print</span><span class="p">(</span><span class="n">floordiv</span><span class="p">,</span> <span class="n">mod</span><span class="p">)</span>
<span class="o"><</span><span class="n">MagicMock</span> <span class="nb">id</span><span class="o">=</span><span class="s">'139748278535024'</span><span class="o">></span> <span class="o"><</span><span class="n">MagicMock</span> <span class="nb">id</span><span class="o">=</span><span class="s">'139748278597184'</span><span class="o">></span>
<span class="o">>>></span> <span class="n">floordiv</span><span class="p">,</span> <span class="n">mod</span> <span class="o">=</span> <span class="nb">divmod</span><span class="p">(</span><span class="n">MagicMock</span><span class="p">(),</span> <span class="n">MagicMock</span><span class="p">())</span>
<span class="o">>>></span> <span class="k">print</span><span class="p">(</span><span class="n">floordiv</span><span class="p">,</span> <span class="n">mod</span><span class="p">)</span>
<span class="o"><</span><span class="n">MagicMock</span> <span class="nb">id</span><span class="o">=</span><span class="s">'139748278678624'</span><span class="o">></span> <span class="o"><</span><span class="n">MagicMock</span> <span class="nb">id</span><span class="o">=</span><span class="s">'139748278741744'</span><span class="o">></span>
</code></pre></div></div>
<p>Nice!</p>
<p>A few updates to the <code class="language-plaintext highlighter-rouge">unittest.mock</code> tests later, I was able to confirm that I hadn’t broken anything and raise a pull request. I haven’t had any activity on this yet but will hopefully do so at some point.</p>
<h3 id="removing-asyncore-from-tests">Removing asyncore from tests</h3>
<p><a href="https://docs.python.org/3/library/asyncore.html"><code class="language-plaintext highlighter-rouge">asyncore</code></a> was one of the modules used in Python to implement asynchronous behaviour before <a href="https://docs.python.org/3/library/asyncio.html"><code class="language-plaintext highlighter-rouge">asyncio</code></a> came along, and has been deprecated since Python 3.6, but is still used in some tests. <a href="https://bugs.python.org/issue28533">Issue 28533</a> covers removing these mentions and reimplementing the internals of a handful of test files using <code class="language-plaintext highlighter-rouge">asyncio</code>.</p>
<p>This issue was recommended to be broken into subissues for each test file in question, so spotting a trivial one I could get done before heading home, I raised <a href="https://bugs.python.org/issue38866">Issue 38866</a> and removed a mention from the <a href="https://docs.python.org/3/library/pyclbr.html"><code class="language-plaintext highlighter-rouge">pyclbr</code></a> test file. Third PR raised, and it’s since been reviewed! I’ll probably work on some other test files next time.</p>
<h2 id="thoughts">Thoughts</h2>
<p>I think my third day of contribution went pretty well, it felt great to get some PRs raised - hopefully these don’t end up blocked on review. I think I’ll keep looking out for issues to work on before next time, because I spent a non-zero percentage of the day looking for issues, which is not the most efficient use of time. Still enjoying and feeling good about everything!</p>Jackson RileyNot one, dear reader, not two, but three pull requests were submitted today. What a rush.First Subinterpreters PR2019-11-21T00:00:00+00:002019-11-21T00:00:00+00:00/2019/11/21/LewisGaul<p>I decided it would be good to have a couple of days of EnHackathon in a row this
time for continuity. I’ve taken a step back from helping to find things for
other participants to work on - now that we’re past the first couple of days
people are clearer on the process.</p>
<p>I continued to focus on issues in the subinterpreters project, with continued
help from Phil and Ben. I think it’s been nice for all of us to have people to
work with and bounce questions and ideas off!</p>
<h2 id="subinterpreters-continued">Subinterpreters Continued</h2>
<p>Picking up where I left off <a href="../12/LewisGaul.html">last time</a>, I spent a fair
amount of time making final improvements to the <code class="language-plaintext highlighter-rouge">channels_list_interpreters()</code>
API I’d been working on. I then started taking a look into a couple of other
issues.</p>
<p>I’d also recommend reading <a href="./PhilConnell.html">Phil’s post</a> about the issue he
was working on.</p>
<h3 id="new-list-channel-interpreters-api">New ‘List Channel Interpreters’ API</h3>
<p><a href="https://github.com/ericsnowcurrently/multi-core-python/issues/52">https://github.com/ericsnowcurrently/multi-core-python/issues/52</a><br />
<a href="https://bugs.python.org/issue38880">https://bugs.python.org/issue38880</a><br />
<a href="https://github.com/python/cpython/pull/17323">https://github.com/python/cpython/pull/17323</a></p>
<p>Having not heard from Eric since last time, I decided to plough ahead with my
interpretation of the implementation required. There were a few concrete things
left to finish off:</p>
<ul>
<li>Make the API accept an argument determining whether to list interpreters for
the send of receive end of the channel.</li>
<li>Do a git rebase to get rid of the PEP 554 changes I originally based my branch
off.</li>
<li>Add tests (in Lib/test/test__xxsubinterpreters.py).</li>
</ul>
<p>It didn’t take too long to get those things finished off with Ben’s help. I then
asked Phil to take a look through in case he had any thoughts, which it turned
out he did! Most of his comments were related to details of the C implementation.
Going through his points took a bit of time (e.g. working out how to define
<code class="language-plaintext highlighter-rouge">PY_INT64_T_MAX</code> without assuming <code class="language-plaintext highlighter-rouge">stdint.h</code> is available!), but in a way they
were an interesting exercise in C!</p>
<p>I then raised my changes as
<a href="https://github.com/python/cpython/pull/17323">my first CPython PR</a>! Upon
review, Victor Stinner requested that I revert the addition of the
<code class="language-plaintext highlighter-rouge">PY_INT64_T_MAX</code> macro in favour of <code class="language-plaintext highlighter-rouge">INT64_MAX</code> in <code class="language-plaintext highlighter-rouge">stdint.h</code>, but Eric has
since echoed Phil’s question of whether we can assume <code class="language-plaintext highlighter-rouge">stdint.h</code> will always be
available…</p>
<h3 id="other-issues">Other Issues</h3>
<p>After getting the ‘list channel interpreters’ PR up, I then started looking in the
<a href="https://github.com/ericsnowcurrently/multi-core-python/issues/">subinterpreters repo</a>
for something to work on next. I started looking into the following:</p>
<ul>
<li><a href="https://bugs.python.org/issue37292">issue37292</a> -
This issue suggests it should be possible to unpickle an object in a
subinterpreter when the object’s class is defined in the <code class="language-plaintext highlighter-rouge">__main__</code> namespace.
Initially it seems surprising that this should be expected to work, but it
seems something like this is possible with <code class="language-plaintext highlighter-rouge">subprocess</code>. My next step is to
work out how it’s done with <code class="language-plaintext highlighter-rouge">subprocess</code> and see if that approach applies in
the same way to subinterpreters.</li>
<li><a href="https://bugs.python.org/issue36225">issue36225</a> -
This issue was easy to get started on - the BPO issue has detailed instructions
from Nick Coghlan for how to write a test that reproduces the problem. The
problem is that the embedding API
<a href="https://docs.python.org/3/c-api/init.html#c.Py_EndInterpreter">states that</a>
subinterpreters will be implicitly be shut down when <code class="language-plaintext highlighter-rouge">Py_FinalizeEx()</code> is
called, when in reality if any subinterpreters are left alive when calling
this it currently causes a <code class="language-plaintext highlighter-rouge">Py_FatalError</code>. Reproducing the issue was the easy
part - next time I will look into fixing it!</li>
</ul>
<h2 id="summary">Summary</h2>
<p>I feel like the task to add the <code class="language-plaintext highlighter-rouge">channel_list_interpreters()</code> API was an
excellent first issue to work on, and I feel lucky to have found it - it’s
taught me a lot about CPython’s C API while also being very satisfying to work
on and introducing me to the subinterpreters module. It’s also good to have been
reminded of how to tackle bug fixes: first work out how to reproduce the
issue, then narrow down the problem and identify the culpable function (perhaps
using a crash backtrace, or tools such as GDB). I now feel more capable of
picking up issues and having a hope of being able to make progress!</p>
<h2 id="whats-next">What’s Next?</h2>
<p>Personally I have one work day left to spend on EnHackathon, although there may
be a couple of people who haven’t taken part every day that may want to do a day
or two without me.</p>
<p>I’m wondering whether to encourage more people in the company to join us for my
last day, with the aim of getting people started to lower the barrier to
contributing in the future. This needs to be weighed up against the level of
productivity we can expect if people only have one day to spend on it.</p>
<p>There’s also the question of whether/how EnHackathon will be happening next year.
Some people have suggested it would be good to do it early in the year while
this month’s work is fresh in our minds, which seems like a good suggestion to
me. I think it’s unlikely I’ll have such a leading role next year, and may
choose not to take part at all (I haven’t decided yet, but might like to use
the time to do an entirely different type of volunteering). Regardless of the
future of EnHackathon, I’ll definitely be continuing with CPython contribution
in my spare time.</p>Lewis GaulI decided it would be good to have a couple of days of EnHackathon in a row this time for continuity. I’ve taken a step back from helping to find things for other participants to work on - now that we’re past the first couple of days people are clearer on the process.D[a]emonic threads2019-11-21T00:00:00+00:002019-11-21T00:00:00+00:00/2019/11/21/PhilConnell<p>Lacking a solid plan-in-advance for the day, I decided on the (seemingly
masochistic!) plan to look into the odd post-finalization crash I started on
<a href="../21/PhilConnell.html">last time</a>.</p>
<p>Recapping my previous summary:</p>
<ul>
<li>
<p>The <a href="https://bugs.python.org/issue33608">subinterpreter patch</a> caused crashes
during shutdown of multithreaded processes with long-lived daemon threads.</p>
</li>
<li>
<p>The crashes were seen intermittently on a handful of the Python
<a href="https://buildbot.net/">Buildbot</a> fleet, mostly on FreeBSD and never on
Linux!</p>
</li>
</ul>
<h2 id="victor-rides-to-the-rescue">Victor Rides To The Rescue</h2>
<p>So the first obvious question was: had anyone provided a core+symbols, or
decoded backtraces?</p>
<p>Luckily, on one of the linked issues <a href="https://bugs.python.org/issue36114#msg337090">Victor Stinner had
posted exactly that</a>:</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>Thread 2 (LWP 101669):
#0 0x0000000801e0495d in CRYPTO_free () from /usr/local/lib/libcrypto.so.11
#1 0x0000000801de09bc in ?? () from /usr/local/lib/libcrypto.so.11
#2 0x0000000801e005b7 in OPENSSL_cleanup () from /usr/local/lib/libcrypto.so.11
#3 0x0000000800825ab1 in __cxa_finalize () from /lib/libc.so.7
#4 0x00000008007b2791 in exit () from /lib/libc.so.7
#5 0x00000008004ca84e in Py_Exit (sts=0) at Python/pylifecycle.c:2166
#6 0x00000008004d6ffb in handle_system_exit () at Python/pythonrun.c:641
#7 0x00000008004d6b07 in PyErr_PrintEx (set_sys_last_vars=1) at Python/pythonrun.c:651
#8 0x00000008004d698e in PyErr_Print () at Python/pythonrun.c:547
#9 PyRun_SimpleStringFlags (command=0x80150de38 "from multiprocessing.spawn import spawn_main; spawn_main(tracker_fd=4, pipe_handle=9)\n", flags=0x7fffffffe860) at Python/pythonrun.c:462
#10 0x00000008004f7a39 in pymain_run_command (command=<optimized out>, cf=0x0) at Modules/main.c:527
#11 pymain_run_python (interp=<optimized out>, exitcode=<optimized out>) at Modules/main.c:804
#12 pymain_main (args=<optimized out>) at Modules/main.c:896
#13 0x00000008004f84f7 in _Py_UnixMain (argc=<optimized out>, argv=0x801bdb848) at Modules/main.c:937
#14 0x0000000000201120 in _start (ap=<optimized out>, cleanup=<optimized out>) at /usr/src/lib/csu/amd64/crt1.c:76
Thread 1 (LWP 100922):
#0 0x000000080047ca84 in take_gil (tstate=0x80262ba10) at Python/ceval_gil.h:216
#1 0x000000080047d074 in PyEval_RestoreThread (tstate=0x80262ba10) at Python/ceval.c:281
#2 0x00000008004f3325 in _Py_write_impl (fd=5, buf=0x8025e3e00, count=29, gil_held=1) at Python/fileutils.c:1558
#3 0x00000008005051aa in os_write_impl (module=<optimized out>, fd=<optimized out>, data=<optimized out>) at ./Modules/posixmodule.c:8798
#4 os_write (module=<optimized out>, args=0x8015b3ba8, nargs=<optimized out>) at ./Modules/clinic/posixmodule.c.h:4182
#5 0x0000000800385a5d in _PyMethodDef_RawFastCallKeywords (method=<optimized out>, self=0x80127a5f0, args=<optimized out>, nargs=2, kwnames=<optimized out>) at Objects/call.c:653
#6 0x00000008003847de in _PyCFunction_FastCallKeywords (func=0x8012820c0, args=0x0, nargs=0, kwnames=0xdbdbdbdbdbdbdbdb) at Objects/call.c:732
#7 0x000000080048e1f5 in call_function (pp_stack=0x7fffdf7faf98, oparg=<optimized out>, kwnames=0x0) at Python/ceval.c:4673
#8 0x00000008004892fa in _PyEval_EvalFrameDefault (f=0x8015b3a00, throwflag=<optimized out>) at Python/ceval.c:3294
#9 0x000000080048f03a in PyEval_EvalFrameEx (f=<optimized out>, throwflag=<error reading variable: Cannot access memory at address 0x0>) at Python/ceval.c:624
#10 _PyEval_EvalCodeWithName (_co=<optimized out>, globals=<optimized out>, locals=<optimized out>, args=<optimized out>, argcount=2, kwnames=0x0, kwargs=0x8025e25f8, kwcount=0, kwstep=1, defs=0x802567798,
defcount=1, kwdefs=0x0, closure=0x0, name=0x8017e2930, qualname=0x80255b7c0) at Python/ceval.c:4035
#11 0x0000000800384690 in _PyFunction_FastCallKeywords (func=<optimized out>, stack=<optimized out>, nargs=0, kwnames=<optimized out>) at Objects/call.c:435
#12 0x000000080048e35f in call_function (pp_stack=0x7fffdf7fb2f0, oparg=<optimized out>, kwnames=0x0) at Python/ceval.c:4721
</code></pre></div></div>
<p>Thread 2 is the main thread, and has just finalized the interpreter and called
<code class="language-plaintext highlighter-rouge">exit()</code>:</p>
<div class="language-c highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="kt">void</span> <span class="n">_Py_NO_RETURN</span>
<span class="nf">Py_Exit</span><span class="p">(</span><span class="kt">int</span> <span class="n">sts</span><span class="p">)</span>
<span class="p">{</span>
<span class="k">if</span> <span class="p">(</span><span class="n">Py_FinalizeEx</span><span class="p">()</span> <span class="o"><</span> <span class="mi">0</span><span class="p">)</span> <span class="p">{</span>
<span class="n">sts</span> <span class="o">=</span> <span class="mi">120</span><span class="p">;</span>
<span class="p">}</span>
<span class="n">exit</span><span class="p">(</span><span class="n">sts</span><span class="p">);</span> <span class="cm">/* <-- frame 5 of thread 2 is here */</span>
<span class="p">}</span>
</code></pre></div></div>
<p>Thread 1 is the villain of our story – it has just finished a blocking
<code class="language-plaintext highlighter-rouge">write()</code> call and so decided to reacquire the GIL:</p>
<div class="language-c highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="n">_Py_write_impl</span><span class="p">(</span><span class="kt">int</span> <span class="n">fd</span><span class="p">,</span> <span class="k">const</span> <span class="kt">void</span> <span class="o">*</span><span class="n">buf</span><span class="p">,</span> <span class="kt">size_t</span> <span class="n">count</span><span class="p">,</span> <span class="kt">int</span> <span class="n">gil_held</span><span class="p">)</span>
<span class="p">{</span>
<span class="p">...</span>
<span class="n">Py_BEGIN_ALLOW_THREADS</span>
<span class="n">errno</span> <span class="o">=</span> <span class="mi">0</span><span class="p">;</span>
<span class="cp">#ifdef MS_WINDOWS
</span> <span class="n">n</span> <span class="o">=</span> <span class="n">write</span><span class="p">(</span><span class="n">fd</span><span class="p">,</span> <span class="n">buf</span><span class="p">,</span> <span class="p">(</span><span class="kt">int</span><span class="p">)</span><span class="n">count</span><span class="p">);</span>
<span class="cp">#else
</span> <span class="n">n</span> <span class="o">=</span> <span class="n">write</span><span class="p">(</span><span class="n">fd</span><span class="p">,</span> <span class="n">buf</span><span class="p">,</span> <span class="n">count</span><span class="p">);</span>
<span class="cp">#endif
</span> <span class="cm">/* save/restore errno because PyErr_CheckSignals()
* and PyErr_SetFromErrno() can modify it */</span>
<span class="n">err</span> <span class="o">=</span> <span class="n">errno</span><span class="p">;</span>
<span class="n">Py_END_ALLOW_THREADS</span> <span class="cm">/* <-- frame 2 of thread 1 is here */</span>
</code></pre></div></div>
<p>However, we’ve just finalized the interpreter in Thread 2, cleaning up all of
Python’s runtime state. So, unsurprisingly, Thread 1’s attempt to access its
threadstate object doesn’t end well:</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>(gdb) p tstate
$2 = (PyThreadState *) 0x8027e2050
(gdb) p *tstate
$3 = {
prev = 0xdbdbdbdbdbdbdbdb,
next = 0xdbdbdbdbdbdbdbdb,
interp = 0xdbdbdbdbdbdbdbdb,
...
}
(gdb) p _PyRuntime.gilstate.tstate_current
$4 = {
_value = 0
}
</code></pre></div></div>
<h2 id="reproducing-the-crash">Reproducing The Crash</h2>
<p>Interestingly, there’s nothing about the sequence of events above that has
anything do to with Eric’s subinterpreter changes. So this looks a lot like a
baseline race condition that’s simply made (more) likely. Can we reproduce it?</p>
<p>Some hacking about later (and after <strong>much</strong> relearning of how to embed
Python…) I eventually came up with some crashing code (hurrah :)):</p>
<div class="language-c highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="cp">#include <assert.h>
#include <pthread.h>
#include <time.h>
</span>
<span class="cp">#include <Python.h>
</span>
<span class="kt">void</span> <span class="o">*</span>
<span class="nf">evil_main</span> <span class="p">(</span><span class="kt">void</span> <span class="o">*</span><span class="n">_</span><span class="p">)</span>
<span class="p">{</span>
<span class="n">PyGILState_STATE</span> <span class="n">gilstate</span><span class="p">;</span>
<span class="n">gilstate</span> <span class="o">=</span> <span class="n">PyGILState_Ensure</span><span class="p">();</span>
<span class="n">printf</span><span class="p">(</span><span class="s">"OTHER: acquired GIL</span><span class="se">\n</span><span class="s">"</span><span class="p">);</span>
<span class="n">Py_BEGIN_ALLOW_THREADS</span>
<span class="n">printf</span><span class="p">(</span><span class="s">"OTHER: released GIL</span><span class="se">\n</span><span class="s">"</span><span class="p">);</span>
<span class="n">sleep</span><span class="p">(</span><span class="mi">2</span><span class="p">);</span>
<span class="n">printf</span><span class="p">(</span><span class="s">"OTHER: attempt to acquire GIL...crash!</span><span class="se">\n</span><span class="s">"</span><span class="p">);</span>
<span class="n">Py_END_ALLOW_THREADS</span>
<span class="cm">/* Dead code */</span>
<span class="n">assert</span><span class="p">(</span><span class="mi">0</span><span class="p">);</span>
<span class="n">PyGILState_Release</span><span class="p">(</span><span class="n">gilstate</span><span class="p">);</span>
<span class="k">return</span> <span class="p">(</span><span class="nb">NULL</span><span class="p">);</span>
<span class="p">}</span>
<span class="kt">int</span>
<span class="nf">main</span> <span class="p">(</span><span class="kt">int</span> <span class="n">argc</span><span class="p">,</span> <span class="kt">char</span> <span class="o">**</span><span class="n">argv</span><span class="p">)</span>
<span class="p">{</span>
<span class="kt">int</span> <span class="n">rc</span><span class="p">;</span>
<span class="n">pthread_t</span> <span class="n">evil</span><span class="p">;</span>
<span class="cm">/* Suppress warnings from get_path.c */</span>
<span class="n">Py_FrozenFlag</span> <span class="o">=</span> <span class="mi">1</span><span class="p">;</span>
<span class="cm">/* Don't try to import site */</span>
<span class="n">Py_NoSiteFlag</span> <span class="o">=</span> <span class="mi">1</span><span class="p">;</span>
<span class="n">Py_SetPath</span><span class="p">(</span><span class="s">L"./Lib"</span><span class="p">);</span>
<span class="n">Py_Initialize</span><span class="p">();</span>
<span class="n">PyEval_InitThreads</span><span class="p">();</span>
<span class="n">rc</span> <span class="o">=</span> <span class="n">pthread_create</span><span class="p">(</span><span class="o">&</span><span class="n">evil</span><span class="p">,</span> <span class="nb">NULL</span><span class="p">,</span> <span class="n">evil_main</span><span class="p">,</span> <span class="nb">NULL</span><span class="p">);</span>
<span class="n">assert</span><span class="p">(</span><span class="n">rc</span> <span class="o">==</span> <span class="mi">0</span><span class="p">);</span>
<span class="n">Py_BEGIN_ALLOW_THREADS</span>
<span class="n">printf</span><span class="p">(</span><span class="s">"MAIN: allow other thread to execute</span><span class="se">\n</span><span class="s">"</span><span class="p">);</span>
<span class="n">sleep</span><span class="p">(</span><span class="mi">1</span><span class="p">);</span>
<span class="n">Py_END_ALLOW_THREADS</span>
<span class="n">Py_Finalize</span><span class="p">();</span>
<span class="n">printf</span><span class="p">(</span><span class="s">"MAIN: interpreter finalized</span><span class="se">\n</span><span class="s">"</span><span class="p">);</span>
<span class="n">sleep</span><span class="p">(</span><span class="mi">10</span><span class="p">);</span>
<span class="cm">/* Dead code */</span>
<span class="n">assert</span><span class="p">(</span><span class="mi">0</span><span class="p">);</span>
<span class="k">return</span> <span class="p">(</span><span class="mi">0</span><span class="p">);</span>
<span class="p">}</span>
</code></pre></div></div>
<p>The intention is to force the same kind of race condition:</p>
<ul>
<li>Spawn a thread and allow it to acquire the GIL.</li>
<li>Drop the GIL in the thread and execute a blocking call.</li>
<li>Meanwhile, finalize Python in the main thread.</li>
<li>When the other thread wakes and tries to reacquire the GIL, it crashes!</li>
</ul>
<p>This is nicely illustrated by valgrind:</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>$ valgrind --suppressions=Misc/valgrind-python.supp fini_crash
==266836== Memcheck, a memory error detector
==266836== Copyright (C) 2002-2017, and GNU GPL'd, by Julian Seward et al.
==266836== Using Valgrind-3.15.0 and LibVEX; rerun with -h for copyright info
==266836== Command: fini_crash
==266836==
MAIN: allow other thread to execute
OTHER: acquired GIL
OTHER: released GIL
MAIN: interpreter finalized
OTHER: attempt to acquire GIL...crash!
==266836== Thread 2:
==266836== Invalid read of size 8
==266836== at 0x15607D: PyEval_RestoreThread (ceval.c:389)
==266836== by 0x15479F: evil_main (in /home/phconnel/dev/cpython/fini_crash)
==266836== by 0x48B94CE: start_thread (in /usr/lib/libpthread-2.30.so)
==266836== by 0x4B232D2: clone (in /usr/lib/libc-2.30.so)
==266836== Address 0x4d17270 is 16 bytes inside a block of size 264 free'd
==266836== at 0x48399AB: free (vg_replace_malloc.c:540)
==266836== by 0x1773FF: tstate_delete_common (pystate.c:829)
==266836== by 0x1773FF: _PyThreadState_Delete (pystate.c:848)
==266836== by 0x1773FF: zapthreads (pystate.c:311)
==266836== by 0x1773FF: PyInterpreterState_Delete (pystate.c:321)
==266836== by 0x174920: finalize_interp_delete (pylifecycle.c:1242)
==266836== by 0x174920: Py_FinalizeEx.part.0 (pylifecycle.c:1400)
==266836== by 0x15487B: main (in /home/phconnel/dev/cpython/fini_crash)
==266836== Block was alloc'd at
==266836== at 0x483877F: malloc (vg_replace_malloc.c:309)
==266836== by 0x178D7C: new_threadstate (pystate.c:557)
==266836== by 0x178D7C: PyThreadState_New (pystate.c:629)
==266836== by 0x178D7C: PyGILState_Ensure (pystate.c:1288)
==266836== by 0x154759: evil_main (in /home/phconnel/dev/cpython/fini_crash)
==266836== by 0x48B94CE: start_thread (in /usr/lib/libpthread-2.30.so)
==266836== by 0x4B232D2: clone (in /usr/lib/libc-2.30.so)
</code></pre></div></div>
<p>This looks exactly like the crash reported by Victor but reproduced on an
unmodified <code class="language-plaintext highlighter-rouge">master</code> checkout of CPython!</p>
<h2 id="can-we-fix-it">Can We Fix It?</h2>
<p>In the case where the non-main thread is using <code class="language-plaintext highlighter-rouge">Py_BEGIN_ALLOW_THREADS</code> and
<code class="language-plaintext highlighter-rouge">Py_END_ALLOW_THREADS</code> the problem is that the threadstate object is stored in
a pointer by the <code class="language-plaintext highlighter-rouge">BEGIN</code> macro but then quietly invalidated before the <code class="language-plaintext highlighter-rouge">END</code>
macro attempts to use it.</p>
<p>So realistically the only way to fix this (without somehow banning this
behaviour in general) is by adding a layer of indirection. My patch on <a href="https://bugs.python.org/issue33608">issue
33608</a> does this, wrapping the threadstate
pointer in a structure with references in both directions between the wrapper
and threadstate:</p>
<div class="language-c highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="c1">// The PyThreadState typedef is in Include/pystate.h.</span>
<span class="k">struct</span> <span class="n">_ts</span> <span class="p">{</span>
<span class="p">...</span>
<span class="cm">/* (Optional) wrapper that will survive this thread state being freed, used
* by daemon threads that wake up after interpreter finalization.
*/</span>
<span class="k">struct</span> <span class="n">_ts_wrapper</span> <span class="o">*</span><span class="n">wrapper</span><span class="p">;</span>
<span class="p">...</span>
<span class="p">};</span>
<span class="c1">// The PyThreadStateWrapper typedef is in Include/pystate.h.</span>
<span class="k">struct</span> <span class="n">_ts_wrapper</span> <span class="p">{</span>
<span class="n">PyThreadState</span> <span class="o">*</span><span class="n">ts</span><span class="p">;</span>
<span class="p">};</span>
</code></pre></div></div>
<p>This gives a way to invalidate the state held by the non-main thread: set the
<code class="language-plaintext highlighter-rouge">ts</code> pointer to <code class="language-plaintext highlighter-rouge">NULL</code> in the wrapper if the Python runtime is being finalized.</p>
<h2 id="open-questions">Open Questions</h2>
<p>There are a few of these! I’m particularly interested in what the others on
<a href="https://bugs.python.org/issue33608">the issue</a> think of this direction:</p>
<ol>
<li>Is this an acceptable change in general? (For what it’s worth, I don’t see
an obvious alternative!)</li>
<li>What’s needed to productize the patch? There are a couple of standout things
for me:
<ul>
<li>There’s one race condition that still needs to be fixed (what happens if
finalization is triggered while the non-main thread is attempting to
acquire the GIL?)</li>
<li>What’s the best approach for adding tests for this case that can be
maintained long-term?</li>
</ul>
</li>
<li>To what extent does this actually fix the problem exposed by Eric’s pending
calls changes? It certainly matches Victor’s particular backtrace (and I’m
fairly confident entirely addresses that instance) but are there other
cases?</li>
</ol>Phil ConnellLacking a solid plan-in-advance for the day, I decided on the (seemingly masochistic!) plan to look into the odd post-finalization crash I started on last time.