Fix crashes during interpreter shutdown on all Python versions (3.2.6)#500
Conversation
Backport of PR python-greenlet#499 (master) to maint/3.2 for greenlet 3.2.6, with all shutdown guards made unconditional across Python 3.9-3.13. The previous backport (3.2.5 / PR python-greenlet#495) only guarded Python < 3.11, but the vulnerability exists on ALL Python versions: Py_IsFinalizing() is set AFTER atexit handlers complete inside Py_FinalizeEx. Two independent guards now protect all shutdown phases: 1. g_greenlet_shutting_down — atexit handler registered at module init (LIFO = runs first). Covers the atexit phase where Py_IsFinalizing() is still False. 2. Py_IsFinalizing() — covers the GC collection and later phases. A compatibility shim maps to _Py_IsFinalizing() on Python < 3.13. These guards are checked in mod_getcurrent, PyGreenlet_GetCurrent, GreenletChecker, MainGreenletExactChecker, ContextExactChecker, clear_deleteme_list, ThreadState destructor, _green_dealloc_kill_started_non_main_greenlet, and AddPendingCall. Additional hardening: - clear_deleteme_list() uses std::swap (zero-allocation) - deleteme vector uses std::allocator (system malloc) - ThreadState uses std::malloc/std::free - clear_deleteme_list() preserves pending Python exceptions TDD-certified: tests fail on greenlet 3.3.2 and pass with the fix across Python 3.10-3.14. Docker verification on Python 3.9 and 3.10 confirms GUARDED on the maint/3.2 branch. Also fixes: - SPDX license identifier: Python-2.0 -> PSF-2.0 - test_dealloc_catches_GreenletExit_throws_other: use sys.unraisablehook for pytest compatibility - test_version: skip gracefully on old setuptools - Flaky USS memory test on Windows Made-with: Cursor
bc4b8fb to
00feede
Compare
Port the maintainer's (jamadden) follow-up refinements from master to the maint/3.2 backport branch: - Refactor atexit registration to use NewReference/Require framework instead of nested ifs and manual decrefs. Crash safety is not optional. (38bf3d7) - Namespace g_greenlet_shutting_down as static in namespace greenlet instead of extern across multiple files. (c545379) - Encapsulate the dual-guard pattern in IsShuttingDown() function, replacing all g_greenlet_shutting_down || Py_IsFinalizing() checks. (879a868, fbb4bcd) - Add comment explaining why g_greenlet_shutting_down does not need std::atomic<int>. (c79fb07) - Add comments on deliberate leaking and exception safety in clear_deleteme_list. (6c517cb) - Comment cleanup and formatting. (fcf6f72, 98e8fb0 partial) Skipped (not applicable to maint/3.2): - CI: bump docker/setup-qemu-action (7b56329) — different CI config - test_greenlet: revert getattr for _is_gil_enabled (98e8fb0) — test does not exist on maint/3.2 All 21 shutdown tests pass, full suite 159 passed / 1 skipped. Made-with: Cursor
|
@jamadden I ported back all your latest changes on #499 Ported (7 of 8 commits):
Skipped (2 items):
Verification: Build succeeded, all 21 shutdown tests passed, full suite 159 passed / 1 skipped. |
|
I'm extremely reluctant to backport this. Not only is the scope larger than the previous backported change, but it introduces new dependencies and, importantly, changes the way certain public APIs operate. I'm not sure any of that is appropriate for a legacy maintenance branch. |
|
@jamadden I understand - is there anything I could adjust on this PR in your opinion that could make it more "mergeable"? otherwise for older Python versions they can always use Greenlet 1.x to avoid crashes (as far as I know) but it would have been nice to have a stable 3.x version |
Summary
Backport of PR #499 to
maint/3.2for greenlet 3.2.6, targeting Python 3.9-3.13.The previous backport (3.2.5 / PR #495) only guarded Python < 3.11 (
#if !GREENLET_PY311), but the vulnerability exists on all Python versions:Py_IsFinalizing()is set AFTER atexit handlers complete insidePy_FinalizeEx. This PR removes all version guards and makes the fix unconditional.Design
Two independent guards now protect all shutdown phases:
g_greenlet_shutting_down— an atexit handler registered at module init (LIFO = runs first) sets this flag. Covers the atexit phase ofPy_FinalizeEx, wherePy_IsFinalizing()is stillFalseon all Python versions.Py_IsFinalizing()— covers the GC collection and later phases ofPy_FinalizeEx. A compatibility shim maps to_Py_IsFinalizing()on Python < 3.13.These guards are checked in
mod_getcurrent,PyGreenlet_GetCurrent,GreenletChecker,MainGreenletExactChecker,ContextExactChecker,clear_deleteme_list(),ThreadState::~ThreadState(),_green_dealloc_kill_started_non_main_greenlet, andThreadState_DestroyNoGIL::AddPendingCall.What changed vs the previous PR #500
The previous version of this PR had all guards wrapped in
#if !GREENLET_PY311, making them completely inactive on Python 3.11-3.13. This update:#if !GREENLET_PY311guards — fixes are now unconditional across Python 3.9-3.13GreenletChecker,MainGreenletExactChecker,ContextExactCheckernow have shutdown protection (these were missing entirely)getcurrent()still returns valid objects before greenlet's cleanup (guards against over-blocking)What changed (files)
C++ shutdown guards (8 files)
PyModule.cppg_greenlet_shutting_down+ atexit handler made unconditionalCObjects.cppPyGreenlet_GetCurrentguard made unconditionalPyGreenlet.cppmurder_in_place()guard made unconditional, addedg_greenlet_shutting_downTThreadState.hppclear_deleteme_list()+ destructor guards made unconditionalTThreadStateDestroy.cppAddPendingCallguard extended withg_greenlet_shutting_downgreenlet.cppgreenlet_refs.hppGreenletChecker+ContextExactCheckergreenlet_internal.hppMainGreenletExactCheckerAdditional hardening
clear_deleteme_list()usesstd::swap(zero-allocation) instead of copying thePythonAllocator-backed vectordeletemevector usesstd::allocator(systemmalloc) instead ofPyMem_MallocThreadStateusesstd::malloc/std::freeinstead ofPyObject_Mallocclear_deleteme_list()preserves any pending Python exception around its cleanup loopTests (3 files)
test_interpreter_shutdown.py— verified RED on greenlet 3.3.2 and GREEN with fix across Python 3.10-3.14getcurrent()still returns valid objects before cleanuptest_dealloc_catches_GreenletExit_throws_other— usesys.unraisablehookinstead of stderr capture (pytest compatibility)test_version— skip gracefully on old setuptoolsOther
Python-2.0→PSF-2.0TDD verification
requires-python >= 3.10)Docker verification on Python 3.9 and 3.10 confirms both Test A (GC finalization) and Test B (atexit phase) return GUARDED on this branch.
Test plan
Relationship to other PRs
master): Universal fix for 3.3.3, this PR is the backport