gh-140797: Forbid capturing groups in re.Scanner lexicon patterns by Abhi210 · Pull Request #140944 · python/cpython

Abhi210 · 2025-11-03T11:16:30Z

This PR adds validation to re.Scanner.__init__ that rejects lexicon patterns containing capturing groups. If a user-supplied pattern contains any capturing groups, Scanner now raises ValueError with a clear message advising the use of non-capturing groups (?:...) instead. Further, tests were added to assert ValueError for lexicons containing capturing groups and a passing test for non-capturing group.

Issue: Crash when using multiple capturing groups in re.Scanner #140797

Lib/re/__init__.py

Lib/test/test_re.py

Lib/re/__init__.py

It adds validation to re.Scanner.init that rejects lexicon patterns containing capturing groups. If a user-supplied pattern contains any capturing groups, Scanner now raises ValueError with a clear message advising the use of non-capturing groups (?:...) instead.

wjssz

After you update the patch, you may ask core developer Serhiy to review.

Lib/re/__init__.py

Lib/test/test_re.py

wjssz

It seems that you are a beginner, please be patient.
After my review is completed, then ask core developer Serhiy to review.

wjssz · 2025-11-04T05:43:32Z

Lib/re/__init__.py

        for phrase, action in lexicon:
+            sub_pattern = _parser.parse(phrase, flags)
+            if sub_pattern.state.groups != 1:
+                raise ValueError(


You can write in one line. A line should <= 80 characters.

raise ValueError("Can not use capturing groups in re.Scanner.")

Thank you for the suggestion. I have resolved it now

wjssz · 2025-11-04T05:48:37Z

Lib/test/test_re.py

+
+        #Capturing group throws an error
+        lex = [("(a)b", None)]
+        with self.assertRaises(ValueError):


You may check exception message.

msg = "Can not use capturing groups in re.Scanner" with self.assertRaisesRegex(ValueError, msg): ...

Thank you for the suggestion! I have resolved it now. Need to learn a lot!

wjssz · 2025-11-04T06:44:45Z

Lib/test/test_re.py

                           'op+', 'bar'], ''))

+    def test_bug_140797(self):
+        #bug 140797: remove capturing groups compilation form re.Scanner


Please add a space after # in comments.

- #Capturing group throws an error + # Capturing group throws an error

And add a space after , in functions arguments.

- with self.assertRaisesRegex(ValueError,msg): + with self.assertRaisesRegex(ValueError, msg):

Then looks good to me.

Oops! Thank you again! Resolved

wjssz · 2025-11-04T06:58:48Z

Lib/test/test_re.py

                         (['sum', 'op=', 3, 'op*', 'foo', 'op+', 312.5,
                           'op+', 'bar'], ''))

+    def test_bug_140797(self):


My negligence, please use test_bug_gh140797 as the name.
If no "gh", It may refer to the previous bug tracker.
Sorry for this.

- # bug 140797: remove capturing groups compilation form re.Scanner + # gh140797: capturing groups is not allowed in re.Scanner

Done! Thank you again for your time and suggestions!

wjssz · 2025-11-04T07:12:54Z

Lib/test/test_re.py

+        # Capturing group throws an error
+        lex = [("(a)b", None)]
+        with self.assertRaisesRegex(ValueError, msg):
+            Scanner(lex)


This saves a line...
My fault again.

- Scanner(lex) + Scanner([("(a)b", None)])

Done! Testing sure takes the time 😂

wjssz · 2025-11-04T07:29:55Z

I have checked basic problems, @serhiy-storchaka please review.

serhiy-storchaka

Thank you for your PR @Abhi210 and thank you for your review @wjssz.

There are some minor nitpicks, overall LGTM.

serhiy-storchaka · 2025-11-04T08:01:04Z

Lib/re/__init__.py

        for phrase, action in lexicon:
+            sub_pattern = _parser.parse(phrase, flags)
+            if sub_pattern.state.groups != 1:
+                raise ValueError("Can not use capturing groups in re.Scanner")


"Cannot". This is the most commonly used variant.

Done! Thank you

Lib/test/test_re.py

serhiy-storchaka · 2025-11-04T08:07:31Z

Misc/NEWS.d/next/Library/2025-11-03-16-23-54.gh-issue-140797.DuFEeR.rst

@@ -0,0 +1,4 @@
+The re.Scanner class now forbids regular expressions containing capturing


Mention that that class is undocumented. You can also use some formatting, even if the link does not work: :class:`!re.Scanner`.

Done! Thank you 😊

…s- Rev-7

serhiy-storchaka

Two more nitpicks and LGTM. 👍

Lib/test/test_re.py

Co-authored-by: Serhiy Storchaka <[email protected]>

miss-islington-app · 2025-11-04T10:54:56Z

Thanks @Abhi210 for the PR, and @serhiy-storchaka for merging it 🌮🎉.. I'm working now to backport this PR to: 3.14.
🐍🍒⛏🤖

miss-islington-app · 2025-11-04T10:54:56Z

Thanks @Abhi210 for the PR, and @serhiy-storchaka for merging it 🌮🎉.. I'm working now to backport this PR to: 3.13.
🐍🍒⛏🤖

…ns (pythonGH-140944) (cherry picked from commit fa9c3ee) Co-authored-by: Abhishek Tiwari <[email protected]>

bedevere-app · 2025-11-04T10:55:05Z

GH-140982 is a backport of this pull request to the 3.14 branch.

…ns (pythonGH-140944) (cherry picked from commit fa9c3ee) Co-authored-by: Abhishek Tiwari <[email protected]>

bedevere-app · 2025-11-04T10:55:11Z

GH-140983 is a backport of this pull request to the 3.13 branch.

…rns (GH-140944) (GH-140983) (cherry picked from commit fa9c3ee) Co-authored-by: Abhishek Tiwari <[email protected]>

…rns (GH-140944) (GH-140982) (cherry picked from commit fa9c3ee) Co-authored-by: Abhishek Tiwari <[email protected]>

3.14 changed behavior of re.Scanner() capture groups, breaking cmake-format. Change was also backported to 3.13. python/cpython#140944

… lexicon patterns (pythonGH-140944) (pythonGH-140983)" This reverts commit ee894d2.

…r lexicon patterns (GH-140944) (GH-140983)" (GH-142231) Revert "[3.13] gh-140797: Forbid capturing groups in re.Scanner lexicon patterns (GH-140944) (GH-140983)" This reverts commit ee894d2.

… lexicon patterns (pythonGH-140944) (pythonGH-140982)" This reverts commit e5266fc.

…on patterns (GH-140944) (GH-140982)" (#142309)

…ns (pythonGH-140944)

Required as of Python 3.15. See: python/cpython#140944

bedevere-app bot added the awaiting review label Nov 3, 2025

bedevere-app bot mentioned this pull request Nov 3, 2025

Crash when using multiple capturing groups in re.Scanner #140797

Closed

Abhi210 force-pushed the fix-scanner-capturing-groups-backup branch from 12bf67b to e9e76d0 Compare November 3, 2025 11:16

wjssz reviewed Nov 3, 2025

View reviewed changes

Lib/re/__init__.py Show resolved Hide resolved

Lib/test/test_re.py Outdated Show resolved Hide resolved

Lib/test/test_re.py Outdated Show resolved Hide resolved

serhiy-storchaka reviewed Nov 3, 2025

View reviewed changes

Lib/re/__init__.py Outdated Show resolved Hide resolved

Abhi210 force-pushed the fix-scanner-capturing-groups-backup branch from e9e76d0 to 12bf67b Compare November 3, 2025 13:13

Abhi210 force-pushed the fix-scanner-capturing-groups-backup branch from 0ebfddd to 29db6ca Compare November 3, 2025 14:00

wjssz reviewed Nov 3, 2025

View reviewed changes

Lib/re/__init__.py Outdated Show resolved Hide resolved

Lib/re/__init__.py Outdated Show resolved Hide resolved

Lib/test/test_re.py Outdated Show resolved Hide resolved

Abhi210 force-pushed the fix-scanner-capturing-groups-backup branch from f1d2c8b to 81f3675 Compare November 4, 2025 05:39

wjssz reviewed Nov 4, 2025

View reviewed changes

Abhi210 force-pushed the fix-scanner-capturing-groups-backup branch from 81f3675 to 401969c Compare November 4, 2025 06:35

wjssz reviewed Nov 4, 2025

View reviewed changes

Abhi210 force-pushed the fix-scanner-capturing-groups-backup branch from 401969c to db0ea4a Compare November 4, 2025 06:49

wjssz reviewed Nov 4, 2025

View reviewed changes

Abhi210 force-pushed the fix-scanner-capturing-groups-backup branch from db0ea4a to 0f9a934 Compare November 4, 2025 07:05

wjssz reviewed Nov 4, 2025

View reviewed changes

Abhi210 force-pushed the fix-scanner-capturing-groups-backup branch from 0f9a934 to 0197f65 Compare November 4, 2025 07:14

serhiy-storchaka reviewed Nov 4, 2025

View reviewed changes

pythongh-140797 forbid capturing groups in re.Scanner lexicon pattern…

feaee4e

…s- Rev-7

Abhi210 force-pushed the fix-scanner-capturing-groups-backup branch from 0197f65 to feaee4e Compare November 4, 2025 08:31

serhiy-storchaka approved these changes Nov 4, 2025

View reviewed changes

Lib/test/test_re.py Outdated Show resolved Hide resolved

Lib/test/test_re.py Outdated Show resolved Hide resolved

bedevere-app bot added awaiting merge and removed awaiting review labels Nov 4, 2025

Abhi210 and others added 2 commits November 4, 2025 15:29

Update Lib/test/test_re.py

d1d582e

Co-authored-by: Serhiy Storchaka <[email protected]>

Update Lib/test/test_re.py

f4f7d4d

Co-authored-by: Serhiy Storchaka <[email protected]>

serhiy-storchaka merged commit fa9c3ee into python:main Nov 4, 2025
46 checks passed

bedevere-app bot removed the awaiting merge label Nov 4, 2025

serhiy-storchaka added the needs backport to 3.13 bugs and security fixes label Nov 4, 2025

serhiy-storchaka added the needs backport to 3.14 bugs and security fixes label Nov 4, 2025

bedevere-app bot removed the needs backport to 3.14 bugs and security fixes label Nov 4, 2025

bedevere-app bot removed the needs backport to 3.13 bugs and security fixes label Nov 4, 2025

serhiy-storchaka pushed a commit that referenced this pull request Nov 4, 2025

[3.13] gh-140797: Forbid capturing groups in re.Scanner lexicon patte…

ee894d2

…rns (GH-140944) (GH-140983) (cherry picked from commit fa9c3ee) Co-authored-by: Abhishek Tiwari <[email protected]>

serhiy-storchaka pushed a commit that referenced this pull request Nov 4, 2025

[3.14] gh-140797: Forbid capturing groups in re.Scanner lexicon patte…

e5266fc

…rns (GH-140944) (GH-140982) (cherry picked from commit fa9c3ee) Co-authored-by: Abhishek Tiwari <[email protected]>

Abhi210 deleted the fix-scanner-capturing-groups-backup branch November 11, 2025 05:20

eli-schwartz mentioned this pull request Dec 3, 2025

re.Scanner lexicon patterns with capturing groups abort the program with a traceback #142220

Closed

Gabrielcarvfer added a commit to Gabrielcarvfer/ns-3-dev that referenced this pull request Dec 3, 2025

ci: Pin cmake-format to Python 3.12

c7c2bc1

3.14 changed behavior of re.Scanner() capture groups, breaking cmake-format. Change was also backported to 3.13. python/cpython#140944

serhiy-storchaka added a commit to serhiy-storchaka/cpython that referenced this pull request Dec 3, 2025

Revert "[3.13] pythongh-140797: Forbid capturing groups in re.Scanner…

24e5fa5

… lexicon patterns (pythonGH-140944) (pythonGH-140983)" This reverts commit ee894d2.

serhiy-storchaka added a commit to serhiy-storchaka/cpython that referenced this pull request Dec 3, 2025

Revert "[3.13] pythongh-140797: Forbid capturing groups in re.Scanner…

3afcd4c

… lexicon patterns (pythonGH-140944) (pythonGH-140983)" This reverts commit ee894d2.

hugovk added a commit to hugovk/cpython that referenced this pull request Dec 5, 2025

Revert "[3.14] pythongh-140797: Forbid capturing groups in re.Scanner…

e149e66

… lexicon patterns (pythonGH-140944) (pythonGH-140982)" This reverts commit e5266fc.

hugovk added a commit that referenced this pull request Dec 5, 2025

[3.14] Revert "gh-140797: Forbid capturing groups in re.Scanner lexic…

4cb6cbb

…on patterns (GH-140944) (GH-140982)" (#142309)

StanFromIreland pushed a commit to StanFromIreland/cpython that referenced this pull request Dec 6, 2025

pythongh-140797: Forbid capturing groups in re.Scanner lexicon patter…

3e3a0af

…ns (pythonGH-140944)

luk1337 added a commit to luk1337/quodlibet that referenced this pull request Jan 23, 2026

Use non-capturing group in PatternLexer

598b6cd

Required as of Python 3.15. See: python/cpython#140944

luk1337 mentioned this pull request Jan 23, 2026

Use non-capturing group in PatternLexer quodlibet/quodlibet#4845

Merged

4 tasks

luk1337 added a commit to luk1337/quodlibet that referenced this pull request Jan 23, 2026

Use non-capturing group in PatternLexer

013b70d

Required as of Python 3.15. See: python/cpython#140944

luk1337 added a commit to luk1337/quodlibet that referenced this pull request Jan 23, 2026

Use non-capturing group in PatternLexer

a30af16

Required as of Python 3.15. See: python/cpython#140944

luk1337 added a commit to luk1337/quodlibet that referenced this pull request Jan 23, 2026

Use non-capturing group in PatternLexer

5fa42fd

Required as of Python 3.15. See: python/cpython#140944

lazka pushed a commit to quodlibet/quodlibet that referenced this pull request Jan 24, 2026

Use non-capturing group in PatternLexer

5cecd9b

Required as of Python 3.15. See: python/cpython#140944

		@@ -0,0 +1,4 @@
		The re.Scanner class now forbids regular expressions containing capturing

Uh oh!

Conversation

Abhi210 commented Nov 3, 2025 • edited by bedevere-app bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

wjssz left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

wjssz left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

wjssz commented Nov 4, 2025

Uh oh!

serhiy-storchaka left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

serhiy-storchaka left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

miss-islington-app bot commented Nov 4, 2025

Uh oh!

miss-islington-app bot commented Nov 4, 2025

Uh oh!

bedevere-app bot commented Nov 4, 2025

Uh oh!

bedevere-app bot commented Nov 4, 2025

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Abhi210 commented Nov 3, 2025 •

edited by bedevere-app bot

Loading