Skip to content

Commit b646547

Browse files
committed
Issue python#2333: Backport set and dict comprehensions syntax.
1 parent 0ca7452 commit b646547

26 files changed

Lines changed: 1294 additions & 350 deletions

‎Doc/reference/expressions.rst‎

Lines changed: 60 additions & 34 deletions
Original file line numberDiff line numberDiff line change
@@ -205,74 +205,100 @@ block, nesting from left to right, and evaluating the expression to produce a
205205
list element each time the innermost block is reached [#]_.
206206

207207

208+
.. _comprehensions:
209+
210+
Displays for sets and dictionaries
211+
----------------------------------
212+
213+
For constructing a set or a dictionary Python provides special syntax
214+
called "displays", each of them in two flavors:
215+
216+
* either the container contents are listed explicitly, or
217+
218+
* they are computed via a set of looping and filtering instructions, called a
219+
:dfn:`comprehension`.
220+
221+
Common syntax elements for comprehensions are:
222+
223+
.. productionlist::
224+
comprehension: `expression` `comp_for`
225+
comp_for: "for" `target_list` "in" `or_test` [`comp_iter`]
226+
comp_iter: `comp_for` | `comp_if`
227+
comp_if: "if" `expression_nocond` [`comp_iter`]
228+
229+
The comprehension consists of a single expression followed by at least one
230+
:keyword:`for` clause and zero or more :keyword:`for` or :keyword:`if` clauses.
231+
In this case, the elements of the new container are those that would be produced
232+
by considering each of the :keyword:`for` or :keyword:`if` clauses a block,
233+
nesting from left to right, and evaluating the expression to produce an element
234+
each time the innermost block is reached.
235+
236+
Note that the comprehension is executed in a separate scope, so names assigned
237+
to in the target list don't "leak" in the enclosing scope.
238+
239+
208240
.. _genexpr:
209241

210242
Generator expressions
211243
---------------------
212244

213245
.. index:: pair: generator; expression
246+
object: generator
214247

215248
A generator expression is a compact generator notation in parentheses:
216249

217250
.. productionlist::
218-
generator_expression: "(" `expression` `genexpr_for` ")"
219-
genexpr_for: "for" `target_list` "in" `or_test` [`genexpr_iter`]
220-
genexpr_iter: `genexpr_for` | `genexpr_if`
221-
genexpr_if: "if" `old_expression` [`genexpr_iter`]
251+
generator_expression: "(" `expression` `comp_for` ")"
222252

223-
.. index:: object: generator
253+
A generator expression yields a new generator object. Its syntax is the same as
254+
for comprehensions, except that it is enclosed in parentheses instead of
255+
brackets or curly braces.
224256

225-
A generator expression yields a new generator object. It consists of a single
226-
expression followed by at least one :keyword:`for` clause and zero or more
227-
:keyword:`for` or :keyword:`if` clauses. The iterating values of the new
228-
generator are those that would be produced by considering each of the
229-
:keyword:`for` or :keyword:`if` clauses a block, nesting from left to right, and
230-
evaluating the expression to yield a value that is reached the innermost block
231-
for each iteration.
232-
233-
Variables used in the generator expression are evaluated lazily in a separate
234-
scope when the :meth:`next` method is called for the generator object (in the
235-
same fashion as for normal generators). However, the :keyword:`in` expression
236-
of the leftmost :keyword:`for` clause is immediately evaluated in the current
237-
scope so that an error produced by it can be seen before any other possible
257+
Variables used in the generator expression are evaluated lazily when the
258+
:meth:`__next__` method is called for generator object (in the same fashion as
259+
normal generators). However, the leftmost :keyword:`for` clause is immediately
260+
evaluated, so that an error produced by it can be seen before any other possible
238261
error in the code that handles the generator expression. Subsequent
239-
:keyword:`for` and :keyword:`if` clauses cannot be evaluated immediately since
240-
they may depend on the previous :keyword:`for` loop. For example:
241-
``(x*y for x in range(10) for y in bar(x))``.
262+
:keyword:`for` clauses cannot be evaluated immediately since they may depend on
263+
the previous :keyword:`for` loop. For example: ``(x*y for x in range(10) for y
264+
in bar(x))``.
242265

243-
The parentheses can be omitted on calls with only one argument. See section
266+
The parentheses can be omitted on calls with only one argument. See section
244267
:ref:`calls` for the detail.
245268

246-
247269
.. _dict:
248270

249271
Dictionary displays
250272
-------------------
251273

252274
.. index:: pair: dictionary; display
253-
254-
.. index::
255-
single: key
256-
single: datum
257-
single: key/datum pair
275+
key, datum, key/datum pair
276+
object: dictionary
258277

259278
A dictionary display is a possibly empty series of key/datum pairs enclosed in
260279
curly braces:
261280

262281
.. productionlist::
263-
dict_display: "{" [`key_datum_list`] "}"
282+
dict_display: "{" [`key_datum_list` | `dict_comprehension`] "}"
264283
key_datum_list: `key_datum` ("," `key_datum`)* [","]
265284
key_datum: `expression` ":" `expression`
266-
267-
.. index:: object: dictionary
285+
dict_comprehension: `expression` ":" `expression` `comp_for`
268286

269287
A dictionary display yields a new dictionary object.
270288

271-
The key/datum pairs are evaluated from left to right to define the entries of
272-
the dictionary: each key object is used as a key into the dictionary to store
273-
the corresponding datum.
289+
If a comma-separated sequence of key/datum pairs is given, they are evaluated
290+
from left to right to define the entries of the dictionary: each key object is
291+
used as a key into the dictionary to store the corresponding datum. This means
292+
that you can specify the same key multiple times in the key/datum list, and the
293+
final dictionary's value for that key will be the last one given.
294+
295+
A dict comprehension, in contrast to list and set comprehensions, needs two
296+
expressions separated with a colon followed by the usual "for" and "if" clauses.
297+
When the comprehension is run, the resulting key and value elements are inserted
298+
in the new dictionary in the order they are produced.
274299

275300
.. index:: pair: immutable; object
301+
hashable
276302

277303
Restrictions on the types of the key values are listed earlier in section
278304
:ref:`types`. (To summarize, the key type should be :term:`hashable`, which excludes

‎Grammar/Grammar‎

Lines changed: 8 additions & 8 deletions
Original file line numberDiff line numberDiff line change
@@ -100,13 +100,13 @@ arith_expr: term (('+'|'-') term)*
100100
term: factor (('*'|'/'|'%'|'//') factor)*
101101
factor: ('+'|'-'|'~') factor | power
102102
power: atom trailer* ['**' factor]
103-
atom: ('(' [yield_expr|testlist_gexp] ')' |
103+
atom: ('(' [yield_expr|testlist_comp] ')' |
104104
'[' [listmaker] ']' |
105105
'{' [dictorsetmaker] '}' |
106106
'`' testlist1 '`' |
107107
NAME | NUMBER | STRING+)
108108
listmaker: test ( list_for | (',' test)* [','] )
109-
testlist_gexp: test ( gen_for | (',' test)* [','] )
109+
testlist_comp: test ( comp_for | (',' test)* [','] )
110110
lambdef: 'lambda' [varargslist] ':' test
111111
trailer: '(' [arglist] ')' | '[' subscriptlist ']' | '.' NAME
112112
subscriptlist: subscript (',' subscript)* [',']
@@ -115,8 +115,8 @@ sliceop: ':' [test]
115115
exprlist: expr (',' expr)* [',']
116116
testlist: test (',' test)* [',']
117117
dictmaker: test ':' test (',' test ':' test)* [',']
118-
dictorsetmaker: ( (test ':' test (',' test ':' test)* [',']) |
119-
(test (',' test)* [',']) )
118+
dictorsetmaker: ( (test ':' test (comp_for | (',' test ':' test)* [','])) |
119+
(test (comp_for | (',' test)* [','])) )
120120

121121
classdef: 'class' NAME ['(' [testlist] ')'] ':' suite
122122

@@ -125,15 +125,15 @@ arglist: (argument ',')* (argument [',']
125125
|'**' test)
126126
# The reason that keywords are test nodes instead of NAME is that using NAME
127127
# results in an ambiguity. ast.c makes sure it's a NAME.
128-
argument: test [gen_for] | test '=' test
128+
argument: test [comp_for] | test '=' test
129129

130130
list_iter: list_for | list_if
131131
list_for: 'for' exprlist 'in' testlist_safe [list_iter]
132132
list_if: 'if' old_test [list_iter]
133133

134-
gen_iter: gen_for | gen_if
135-
gen_for: 'for' exprlist 'in' or_test [gen_iter]
136-
gen_if: 'if' old_test [gen_iter]
134+
comp_iter: comp_for | comp_if
135+
comp_for: 'for' exprlist 'in' or_test [comp_iter]
136+
comp_if: 'if' old_test [comp_iter]
137137

138138
testlist1: test (',' test)*
139139

‎Include/Python-ast.h‎

Lines changed: 21 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -186,10 +186,10 @@ struct _stmt {
186186

187187
enum _expr_kind {BoolOp_kind=1, BinOp_kind=2, UnaryOp_kind=3, Lambda_kind=4,
188188
IfExp_kind=5, Dict_kind=6, Set_kind=7, ListComp_kind=8,
189-
GeneratorExp_kind=9, Yield_kind=10, Compare_kind=11,
190-
Call_kind=12, Repr_kind=13, Num_kind=14, Str_kind=15,
191-
Attribute_kind=16, Subscript_kind=17, Name_kind=18,
192-
List_kind=19, Tuple_kind=20};
189+
SetComp_kind=9, DictComp_kind=10, GeneratorExp_kind=11,
190+
Yield_kind=12, Compare_kind=13, Call_kind=14, Repr_kind=15,
191+
Num_kind=16, Str_kind=17, Attribute_kind=18,
192+
Subscript_kind=19, Name_kind=20, List_kind=21, Tuple_kind=22};
193193
struct _expr {
194194
enum _expr_kind kind;
195195
union {
@@ -234,6 +234,17 @@ struct _expr {
234234
asdl_seq *generators;
235235
} ListComp;
236236

237+
struct {
238+
expr_ty elt;
239+
asdl_seq *generators;
240+
} SetComp;
241+
242+
struct {
243+
expr_ty key;
244+
expr_ty value;
245+
asdl_seq *generators;
246+
} DictComp;
247+
237248
struct {
238249
expr_ty elt;
239250
asdl_seq *generators;
@@ -458,6 +469,12 @@ expr_ty _Py_Set(asdl_seq * elts, int lineno, int col_offset, PyArena *arena);
458469
#define ListComp(a0, a1, a2, a3, a4) _Py_ListComp(a0, a1, a2, a3, a4)
459470
expr_ty _Py_ListComp(expr_ty elt, asdl_seq * generators, int lineno, int
460471
col_offset, PyArena *arena);
472+
#define SetComp(a0, a1, a2, a3, a4) _Py_SetComp(a0, a1, a2, a3, a4)
473+
expr_ty _Py_SetComp(expr_ty elt, asdl_seq * generators, int lineno, int
474+
col_offset, PyArena *arena);
475+
#define DictComp(a0, a1, a2, a3, a4, a5) _Py_DictComp(a0, a1, a2, a3, a4, a5)
476+
expr_ty _Py_DictComp(expr_ty key, expr_ty value, asdl_seq * generators, int
477+
lineno, int col_offset, PyArena *arena);
461478
#define GeneratorExp(a0, a1, a2, a3, a4) _Py_GeneratorExp(a0, a1, a2, a3, a4)
462479
expr_ty _Py_GeneratorExp(expr_ty elt, asdl_seq * generators, int lineno, int
463480
col_offset, PyArena *arena);

‎Include/graminit.h‎

Lines changed: 4 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -64,7 +64,7 @@
6464
#define power 317
6565
#define atom 318
6666
#define listmaker 319
67-
#define testlist_gexp 320
67+
#define testlist_comp 320
6868
#define lambdef 321
6969
#define trailer 322
7070
#define subscriptlist 323
@@ -80,9 +80,9 @@
8080
#define list_iter 333
8181
#define list_for 334
8282
#define list_if 335
83-
#define gen_iter 336
84-
#define gen_for 337
85-
#define gen_if 338
83+
#define comp_iter 336
84+
#define comp_for 337
85+
#define comp_if 338
8686
#define testlist1 339
8787
#define encoding_decl 340
8888
#define yield_expr 341

‎Include/opcode.h‎

Lines changed: 3 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -147,6 +147,9 @@ extern "C" {
147147
/* Support for opargs more than 16 bits long */
148148
#define EXTENDED_ARG 145
149149

150+
#define SET_ADD 146
151+
#define MAP_ADD 147
152+
150153

151154
enum cmp_op {PyCmp_LT=Py_LT, PyCmp_LE=Py_LE, PyCmp_EQ=Py_EQ, PyCmp_NE=Py_NE, PyCmp_GT=Py_GT, PyCmp_GE=Py_GE,
152155
PyCmp_IN, PyCmp_NOT_IN, PyCmp_IS, PyCmp_IS_NOT, PyCmp_EXC_MATCH, PyCmp_BAD};

‎Include/symtable.h‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -42,6 +42,7 @@ typedef struct _symtable_entry {
4242
an argument */
4343
int ste_lineno; /* first line of block */
4444
int ste_opt_lineno; /* lineno of last exec or import * */
45+
int ste_tmpname; /* counter for listcomp temp vars */
4546
struct symtable *ste_table;
4647
} PySTEntryObject;
4748

‎Lib/compiler/ast.py‎

Lines changed: 45 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -890,6 +890,51 @@ def getChildNodes(self):
890890
def __repr__(self):
891891
return "ListCompIf(%s)" % (repr(self.test),)
892892

893+
class SetComp(Node):
894+
def __init__(self, expr, quals, lineno=None):
895+
self.expr = expr
896+
self.quals = quals
897+
self.lineno = lineno
898+
899+
def getChildren(self):
900+
children = []
901+
children.append(self.expr)
902+
children.extend(flatten(self.quals))
903+
return tuple(children)
904+
905+
def getChildNodes(self):
906+
nodelist = []
907+
nodelist.append(self.expr)
908+
nodelist.extend(flatten_nodes(self.quals))
909+
return tuple(nodelist)
910+
911+
def __repr__(self):
912+
return "SetComp(%s, %s)" % (repr(self.expr), repr(self.quals))
913+
914+
class DictComp(Node):
915+
def __init__(self, key, value, quals, lineno=None):
916+
self.key = key
917+
self.value = value
918+
self.quals = quals
919+
self.lineno = lineno
920+
921+
def getChildren(self):
922+
children = []
923+
children.append(self.key)
924+
children.append(self.value)
925+
children.extend(flatten(self.quals))
926+
return tuple(children)
927+
928+
def getChildNodes(self):
929+
nodelist = []
930+
nodelist.append(self.key)
931+
nodelist.append(self.value)
932+
nodelist.extend(flatten_nodes(self.quals))
933+
return tuple(nodelist)
934+
935+
def __repr__(self):
936+
return "DictComp(%s, %s, %s)" % (repr(self.key), repr(self.value), repr(self.quals))
937+
893938
class Mod(Node):
894939
def __init__(self, leftright, lineno=None):
895940
self.left = leftright[0]

‎Lib/compiler/pyassem.py‎

Lines changed: 3 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -685,7 +685,9 @@ def findDepth(self, insts, debug=0):
685685
effect = {
686686
'POP_TOP': -1,
687687
'DUP_TOP': 1,
688-
'LIST_APPEND': -2,
688+
'LIST_APPEND': -1,
689+
'SET_ADD': -1,
690+
'MAP_ADD': -2,
689691
'SLICE+1': -1,
690692
'SLICE+2': -1,
691693
'SLICE+3': -2,

‎Lib/compiler/pycodegen.py‎

Lines changed: 49 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -589,6 +589,55 @@ def visitListComp(self, node):
589589
self.emit('JUMP_ABSOLUTE', start)
590590
self.startBlock(anchor)
591591

592+
def visitSetComp(self, node):
593+
self.set_lineno(node)
594+
# setup list
595+
self.emit('BUILD_SET', 0)
596+
597+
stack = []
598+
for i, for_ in zip(range(len(node.quals)), node.quals):
599+
start, anchor = self.visit(for_)
600+
cont = None
601+
for if_ in for_.ifs:
602+
if cont is None:
603+
cont = self.newBlock()
604+
self.visit(if_, cont)
605+
stack.insert(0, (start, cont, anchor))
606+
607+
self.visit(node.expr)
608+
self.emit('SET_ADD', len(node.quals) + 1)
609+
610+
for start, cont, anchor in stack:
611+
if cont:
612+
self.nextBlock(cont)
613+
self.emit('JUMP_ABSOLUTE', start)
614+
self.startBlock(anchor)
615+
616+
def visitDictComp(self, node):
617+
self.set_lineno(node)
618+
# setup list
619+
self.emit('BUILD_MAP', 0)
620+
621+
stack = []
622+
for i, for_ in zip(range(len(node.quals)), node.quals):
623+
start, anchor = self.visit(for_)
624+
cont = None
625+
for if_ in for_.ifs:
626+
if cont is None:
627+
cont = self.newBlock()
628+
self.visit(if_, cont)
629+
stack.insert(0, (start, cont, anchor))
630+
631+
self.visit(node.value)
632+
self.visit(node.key)
633+
self.emit('MAP_ADD', len(node.quals) + 1)
634+
635+
for start, cont, anchor in stack:
636+
if cont:
637+
self.nextBlock(cont)
638+
self.emit('JUMP_ABSOLUTE', start)
639+
self.startBlock(anchor)
640+
592641
def visitListCompFor(self, node):
593642
start = self.newBlock()
594643
anchor = self.newBlock()

0 commit comments

Comments
 (0)