Python list comprehensions and generators — Введение в Python

List comprehensionsand generators in Python

What is a list comprehension

• An expression that creates a collection based on another collection
• Produces a list in general
• Short clear syntax
• Supports predicate clause to filter values
• Might be lazy
• One of the most powerful Python features

Just to compare

numbs = [1, 2, 3, 4, 5]
result = []
for x in numbs:
if x > 3:
y = x * x
result.append(y)

numbs = [1, 2, 3, 4, 5]
result = [x * x for x in numbs if x > 3]

List comprehension in depth

[stmt for var in iterable if predicate]

where:

• stmt — any statement,
• var — variable(s) on each iteration step,
• iterable — any iterable value,
• predicate — True/False statement with var in scope

[x for x in [1, 2, 3]]                           [1, 2, 3]
[x * x for x in [1, 2, 3]]                       [1, 4, 9]
[x for x in [1, 2, 3, 4, 5] if x % 2 == 0]       [2, 4]
[x + y for x, y in ((1, 2), (2, 3), (3, 4))]     [3, 5, 7]
strs = ['foo', 'bar', 'test']
[x.upper() for x in strs if len(x) < 4]          ['FOO', 'BAR']

users = [{'name': 'ivan', 'age': 29},
         {'name': 'juan', 'age': 31}]
[user['name'] for user in users if user['age'] > 30]       ['juan']

mydict = {'foo': 'test', 'bar': None}
[k for (k, v) in mydict.iteritems() if v is not None]      ['foo']

List comprehension — conclusion

• Square brackets syntax
• Creates list
• Computes all values when created
• Aware of long lists (save memory)
• Use it rather than cycles

Set comprehensions

• Same as a list comprehension, but...
• Curly brackets syntax
• Returns a set as a result, so...
• Guaranties unique elements

{x for x in [1, 2, 3]}                      {1, 2, 3}
{x for x in [1, 2, 2, 2, 3, 1]}             {1, 2, 3}
list({x for x in [5, 2, 1, 1, 2, 5]})       [1, 2, 5]

users = [{'age': 20}, {'age': 90}]
{user for user in users if user['age'] > 30}
>>> TypeError: unhashable type: 'dict'

Dict comprehension

• Curly brackets syntax
• key: value statement

{k: v for (k, v) in [('foo', 1), ('bar', 2)]}
# {'bar': 2, 'foo': 1}

# fix string keys
mydict = {'1': 'value1', '2': 'value2'}
{int(k): v for (k, v) in mydict.iteritems()}
# {1: 'value1', 2: 'value2'}

# reverse a dictionary
{v: k for k, v in mydict.iteritems()}
# {'value1': '1', 'value2': '2'}

# drop keys with None values
mydict = {'foo': 42, 'bar': None}
{k: v for k, v in mydict.iteritems() if v is not None}
# {'foo': 42}

Generator object

• Round brackets syntax
• It's not a tuple!
• Lazy sequence
• Each element computes only when it requires
• Saves memory and CPU

Lazy!

data = [1.0 / x for x in [3, 2, 1, 0]]
>>> ZeroDivisionError: float division by zero

data = (1.0 / x for x in [3, 2, 1, 0])
for x in data:
    print x

>>> 0.333333333333   0.5   1.0
>>> ZeroDivisionError

Iteration only

• you cannot access value by index
• you don't know the length
• you can pass through it only one time
• you cannot iterate backwards
• pass it to list/tuple/set to get a common collection

gen = (x for x in [1, 2, 3])
print gen
<generator object <genexpr> at 0x105a12e60>

print gen[1]             TypeError: 'generator' object has no attribute '__getitem__'
len(gen)                 TypeError: object of type 'generator' has no len()
data = list(gen)         [1, 2, 3]

for x in gen:
    print x
# will print nothing! Why?

range vs xrange

range(1, 10000)
>>> [1, 2, 3, ..., 9998, 9999, 10000]

xrange(1, 10000)
>>> xrange object

Generators are everywhere

• Mathematical sequence (Fibonacci, progressions)
• Network calls results = (get_data_from_server(user) for user in (...))
• ORM: Django, SQLAlchemy (querysets)
• Parsers (json, xml)
• File readers
• Almost everything in Py3 became lazy

Look for itertools module

chain('ABC', 'DEF')                        A B C D E F
combinations('ABCD', 2)                    AB AC AD BC BD CD
compress('ABCDEF', [1,0,1,0,1,1])          A C E F
count(2.5, 0.5)                            2.5  3.0  3.5 ...
cycle('ABCD')                              A B C D A B C D A B C D
izip_longest('ABCD', 'xy', fillvalue='-')  Ax By C- D-
product('ABCD', 'xy')                      Ax Ay Bx By Cx Cy Dx Dy
imap, islice, ifilter, etc...
tonnes of recipes

Next lesson

• How does iteration work under cover
• Create generators using functions
• Yield operator