Python yield and iteration in depth

Yield operator in Python

The problem

• Generator creates lazy seq, but...
• The syntax is difficult for long expressions
• Extra variables problem

Two ways

• Return generator from a function
• Use yield operator in a function

What is yield

• When yield inside a function, it returns generator
• Function evaluates only when iteration starts
• Yield returns a result as a generator element
• Yield stops function until the next iteration
• The function starts from the previous yield

Usage

def foo():
    do_taks1()
    do_task2()
    ...
    yield el1
    yield el2
    ...

data = foo()
print data
>>> <generator object...

list(data)
>>> [el1, el2]

Common rules

• The number of elements is equal to number if yields
• You can use yield with cycles
• raise StopIteration() to break immediately
• Other rules applied to generators are true

def foo():
    1/0
    yield 42

data = foo()
# no exception

def get_ints(file_path):
    f = open(file_path, 'r')
    for line in f:
        yield int(line)
    f.close()

for n in get_ints('~/user_ids.txt'):
    print n

def fib(a=1, b=2, limit=100):
    for _ in xrange(limit):
        a, b = b, a + b
        yield b

list(fib())
>>> [3, 5, 8, ...]

def foo(limit=10):
   i = 0
   while True:
       yield i
       i += 1
       if i > limit:
           raise StopIteration()

How does iteration work in depth

• Python never iterates the object itself
• Python forces an object into a generator (if possible)
• Iterate generator stands for calling gen.next() method
• (or next(gen) function in Py3)
• Catch StopIteration error, stop if it occurred

iter([1, 2, 3])              <list iterator at 0x0000...
iter('foo bar baz')          <iterator object...
iter({1, 2, 3})              <setiterator at 0x000...

iter(obj) the same is obj.__iter__()

[1, 2, 3].__iter__()         <listiterator...

iter(1234567)                TypeError: 'int' object is not iterable

Why need iter

• Implement own iteration rules for...
• Databases, ORM
• Structed data (CSV, XML)
• Network resources

How does Python iterate a generator

gen.next()        1
gen.next()        2
gen.next()        3
gen.next()        StopIteration error

Why need manual iteration?

• Perhaps you won't use it
• It's just an interface
• Some tricky situations
• itertools library