StopIteration in read_html

[cancan101]

pd.read_html("http://www.sec.gov/Archives/edgar/data/1174922/000119312513329016/d548026d10q.htm",header=[1,2,3])

----> 1 pd.read_html("http://www.sec.gov/Archives/edgar/data/1174922/000119312513329016/d548026d10q.htm",header=[1,2,3])

/home/alex/git/pandas/pandas/io/html.pyc in read_html(io, match, flavor, header, index_col, skiprows, infer_types, attrs, parse_dates, tupleize_cols, thousands)
    838                          'data (you passed a negative value)')
    839     return _parse(flavor, io, match, header, index_col, skiprows, infer_types,
--> 840                   parse_dates, tupleize_cols, thousands, attrs)

/home/alex/git/pandas/pandas/io/html.pyc in _parse(flavor, io, match, header, index_col, skiprows, infer_types, parse_dates, tupleize_cols, thousands, attrs)
    710     return [_data_to_frame(table, header, index_col, skiprows, infer_types,
    711                            parse_dates, tupleize_cols, thousands)
--> 712             for table in tables]
    713 
    714 

/home/alex/git/pandas/pandas/io/html.pyc in _data_to_frame(data, header, index_col, skiprows, infer_types, parse_dates, tupleize_cols, thousands)
    600                     skiprows=_get_skiprows(skiprows),
    601                     parse_dates=parse_dates, tupleize_cols=tupleize_cols,
--> 602                     thousands=thousands)
    603     df = tp.read()
    604 

/home/alex/git/pandas/pandas/io/parsers.pyc in TextParser(*args, **kwds)
   1173     """
   1174     kwds['engine'] = 'python'
-> 1175     return TextFileReader(*args, **kwds)
   1176 
   1177 

/home/alex/git/pandas/pandas/io/parsers.pyc in __init__(self, f, engine, **kwds)
    485             self.options['has_index_names'] = kwds['has_index_names']
    486 
--> 487         self._make_engine(self.engine)
    488 
    489     def _get_options_with_defaults(self, engine):

/home/alex/git/pandas/pandas/io/parsers.pyc in _make_engine(self, engine)
    601             elif engine == 'python-fwf':
    602                 klass = FixedWidthFieldParser
--> 603             self._engine = klass(self.f, **self.options)
    604 
    605     def _failover_to_python(self):

/home/alex/git/pandas/pandas/io/parsers.pyc in __init__(self, f, **kwds)
   1291             self.data = f
   1292 
-> 1293         self.columns = self._infer_columns()
   1294 
   1295         # we are processing a multi index column

/home/alex/git/pandas/pandas/io/parsers.pyc in _infer_columns(self)
   1485 
   1486                 while self.pos <= hr:
-> 1487                     line = self._next_line()
   1488 
   1489                 this_columns = []

/home/alex/git/pandas/pandas/io/parsers.pyc in _next_line(self)
   1544                 line = self.data[self.pos]
   1545             except IndexError:
-> 1546                 raise StopIteration
   1547         else:
   1548             while self.pos in self.skiprows:

StopIteration: 

[jreback]

@cpcloud close this?

[cancan101]

@jreback What do you mean by close this? As in, are you not able to replicate issue?

[jreback]

@cancan101 is this the same issues in #5129 ?

[cancan101]

@jreback At this moment I don't think so; still waiting on fix for that issue to know for sure.

[cancan101]

Should this be fixed for 0.13?

[cancan101]

Here is another (simpler) page with problems:
pd.read_html("http://biz.yahoo.com/research/earncal/20131216.html", skiprows=[0,1], header=0)

[jtratner]

If you have time, can you try slimming down the page to figure out an even
more minimal example?

[cancan101]

pd.read_html(StringIO("<table><tbody><tr><td>US</td></tr></tbody></table>"), skiprows=[0], header=0)

[cancan101]

It appears to be an issue with the text parser trying to read multiple lines as specified by either skip rows or headers when there are in fact not enough lines in the specified table. This is my hacked fix: cancan101@142ca62

[cancan101]

@cpcloud Do you have any bandwidth on this?

[cancan101]

@cpcloud Any idea on this?

[cpcloud]

Hey @cancan101 sorry for the incredible lag on my part. Crazy busy (as I'm sure you are too). I haven't had a chance to really dig in here. I can take a deeper look this weekend.

[cpcloud]

@cancan101 On second thought, it looks like you might know better than I. Want to take a crack at it? I know that the TextParser code can be intimidating, but it's balanced out by the elation you feel when you finally fix the issue 😄 LMK. If you don't want to, that's fine too.