Ticket #366 (closed defect: fixed)
Position bug in RubyLexer's heredoc handling
| Reported by: | thatch | Owned by: | thatch |
|---|---|---|---|
| Priority: | major | Milestone: | 1.0 |
| Component: | lexers | Keywords: | |
| Cc: |
Description
I already have a fix ready. Here is a real world example of how this confuses the irb lexer.
Source:
irb(main):001:0> puts <<END irb(main):002:0" a irb(main):003:0" b irb(main):004:0" c irb(main):005:0" END a b c => nil
Buggy output (notice all the prompts at the end of the first line):
irb(main):001:0> puts <<ENDirb(main):002:0" irb(main):003:0" irb(main):004:0" a b c irb(main):005:0" END a b c => nil
Test script:
from pygments.lexers.agile import RubyLexer
curcode= u'puts <<XYZ\na\nb\nXYZ\n'
r = RubyLexer().get_tokens_unprocessed(curcode)
pos = 0
for i, t, v in r:
if i != pos:
print "continuity error", pos, i, t, repr(v)
pos += len(v)
print "done"
This prints:
continuity error 10 15 Token.Text u'\n' done
There's a newline whose position is being incorrectly reported, which causes all the insertions to happen too early.
Change History
Note: See
TracTickets for help on using
tickets.