Pygments

generic syntax highlighter


Ticket #366 (closed defect: fixed)

Opened 2 years ago

Last modified 2 years ago

Position bug in RubyLexer's heredoc handling

Reported by: thatch Owned by: thatch
Priority: major Milestone: 1.0
Component: lexers Keywords:
Cc:

Description

I already have a fix ready. Here is a real world example of how this confuses the irb lexer.

Source:

irb(main):001:0> puts <<END
irb(main):002:0" a
irb(main):003:0" b
irb(main):004:0" c
irb(main):005:0" END
a
b
c
=> nil

Buggy output (notice all the prompts at the end of the first line):

irb(main):001:0> puts <<ENDirb(main):002:0" irb(main):003:0" irb(main):004:0" 
a
b
c
irb(main):005:0" END
a
b
c
=> nil

Test script:

from pygments.lexers.agile import RubyLexer
curcode= u'puts <<XYZ\na\nb\nXYZ\n'
r = RubyLexer().get_tokens_unprocessed(curcode)

pos = 0
for i, t, v in r:
    if i != pos:
        print "continuity error", pos, i, t, repr(v)
    pos += len(v)
print "done"

This prints:

continuity error 10 15 Token.Text u'\n'
done

There's a newline whose position is being incorrectly reported, which causes all the insertions to happen too early.

Change History

Changed 2 years ago by thatch

  • status changed from new to accepted

Fix in my branch as [c3d87fc41a23] and test in [72573bca3fd0].

Changed 2 years ago by thatch

  • status changed from accepted to closed
  • resolution set to fixed

merged.

Note: See TracTickets for help on using tickets.