IllegalStateException by trying to delete the pipe symbol at the EOF

[Chris2011]

See here my little gif, hope it is clear what I want to do:

At the end of the file, I try to remove first the unnecessary whitespaces and then, when I try to delete the pipe '|' I got the IllegalStateExceptipn. This is related to this PR discussion at my fork: #2

And this ticket is a duplicate of this #23

[Chris2011]

This is the Stacktrace:

java.lang.IllegalStateException: Lexer development.downright.cucumberbeans.antlr.integration.IntegrationGherkinLexer@6a5ff6ff
  returned null token but lexerInput.readLength()=48
  lexer-state: null
  tokenStartOffset=1547, readOffset=1595, lookaheadOffset=1596
  Chars: "| third     | Chuck Norris| run for his life   \n" - these characters need to be tokenized.
Fix the lexer to not return null token in this state.
	at org.netbeans.lib.lexer.LexerInputOperation.checkLexerInputFinished(LexerInputOperation.java:457)
	at org.netbeans.lib.lexer.LexerInputOperation.nextToken(LexerInputOperation.java:217)
	at org.netbeans.lib.lexer.inc.TokenListUpdater.relex(TokenListUpdater.java:627)
	at org.netbeans.lib.lexer.inc.TokenListUpdater.updateRegular(TokenListUpdater.java:280)
	at org.netbeans.lib.lexer.inc.TokenHierarchyUpdate$UpdateItem.update(TokenHierarchyUpdate.java:350)
	at org.netbeans.lib.lexer.inc.TokenHierarchyUpdate.processLevelInfos(TokenHierarchyUpdate.java:225)
	at org.netbeans.lib.lexer.inc.TokenHierarchyUpdate.updateImpl(TokenHierarchyUpdate.java:196)
	at org.netbeans.lib.lexer.inc.TokenHierarchyUpdate.update(TokenHierarchyUpdate.java:134)
	at org.netbeans.lib.lexer.TokenHierarchyOperation.textModified(TokenHierarchyOperation.java:605)
	at org.netbeans.spi.lexer.TokenHierarchyControl.textModified(TokenHierarchyControl.java:96)
	at org.netbeans.lib.lexer.inc.DocumentInput.textModified(DocumentInput.java:152)
	at org.netbeans.lib.lexer.inc.DocumentInput.removeUpdate(DocumentInput.java:145)
	at org.netbeans.lib.editor.util.swing.PriorityDocumentListenerList.removeUpdate(PriorityDocumentListenerList.java:116)
	at javax.swing.text.AbstractDocument.fireRemoveUpdate(AbstractDocument.java:259)
	at org.netbeans.editor.BaseDocument.fireRemoveUpdate(BaseDocument.java:1662)
	at org.netbeans.editor.BaseDocument.handleRemove(BaseDocument.java:1047)
	at org.netbeans.editor.BaseDocument$FilterBypassImpl.remove(BaseDocument.java:2651)
	at javax.swing.text.DocumentFilter.remove(DocumentFilter.java:79)
	at org.openide.text.CloneableEditorSupport$DocFilter.remove(CloneableEditorSupport.java:2389)
	at org.netbeans.editor.BaseDocument.remove(BaseDocument.java:960)
	at org.netbeans.editor.BaseKit$DeleteCharAction$3.run(BaseKit.java:2184)
	at org.netbeans.editor.GuardedDocument.runAtomicAsUser(GuardedDocument.java:356)
	at org.netbeans.editor.BaseKit$DeleteCharAction.actionPerformed(BaseKit.java:2176)
	at org.netbeans.modules.csl.core.CslEditorKit$GsfDeleteCharAction.actionPerformed(CslEditorKit.java:395)
	at org.netbeans.editor.BaseAction.actionPerformed(BaseAction.java:339)
	at javax.swing.SwingUtilities.notifyAction(SwingUtilities.java:1663)
	at javax.swing.JComponent.processKeyBinding(JComponent.java:2882)
	at javax.swing.JComponent.processKeyBindings(JComponent.java:2929)
	at javax.swing.JComponent.processKeyEvent(JComponent.java:2845)
	at java.awt.Component.processEvent(Component.java:6310)
	at java.awt.Container.processEvent(Container.java:2236)
	at java.awt.Component.dispatchEventImpl(Component.java:4889)
	at java.awt.Container.dispatchEventImpl(Container.java:2294)
	at java.awt.Component.dispatchEvent(Component.java:4711)
	at java.awt.KeyboardFocusManager.redispatchEvent(KeyboardFocusManager.java:1954)
	at java.awt.DefaultKeyboardFocusManager.dispatchKeyEvent(DefaultKeyboardFocusManager.java:806)
	at java.awt.DefaultKeyboardFocusManager.preDispatchKeyEvent(DefaultKeyboardFocusManager.java:1074)
	at java.awt.DefaultKeyboardFocusManager.typeAheadAssertions(DefaultKeyboardFocusManager.java:945)
	at java.awt.DefaultKeyboardFocusManager.dispatchEvent(DefaultKeyboardFocusManager.java:771)
	at java.awt.Component.dispatchEventImpl(Component.java:4760)
	at java.awt.Container.dispatchEventImpl(Container.java:2294)
	at java.awt.Window.dispatchEventImpl(Window.java:2746)
	at java.awt.Component.dispatchEvent(Component.java:4711)
	at java.awt.EventQueue.dispatchEventImpl(EventQueue.java:758)
	at java.awt.EventQueue.access$500(EventQueue.java:97)
	at java.awt.EventQueue$3.run(EventQueue.java:709)
	at java.awt.EventQueue$3.run(EventQueue.java:703)
	at java.security.AccessController.doPrivileged(Native Method)
	at java.security.ProtectionDomain$JavaSecurityAccessImpl.doIntersectionPrivilege(ProtectionDomain.java:80)
	at java.security.ProtectionDomain$JavaSecurityAccessImpl.doIntersectionPrivilege(ProtectionDomain.java:90)
	at java.awt.EventQueue$4.run(EventQueue.java:731)
	at java.awt.EventQueue$4.run(EventQueue.java:729)
	at java.security.AccessController.doPrivileged(Native Method)
	at java.security.ProtectionDomain$JavaSecurityAccessImpl.doIntersectionPrivilege(ProtectionDomain.java:80)
	at java.awt.EventQueue.dispatchEvent(EventQueue.java:728)
	at org.netbeans.core.TimableEventQueue.dispatchEvent(TimableEventQueue.java:159)
[catch] at java.awt.EventDispatchThread.pumpOneEventForFilters(EventDispatchThread.java:201)
	at java.awt.EventDispatchThread.pumpEventsForFilter(EventDispatchThread.java:116)
	at java.awt.EventDispatchThread.pumpEventsForHierarchy(EventDispatchThread.java:105)
	at java.awt.EventDispatchThread.pumpEvents(EventDispatchThread.java:101)
	at java.awt.EventDispatchThread.pumpEvents(EventDispatchThread.java:93)
	at java.awt.EventDispatchThread.run(EventDispatchThread.java:82)

[Chris2011]

And this is the answer of a helping guy, your created the lexer for C#:

Hi Chris,

The UNKNOWN_CHAR should definitely be added as the last token in the lexer. The error you described is more like the one of the UNTERMINATED_STRING. You expect at T_ROW a token as a whole line including a closing pipe. Same with PHOLDER and DOCSTR. You should only define individual tokens and no rules in the lexer. That should happen in the parser. Then it works with the error handling.

Example for the replacement of T_ROW:
Under "Lexer rules":
PIPE: '|'
CELL_CONTENT: (~ ('|' | '\ n' | '\ r')) *
Under "Parser rules" you can then use the tokens in the new rule t_row:
t_row: PIPE (CELL_CONTENT PIPE) + NL
(Originally, your rule also contained ('' | '\ t') *. That is unnecessary now. Whitespaces are automatically hidden by your token WS in the channel HIDDEN and so do not appear in the parser anymore. Simplifies the writing of parser rules.)
Similarly, you must proceed for all your other tokens that can be closed.

So it could be that this can fix the problem. I didn't try it yet because I don't understand the whole lexer stuff. So a more advanced user can have a look maybe.