FilesExpand file tree

master

/

python_tokenizer.py

Copy path

Blame

More file actions

Blame

More file actions

Latest commit

 

History

History

History

18 lines (13 loc) · 370 Bytes

master

/

python_tokenizer.py

Copy path

Top

File metadata and controls

• Code
• Blame

18 lines (13 loc) · 370 Bytes

Raw

Copy raw file

Download raw file

Open symbols panel

Edit and raw actions

1

2

3

4

5

6

7

8

9

10

11

12

13

14

15

16

17

18

import gate, re

@gate.executable

def tokenize(document, outputAS):

p = re.compile('\w+')

for token in re.finditer(p, document.text):

outputAS.add(

token.start(),

token.end(),

"Token.py2",

{"string":token.group(), "rule":"py"})

return document

if __name__ == "__main__":

tokenize.start()