Hello World!

It's a blog about gathering text data, exploring and processing it, and eventually making sense of it effectively.

Tokenization with Python and NLTK

November 6, 2017

Tokenization is the process of splitting up text into independent blocks that can describe syntax and semantics. Even though text can be split up into paragraphs, sentences, clauses, phrases and words, but the most popular ones are sentence and word tokenization.