Natural Language Processing (NLP)

Natural Language Processing (NLP) can be broadly divided into 2 parts:

1. NLU is natural language understanding like in Chatbots, Intelligent assistants like Siri and Alexa
2. NLG is Natural language generation, it includes generation of text from image, generation of image from text, etc.

Once you know core idea behind any language, then we can write Python program for the Text Mining or Information Retrieval

Steps in any NLP Program:

1. Text Pre-Processing
   • Word Tokenization, or
   • Sentence Tokenization
   • mixed case to lower case
   • punctuation removal
   • stopwords removal
   • number removal, only if needed
   • Stemming, or
   • Lemmatization
2. Capturing Features Vectors from text (Text Data Feature Engineering)
   • BOW (bag of words)
   • TF-IDF (term frequency – Inverse document frequency)
   • Word Embeddings (word2vec model)
   • doc2vec model
   • CBOW and skip-grams
3. Final step is to use the AI-ML Model appropriate for the task (i.e. Usecases/Applications of NLP)
   • For Spam Detection, Fake News Detection, Sentiment Analysis, etc. (Classification Models like NB, Logit, KNN, SVM, DTree, etc.)
   • PoS (Part of Speech) Tagging and NER (Named Entity Recognition)
   • Topic Modelling or Document tagging (via LDA, QDA)
   • Article Summarization
   • Machine Translation and Transliteration

---

Common Python Packages for NLP:

1. NLTK
2. Spacy
3. Gensim (Word2Vec models)
4. sklearn

---