NLP Basics

NLP (i.e. Natural Language Processing) is the most important field of Data Science nowadays (as of 2023) where we work with Text Data day in and out, with social media becoming the new norm.

So the need to process and mine this unstructured data becomes quite meaningful and business critical for every industry. Here we uses Word Embeddings (like Word2Vec, Doc2Vec, etc.) and Neural Networks to gain insights, also nowadays the Generative AI and use of Large Language Models (LLMs) models like BERT, T5, GPT 3/3.5/4 are disrupting the market.

This field is broadly divided into 2 parts:

• NLU (Natural Language Understanding)
  • Traditional Text Mining Tasks
• NLG (Natural Language Generation)
  • More of Generative AI paradigms like OpenAI’s ChatGPT

Here we will talk about NLU, where we try to understand the Textual Data and make retrieve some information out of it.

There are multiple steps involved in NLU –

• Text Data Gather (Web Scrapping, Social Media APIs, etc.)
• Text Data Pre-Processing
• Text to Numeric conversion (needed as Computers understand Numbers Only)
  • BOW
  • TF-IDF
  • One-Hot Encoding
  • Word Embeddings (Word2Vec)
    • CBOW
    • SkipGram

Some of the Applications of NLU are:

• POS Tagging and NER (Named Entity Recognition)
• Text Classification (Spam vs Ham)
  • (e.g. Google uses this to filter Spams from general emails)
• Sentiment Analysis
  • (Twitter Data, etc.)
• Article Summarization
• Topic Modelling
  • To extract meaningful topics from the corpus of documents
  • These Topics are groups of important/relevant keywords
  • Models –
    • Traditional – LSA, LDA (Latent Dirichlet Allocation)
      • Latent means Hidden, Dirichlet means Prior Probability Distribution (Word to Topic and Topic to Document)
    • Advance – Top2Vec and BERTTopic