Legendary Logo


Data Collection Pipeline
Business Need
  • Monitor specific trends in the industry and discover new ideas around key marketing segments;help marketing develop ideas to generate awareness for internal brand products/movies.
  • Develop a data collection pipeline (DCP), to analyze and monitor how competitor productscompare and are impacting audiences and company marketing plans.
  • This DCP process and method needs to scrape, collect and store web data on a day-to-day basis.
  • Make the findings available to the client via a web-based dashboard for immediate access.
  • Set up an AWS instance (server to run python script), to house data from select sources.
  • Write a python script to scrap and access API’s to collect movie data and sentiment around thosetitles and store data in AWS cloud database for easy access with SQL.
  • Develop special collection criteria (naming convention strategy), for client’s products, key words,and sentiment, as well as a Global data dictionary and definitions for the entire enterprise data collection process.
  • Scrape a list of Data Sources and URLS and specific details for analysis around title reviews, sentiment, and popularity.
  • Visualize; Create a web-based dashboard to study sentiment, popularity and other stakeholder KPIs.
  • An efficient DCP helped scrape all relevant websites and collect data to be stored in AWS on a daily basis.
  • A simple-to-read dashboard provided a variety of metrics and KPIs for monitoring title popularity and daily sentiment.
  • Stakeholders can now compare their data with industry content using buzz, sentiment and key metrics.
  • This data provided crucial insight to help develop future products, guide marketing decisions and plan campaigns.
Python, AWS, SQL
Truemetrics has hired a fantastic team of web analytics professionals. ABC Family’s success and traffic growth was in part due to their expert analysis and attention to detail.
Josh Auffret -
Senior Producer, ABC Television Group
Ready to get started?
Connect with us