These days, we are seeing an increasing trend towards automation, as people look for ways to stop wasting time on repetitive tasks and focus on analysis. It’s an evolution in both the market and our daily work habits. However, it can still be complicated to navigate the jungle of data, scripts, languages, formulas and everything that comes with it.
At Adviso, we think data is essential when it comes to decisions about long-term digital activities, and strategies for reaching the right audiences at the right times. The good news? Everything you need to make those choices more easily is available online.
To better understand the scripts and methods presented here, you’ll need some basic knowledge and understanding of programming languages and spreadsheet formulas. This knowledge will allow you to better read and understand the article.
To recreate the scripts and examples shared here, you’ll need the following tools:
Note: The tools presented below are the most popular and easy to use on the market, but it’s entirely possible to use other, similar tools.
Jupyter Notebook is available on Jupyter and allows you to launch and use the Python or R programming languages visually. There are two possible ways to use it, but we’d recommend skipping directly to Anaconda, a software that has an integrated Jupyter Notebook “module.”
The main advantage of using Anaconda and Jupyter Notebook is the fact that you can manage work environments for, for example, different versions of Python.
To use Jupyter Notebook, you’ll need to install the software locally, unlike Google Colab. This might be an advantage if you’re on the go without the Internet, but requires a good computer for bigger calculations.
Google Colab is Google’s online data science platform. It’s integrated with the Google Drive suite and allows you to work from Google’s servers. In terms of functionality, it’s very similar to Jupyter Notebook.
All you’ll need is an Internet connection and a Google account to access the interface. All the heavy lifting for your calculations is done directly on Google’s servers.
By adapting a script available on Google Trends, we’re able to “replace” and complement keyword generation tools.
To do this, we’ll use the Jupyter Notebook tool and the Python programming language, as well as the pytrends (the equivalent of an API for Google Trends) and panda (for the manipulation of data via Python) libraries.
>> To see the result, click here.
Below, we’ll review the steps for setting up and using the work environment (installation, script and results).
First off, install pytrends using the command below:
pip install pytrends
Then install panda using the same command, but specifying the panda library:
pip install panda
Then, to import the libraries, copy/paste the code below and launch it in Jupyter Notebook or Google Colab:
import pandas as pd from pytrends.request import TrendReq pytrend = TrendReq()
We’ll begin using this script via a first function that allows us to get keyword suggestions based on Google Trends. It’s important to know that this data needs to be compared in other tools (Google Keyword Planner, Keywordtool.io, Ahrefs or SEMRush for example) if you want the volume of searches to be representative of the market you’re targeting.
keywords = pytrend.suggestions(keyword='mot clé')
The keywords variable is created via the pytrend.suggestions function.
All you have to do is insert your keyword into the script and launch it. You can find examples further down in the article.
df_kw = pd.DataFrame(keywords)
We’ll create the framework for the dataset df_kw based on the variable created. You can give the dataset any name you want.
It happens often that there’s “noise” in the results, so we’ll delete one of the columns (the ‘mid’ column) using the drop module.
After analyzing the keyword suggestions given, it might be interesting to get a perspective on current trends. That’s what Pytrends can allow you to do, which we will look at in the next paragraph.
df_popsearch = pytrend.trending_searches(pn='pays')
The trending_searches module in pytrend allows us to create a framework for our dataset called df_popsearch (you can name your dataset whatever you’d like). Searches can be done by country via the pn parameter. The country should be written out in letters, not in ISO code.
To validate the result, we’ll display only the beginning via the head()module.
Daily search trends are good, but Google Trends also has a hot_topic module that allows us to see what Internet users are searching for most right now (worldwide, in a particular category, by country, etc.).
pytrend.build_payload(kw_list=['keyword 1’, ‘keyword 2'])
You can search for hot topics related to a keyword or keyword group.
related_queries = pytrend.related_queries()
We’ll integrate the pytrend.related_queries() function into a list named related_queries. You can give this list any name you want.
Once it’s been set up, you have to launch the line below for values to be returned. You’ll see a distinction between top searches and rising searches.
You can also access “top” and “rising” information related to a particular keyword, like this:
Google Trends is known for its Trending Topics option, probably the most used function on the website. We can explore it in this example thanks to pytrends.
df_todaytt = pytrend.today_searches(pn='CA')
This bit of code allows us to see the Trending Topics for the day. You can manage the geographic part via the variable pn= (here it’s CA for Canada). We’ll save it all in a dataset called df_todaytt.
After all these steps full of code and possible uses for pytrends, I’m sure you’re anxious to know what exactly comes out of all this. You’re in the right place.
You of course need to filter out the noise that sneaks into these queries (Seoul or Securities and Exchange Organization of Iran), because Google Trends is not very qualitative in its suggestions (and our goal is to get ideas from the market).
The trends shown above are those for the beginning of April 2020 in Canada according to Google Trends.
Note: If you launch this script yourself, you might get different results since search trends are related to seasonality and to the country targeted.
There’s still some noise in this query; the strength of the example is that it also illustrates the importance of the human behind the automation of tasks via scripts.
To continue the analysis, you can export the data into a .csv file, for example:
dataframe.to_csv(‘./path/export/document.csv’, index = False, header = True)
We’ll use .to_csv and use the example of a dataset called dataframe.
The index = False and header = True parameters respectively allow you to remove the first column with the numbers 0 to N and to keep the table headers. Note that these parameters are case-sensitive, so be sure to specify False and True with a capital letter at the beginning of the word.
This technique can be used at any time to complement and reinforce your digital marketing strategy. But we also see a not-negligible value in employing it during this crisis, allowing you to better analyze the impact of COVID-19 on users’ searches.
These are keywords, related to the coronavirus in this case, with an order of importance where 100 is the highest and 0 is the lowest. The most queried terms are separate from the terms that have increased the most in search volume over time.
These are keywords classified by topic, in this case related to “coronavirus.”
You can do the same thing to find topics related to e-commerce or delivery.
The global trend right now is definitely searching for recipes to make your own bread at home.
When we search for topics related to “baking your own bread” we are able to identify expressions that might enrich the vocabulary of the pages of a cooking or online shopping site. It’s practical both for websites (development of a pillar page or group of pages) or for a YouTube channel during the launch of a video.
If we do the same thing for cake recipes, the information that emerges is very interesting:
Google Trends is a good tool that Google provides free, but like all tools, it has its limits: keyword comparison, comparison of 5 terms maximum, linguistic limitations for Quebec French for searches in Canada…
What’s more, an essential point to consider when it comes to Google Trends is that these are relative comparisons of keywords scaled from 0 to 100. It can be complex to interpret, which is why it’s recommended that you use an external tool (Google Search Console, Ahref, SEM Rush, etc.) to access the full volume of queries made. For example:
Additionally, you can’t forget to contextualize terms within the global positioning strategy for your brand or site. A keyword might see an increase in volume, but not be of any real interest in terms of ranking, for example during the current time of quarantine, don’t think covid, think more tips, delivery, help, etc.
In the end, you’ll find the same limits in pytrends as you do directly in Google Trends (5 keywords, comparisons, etc.) with some even more marked limits (geographic targeting).
The advantage of this API is that it allows you to automate part or all of your research, allowing you to make better marketing decisions. By automating, you’ll be able to connect several tools and several APIs so you can extract data faster and make decisions based on verified data.