Why We Build Our Own Optimization Software

Over the past few years we’ve been quietly developing internal optimization software for the advertising channels we manage for our clients. We even recently became a badged Facebook Marketing Partner thanks to the software we’ve built on top of the Facebook Marketing API. When I tell people that we develop our own technology I’m often met with confusion. Digital marketing agencies traditionally have either leveraged third-party optimization software or none at all, depending on their approach and workflow. However, as practitioners who manage paid advertising for several companies across several verticals, we know the first-hand efficacy of the ad platforms better than third-party software companies and even the platforms themselves. We know the platforms’ capabilities and limitations at any given moment, and these capabilities and limitations change quite rapidly. This gives us a very unique perspective, and when paired with software development capabilities, makes for a powerful combination that we’ve found can often outperform the platform’s native optimization features and off-the-shelf optimization software.

What Even Is Optimization?

Before diving deeper into why we build our own tech, I want to talk about some problems with optimization. Optimization is one of those ominous terms when it comes to digital marketing that has many different meanings. For performance marketers, optimization typically breaks down into a few core areas for each channel:

Audience optimization (are we targeting the right people, considering the goals at hand?)

Creative optimization (are we emphasizing creatives that perform best for a given audience and pausing poor performing creatives?)

Bid optimization (are we paying the optimal amount to acquire a customer for a given audience?)

Budget optimization (are we optimally allocating budgets to the audiences that perform best?)

In concept, these sound like very commonsensical questions for marketers to ask themselves and act on. In practice, they are very rarely well-thought out or routinely executed on. I attribute part of the issue to a lack of training. To really master paid marketing you have to build a great foundational understanding of how the advertising platforms work and how to manipulate them. Unfortunately, most practitioners don’t develop this foundation and just jump right in. This creates a massive knowledge debt that gets worse with time as marketers typically continue down the path they started on and never go back to learn what they skipped over.

Surprisingly, this is even a problem in the agency world where training is supposedly abundant. Even for the most senior agency employees we hire, we have found that we need to teach this foundation to ensure they didn’t miss anything critical along their career. On that note, I should mention that we are hiring 🙂

The Rise of Machine Learning and Automated Optimization Features

There is another major reason that rigorous optimization is not emphasized by most performance marketers and that is platform improvement, which has led to platform over-reliance. The major advertising platforms like Google and Facebook have made huge strides (especially within the past few years) to eliminate much of the manual optimization work required by marketers. They have done this by introducing automated, machine-learning optimization features. This is great since it gives you more leverage as a marketer and allows you to focus on other high-value activities.

The problem, though, is that many marketers blindly believe they will achieve optimal results by using automated optimization features from the platforms, which is not always the case. A good performance marketer will tell you that that these automated features have historically taken a lot of time to mature before they can be wholly trusted. I refer to the time it takes for automated optimization features to mature as the “learning gap.” Understanding this learning gap allows you to take advantage of the inefficiencies that are created by automated features while they mature, in order to get an edge on optimization.

The Opportunity

he above issues are somewhat compounding as you might imagine. Marketers who don’t understand the fundamental optimization levers that exist will naturally gravitate towards automated tools to find the path of least resistance. These marketers are at a two-fold disadvantage because they do not have a disciplined manual optimization workflow to use in the absence of automated tools and they lack the knowledge needed to challenge the efficacy of automated features.

Before I started Growth Pilots in 2014, I was a paid marketing consultant and observed the above issues first hand when I would talk to companies and look at their accounts. There was very little sophistication being applied to manual optimization processes. The automated features offered by the platforms at that time were sparse and those that existed were very immature. It was the perfect storm, and I saw this as an opportunity.

Moving From Manual Optimization Workflow to Software-Driven Optimization

I developed a very thorough playbook that was essentially a rules-based system our team members could follow to regularly assess performance and then make “algorithmic” adjustments on a very granular level to improve account efficiency. This optimization cadence worked like magic. We would take over an account and see almost instant performance gains that allowed us to unlock growth.

After several months I realized this process was so repeatable and consistent that we went the extra stretch and decided to build software on top of the Google AdWords and Bing Ads APIs to fully automate it. The goal with our software was two-fold: 1) for manual bid optimization, it would automate the process of analyzing keyword performance and determining an optimal bid for that keyword using historical conversion data. 2) for automated bidding via Google Conversion Optimizer, it would analyze ad group performance at the current target CPA bid and adjust the target CPA bid based on goal under/over achievement. Our tool worked very well and saved us at least 80% of the time we previously spent bidding manually.

Based on the efficiency gains we saw with SEM we followed suit on Facebook, building optimization software that would adjust bids and budget allocations amongst ad sets based on achieved performance vs desired performance. What was interesting was that our technology was (and still is slightly) more effective for Facebook ads than for Google ads. The reason for this is that Google’s automated features had a head start given their age and thus had a longer time to iterate and improve. This eventually eroded some of the value that our tool provided (i.e. the learning gap was closing up). Facebook was still maturing and the learning gap was much wider so we saw much better efficiency from using our optimization technology there. However, Facebook has also seen big improvements in their automated bidding functionality and they recently started rolling out budget optimization tools similar to what we’ve built – so the value of our current technology will certainly erode at some point in the future.

You might be thinking this is a bad thing – but it’s actually an intended consequence of our technology development strategy. In fact, it’s great validation for us when the platforms improve or build something that our software does because then we know we are on the right track in our optimization workflow. We have no desire or ability to compete with the massive amounts of data and engineering resources the major ad platforms have. Their features, once fully baked, will be far better than anything developed by a third-party. We will continue adapting our software to take advantage of the learning gaps that present themselves and we will happily retire our features whose value is diminished by the advancement of the platforms. As marketers, we are faced with a constantly evolving landscape and that requires adapting our mindset and techniques to keep up with our environment.