Tips from the 2012 X Change Testing Huddle

This remains one of my favorite huddles at Semphonic's X Change conference and last year was no exception. Here is a summary of the most interesting points.

1. Losses are Good. 
This is how learnings happen. A common ratio is 1 test out of 10 will result in an actual change, though Dell's rate is higher at 1 in 3 which may reflect their ability to follow the guidelines below. Otherwise the test change is not significant. I have heard from Tim Ash (who was not at this huddle but has written a great book on Landing Page Optimization) that making bigger changes can help drive this ratio down.

2. How do you Decide what to Test?
In most cases teams upstream decide, but consider giving the following guidance. What is the level of revenue impact? What is the level of learning possible (is it an isolated variable)? What is the level of effort to run the test? How deep is the customer insight (will it cross-correlate to other areas)? And finally, the golden rule of testing: Don't test what you cannot implement. 

3. Have a Slate of Tests 
This is a previous learning from Dylan's huddle back in 2010. Always have a slate of tests in the cue. This can be helpful in declaring closure because "you need the traffic for the next test" or if you need to discard a test (see #6 below). The new insight is to hold out spots for re-tests. Ideally, every test would be conclusive and unquestioned, but this is simply not reality.

4. Testing Wiki
Once tests get going it's easy to forget exactly what was tested and the outcome. To avoid repetitive tests, document the results. This includes failed tests (where the change was not significant) as well as successful tests. Include elements such as pathing, navigation, page function, customer segment, targeting, banners. Also list the top 10 most impactful at the top and the latest learnings.


5. A/B then MVT
Start with A/B testing then move to Multivariate. Multivariate testing in Test and Target is not good because it uses the Taguchi method which is to say it's not truly random by hour, it's random by day. Test & Target also has difficulty with mass segmentation. Consider running A A B tests initially to control for stability.

6. Throw out the Test
When the data is between SiteCatalyst (web tracking tool) and Test & Target (testing tool) is off by more than 15% throw out the test. Identify data issues like this within the first 3 days. Tests often need 10 days of stability before you call it (95% confidence). Dell runs tests for about 21 days. Intuit runs for 14 days. It may take two weeks to get a test into the launch cue so identify data problems early.

7. Initial Test Analysis
Run a quick initial analysis within 4 days and a deep dive analysis within 2 weeks once the test completes. Front load the analysis by asking "What do you need to make a decision?" Do not roll out preliminary numbers before the test is complete.

Think you might like to attend a huddle? Then come to X Change! Contact me for a discount code.

High Powered Social Media Metrics

Here is a great list of powerful metrics to use on social media. I've culled these from Gary Angel's Semphonic presentations which are available here.  

Community Metrics (like Facebook or Google+)

•Incremental Reach: New visitors to the Community from Non-Branded Natural Search

•Effective Reach: Direct Community Traffic + Non-Competitive Search Traffic Visits/ All Visits

•Community Independence: Visits sourced on a main site that drive to a community / Total community visits

•Community Integration:  Visits sourced on the community that drive to a main site.

Conversion Metrics

•Direct Marketing Effectiveness: Visits that include community and a main site measure of success/ all community visits

•Halo Marketing Effectiveness: Visitors that include community and a main site measure of success / all visitors

Product Metrics

•Issue Rate: Problems by Feature of Application / Total Problems

•Feature Requests: Request count by Feature

Overlooked, Underloved, and Unknown Analytics

I am very excited to be attending the upcoming OMS Seattle 2011 event. I will be hosting a presentation titled “How Sexy is your Data?” where I will talk about using the new Google Dashboards and other V5 goodies. Also, I will cover some fundamentals of good data visualization and tracking of social media traffic.

• Did you know you can create a segment for fans vs non-facebook fans and see how they perform on your site?
• Did you know you can classify or create groups in the marketing funnels reports to make the analysis more useful?
• Do you know why you can't see campaigns in GA when your agency says they've added tracking?

Find out this and more at my OMS Seattle session scheduled for June 23, 2010 at 11:30 am. I also have one free pass left if you haven't registered yet. DM me on Twitter under @ahartsoe to get it.

And finally, there will be one-on-one sessions with Semphonic that I would like to encourage you to sign up for. In these special 30 minute sessions we will answer your specific questions about tracking, reporting, and other measurement needs fulfilled in Google Analytics. To sign up you can visit us here or e mail SemphonicOMS@gmail.com.

See you there!

Measurement Framework for SEO Analytics – Part Three

The third and final installment of this series focuses on Types of Analysis.  By understanding and utilizing all types of analysis you are able to fully optimize your sites potential. This posting will cover things such as high level analysis, analyzing over time, and analyzing quality. 

High level analysis includes SEM traffic, the volume and percentage of traffic to your site that Search accounts for, and the relative importance of Organic vs. Paid Search Marketing in terms of traffic and conversion.

The conventional wisdom: adding PPC will boost organic click through.

The harsh reality: it will often lessen it. Semphonic has done repeated tests on the claim that by adding PPC it will boost organic clicks. While the results vary dramatically, the most common case is just the reverse. PPC will often cannibalize organic clicks thereby overstating the effectiveness of your PPC and badly skewing your bid strategies.

The lesson: don’t believe the conventional wisdom:  Measure it for yourself. 

Traffic and conversion contribution by Engine (organic and paid) are useful for SEO efforts. Changes in percentage of site traffic and share of traffic by engine (especially organic traffic) tracks SEO effectiveness over time. 

To analyze channels it is important to consider several aspects. Organic Cannibalization tracks the impact on organic traffic of starting or stopping a paid campaign; measure the support or cannibalization by search term. PCC self cannibalization measures the interaction over time between visitors who source from PCC multiple times. This is usually measured by Search Term or Ad Group. SEM/ Media interaction measures the over time relationship between SEM sourcing and alternative online sources including banners and direct. Brand impact measures the impact of mass-media advertising on Search usage. 

The conventional wisdom: each channel can take care of itself, if it’s ROI it’s decent, that’s good enough.

The harsh reality: unless you know how each channel compares, you have no way of intelligently allocating your resources. By measuring channel performance and engagement you can tell which channels have reached (or exceeded) their optimum scale.  

To analyze over time consider sales cycle’s that measure the time “tail” of SEM sourced visitors. Also lifetime value measures the repeat conversions and customer quality of SEM converters. It is essential to track the over-time interactions and the success of PPC and SEO if you intend to allocate resources between them with any degree of accuracy. 

Analyzing quality includes Engagement: measures the quality of SEM visitors (by Search Terms or Ad Groups) in terms of conversions, proxies or site engagement. Creative Conversion tracks conversions (or engagement) by Creative to assist in ad optimization. Entry page tracks the differential in Conversion/ Engagement by Entry Page.  It is  used to track the differential between PPC and Organic quality by Search Term. 

The conventional wisdom: If I don’t have conversions on my site I should optimize to clicks; common thinking on Pharmaceutical sites.

The harsh reality: no single strategy for PPC is worse than machine optimizing for clicks, with a close second being Agency optimizing for clicks. Optimizing for clicks will invariably drive you lots of poorly qualified and useless traffic. There is a better way. And this may be the most pressing and deadly sin for big pharmaceuticals companies.

The lesson: if you’re lazy, you’re Agency will probably be lazy too. 

In summary I would like to point out a few key aspects to remember. 

1. Make sure you are optimizing the right measures.
2. Make sure you can carry through as much information as possible from the purchase side to the Web Analytics Tool.
3. Many key capabilities are difficult to get from even advanced WA tools. Make sure you think about what you want and see if you can figure out a way to get it. Don’t simply rely on the “stock” reports.
4. There are a range of analytics projects that can help drive SEM success. Try one!

In case you missed it, here are part one and part two:
Measurement Framework for SEO Analytics- Part One
Measurement Framework for SEO Analytics- Part Two

Measurement Framework for SEO Analytics- Part Two

Part two focuses on Coding for Measurement. To effectively use the web analytics tool you need to ensure that the key variables are getting passed from your search program. This may take the form of a campaign code, i.e. t:knc:c:gtlifinsbro:ad:lifins:se:msn:k:lifeinsurance:m:ext. Parameters such as identifying paid campaigns, ad group, purchased keywords and create are included in the comprehensive tracking of inputs.

For a successful infrastructure, it is essential that your web analytics solution e setup to recognize campaigns when they are properly coded.

• Omniture:  Pull the s.campaign variable by using the GetQueryParm plugin.  Set tracking code id in admin.s.campaign=getQueryParam(“cmpid”)
• Webtrends:  Automatically detects wen it sees the parameter “wt.mc_id”= in the query string.  Must have campaign module enabled.
• Google:  You will need to tell your developer what variables and values you want to capture.  If you are only tracking AdWords, just use the automated tracking function within AdWords.  No need to customize.  Google accepts the following variables in the query string.  The red variables are the most important:
• utm_campaign – the name of your marketing campaign.  Keep it short and general
• utm_medium – method of distribution.  This might be all PPC now but ou could extend this tracking to email campaigns, banner ads, etc.
• utm_source – who are you partnering with to push the message?  Google, Yahoo, MSN, for example
• utm_content – ad version.  Often used for testing one version of text over another.
• utm_term – keyword.  Google analytics will pull Adwords data through, but other engines will be lost unless you do further integrations. 

For Google Apps campaigns, any link anywhere which is part of your campaign should be tagged.

a PPC campaign run on Google might look like this:

http://www.yoursite.com/landing_page.php?utm_campaign=sample-campaign&utm_medium=ppc&utm_source=google

a PPC campaign on Yahoo might look like this:

http://www.yoursite.com/landing_page.php?utm_campaign=sample-campaign&utm_medium=ppc&utm_source=yahoo&utm_term=dental-insurance   

an email newsletter campaign might look like this:

http://www.yoursite.com/landing_page.php?utm_campaign=sample-campaign&utm_medium=news&utm_source=campaigner&utm_term=dental-insurance   

Reporting

Reports are not analysis.  They are output created to visualize the data. 

SEM Capabilities

PPC Buying Optimization:

Key Capabilities

• For Broad Match

–         Track results by both actual search term used and search term purchased

• For Ad Networks

–         Track Content Match Source – critical for remnant networks

• For long-tail analysis

–         Ability to collapse search terms and analyze them as a unit

• Program Timing

–         Day Parting and Time Parting in the WA tool

• Cost Integration

–         Carry the key purchase data (impressions, clicks, cost, position) to the behavioral data

• Creative Optimization by Success

–         Analyze keyword by creative to conversion

Site Behavioral Reports

• Ability to Path over time at the Event Level

            -   X Visitors Entered on PPC Search – Y Re-Entered on Visitor Search – Produced a Success

• Over-Time Report (Behavior of Visitors who entered on PPC during a particular time period in subsequent time periods)
• Page Performance by Entry Type Report

-           (SEO, PPC, Other Campaign, Direct, Previous Page)