The truth behind A/B testing no one ever told you before
We are done with the basics of campaigns now it's time for us to talk about A/B split campaigns. The idea behind A/B testing is to test out 2 variations to see what works better. In this lecture we will cover everything you need to know about this topic and more then that when is it a good idea to actually use it.
Video Transcript
00:00Welcome back we owe you a split test campaign so we're going to do that right away.
00:04So open up your create campaign in your campaigns box and select a B split campaign.
00:10Once you do that it's going to open up your split campaign menu.
00:13Now if you have more than one list then it's going to ask you who the recipients are in
00:17the next tab.
00:18But for now we need to figure out what do we want to test.
00:23Do we want to test the subject lines?
00:25Do we want to test who it's from?
00:28Do we want to test the delivery date?
00:30In my case I want to test the subject line.
00:33So I'm going to leave the subject line set up.
00:35Now we're going to read here basically how should we split the campaign.
00:39The campaign by default is going to be split to be 20% will get the test and the rest
00:44will get everything.
00:45Or if we wanted to we could have raised the test amount and really you would only raise
00:50the amount if you thought that you don't have enough users to make a significant decision
00:57based off of this test.
00:58Now in our case we don't think we have enough users.
01:02We have here about 2500 users so it might even make our split test maybe 24% and really
01:08it's up to you.
01:09And the split test will basically be sent out to 24% of our list.
01:13Those 24% would then based on their reaction to our email would then based on that decide
01:18who will get what will our other 76% of our users will get.
01:23So basically how will the user be chosen?
01:26After sending the test segment the remaining 76% of our list will be sent the best performing
01:31version.
01:32Alright so literally it all depends on how much we set up here so we're going to leave
01:36that at 24% and now we can decide what is our consideration.
01:42Now again as we were saying earlier one of the biggest and most important things with
01:46the list is making sure that people open it.
01:49Sometimes they're not going to click on it as much as you want and because we don't
01:52have enough users when you're looking at down the line do you want to test the base
01:56on how many opens or do you want to test the base on how many clicks which is a really
02:00important factor do you want to ask yourself.
02:03Now we can decide basically we can decide by open rate by click rate or something manual.
02:10Now I personally prefer or the open rate or the click rate really it's hard to tell
02:15what the smaller your list is the open rate is more important.
02:19The bigger your list is the click rate is more important but for our case for this
02:24test itself we're going to decide based on our open rate because we really are going
02:28to decide what our title is and if our title is attractive enough for them to read on and
02:33find out more information about our offer.
02:35We're not going to worry about the click rate we could do a test like that in the
02:38future for a future campaign.
02:40We then need to figure out after how much time do we want the campaign to then be sent
02:45out to all the rest of our users.
02:47Do we want it to be hours do we want it to be days.
02:49In our case because now it's 8 a.m. Eastern time I kind of want it to be sent out around
02:558 a.m. Eastern time so I'm going to wait one day for it to all propagate and figure
02:59out which one is the best base based on those 24 percent of those users.
03:03Alright so I'm happy with these results and I'm going to click on next.
03:08Once I click on next it's time for me to decide which campaign I want to use.
03:11In this case I want to go for my zero to geek campaign which has two thousand three
03:15hundred and thirty two recipients.
03:18Now because my email itself is really targeting to try to get users to upgrade from a free
03:23membership to become our paying members I want to save I want to send it to a segment.
03:29So I want to send it to a segment itself but I don't have any segments.
03:32So I might want to create a segment or would I want to create one yeah I want to create
03:38one.
03:39So I'm going to click on to create a segment and once I click onto segment I could add
03:44here some sort of a condition and that condition could basically be my segmentation.
03:50Now I already prepared in advance and I want to show you basically the steps involved with
03:55segmentation.
03:56Now the first step is I need to be able to recognize between the different users.
04:00In my case really the most important factor was are they paying me did they pay me my
04:05lifetime membership or they didn't pay the lifetime membership.
04:09So I basically created a group and in my groups there basically I created a new group
04:14and manually I've imported into the specific group in these in this email list.
04:19So if we view the group I basically created a group and in that group I basically put
04:24inside our life members right inside of that group.
04:27So once basically what I've done was I created a group and then I selected basically I wanted
04:33it to be private because I don't want users having access to it.
04:37I could have had users have a selection of either or type of options but in our case
04:41I don't want our users to gain any access to this group information.
04:46The next step was I added inside information basically who do I want to send it to and
04:51I split it into two different groups.
04:54Once I've added the group so for example let's say I'm happy with these groups and
04:58I'm going to save it for a second here and I could delete that.
05:02So let's put here a title so my group and let's say is group one.
05:08So you want it to be a little bit more descriptive and this might be group two.
05:12I'm going to delete this one and I'll click on save.
05:15Now when I've done that I could start importing into these groups.
05:20Basically I could be done for now or I could click on import into these groups and then
05:24I could actually import specifically to a specific group and when I'm doing my imports
05:29it would basically ask me to what group do I want to add those users into.
05:35So that's how we added the group.
05:36So let's go back into our campaign and literally what we want to do is we want to go back into
05:41that splitting because we want to use that and I'm going to click onto this edit.
05:45We want to use the group that we have to use it for our filtering.
05:49So again I'm going to click onto zero to geek and I want to create send to a new
05:53segment and the subscriber match all group and I'm going to select one of my user
06:04type groups and I'm going to select well none of our life members.
06:19So basically I'm basically disallowing anyone that's inside of our user group life because
06:23they already are completely inside and they don't need to get this email.
06:28So once I click on next that the email campaign should be sent out when we're going to send
06:32our campaign.
06:33It should be sent out to our recipients of that campaign would be explicitly connected
06:38to our segment subscribers match all of the following and basically user group it's not
06:44of life.
06:45So basically we did the exact opposite literally anyone that's in our life membership will
06:49not get the email and everyone else will.
06:51All right so let's move on to the setup and in our setup I'm going to start off
06:55with basically selecting a name for my campaign.
06:59This campaign name I'll call Facebook and life because we're going to be talking about
07:08a free extra for users that logged in with Facebook and it's also a reminder that it's
07:13a conversion to our life membership.
07:15So it's just an internal name for us.
07:18Now we want to figure out with a title for our email for our campaign.
07:22So if I just take a peek at my document that I've created really one of it is about
07:26our lifetime membership and one of them is about Facebook.
07:31So how about we talk first about the Facebook part where have you logged in with Facebook
07:35question mark and then the other one might be a more explicitly directed towards our
07:41actual offer itself which is the lifetime offer because we have your two offers one is we're
07:46saying that if you logged in with Facebook you'll get a free access start free access
07:49for 24 48 hours and if you want to become one of our paying members then you will get
07:55access for life if you pay us $299.
07:58All right so if I go back inside of your into my browser so I'm going to select your
08:02first of all maybe group one will be have you logged in with Facebook.
08:12All right so that's going to be our have you logged in with Facebook that's enough of a
08:15question if they remember zero to geek even if not it might make them curious about finding
08:20out who we are again because it might have been a year or six months since they joined
08:23us but many of these users should remember who we are hopefully and our next one will
08:29be our second message will be zero to geek limited time left to join us for life right
08:40so limited time left to join us for life and again it's enough information to be a little
08:45bit interesting to hopefully make people open it up not empty of substance completely but
08:51doesn't have enough information to make users basically disengaged already from the first
08:57step so it's kind of like a question have you logged in with Facebook and then they might
09:00not know now and do research your subject line so for example if we click on the research
09:06tool which we've seen in the A B testings before we could then look at that labs project
09:11and basically say does Facebook is maybe Facebook a bad keyword let's search and find
09:18out so there's no net results for Facebook logged logged in let's see if there's anything
09:24negative about logged in nothing so we aren't really saying that they have enough information
09:29on so that's fine is limited time a good keyword so limited time is not bad not good it's kind
09:38of like more in the negative side now is it join us is join us a good one or a bad one please
09:47join us is seems like to be a really really really powerful so please join us so how about
10:02we try to figure out to use in our secondary test one is whatever we felt like putting I'm
10:07all of a fan of trying to put that please join us so how about instead of limited time
10:12please join us you absolutely deserve it all right and I'm just gonna have to test my
10:24spelling because I'm not a big spelling bee so no errors that's always good right cool all
10:31right so please join us you absolutely deserve it it's kind of like an interesting I would be
10:37curious and we use the word absolute because one of our titles is also called absolute HTML
10:41so it kind of like works really nicely it's not saying you would see usually in a spam email
10:47please join us you absolutely deserve it is it's kind of interesting okay so we set up our
10:52message a and we set up a message B which both of them are going to be tested and then basically
10:56the one that wins will be the one that will be used for there on for the remainder what was it
11:02a 66% of our users or was it 76% of our users once we finish up those 24% perfecto let's move
11:10on all right now it's time for us to pick a template I'm gonna go to one of my templates
11:17so I'm gonna pick my zero to geek base template once I pick my zero to geek base base base base
11:24template I'm just gonna fill up the information it shouldn't take me too long because I have
11:28already my copy ready and configured so I'm just gonna go right into edit my copy area and I am just
11:36gonna just gonna copy right here the copy here and I'm just gonna paste all that right in there
11:45so we have our copy all set and configured this we might want to set up as a title so I'm gonna
11:50come right here and I'm gonna set up a title right here have you logged in with Facebook if
11:56not just click on login beautiful want to get rid of a little bit of extra lines there I'm gonna
12:01click on save let's see if Facebook is written differently Facebook is in the lowercase the book
12:06oh I always thought it was book would be uppercase quicker login all right so we were learning that's
12:16that's always good when you're the actual word learning is with a typo wish all right so save
12:22now we have our our actual meat of our content inside of our application already all that's
12:28left is maybe a highlighted area and the highlighted area is going to be of speaking of all the way
12:34which will have our actual offer and notice how I put my offers always as a side note and it's
12:38never really the main attraction almost all the time the main reason is because I really want
12:43users to continue and engage with our product as long as a couple of people pay us and
12:47along as we're making enough money out of our emails there's no point in overly marketing things
12:52without giving things to users and we're happy to give things to users as long as we could afford
12:56it all right so let me just pop right pop with that content right in there and I'm just gonna
13:02find our headline for every single one of our development tendos and future tells can be yours
13:07for 289 for with free unlimited access is your the gig for life no waiting for emails and freebies
13:13this offer is only valid for a month for this month or until we run out of seats our normal
13:18price is 588 a year and a good thing to do is always to cross through if we have a cross through
13:22here yeah no cross through but let's see if maybe we have that in the styles nope all right so I'm not
13:28gonna worry about it it's always I like putting cross through is on big numbers the one of our
13:32supporters now and gain access forever click to learn more all right so one thing that I want you
13:38to notice is also once we wrote down our email is two things or I'm not gonna worry about the
13:42bottom because it's gonna be the standard that we have all the time notice when you scan
13:47through and read it is one is that it visually is readable and the second thing is imagine that
13:52you're a regular user so if I was a regular user reading a newsletter I would read have you logged
13:57in with Facebook get a free day pass no credit card needed log in and start learning for free
14:03that's all I would really read why because I bolded this this is bold that is bold and that's
14:08really the main message and we going down speaking of all the way it's kind of like a tease it makes
14:13people think hey the interesting what's that be one of our supporters now and gain access for
14:17forever click to learn more so this is really all I want the users to read but sometimes you
14:23need a little bit more meat inside of your email for the email to be accepted now don't forget
14:28also that your preview so you want to put something in your preview is um by logging
14:34in with Facebook you can get but logged in already then grab everything we ever made for free for
14:49the next few days support us and we will support back with our best offer ever alright so we got
15:10here a little bit of copy it's a teaser it gives a little bit more information logged in already then
15:15grab everything we ever made for free for for the next few days support us and we will support back
15:21with our best offer ever so it's it's it's enough information to give the user again a lead-in to
15:27what they're going to be getting but it doesn't give them all the details but it's enough of a
15:31tease to make the user say hey what what's what's this all about alright and really that's
15:35the art of this now again I'm not going to send the email out and you're not going to see me
15:39send the full thing out because I am going to send it on my own but I do want to do a couple of
15:43sub before and the first thing is I want to check how it looks like in a pop-up preview I want to
15:48check how it looks like with a push to mail chip mobile not as much but I do want to run an
15:52inspection and this one I want a thorough one because we have over 2,000 members so the
15:56better it is the better it looks so this looks great to me so what I'm going to do now is I'm
16:01going to run an inbox inspection
