An interview about our social proof experiment

After I read Sara's post about our social proof growth experiment, I wanted to know even more. Fortunately the engineer who designed the test was just one scheduled-last-minute Zoom call away. Also fortunately, it was my good pal Ethan.

The original plan was to record a video, then split it up as a 90-part series on TikTok. But in the end, it made more sense to just post the audio. It's a perfect summertime complement to hiking, gardening, or — my personal favorite — swiping through Instagram from the sofa with the A/C set to 68.

Listen to the recording below or scroll a little further for the transcript color-coded in the style of AOL Instant Messenger.

Brad:
Hello, testing, this meeting is being recorded. Okay, Ethan, thank you so much. We're probably not going to post this video and I'm sorry for the background noise here on my balcony, but we're going to talk today about an experiment that you did with our demo request form.

Ethan:
Yeah, yeah, totally.

Brad:
So we had a form and it was doing okay. And then we decided to put some logos underneath or to the left of the inputs. Is that right?

Ethan:
Yeah. I mean, so prior to this, we had one quote which was visible to everyone and then a series of logos that were equally visible to everyone. So we did have a quote, and we did have logos next to our form for social proof.

Brad:
What was the quote out of curiosity, or who was it from?

Ethan:
I don't even remember at this point.

Brad:
Irrelevant. It doesn't matter!

Ethan:
Well, we have this long list of, of array of quotes that we can pull from. We use them all pretty interchangeably across the site.

Brad:
And then you decided — who made, actually, how did the decision come to be? Like, where does like your team, the growth engineering team, how do y'all decide, hey, let's do something. We have this quote, we have these logos, we're going to try to make them more relevant because we know the types of people who are coming to the site and potentially filling out the form.

Brad:
But, who, where does that idea spark? Was it from you? Was it from the team talking together? Was it from the marketing team overall?

Ethan:
Sure. You know, I mean, I have no idea and I don't want to take any kind of ownership for the amazing ideas that come out. But as a whole, collectively, we have such a phenomenal team where anyone can throw ideas at the engineering team.

Ethan:
We'll put it into our queue, we'll triage it. And then take a step back and try to ask a question. Do we think that this particular improvement could improve things or have a negative impact? And we try to measure and we write out what we think the impact would be. What's our estimate, what's our hypothesis. What do we expect to gain? What areas is that going to impact?

Ethan:
So we kind of have this running list of triage-able, creative, amazing ideas, and it's a pretty long list. And then periodically as a growth engineering team, we review that list and then we bump a bunch up to the top, which align with what we're trying to work on.

Brad:
In general, like out of all the tests that y'all run, how many would you say end up having like a positive result like the one that we're going to talk about here today. And how many are duds?

Ethan:
Oh, man. I don't even know. I could quantify that. I'd have to find some data, which I don't have in front of me. It's a mixed bag. Sometimes there, you throw a change up there and we look at the report and it's statistically insignificant. It's not one way or the other.

Ethan:
And some that we're like, well, we don't know, you know, I didn't really do anything with that. Sometimes I feel like pretty rarely does it have a negative impact, but sometimes it does. And then we're tracking those reports and then usually within a day or two, if we notice the changes really bad and we'll pull it revert back real fast. And if we're unsure and we need some more time, then we kind of track the pulse on that. And then, respond accordingly.

Brad:
So in this case, somebody had the idea, we don't know who — could have been you, could have been some of the team, sort of, who knows, you had this list of ideas. And then somebody thought like, "Hey, why don't we work on this one? Why don't we change these logos? We know we can find more relevant logos depending on who the company is." And you went and found a better quote.

Brad:
But you started with logos, is that right?

Ethan:
Yes. Yup, okay.

Brad:
And you decided to change the logos next to the demo request form based on what — was it industry, what, was it company size? Was it — what was it?

Ethan:
We've come up with a criteria for our marketing site where we are targeting five different kinds of companies, but the industry isn't necessarily specific. But these are a combination of how big we think the company is, where they are in their growth trajectory, based on a bunch of different factors. And then we'll market to them based on those kind of general groupings.

Ethan:
So the analogy that I like to talk about is, I like to talk about it in that healthcare perspective. If you are selling, say a health care aid HR tool, you probably have two very different groups. One being like IBM, huge infrastructure. And then you've got mom and pop business with a really small team where you might have one person dedicated to HR. And if you can compartmentalize those two groups, then you take a step back from a marketing purpose perspective and you think, well, I'm going to talk about big, scalable infrastructure that rolls out across large companies and helps HR teams collaborate effectively and optimize their, you know, their time.

Ethan:
And so we're a big, big business. And then the other verbiage is going to be focusing on, you know, acting as if you were a Fortune 500 company with really small agile seat. So we're based on what your product is and then who you think are your different demographics and what, you know, so it could be company size. It could be whether they're a small B2B startup, whether they're open source, right? Those are some of ours that we use to kind of imagine who those customers are and what kind of questions and contexts they might be viewing ourselves.

Brad:
So you started with five groups. Did you feel like five was a good number?

Ethan:
Yeah. Five is good because, I mean, it's hard. You could break this down as much as you want. We can have a thousand groups, but then that becomes really, it's just the infrastructure to manage a thousand quotes and a thousand groups of logos. We just couldn't do with our size team. So five is good. That's a manageable group. We can come up with five quotes, five sets of logos. And then you're also measuring that across those industries. So five is manageable based on our team size, we think.

Brad:
First, you had to like design it, right? And that's, that was like your role, like that's part of it. Like you had to make — it's not easy to get all the logos to look good, right. The logos, they're all different Heights and widths and like, you're fitting it in this spot and it's not necessarily, it's not as easy as just saying, like put the logos in there. Like you had to make sure that it looks good.

Ethan:
Yeah, because some, some logos are tall and skinny and some are wide and chunky. And so you have to tweak them a little bit here and there based on a group.

Brad:
And after that rolled out, you, we saw that there was like a success - it works. I'm seeing here, I'm seeing here in our notes doc that there was a 34% lift in the number of demos requested in the first few months. So it was like a big, a big win.

Ethan:
It was a big win. And it's interesting because I like to think about using actual information is like, this is, this is like responsive 2.0, right. We used to design for desktops and laptops and things like that. Now we're designing based on the person who they are - we're revealing that information so that it's personalized based on the context of that person in their role. And so this is a lot of new ground.

Ethan:
We don't know some of the answers to these questions and different companies are running similar results, or we're running similar tests but coming out with different results. And so I think what's bubbling up is this idea that, one, big name brands are going to win over terrible brands. If you show Nike and you show mom and pop Shoe Sole, like, that's it, they're not going to know who those are, even if they're both in the apparel industry or not in the industry, like it's just going to outpace it.

Ethan:
So you have to first, I think, make sure that the logos are really, really established logos across every industry, because if they're not established logos, then that's actually going to have a negative impact. And where, what we've tried to do is really put forward excellent logos based on those segment groups. And so we had kind of a answer to question one and, you know, personalization is really impactful or can be, but it's how you do it is also equally as important. And so, yeah, so again, this is a hypothesis. We thought it would be a mixed bag. It could have none. And we really weren't totally sure until seeing those results, which is, which is really nice.

Brad:
And I don't know who did it, but somebody probably had to go and contact all of these logos and all of these companies to make sure that we could use them in this context. And that was probably more effort than you would think as well.

Ethan:
Oh, totally. Yeah, I mean, we had somebody on that side who focuses on customer relations and we've got amazing companies that we ask for feedback based on our product. And if their quotes are great, we're over asking for permission to share their quotes and logos, et cetera.

Brad:
How is it decided — you, you ran the test, it was out for a couple months. It was a big success. You did five different sets of logos. It would increase the demos by a third. How did you make the decision to say, "hey, we can make this even better. We can push this even further" and we can like try something, you know, we're not satisfied here. We're going to try to change the quotes as well.

Ethan:
Yeah, yeah. I mean, no one, I think, is anyone ever satisfied, right? In the web industry, we're always pushing the bounds. What could we make better? What new technology can we use? How can we optimize? And that's just not, that's not just within our industry is the web, right. That's also sales is doing it. CEOs are doing it. Everyone is asking that question.

Ethan:
So for us, we're like, great love these results. Let's keep the momentum going. Right. How can we push the balance more? What can we continue off of that? And I think that the lesson that we learned was, yes, we're confirming that does have a direct impact on the form submits and that's making sure that it's relevant. So now reapplying that again, how can we increase relevance to the customer even more?

Ethan:
And I think that that's probably what leads into the next question, which is like the quotes, right?

Brad:
Yeah. The quotes, the quotes came in and we found kind of quotes that paired with those same five groups from the start. I'm sure we had to, did we have those on hand or did we have to like, get those?

Ethan:
It was a mix. We had some of them and I think some, we, you know, we pulled in.

Brad:
I mean, you put the quotes, you added the quotes, below the logo, I think, or above. You got a nice, big quote, five relevant people to the groups that y'all had selected. And then what happened after that? What were the, what were the results?

Ethan:
Again, the results were excellent. I don't have the numbers right in front of me.

Brad:
It might've been, it might've been 84%.

Ethan:
It was really high, which again, you know, this is the question, it's always a bit risky. Because we're hoping, and our hypothesis, again, built off of the logos was: relevance does have a factor. So let's hope that that's great. And let's hope that continues to build, but you do run the risk of potentially alienating people by using the wrong quote.

Ethan:
And so it might be from some CEO at one company that has one particular angle and changing it for that group may not have been the best choice. So it was re- it was, we just didn't know until rolling it out. But you have to take those risks. And I guess that's kind of what our — I'm really, I'm really, really happy.

Ethan:
And I'm proud to work at Clearbit where we're not just supported, but we're encouraged to take those risks, and having a leadership team that is like, yeah, you think that you believe, but this is going to impact things. You're excellent in your role. And I'm going to trust you to make that decision, even if it means for a couple of days, we negatively impact our leads. The benefit of that is going to grow. And it's encouraging to have leadership that believes in you, in your role, right.

Brad:
It's fortunate in this case that not only was everyone behind you and not only did you have the opportunity to do it, but it actually worked. So now that you've like proven that this sort of concept is successful, how, you know, how can you use this in like other places? I know that I've seen, you mentioned potentially, and in customer testimonials, or we can use this in paid ads or onboarding, where else do you see the opportunities for something similar?

Ethan:
I mean, everything from verbiage and how we describe our products, you know, we might be talking about why we're trying to sell one particular tool, whether it's in a header or a product description, we're talking about problems. And then how are our products, our solutions to those products or problems.

Ethan:
And if we're talking about the wrong problems, then that's going to put this dissidence in what the person is reading. So the better we can align those, the better that our product is going to appear. And so, you know, within the marketing verbiage on the site, you know, also within email campaigns, advertising is a huge one as well. Being able to show the same kind of message and verbiage across people's social channels like Facebook, you know, with Clearbit and our advertising platform. So it's, the potential is, it's more a limitation of our ability to implement than it is the infinite options out there.

Brad:
That makes sense. Yeah, I think it's inspiring. I think what y'all do is really great work and I'm just impressed with the risks you take and I'm impressed with the results, the good ones and the bad ones, you know, I think it, I think it's just wonderful.

Brad:
And one of the things that I think is like, you know, super wonderful, just kind of the way that you think personally knowing you a little bit, is that, are we trying to sell this software? Of course! Is it our job to make sure people fill out that form, request those demos and get to our sales team? I think so. But I think the way that you look at problems isn't necessarily just about like sales and numbers and that's all great. But when we talk about relevancy, I think you've created a great amount of relevance for the potential customers. But I think what you've also done is show a great amount of empathy for these people who are software buyers, but like they're still people.

Brad:
Would you be able to talk, just lastly, a little bit about how this sort of experiment showed our own empathy in the people who are visiting our site, even for a brief moment?

Ethan:
Absolutely. So, you know, I, as a web designer and engineer years ago, wrapping my mind around responsive design was this paradigm shift between making our site accessible to a whole group of people, right? There was, you could make a desktop website and then as it scales down to a tablet and then down to a phone, what you're doing is you're acknowledging the limitations and benefits of that particular user in that place in time. You know, a phone is only so big and that's viewed as a, you know, sometimes as a limitation in accessing information on your website, but if you adjust it down, you think about what context should be shown first and how to communicate it effectively.

Ethan:
You're inviting that person in and giving them all the first class experience. They're not a second class citizen in your website. They're, you're treating them just like every single person that comes to your website.

Ethan:
And I think if we are, we're trying to strive for that regularly. How can we look at a person and understand where they're coming from? And then treat them like first class citizens and roll out the red carpet. And so that's responsible design that's accessibility and readability. That also extends into the context of the person in our role. We are not just presenting somebody to try and sell, we're articulating it so that they can understand why it's important to them and their role that's actually going to help them then sell that internally.

Ethan:
We're empowering them to go to their leadership and be like, "Hey, here's our problem. We're a small team. And we want to operate like a Fortune 500 company." That managers would be like, "Great. That's exactly, that's our problem. This is the solution for it." So I'm empowering them as well. So we're just trying to become better at understanding the context of the user and changing our content to do that. And personalizing it. It's just an extension of that.

Brad:
Ethan, I think this was great. I hope people like it.

Ethan:
I mean, I love it. I love it. You know, and again, this goes back to, we have a phenomenal team. I am only so good as a designer and engineer as having a copy team that is going to wordsmith magic words for me to put onto the page. It's going to communicate that message. And then having an engineering team that is building a really, really powerful Reveal program that's going to identify what company they are, you know, so those engineers are brilliant.

Ethan:
So really it's a Captain Planet moment where you get all these phenomenal people together and having those resources is just, it's just tremendous. I'm all the better for the people that I work with. And we're all included.

Brad:
We're all the better for you. And can't wait to see what y'all do next. It's always exciting. Thanks, Ethan.