Ten years ago, when CasperVend was still in its infancy, someone came into my store and sent me an IM. He said, "I hope you never forget your roots, and that you never become a huge careless corporate entity like Hippo" - I laughed, and said "Oh don't be silly, I'm just a small operator, that will never happen".

You see, I originally created CasperVend just for me. I wasn't happy with any of the solutions available on the market, and I needed something to sell my gadgets with. So I created CasperVend. It ran with a single script, and had lots of features nobody had seen before - so it soon attracted the attention of other merchants, who asked me to release it.

Today, CasperVend services the majority of in-world sales, often with over a million transactions processed in a single day. Since we've been running, many competing systems have come and gone. We've seen the demise of the giant Hippo, which was left without the love and care it needed for many years. Other strong contenders such as E2V have risen and fallen. We have become the behemoth that I never had any desire to become.

But throughout that time, we've remained absolutely true to our roots. Low impact, easy to use scripts, and reliable services. The only thing that has changed is the number of customers we service, and the corresponding impact on our time.

Recently, there have been some rumblings on social media. "CasperVend is losing everyone's transactions!" "They call their customers idiots!" "They abuse their customers!" "THEY JUST DON'T GIVE A FUCK!" "Casper kicked my cat and defiled my fair maiden!" (Okay, maybe not that last one).

The problem with social media, though, is that it is a massive enabler of confirmation bias, and that it provides a giant soap box for the loudest people - and people only shout loudly when they are unhappy. You only have to see the results of recent elections and referenda to see what a catastrophic effect this confirmation bias can have.

However, of course I could not just ignore these rumblings. So, I set up to gather actual factual information via a survey with some of our customers. No rumour, no bias, just hard statistics. I'm going to go through each of the survey results in turn and talk about each one.

Overall experience

The first question asked "How would you rate your overall experience with CasperVend"?

It's very pleasing to see such an overwhelmingly positive response to this question. However, of course, it's important not to grow complacent with this kind of feedback - clearly there are a small number of people who are unhappy, and we will be actively working to improve this even further.

Delivery reliability

This was the main topic of the gossip on social media, so of course, this is the area we were most concerned about.

Firstly, we asked, "How would you rate the overall delivery reliability of CasperVend?"

Absolutely nobody selected "Poor" or "Terrible" for this category, which is absolutely fantastic. However, again, there is definitely room for improvement - we strive for every experience to be "Excellent".

To get a clearer picture, we also asked "Are you aware of any transactions of YOURS which have been lost by CasperVend within the last three months, that weren't a result of a misconfiguration at your end?"

This implies that 90% of our customers haven't experienced even a single lost transaction within the last three months - which is great! However, I certainly do feel that if 10% of our customers have genuinely experienced even one lost transaction, then we definitely have work to do.

So, I set about personally contacting every customer who answered "Yes" to this question, to get more information. (As of writing I've only got through about 80% of you, sorry - I will continue though!).

I asked: "Hey there! Thank you for filling out the CasperVend survey. I notice that you said "Yes" to having experience lost transactions within the last three months. I was wondering if you could share the details of this so that I may investigate?"

Here is the breakdown of the responses I received:

So, in reality approximately half of those who answered "Yes" have actually experienced a lost transaction. So, let's focus on those.

Most of the people who legitimately experienced a lost transaction said that it was at most one lost transaction per month. Of the people who experienced more regular issues, it seems to happen during flash sales or crowded events. Gachas were commonly mentioned as a source of the failures.

Out of the people who selected "Yes" to the lost transactions question, 78% said the overall reliability was "Good" or "Excellent". 23% said "Average", and nobody said "Poor" or "Terrible".

However, what follows is possibly the most concerning statistic. Out of all the people who reported that they have lost transactions during this period, only one of them contacted our support team about the issue.

I'll address the implications of these statistics in the conclusion.

General satisfaction

We asked "Would you recommend CasperVend to your friends or colleagues?"

No problems here. A whopping 96% of you would recommend us - Thank you, everyone!

Next, we asked "Are you considering moving to a different vending system?"

6% said that they were considering a switch. This is pretty standard customer attrition. Obviously, we cannot service absolutely everyone's needs and some people grow to need features we do not currently provide.

So, how many are thinking of leaving because they are unhappy? Out of those who are thinking of leaving us:

39% said they would not recommend us (2.3% of all responders)
53% rated us "Average" or below (3.18% of all responders).
17% rated us "Poor" or below (1.02% of all responders).

So, of the people thinking of leaving us, around half are leaving because they are unhappy with us or our service. One must acknowledge that we simply cannot please everyone, however, of course we'll do everything we can to improve this. It may be only a small number of users, but every customer matters to us.

Customer Support

Now, the elephant in the room. I'm very aware that our support is far from perfect, and I'll get into the reasons why in a bit. But first, the survey results:

57% of participants rated our support "Excellent" or "Good".
5% of participants rated our support "Average".
6% of participants rated our support "Poor" or "Terrible".
32% of participants said "N/A" - they haven't contacted our support team.

Excluding the N/A category, the numbers are:

84% "Excellent" or "Good"
8% "Average"
8% "Poor" or "Terrible" (Note the ratio of poor/terrible to average changed due to rounding).

Yeah, pretty much as expected. Obviously this is an area that we need to focus on, and I will outline some things we will do to improve this in the conclusion.

There have been some stories of me insulting and verbally abusing customers. Some of these stories are false or blown out of proportion, but some of them are true. It has happened. The thing is, I'm not a customer service guy - I'm a developer, a software engineer. There's a reason why Google won't let you speak directly to their developers. We tend to be socially questionable folk, hiding in the shadows and protecting our precious.

The painful fact is, we have a lot of customers to support, and very limited time to do that. I have a few very wonderful people (such as Norsk and Sphynx) who take care of most of the brunt, but we're still stretched pretty thin.

So why don't we hire a large support team, people who are good in this area? Simple economics. Think about it - you pay $10 USD for a vending system. Many of our customers have had their system for ten years. That's a buck a year, and that will continue going down as long as you use your system. I can't hire a large team to provide technical support.

There's also the fact that I despise script-reading technical support - people who don't understand how the system works, and are just there to deflect the bulk of queries - when I have to deal with these people myself, I end up getting very frustrated. When customers receive a reply, I want the reply to be accurate, helpful and technically ept.

So why don't we just charge more money, or implement a premium support tier? Because this goes against everything I believe in. CasperVend is profitable, it always has been, but it's not a cash cow. We've kept our prices as low as possible, because we believe that every merchant should have access good quality tools, whether they're large and successful, or just starting up. I don't believe in a class system, where people only get help when they can afford to pay for it.

The ultimate solution? Please, please, help us to help you. If you refuse to read the manual, if you are being obnoxious, if you are fighting against our support team when we're trying to assist you, we won't be patient. Please, follow the instructions. Please, read the rules. Please, provide the information we need to assist you. If you have exhausted the available resources and you can't find the answers you need, of course we'll help you. When you file a ticket, please provide any relevant information (customer names, SLURLs to vendors, transactionIDs if you have them) - this will help us to help you.

Conclusion

Our focus is, and always has been, on providing an excellent, stable service, and keeping it online. We have achieved that - the numbers speak for themselves - and we'll continue to evolve to ensure that we always maintain that status.

Customer support is equally as important, of course, and the survey has highlighted the truth that ours is not perfect. It's a tricky topic, particularly when you have a large user base and limited resources to play with.

One thing is absolutely clear - you can't make assumptions based on a few loud people on social media. There's so much misinformation, and following that, queue a barrage of accusations and posts from other merchants claiming that the CasperVend world is collapsing into a fiery pile and we routinely cook babies in a stew.

There is one accusation which stuck in my head - that we do not care. That couldn't be the furthest from the truth. However, it's true that I don't care about whining on Facebook. I care about our customers, and implore any customers who are unhappy with our service to contact our support team. We will listen to you there.

Moving forward

Delivery Reliability

Clearly, given the massive quantity of transactions that we process, we are doing extremely well for delivery reliability. However, it does seem to be true that a very small number of transactions are being lost.

Here's what happens when you pay a vendor:

1. The vendor first checks that the amount you've paid covers the cost of the item, and your entitlement to any group discount. If the amount is insufficient, it immediately refunds your money.

2. The vendor then immediately generates a transactionID and issues it to you in local chat. It looks like this:

This URL allows us to track the transaction the whole way through our system, even before it arrives at our servers. No other vendor system operator provides this level of detail.

3. The vendor then attempts to deliver the transaction to three separate servers.

The first two servers are our main application servers, and 99.9999% of the time the transaction goes straight through to these servers, and the transaction gets processed.

The third server is our off-site backup transaction processor. This server is positioned in a different country, at a different host, and with a completely isolated data route to the LL servers. Even if our entire network is down, this server will catch any transactions and will forward them to our application servers once they are reachable again.

4. If the vendor doesn't get an "OK" message back from any of the three servers, it falls back to e-mail. E-mail is the second fallback method, and is the slowest. It'll take a few minutes to get through to us.

The e-mail fallback tends to be used when vendors are hammered at a busy event or flash sale. I spent a while digging through our systems during the past week, and found that our e-mail processor wasn't functioning correctly. As a result I found about 500 transactions which were never processed, and they have now been restored.

So what if a transaction never reaches us, despite all of the above failovers? The fact is, and the numbers show, this is extremely rare. Recently, LL's e-mail servers have been struggling (https://jira.secondlife.com/browse/BUG-225431) and this has definitely had an impact on this. We also know that some people have had a rare issue with vendors freezing, and of course, if the script is stopped then payments won't be handled.

Unfortunately, the main issue we have is that these incidents are simply not being reported to us. Obviously, if we don't know about a problem, we can't do anything about it. It's been suggested that this may be because our support is scary or too time consuming, so we're moving to address that.

So, here's what we're doing in the area of delivery reliability

• We'll be shortly releasing a new vendor script, v2.20, with some memory improvements to combat the rare vendor freezing issue. In addition, this script provides more information in the SL transaction log for outgoing payments, to help users see what's going on.

• We'll add a new simple form for reporting transaction failures, which will gather the information we need quickly and easily, to hopefully encourage more users to report these problems.

• We're going to do exhaustive testing with vendors in very busy environments, particularly gachas, and fix any issues that crop up.

Customer Support

Obviously we want everyone to have a great experience with our customer support, and while it's important to acknowledge that we simply cannot hold everyone's hand - we just don't have the resources - there is definitely more that we can do.

A big problem for us is lack of information. Either issues aren't reported, or we don't get enough information to help properly. In order to be able to help you effectively, we need to know the exact issue. Please be precise and concise. Telling us "it doesn't work" does not help us to understand the problem. If you are getting any messages, please tell us what they say (exactly). Tell us exactly what you are doing, and exactly what is happening. If we ask for clarification, please don't get offended - we are trying to help.

Here's what we'll do to improve our customer service going forward:

• We'll work on more tutorial videos for CasperLet and CasperVend to show how to use the systems effectively.

• We'll improve our support ticket system so that it asks for the information we need directly, rather than depending on customer intuition.

• I'll give our support people better access to the internals of the CasperVend system, so that they can provide better support.

Finale

Wow, you're still here? That was a lot to read, I know! Thank you to all our customers for your overwhelming support. We'll do everything we can to improve and grow from where we are, moving forward.

We're not going anywhere, we've still growing faster than ever before, and we'll continue to be here while other services fade away. <3