Meet the New SearchNode: How One Failure Changed eCommerce Search Landscape

It was Winter of 2017 when we met with one of the TOP 100 European eCommerce companies. It was our 3rd meeting and we were super excited as they have already tested our technology. In emails, they said, that the SearchNode API is really powerful and it looks like they can build powerful search they need.

However, during that 3rd meeting, most of the time was spent on SearchNode comparison with Elasticsearch. They wanted to learn what would be the benefits if they work with SearchNode, rather than build an in-house search team.

It was quite easy for everyone to understand: using SearchNode, excellent search experience would be up and running in 1–2 months, with an analytics tool, admin dashboard and support from our engineers. In comparison, to achieve similar results with Elasticsearch would require 1–2 years and a lot of human capital. So with SearchNode, it should have been much cheaper and awesome search experience would come much faster.

After a couple of days, we got the answer: they have decided to build an in-house search team and construct search by themselves using Elasticsearch. It was difficult for me to understand this business decision, while the advantages were on our side. So I called their CTO, as we have set a nice relationship during meetings.

What he told me, changed everything and dots in my head have been connected:

“Antanas, your technology is awesome: developers liked it, business saw a great potential of search experience. But we need more than a technology, we need experienced search experts working full-time on our search. Our business is different from others, we need more than your API offers, also we don’t know what we will need in the future and it’s a risk to rely on one API. Finally, we want to set a process of continuous search improvements and go deep into the product, have the ability to change it’s core functionality when we need it.”

It was clear to me. Small eCommerce businesses need just an easy to implement search technology with a dashboard to customize a surface of the search (calibrate weights, boosts, add synonyms, protwords, etc.).

Medium-big eCommerce businesses need powerful search technology AND search experts who would be able to implement it based on custom needs and continuously work on it with the ability to change the technology itself. Businesses think: “we will build whatever we need”, rather than “we will build whatever X has to offer us”.

Dashboard Only Gives an Illusion of Control and Independence

Many SaaS eCommerce search solutions come together with a dashboard, which allows easily fix some primitive issues, and implement changes, such as set-up boosts, put redirects, add synonyms, modify ranking and etc. However, usually, this only deals with the surface of the search, as all of the changes need to be implemented one-by-one, and there are thousands of similar cases, which have to be fixed manually.

For example, you notice that when searching for ‘dog food’, search also displays cat food, dog bowls, and dog feeders. Irrelevant products are found because they are all in the same category (Dog Food & Accessories), moreover, in the products description, there is text like ‘<…> bowl is usually used for dog or cat food <…> ”. In this case, the dashboard is used to manually exclude irrelevant results for the specific search query ‘dog food’, so next time user searches it – irrelevant results are not found. But the biggest challenge here is that there are tens or hundreds of thousands of products, with millions of similar cases, and the solution should apply for all of them, not just specifically for ‘dog food’. We call it – scalable search improvements.

Even more, in long-term, manual fixes can create search-debts. For example, if you set up a rule ‘if Adidas shoes are searched, redirect to sports shoes category’ (in case you don’t sell Adidas shoes), later when you start selling Adidas shoes, no one will find them, as the previous redirect rule will still be applicable, and fixing it will require manual action.

Thus, the dashboard creates a smokescreen, allowing online stores to think that they are in control when in reality, it only touches the facade and has a huge potential to create serious challenges in the future. Instead of that, the search should be improved by modifying and changing core relevancy and data processing algorithms, not just individual problems.

Hidden Revenue Behind the Long-Tail Queries

Some eCommerce companies only pay attention to the top frequently searched queries. As a result, they put most of their search improvement efforts there, when actually these queries can take only up to 20% of overall search revenue. This creates a false belief that the search is being continuously improved and that the business is getting the most of it. What they forget is that a huge part of the search volume and revenue lays behind the Head: in the Long-Tail. Even though these queries have a lower frequency but they outnumber the ones in the Head, many of them are never even checked because it’s impossible for a human to manually analyse e.g. 50 000 search terms. Still, those queries are staying there and waiting to be optimized for conversions.

The Long-Tail issue becomes extremely relevant if an eCommerce site has over 50 000 monthly search queries. We analyzed 50 eCommerce websites (our clients), who have above 50k search queries (most of them hundreds of thousands and millions) per month, and we found that on average, TOP 100 queries make only 20% of all search revenue.

This is one of the many pieces of research we did about eCommerce. By combining tons of data, users behavior insights and best practices from the market, we help eCommerce companies benchmark their searches and find low-hanging fruits. If you are interested in getting insights from us, feel free to join our monthly newsletter. I personally guarantee you that it is a spam free subscription, and we will keep it that way.

The New SearchNode vs Existing Alternatives

One of the most popular options for modern eCommerce search is choosing to develop it in-house. This requires lots of human capital since you need to hire a full-time development team, with a very specific background (search development). In addition, to achieve a high-quality search also requires a lot of time, in some cases, even a few years. On the other hand, with the in-house team, your search is much more flexible, as you are not dependent on any service provider, therefore, you can achieve a better quality of your search. You can find out more about common misconceptions of building search in-house in this Video.

Another option is to use a SaaS vendor. It delivers a search of quite a good quality pretty fast, and the costs are relatively low. However, the possibilities are very narrow, as the company is dependent on a solutions provider and it’s product capabilities. Usually, it is a one-size-fits-all product, with basic maintenance support, and limited possibility of search improvements in the admin dashboard. Thus, your search quality is improved slowly, and as was already mentioned, with lots of manual work. Often a big part of these human efforts go to maintaining the current search quality and making sure that all the previous manual adjustments are still relevant today instead of improving it.

What SearchNode offers is a SaaS product and dedicated solution developers, highly skilled in search. The client obtains a high-quality product as fast as with usual SaaS vendor, for relatively low costs. And it gets even better over time, as dedicated developers team improves patterns, instead of a manual case-by-case fixing. They literally work on each client search individually, not just advise or work on the global product. Developers adapt search algorithms to meet the client’s business and language specifics, provide custom-ranking algorithms, optimize long-tail queries, and make data-driven improvements. Even if a client doesn’t know why ‘dog food’ query also finds dog bowls, feeders, and cat foods, SearchNode engineers track the core issue and then implement the best solution, not just for one case, but for all similar cases. That creates scalable improvements and search experience flourishes. Read more about our Solution.

The Dinner is Served… or Not?

Choosing what to do with your on-site search provider is very much like going to the restaurant for dinner.

In the case of the in-house development, you come to the restaurant to have a nice candle light dinner, but as you approach the registration desk, you realize that it’s empty. In fact, the whole restaurant is completely deserted: there is no hostess to show you your table, there is no bartender to fill up your glass, there is no waitress to take your order. If you still want to get that dinner you have to hire the staff yourself, train them how to serve tables and cook the main meals, and then just wait and hope for the best. In such a manner, in order to get your dinner, you will have to put a lot of effort and money, and you can not be sure, that the outcome will be great. On the other hand, the potential for the flavor is boundless.

In the SaaS vendor case, things seem to be fine at first. You get seated to the elegant table, friendly waiter take your order, and you even get a fine glass of drink. As you get relaxed, the waiter comes back to inform you that the restaurant has a cook-dinner-yourself concept. Basically, the restaurant will give you a recipe, provide you with all the ingredients, hand you out all of the best equipment, and will let you enter the restaurants’ kitchen. It is pretty clear, that even with all this, it still remains hard to achieve the level of an experienced chef, and don’t forget that you cannot really improvise or change the original recipe. Thus, with just a SaaS vendor you get a product with limited potential, and still, have to put a moderate amount of effort.

Choosing SearchNode is like getting a full service at the first-rate restaurant. Not only you get a full service and delicious fully-cooked dinner, but we will keep delivering it to your doorstep day after day. Also, meals will even get better, and better every day, as the kitchen will learn more and more about your needs for taste. In this degree, with our solution, you will have a product with unlimited potential, with as little efforts from your side, as possible.

Sincerely,
Antanas Bakšys,
CEO of SearchNode