Why I advised my company to go for Elasticsearch

One day, I was checking out my company's infrastructure cost and cost of other services. I was surprised to see that around $70000 was getting paid to Enterprise edition of Algolia Search. Algolia is search provider, it lets you to index your data and then you search, filter and can use other search features.

The high cost stuck in my mind. I talked to my boss about it. And asked why are we paying so much for searching, why can't we use Elasticsearch. His reply was exactly at the point of speed and its unmatched search capability. He told me that Algolia claim that they are much faster than Elasticsearch (around 200x) in searching. They do not have any latency and their client don't have to store the data and maintain the infrastructure.

I was rigid on cost. Why we pay so much? He said we can reduce the cost, but we cannot get rid of Algolia (only because of its quick response for a search query); you can try to find alternative and see if it fits into our ecosystem. 

I took around two weeks to see that what we can and cannot achieve if move to Elasticsearch. We can have hits data, highlighting search results and facet search capability in Elasticsearch (these the capabilities which we were provided by Algolia as well). But, Algolia had unmatched speed against Elasticsearch. And they charge more for exceeding storage limit.

We had meeting after two weeks, I put my points, both advantages and disadvantages. Our team discussed about the architecture, Elasticsearch cost on AWS and our development approach. We concluded our meeting with a decision to keep both Elasticsearch and Algolia. 

We will use Algolia for last 6 months data. Elasticsearch will be used for historical data (more than 6 months old). In this way, we were able to maintain current capabilities of Algolia and reduce cost due to Elasticsearch implementation.

Below is illustration of cost on AWS for managed Elasticsearch Service and compute costs:

a) t2.medium.elasticsearch 

vCPU=2 Memory (GiB)= 4  Price Per hour = $0.078.
Average Usage (in hours):  2000
Storage Cost: $5.96 per month for 40 GB ($0.149 GB-month of general purpose provisioned storage)
Cost in a Month: 156 + 5.96 = 161.96
$1943.52  in a year

b) Amazon EC2 running Linux/UNIX Spot Instances: $300 in a year

c) DynamoDB: $60 in a year

So, $2303.52 in a year. Based on data let's estimate it would not more than $3500 in a year. You can see there is huge difference between the cost. We could take lower/cheaper package from Algolia and save cost by using AWS Elasticsearch.