This is something that we're sharing because I think it's good for our customers to know what's up and what we're doing behind the scenes. Unfortunately, it's not all sunshine and rainbows and new functionality... sometimes we have to deal with BS like hackers trying to access our systems and take us down. Last week we had one of these episodes. Thankfully, this is not a security breach announcement.
Late last week we noticed that one of our database nodes was getting a lot of traffic to the extent that it took it down. Through our backend developers' investigation we found that a couple of IP addresses in China and Milwaukee (🤷♂️) were doing their best to brute force access our ReThink database which holds our listing data repository. They had focused on one server out of our cloud array and went to town with a few million login attempts an hour to try and access it through SSH. They didn't succeed. They also didn't take us down.
Our dev team was able to work over the weekend and mitigate the issue by cutting off the ssh traffic through our firewall and hardening it up some so other attacks would also not succeed. What they did succeed in doing is crippling one of our ReThink cloud servers to the point that it was pretty screwed up and not connecting with the other server in the cluster. Talk about a cluster, right? (bad jokes) Anyways.. The beauty of a cloud setup like we have is that all the servers in these clusters replicate over all the servers in the cluster, so we didn't actually lose any data. But getting the #3 server back up and connected to the other servers was a real pain in the butt.
The #3 server took a couple of days for our backend team working through the weekend to get back online, connected and replicating properly again.
Why am I even putting this in the changelog? Because this is a good example of a couple of things: 1) Our infrastructure is really secure, and more importantly, 2) this is the kind of thing that comes up that our support team talks about as something taking priority, where it delays tackling other bugfixes and development projects. Sometimes we just have to drop everything and put out a fire like this (that comes out of nowhere). This kind of thing is thankfully is pretty rare.
We appreciate all of our customers understanding and patience when things like this happen. Thankfully this didn't have any direct customer impact, so it went on behind the scenes. But still... know that we couldn't be working harder (and hopefully, smarter) than we are to make Showcase IDX the leading IDX plugin that we all want it to be.