IBM Cloudant pulls plan to fund new foundational layer for CouchDB

The IBM Cloudant software team has decided to pull the plug on its push to create a new database engine underlying Apache CouchDB, the NoSQL document store used by the BBC, Apple and atomic research facility CERN.

In a post on the Apache list, Robert Newson, former IBM Cloudant employee and Apache CouchDB Project Management Committee member, explained that IBM Cloudant had supported a plan to "build a next-generation version of CouchDB using the FoundationDB database engine as its new base."

"They will not be continuing to fund the development of this version and are refocusing their efforts on CouchDB 3.x," he said.

The most recent version of CouchDB is 3.2.1, released in November last year.

Using FoundationDB, an Apache open-source project borne out of Apple, as a foundational support layer would benefit scalability but also recreate consistency within a cluster which the project left behind after the 1.0 release.

However, these improvements would have come at a cost, which was too high for IBM Cloudant in the end, Newson said.

Jan Lehnardt, CouchDB committer and consultant, explained that not all the benefits were set to materialise in the way expected. Firstly, certain API guarantees in 3.x would not be possible to recreate with native FoundationDB features.

"We also learned that operating a FoundationDB cluster is a significant effort that somewhat goes against CouchDB's mostly "just works" nature," Lehnardt said on the thread.

The withdrawal of IBM support for the transition to a new foundation layer in the 4.0 release left the project with some options to mull over, he explained.

Given transitioning to FoundationDB might be worth it for large users of CouchDB it could be worth maintaining two parallel codebases, but then 3.x/4.x naming would not work.

The options were to then give FoundationDB-CouchDB its own separate project name and versioning, with a clear delineation between them.

• Google plays catch-up with JSON support for distributed RDBMS Spanner
• DataStax buries Apache hatchet and launches features to make NoSQL Cassandra faster, safer and more graphable
• Apache Cassandra at 10: Making a community believe in NoSQL
• Mexican tax refund site left 400GB of sensitive customer info wide open
• World-Check terror suspect DB hits the web at just US$6750

"We would have to maintain two projects complete with release management, vulnerability management, the lot. At the moment, CouchDB has just about enough folks contributing to [moving] forward at a reasonable pace.

"Doubling that effort might be tricky. While we had an influx of contributors recently, this would probably need more dedicated planning and outreach," he observed.

It would also mean new API features would have to be implemented twice to maintain overlap, producing another additional burden on contributors. He said there were probably more options and invited further discussion.

Matt Yonkovit, head of open source strategy at consultancy and support firm Percona said there may be hope for FoundationDB underpinning newer versions of CouchDB in the long run, depending on commitment from the community.

"Features that are needed or demanded by the community at large tend to get developed independent of a sponsor, with sponsors merely speeding the roadmap.

"Features for hire only carry on until the money or corporate interest runs out. You will often see some features introduced via this approach later be ripped out or moved to End Of Life status when the money runs out," he said. ®