Last Thursday, the Elasticsearch Netherlands group held its eighth meetup. This meetup was a bit special, however, since it also coincided with the Elasticsearch company meeting. Rather than the traditional two presentation format, the Elasticsearch developers gave a number of lightning talks about what they have been working on.
A big thanks to Trifork for hosting the meetup! Around 65 people showed up, with lots of great discussion before, during and after the event.
Drew Raines – Life after EC2
Elasticsearch talks are often at a very high level: a new feature, a new type of query, how to implement XYZ functionality. Drew’s talk went the other direction and discussed the ramifications of running on bare metal hardware. In particular, he discussed how he and GitHub debugged a performance quirk with their new cluster.
Long story short, they discovered that the default Linux I/O scheduler is atrocious when using SSDs, or RAID over fast disks. After switching their scheduler to Noop or Deadline, they saw a remarkable 550x performance increase!
Check out Drew’s presentation for more details (especially if you are using RAID or SSDs!)
Costin Leau – Real-time data with Hadoop
Costin talked about his new Elasticsearch-Hadoop integration. He highlighted what Hadoop is good at…and where it is lacking. Real-time processing, analytics, and geo operations are all difficult for Hadoop. Elasticsearch, on the other hand, excels at these problems.
Costin explained how Elasticsearch can be integrated with an existing Hadoop installation to provide functionality that is difficult or slow in Hadoop. He then showed examples in several popular Hadoop frameworks and some benchmark data. Costin’s slides are available on Speakerdeck.
Clinton Gormley, Karel Minarik, Honza Kral, Zachary Tong – Unleashing the Clients
The client team gave a short overview of the new language clients. They talked about the motivations behind creating the clients, as well as the need for a unified interface, consistent testing framework and pluggable components.
Britta Weber – Function Score Query
Britta talked about the new Function Score Query. Partially known for its outrageously awesome ticket on GitHub, the function score allows you to tweak scoring with complicated mathematical functions. Britta discussed how it is used and some example scenarios.
Igor Motov – Snapshot and Restore
Alexander Reelsen – Completion Suggester
Uri Boness – Aggregations
Simon Willnauer – Lucene Pipeline
Big thanks to everyone who showed up for the meetup, and everyone who stayed around to chat after the presentations. The Elasticsearch team had a great time presenting and really enjoyed fielding questions about their respective projects.