SearchKick – Remapping an Index

Recently someone updated a Rails Model ID datatype from int to uuid. This has caused issues with ElasticSearch in my local instance:

:exception:
- Searchkick::ImportError
- '{"type"=>"mapper_parsing_exception", "reason"=>"failed to parse [id]", "caused_by"=>{"type"=>"number_format_exception",
"reason"=>"For input string: \"03b42e85-1f09-402c-8f51-89d61c128ffc\""}} on item
with id ''03b42e85-1f09-402c-8f51-89d61c128ffc'''
:exception_object: '{"type"=>"mapper_parsing_exception", "reason"=>"failed to parse
[id]", "caused_by"=>{"type"=>"number_format_exception", "reason"=>"For input string:
\"03b42e85-1f09-402c-8f51-89d61c128ffc\""}} on item with id ''03b42e85-1f09-402c-8f51-89d61c128ffc'''

After some digging I around I was able to resolve this by deleting the Indexes. Rails/SearchKick rebuilt them afterwords.

You can view the available entries with a GET to elasticsearch with “*” as the path.

curl 'localhost:9200/*'

It’s easier to read if you pipe it into jq.

brew install jq
curl 'localhost:9200/*' | jq keys

From the list, identify the object that is failing, mine was organizations_development. Delete the mapping with

curl -XDELETE 'localhost:9200/organizations_development'

From there, elasticsearch will rebuild it and index your data.

Silver Bay Saved My Bacon!

This morning I was attempting to customize DoorKeeper. After an hour of prodding I wanted to wipe all my changes.

1. Git checkout .
* removes changes to known files
2. Git clean -fd
* Removes unchanged files and directories

Except that’s not what I ran. I ran clean with the flags ‘fdx’ which also removes hidden files. So I lost all my .env and config files. Sure I could spend the next hour or two hunting those down.

I though Time Machine tracked these files but I didn’t see them. Silver Bay Tech has a simple blog to follow along and have Time Machine restore these files.

Mac OS X – Restoring Hidden Files and Folders with Time Machine

Thanks Silver Bay!

Papers We Love: Feral Concurrency Control

This months PWL meeting discussed ‘Feral Concurrency Control: An Empirical Investigation of Modern Application Integrity’. The intended audience is database system researchers. It explains how popular application frameworks lean on ORM’s that disregard the database concurrency control solutions. Instead of using the databases system for foreign keys, constraints, and transactions they create a set of rules that perform in the application layer. The paper demonstrates how validations in the application layer give us inconsistent results.

Unique constraints are given as an example. When a field with a unique validation is created or modified in Rails, it will query the database to ensure that value does not already exist.

Applications with minimal traffic can be run with only one Ruby on Rails instance.

Initial Rails Application
User --> Ruby on Rails Application --> Database

As requests against the applicaion increase, we can add more Ruby on Rails instances to handle the traffic.

Ruby on Rails applications scaled to handle growing requests
         Ruby on Rails Application #1
User --> Ruby on Rails Application #2 --> Database
         Ruby on Rails Application #3 

In the experiment an application is bombarded with redundant data. In Figure 2 we see a blue line, the constant. This is the amount of duplicates without validation. The green line is the amount of duplication with validation. We can see with one running instance we have 0 duplicates. As soon as we start adding more instances the amount of duplicates jump.

As a Ruby developer with a history of Rails I was surprised at the accusations coming from the paper and discussion. Rails doesn’t prohibit us from using database concurrency control solutions. In my experience we have always used them.

This was really eye opening. I’m glad I went to this discussion and I’m thankful for the Seattle Chapter of Papers We Love. Many people come to Rails from boot camps or are self-taught. While ORM’s don’t prevent you from using the database it definitely encourages it to be blindly used as a storage system. It’s changed my thinking in how to help grow jr developers and the importance that needs to be placed on what’s happening in db/schema.rb.

Here is a list of resources I will be referencing in future conversations:
* Transactions
* Constraints vs Validations
* Postgres Explain

As a side note, validations where only inconsistent when compared to other records (am I unique, does this key exist). Validations performed consistently when evaluating data in the current record (is this a string and does it have a pattern?).