We’re in the process of building a very database-heavy system to store lots of metric data for our FiveRuns RM-Manage product. We looked at using tools like RRDTool and Berkeley DB but eventually came to the conclusion that a well-tuned mysql instance will work better for the flexibility and ease of maintenance we would like.
So we are using ActiveRecord as our database layer and it does not have a reputation as the speediest of ORM layers but usually that doesn’t matter too much. However in this case our system is so heavyweight that tuning ActiveRecord is vital.
After a quick profile with ruby-prof, it looks like we spend a fair amount of time cloning attributes. Speak of the devil, apparently this issue has been fixed in Rails Edge so hopefully Rails 2.0.3 will feature this fix. In the meantime, I rolled my own insert/update statement creation and found that it improved performance by 40-50%! I could do this because we are inserting large amounts of records and immediately throwing the records away so there is no need to update the model instances in memory.
Still approximately 1/3 of the time is spent just doing string manipulation to create the database statements. I might try using mysql’s multiple insert feature next and see what it can do for me.
3 responses so far ↓
1 Gleb Arshinov // Feb 8, 2008 at 5:47 pm
Mike, try ar-extensions. This was mentioned in the comments of our blogs, and seems like what you need here.
2 Cyril David // Feb 20, 2008 at 6:14 am
I created an in-house plugin that does batch inserts of SQL.
You may check it here
http://code.google.com/p/batched-sql-fu/
3 Ken // Apr 9, 2008 at 5:29 pm
I’ve used ar-extensions for this, too. If you have a bunch of records you need to import, it’s great. You can call AnyClass.import(list_of_attributes) to dump them in one transaction, bypassing AR completely. Importing 15,000 records dropped from 4 minutes to 20 seconds on our test box (admittedly not the fastest).
Leave a Comment